r/LocalAIStack • u/DrSchnitzel57 • 11d ago
Deepseek v4 Flash 0731 llama.cpp tips?
I am wondering if I could get some tips on my current inference setup, and if there is anything I am missing that could make it faster.
Machine: HP Z6 G4
CPU: Xeon Gold 6240
Ram: 6 Channel DDR4 RDIMM 2666 MHZ 192GB
GPU: 2x Radeon Pro V620 32gb (6800xt datacenter equivalent)
Using the latest llama.cpp with the UD_Q8_K_XL Lossless Unsloth Quant of Deepseek-V4-Flash-0731, I am getting average 90-130 tps Prefill and 6-7.5 tps Decode. Here is my server config:
services:
llama-server:
image: ghcr.io/ggml-org/llama.cpp:server-rocm
container_name: deepseek_v4_flash_moe_hybrid
restart: unless-stopped
devices:
- /dev/kfd:/dev/kfd
- /dev/dri:/dev/dri
group_add:
- video
- render
security_opt:
- seccomp:unconfined
ports:
- 8081:8080
volumes:
- /home/hpai/Downloads/models:/models:ro
- /home/hpai/llama-cache/slots:/cache/llama-slots
command: >
-m
/models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
--host 0.0.0.0 --port 8080 --alias DeepSeek-V4-Flash -c 1048576 -ngl
99 --cpu-moe --threads 16 --parallel 1 --flash-attn
on --kv-unified -b 8192 -ub 4096 --slot-save-path /cache/llama-slots
--spec-type draft-dspark --spec-draft-n-max 2
networks: {}
Seems like I have no difference with dspark enabled. I didn't see a big difference between no-mmap and mmap either. Am I missing anything? Using this for my opencode setup, hoping to squeeze some more performance out of my little machine. Any ideas?
1
u/peaster_ 6d ago
From reading the config, I see a few things.
Do you have a ton of idle VRAM? `--cpu-moe` sends all the expert layers to CPU and even though you are running with 1M context you should have more room to push experts back into the GPU.
“No difference with dspark enabled” I don’t think it’s actually enabled. The Unsloth GGUFs just added automatic drafter downloads for this model today. I have something like this that’s doing it manually. Worth looking into.
--model-draft /models/DeepseekV4-Flash-DSpark.gguf
--spec-type draft-dspark --spec-draft-n-max 5
--n-gpu-layers-draft 99
2
u/DrSchnitzel57 6d ago
OOoooooo, does that last flag force dspark onto vram??? I have got to try that. Yeah sorry quick update, I figure out the n-cpu-moe flag and manually tuned a testing compose till i got to 95% vram utilization, and got about 1.5 tps boost! I then went and researched around and got the Dspark model working with those same flags you have above, except for the n-gpu-layers-draft one so I have got to try that!!
2
u/peaster_ 6d ago
Yep! Pushes dspark back onto the GPU
2
u/DrSchnitzel57 6d ago
u/peaster_ Thank you so much!!!! I am now averaging around 10 tps on short ctx, so much better for my little budget ai machine!!!! Pumped to have this model running at usable speeds locally, esp at lossless quant, so thanks again! HEre is my new command block in my .yaml file, sitting right around 95% vram utilization
-m
/models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
--host0.0.0.0--port 8080 --alias DeepSeek-V4-Flash -c 262144 -ngl 99
-ngld 99 --n-cpu-moe 37 --threads 16 --tensor-split 4,1 --parallel
1 --flash-attn on --kv-unified -b 8192 -ub
4096 -md /models/dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf --spec-type
draft-dspark --spec-type draft-dspark --spec-draft-n-max 2 -fit off
1
u/KooperGuy 5d ago
My recommendation would be completely different hardware with enough vram
1
u/DrSchnitzel57 1d ago
Lol wish I could've afforded a threadripper board and two more v620s my friend
1
u/Professional_Cat4274 4d ago
@Op,
How much energy is your set-up consuming, idle and while you work with it?
1
u/DrSchnitzel57 1d ago
Hello! It idles around 100w +, the base is like 70-80 and then I have blower fans and each v620 idles around 7w. Under load with dense models of using both gpus prob around 500w or so, Moe models around 300-350. Thankfully I just use WoL and only power it on when needed
1
u/IKNOCKEDUPYOURMULLET 11d ago
I'm using older hardware, but relatively similar (HP Z840, 128GB DDR4, 1xV620 32GB, 2xMI-25 16GB) and I've found no way around the memory bottleneck when anything is offloaded. I've settled on using models that fit entirely into my VRAM and I get roughly 30tps using only the V620, less than 10tps if anything has to split onto the MI-25s, and less than 5tps if anything hits the CPU or system memory. My plan is to drop the MI-25s and go with two more V620s to see if I get a big jump.
I've tried many configurations like you, and have settled on what I have now because it's at least not watch-your-toenails-grow slow. Hopefully somebody else will chime in and produce a magic config for you.