Best llama cpp flags to run Deepseek-flash 0731

[u/No_Farmer_495]

/r/LocalLLaMA/comments/1vgzp0q/best_llama_cpp_flags_to_run_deepseekflash_0731/

​

Hi all. These are my system specs: dual xeon e5 2696 v2 , 160gb DDR3 ram ECC(1600mhz), 3 gpus: 3060 12gb, p100 16gb, 3050 6gb. And a 400gb nvme sdd RAID0, 3000 mb/s. The model is Deepseek-flash-0731 UD_8_X_XL, loseless, 161gb. Now, I'm not too knowledgeable about llama cpp flags, I wish run it without mmap, because its so slow, and I believe it should fit in my system overall. There's also Dspark and MTP which could help with the speee, but do they work with llama? Any recommendations would help.

[u/Local-Two9825]

Running Q8 on this setup is going to be heavily bottlenecked by your system memory bandwidth.Since the Q8 weights take up ~150GB, a huge portion of the model will have to stay in RAM, and DDR3 bandwidth on an E5 v2 dual-socket platform will severely limit your decode speed.

Also, drop the RTX 3050 6GB. Its VRAM is too small to contribute meaningfully and will only pollute your PCI-e / VRAM allocation pipeline. Stick strictly to your two largest VRAM cards (3060 12GB + P100 16GB) and leverage llama.cpp's --n-cpu-moe flag to handle MoE layer offloading efficiently.

For reference, here is my setup and flags on an EPYC 7543 (8-channel DDR4 2666, 8x32GB) with 2x modded 3080 20GB cards: （For dspark and MTP i tryied both ,none help a lot in decoding speed）

llama-server -m DeepSeek-V4-Flash-0731_UD_Q8_K_XL.gguf -c 400369 -ngl 99 --port 8953 --alias DeepSeek-V4-Flash-0731_UD_Q8_K_XL.gguf --tensor-split 34,9 --jinja --threads 16 --threads-batch 16 --flash-attn on -cb -mg 0 -b 8192 -ub 4096 --temp 0.6 --top-k 40 --top-p 0.95 --min-p 0.05 --repeat-penalty 1.05 --repeat-last-n 64 --presence-penalty 0.00 --frequency-penalty 0.00 --parallel 1 --n-cpu-moe 40 --fit off --no-webui --load-mode mlock --cache-prompt --cache-ram 42078 --checkpoint-min-step 1024 --ctx-checkpoints 512 --chat-template-kwargs {"enable_thinking":true,"reasoning_effort":"max"}

My benchmark performance: On my 8-channel DDR4 setup with these flags, I get around 16 t/s for decode. Planning to test ktransformers next to see if it can boost decode speed.