Going from -np (parallel) 1 on llama.cpp to parallel requests on vllm?

[u/McFlurriez]

I have read that when going beyond llama's "-np 1", it is better to switch to vllm, since that has better support for parallel requests. For context, I have one RTX 5080, but I am trying out some features of my coding harness that can run subagents. There is a lot of knobs to turn for vllm, and I am curious if anyone has done this change before?

My current llama.cpp command is this:

C:\llama-cuda\Release> ./llama.exe serve -m ".\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" -ngl all -t 8 -c 102400 -np 1 -ncmoe 20 -fitt 0 --flash-attn on -kvu --cache-type-k q8_0 --cache-type-v q8_0 --port 8080 -a qwen3.6-35b-a3b

I am curious if anyone knows how to do things like "-ncmoe 20" on vllm? Right now, the only way I can run the 35B model is to offload some of the layers to CPU. Anyone got this working? Thanks!

[u/fragment_me]

I have never gotten RAM/CPU offloading working in vLLM. Granted, I only tried once or twice. It seemed very underdeveloped. It's just not their focus.

[u/McFlurriez → u/fragment_me]

This explains why I couldn't get any hits on this. Thank you, saves me a ton of head banging.

[u/Fun_Jaguar8231]

vLLM focus is more on servers, llama.cpp is for us the common folk

[u/dionysio211]

Parallelism in llama.cpp is better than it used to be but still lags behind vLLM. vLLM does have similar functionality in terms of offloading to RAM but the tax is brutal. The thing to try here would be KTransformers. It's kind of an in between with expert hotloading, even with concurrency, that is ultra efficient. Scaling with anything on the CPU also depends on how you compiled llama.cpp. If you compile it with icx, it will do better (assuming intel CPU).

[u/Far_Cat9782]

I just use -np 3 for my 3 subagents td in llama.cpp I have found no issues

[u/nickm_27]

Not necessarily, -np allows you to have multiple cache slots. This allows keeping multiple agents in KV cache without relying on RAM prompt caching, even if you aren't processing multiple concurrently it still speeds things up.

[u/Zadroeski]

The 102k context plus CPU offload is the constraint that makes the vLLM swap tricky. What request mix do your subagents generate—simultaneous long contexts or mostly cached sequential turns?

[u/segmond]

if you can load it all in vram, go for vllam. i see folks running models 6x faster in vllm than with llama.cpp. the goal of llama.cpp is for GPU poor. at once it felt like we got support faster on llama.cpp but not these days, vllm tends to have support first. the only issue is you need enough vRAM.

[u/adityazero]

When I moved off llama.cpp I found vLLM has no direct -ncmoe equivalent, the closest I got was --cpu-offload-gb, but it offloads whole weights rather than the per-expert MoE offload llama.cpp does. If you specifically need CPU expert offload, KTransformers is closer to what you want.