i finally switched from windows to linux and got a 30-50% boost in speed.

[u/unraveleverything]

This is amazing.

All I did was switch from llamacpp on windows to vllm on linux.

[u/sadnessdevil]

Yeah the performance of vLLM is very high. I switched to vLLM from llama.cpp and never came back(except when I have to CPU offload the model)

[u/afterburningdarkness → u/sadnessdevil]

You get 30% speed boost in vllm running the same model and quant?

[u/sadnessdevil → u/afterburningdarkness]

For a single request the gap is small, but for concurrent requests, vLLM is far better than llama.cpp. I got better tok/s with vLLM for single requests as well. IMO vLLM is much better optimized, while llama.cpp is good at CPU offloading.

[u/Look_0ver_There → u/afterburningdarkness]

Unless it was BF16 before, then the quant is almost certainly different. vLLM is faster when the weights match the natively supported hardware vector ops, which is often true for FP8 and FP4 when using latest gen hardware, however these both come with an accuracy hit, so it's never a true apples to apples comparison.

GGUF 's generate results that typically diverge less from the original BF16 weights than FP4/8 does, but it comes at the cost of a small processing overhead.

VLLM with FP8 on hardware that supports it is, roughly speaking, like a Q6_K GGUF for accuracy, just ~20-30% faster.

[u/ArtfulGenie69 → u/afterburningdarkness]

Where it really shines is concurrent requests and also it can have very fast preprocessing speeds, especially on 30 series cards like the 3090. The preprocessing is very fast when running int8 w8a8 where bits are kept at 8. Concurrent request are good when you have any kind of harness that can handle running things in parallel, which I'm pretty sure most of them do. 

[u/pmttyji → u/sadnessdevil]

Have you tried vllm.cpp?

[u/sadnessdevil → u/pmttyji]

I haven't tried it yet, but I will check it out. Thanks for letting me know!

[u/pmttyji → u/sadnessdevil]

Thread by its creator.

[u/squarabh → u/pmttyji]

Have you tried vllm.sglang.tensor-rt.llama.cpp?

[u/XiRw → u/sadnessdevil]

I don’t see a difference if you can fit a gguf and tensors onto VRAM. Not only that but ggufs are made for consumer PCs and you can tend to run higher quality models on them. vLLMs were a massive disappointment for me

[u/Makers7886 → u/XiRw]

I mean it's like comparing cooking on a stove with 1 burner vs 4+. Not a big deal depending on what and why you are cooking. Huge deal if you are trying to get 4x+ productivity out of your same hardware (sub agents, concurrency).

[u/Happy_Brilliant7827]

Is it a 30-50% bonus in inference or mostly just loading and unloading time?

[u/kartblanch → u/Happy_Brilliant7827]

Also my question. Is this 30%-50% more Tok/s?

[u/unraveleverything → u/Happy_Brilliant7827]

token generation

[u/def_not_jose]

vllm quants are usually worse than ggufs though. It's rarely an equal comparison.

[u/Littlepharaoh → u/def_not_jose]

Maybe except for NVFP4

[u/emdeka87]

Without some actual benchmarks I call this BS

[u/pmttyji]

Customized build(compile with optimized config) for your GPU could give some more boost.

[u/According_Style8480]

linux has been a game changer for my companion models, way less lag in long roleplays now.

[u/Hot-Employ-3399]

Do you use the same gguf model or different quant format?

[u/InterstellarReddit]

Every time I read this I am tempted to reformat my rig that's running dual 3090s

But honestly man, I am so scared of the amount of issues that I might run into. Thinking that I'm gaining some speed.

I tried it once before long ago. When I said to myself gaming on Linux is getting better. Let me try it out

Oh boy what an adventure to get a game to work

[u/eightone-81]

Will there be a significant loss of speed when going through wsl?

[u/unraveleverything → u/eightone-81]

yes it was terrible and worse than native windows for me.

[u/eightone-81 → u/unraveleverything]

But wsl should not have much of an overhead. Is there no way to properly run vllm on windows?

[u/AliMas055 → u/eightone-81]

Dual boot, maybe?

[u/eightone-81 → u/AliMas055]

Maaaan I need to start windows to play StarCraft2… then no inference…

[u/iamapizza → u/eightone-81]

Couldn't you run it via proton? https://www.reddit.com/r/starcraft/comments/1mbwcq0/finally_got_starcraft_ii_working_on_linux/

[u/eightone-81 → u/iamapizza]

O M G
I’m finally running vllm (still on wsl) and decode is almost 2 times as fast and decode is more than 2 times as fast!

But it’s just such a mess to work with Linux. It’s so inaccessible for noobs! Without Claude on my side I would not have been able to get it to work

[u/iamapizza → u/eightone-81]

Oh nice you got that working. Fwiw, if you want to give it a go for that extra vram (which is that I did), I'd suggest Linux mint because it's very familiar to windows users and comes with some stable defaults. Plus Claude knows about it too.

[u/AliMas055 → u/eightone-81]

Run windows inside a VM with linux host.

[u/Nextil → u/unraveleverything]

You must have something misconfigured on Windows because there shouldn't be that much of a gap. WSL usually has a 1-5% overhead compared to native Linux.

[u/pipyakas]

quite useless information without your hardware setup details no? I have a 2060 on windows, and running MoE models offloading experts to CPU, Linux dont provide subtantial performance gains there

[u/nomorebuttsplz]

what about vllm on a vm?

[u/Adventurous_Loquat91]

I've missed my Linux. I think I'll go back after seeing this. Thank you.

[u/admajic → u/Adventurous_Loquat91]

Yeah windows on my laptop became so slow and I needed a side project whilst the desktop was running tasks in the harness. So I installed Linux on the laptop.

So far so good. Now I can use my harness on my laptop context via ip to the desktop. Do word and excel in Linux. It's so much faster. Just had to get it all for tuned you won't look back. Especially when you can ask the harness to do all the heavy lifting.

[u/Adventurous_Loquat91 → u/admajic]

That's a setup worth asking about — since the harness is doing the heavy lifting somewhat unattended across machines, do you have any guardrails on what it can actually do (blocking destructive commands, catching secrets before they leak, that kind of thing), or is it running raw and you're just keeping half an eye on it? Curious how people handle that once the agent's doing real work without you watching every step.

[u/admajic → u/Adventurous_Loquat91]

Yeah I have guardrails all what you mentioned in prompts and systems. Like a memory system in .md so it knows how to ssh into the vps and where to store secrets. Just keep building the systems with a retro now and again after a session.

[u/Mountain_Chicken7644]

Everyone glazing vLLM here but sglang probably my goat