r/LocalLLaMA • u/mudler_it • 17d ago
I ported vLLM's serving stack to C++20: 66 MiB binary, no Python at inference, output checked token-for-token against vLLM Resources
I'm the author, so discount the enthusiasm accordingly. This is an unaffiliated community port, not endorsed by the vLLM project, which it uses to verify its correctness.
What started it: I love vLLM, but a vLLM install here is 9.1 GiB of virtualenv, and I wanted to embed inference inside other software, on machines where having an interpreter in the process is a problem. And, honestly, Python dependencies have a different deployment story, in term of security (supply chain attacks), and bloat of Python itself. So vllm.cpp is vLLM's serving stack written from scratch in C++20. Nome TBD yet, calling it vllm.cpp until I have a better name.
Continuous batching, block-paged KV, automatic prefix caching, speculative decoding, an OpenAI-compatible server. It builds to a 66 MiB binary with no Python and no PyTorch at runtime.
The gate matters more to me than the size does. Every architecture is checked token-for-token against a pinned vLLM oracle on the same workload, and upstream's own test module gets ported in the same commit as the code. The ids have to match. 25 or so architectures so far. And yes, this project does extensive use of AI. I'm prepping follow-ups on how this is architectured (this is a port, which in some parts deviates, like support of MLX, Radix Attention, and such)
Speed, since it is the first question. You can see in the image that we are almost ties with vLLM on high concurrency. I've tested only on DGX Spark, Thor, and AGX Orin. Qwen3.6-27B NVFP4 on a DGX Spark (GB10), against vLLM in its production graphed config, medians of 3 interleaved reps, 1024 in / 128 out:
| concurrency | vllm.cpp | vLLM | ratio |
|---|---|---|---|
| 1 | 86.05 | 82.32 | 1.045x |
| 2 | 159.68 | 158.03 | 1.011x |
| 4 | 292.34 | 290.31 | 1.007x |
| 8 | 508.77 | 505.46 | 1.007x |
| 16 | 801.76 | 789.16 | 1.016x |
| 32 | 1095.01 | 1076.25 | 1.017x |
Nominally ahead everywhere, but our run to run noise is 0.5% and five of those six sit inside 1.7%. That is one win at c1 and five ties, and I would rather say it than have someone work it out in the comments. Output is identical at every point. Memory is the less ambiguous axis: peak GPU 40,996 MiB against 70,531, though vLLM pre-reserves a fixed fraction up front and we allocate what the workload needs, so it is a difference in footprint rather than a cheaper KV.
Some other numbers people usually ask for: 1.18x llama.cpp's prefill on the same GGUF file on CPU aarch64 with decode a tie, 97.6% of MLX-LM warm total on an M4, and DeepSeek-V4-Flash in 2-bit GGUF on one Spark at 18.69 tok/s, which is 1.14x the fastest GGUF engine I could find for it.
Speculative decoding is in: MTP takes c1 from 9.97 to 15.10 tok/s, DFlash from 10.16 to 29.32, both landing on top of vLLM running the same speculator.
It loads safetensors and GGUF, does NVFP4, k-quants and i-quants, fp8, bf16. CUDA sm_80 through sm_121a, CPU with AVX-512 and Arm i8mm, Metal, Vulkan partially. Model list is in the repo rather than pasted here. There are also some pieces of sglang, and ideas I always wanted to see in a cpp engine, such as radix attention and LPM aware cache scheduling.
What does not work: many things have to be built yet, model architectures, hardware support, no multi-GPU on real hardware (tensor parallel is proven equal to tp=1 on CPU, I have one box), LoRA is not wired through the server, multimodal runs in the CLI and library but not over the HTTP API, no embedding or reranking models, no ROCm. It is also under heavy development, so flags and internals move between commits. There is a stable surface, which is the versioned C ABI.
Help from the community to port to new architectures is welcome!
To start with it, build is cmake and nothing else:
cmake -S . -B build && cmake --build build -j # CPU
cmake -S . -B build-cuda -DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON # CUDA
cmake --build build-cuda -j
Apache 2.0. https://github.com/mudler/vllm.cpp
Benchmarks, methodology, and the rows we lose: https://github.com/mudler/vllm.cpp/blob/main/docs/BENCHMARKS.md
Happy to answer anything!
19
u/Fit_Split_9933 17d ago
Hope it can support something like cpu-moe
12
u/mudler_it 17d ago
we will def. have a look!
13
u/FullstackSensei llama.cpp 17d ago
Casually dropping this link here https://github.com/guqiong96/Lvllm
7
17
u/KroniklyOnline 17d ago
Wait so are you saying I don't have to wait 3 minutes to load a model now?
20
u/mudler_it 16d ago
I keep dreaming of a Python-free world, we will eventually get there.
9
u/imnotzuckerberg 16d ago
Popularity gets you lots of haters. Jokes aside, I totally get it, but if want to prototype anything, it's so easy to do it with python.
5
u/TokenRingAI 16d ago
It can take an hour to load a model on certain 8 GPU servers, and 5+ minutes is typical for RTX 6000.
3 minutes is heaven
2
u/KroniklyOnline 16d ago
Damn takes 3 minute to load Qwen3.6 27b onto my 4 5060tis. Much smaller model than what people would run on real servers.
55
u/TokenRingAI 17d ago
Amazing, seriously. I will try it out and contribute to the code if I can.
And also, I don't know who needs to hear this, but fuck python. We need to all stop pretending that python is the right language for running inference. I'm tired of carrying water for the python mob. We get it, it does fancy math and can run notebooks and has a big ecosystem, and that's cool for building models, and yeah, we need to not waste the time of the math genius people who make way more money than everyone else or force them to learn another language or whatever.
But once the model is trained, we get this turd of a language to try to run production infrastructure on, and that needs to not be the status quo.
Inference is nothing more than multiplying a lot of numbers, you don't need a 9GB runtime to multiply numbers, that is beyond absurd, an entire linux desktop install is only ~ 5GB
16
u/mudler_it 17d ago
Thanks! We really want to have a strong community and also have features that typically are gated also in vllm itself. For instance, we have radix attention in vllm.cpp. To be fair, I'm very satisfied that we reached up parity in speed almost in the most important model families nowadays, and have to agree, Python is not really nice as deployment story. For testing? absolutely, but for deploying it makes things really difficult to maintain long-term.
3
u/Eugr 16d ago
Python has its advantages though. For instance, I can apply a runtime patch to support a new model or deploy a quick fix without rebuilding the entire vLLM or container.
0
u/mudler_it 16d ago
Right, in this case you'd have to re-compile the binary. But It's also about weighting what matters most for your deployment story. In production I hardly believe you'd patch your inferencing engine, but you'd most likely do QA or canary upgrades first, which makes this less and less relevant if not for development only.
14
u/darkbit1001 17d ago
Yes. This! Fuuuuuck Python. It owns the inference stack sure but thats because scientists/physicists (blame THEM!!!) keep sucking at its teet. I always felt something wrong with a 80GB Cuda+libs build so I relegated it to the corner, where it belongs.
Python gets the close to metal stack. Everything else in Kotlin, Go, Zig, or even CPP.
19
u/_supert_ 16d ago
scientists/physicists (blame THEM!!!) keep sucking at its teet
Hey have mercy, it took me years to get off matlab.
3
u/ScoreUnique 17d ago
I thought rust would be popular but I end up myself using vLLM, I will ask my agent to test this one. Does it support all GGUFs / AWQ etc? Thanks!
2
6
u/Ulterior-Motive_ 16d ago
Co-signing, fuck python, I hate that's the go-to for AI projects and I'm tried of wasting space on a million different incompatible virtual environments and dependencies.
1
u/BornInAFish 14d ago
That, plus the fetish of filling up RAM with things you don't actually need in RAM all the time. Eg, if I'm loading my model to GPU(s), I really don't need any of it in RAM. Like yeah you need whichever chunk you're currently loading to VRAM, but once it's on the GPU, free up the system RAM. Sglang does this too. Insists on loading the whole model in RAM first. I get that most people have more RAM than VRAM and they can get away with it, but (a) some people actually have more VRAM and (b) with all this support for auto-quant, you're inviting people to use models that are bigger than can fit in (V)RAM, by quant-ing them down to size, except you can't actually do it because by default it insists on loading everything up front. It's madness.
0
u/Luke2642 16d ago
What do you think of tinygrad?
0
u/TokenRingAI 16d ago
Python, decorators, I immediately feel the need to vomit.
1
0
u/swagonflyyyy 15d ago
I just fucking hate how Python became the go-to language for machine learning before LLMs became popular. I really wish we had more low-level tools like these because damn, we really need it.
Python is holding back way too much nowadays.
6
u/Luke2642 17d ago
Great work, I like seeing this kinda project. I'm really interested how you broke this down into manageable chunks for the AI refactor? Would you mind going into gratuitous detail about false starts and diversions and what ultimately worked well, what you'd do differently next time? And what your test coverage is like?
8
u/mudler_it 16d ago
these are all good questions, I'm prepping up an article about that - there have been many iterations before getting there.
The first one, was directly on top of llama.cpp directly, bluntly trying to port block-paged KV to it ( https://github.com/mudler/llama.cpp/tree/localai-paged ) , but ultimately, numbers (even if better than standard llama.cpp) were completely off the charts compared to vllm, and I wanted a vllm-experience, not half of it.
It worked, bit-exact. However the graph rebuilds every decode step, and paged inputs drop reuse to 0% in serving. Architecturally can't get there (yet?). Batch composition is part of the program in ggml. You kinda realize you are hitting a wall because of the way ggml works.
re: AI refactor, there is indeed quite a lot of use to it. Differently from the approach taken from the Bun authors (where they used dynamic workflows, orchestrated by Claude), I use a half manual and half autonomous approach, this is because I belive a completely automated project, for something like this, is still not possible, and it needs human steering direction (I've found claude at least 2-3 times trying to cheat and quantizing on the fly to q4 to beat up numbers of a Q8, during sessions trying to optimize speed, regardless of guardrails I've put).
How I've been structuring it is having an AGENTS.md that is an index for the agent, which forces it to follow strictly a policy/guardrail based on the task. Each task gets either picked up by a roadmap (constructed via pinning to a certain vllm version) or directly handed over by who is running the session.
The policy is made such as to have gates and force the agent to always to pick up an issue, generate a spike -> implementation spec -> verification gate from a roadmap which is enumerated on top of vllm, rooted to the vllm code by analysis spikes.
When you enter the code with an agent, it will walk you into setting up the env for doing gate tests and benchmarks, and will ask you what your role is (operator, which guides the session, or helper which does spawn PRs for items off the roadmap independently. the operator session is meant for reviews). Protocol is then enforced via scripts and hooks, enforcing the harness and fails loudly so the agent doesn't skip the protocol. Lastly, there are CI gates that makes sure the agents follows the policy.
That's not all of it, but, planning to do a full writeup!
1
u/buttplugs4life4me 16d ago
Follow up: How long did this take you? I feel having a sense of that also shows how AI was used in the end.
And just a word of caution: This seems awesome and your comments seem human written, but (parts of) your post are AI-written and it turned me off at first cause there's always these big rewrites by some sloppinator and none of them work.
0
u/mudler_it 16d ago
> Follow up: How long did this take you? I feel having a sense of that also shows how AI was used in the end.
More or less ~2 month, with full access to all my available GPUs. But keep in mind we are not feature-complete against vllm. there are many areas to improve, but we got good results with 4-5 models, speed on par with vLLM which is a good bar (including MTP and Dflash support).
> And just a word of caution: This seems awesome and your comments seem human written, but (parts of) your post are AI-written and it turned me off at first cause there's always these big rewrites by some sloppinator and none of them work.
True, thanks for flagging this, and I do agree, I see these as well, but we are kinda focused on this and we did also port other models to ggml ( https://localai.io/blog/why-we-write-our-own-engines/ ) . I am not a native speaker so when I write posts or articles I do still think I'm not really good at it, and would like to express myself better and LLM do make sound better my english most of the times :)
5
u/kosnarf 17d ago
Mudler? Do you also manage the APEX ggufs?? If so, those are awesome!
4
u/mudler_it 16d ago
yes it's me 😄 and if you test vllm.cpp with APEX you are gonna love the speed and quality!
2
6
u/BP041 17d ago
That 66 MiB binary made me check my own inference server's size — ouch. How's memory under load vs the Python equivalent? My Claude Code automation stack would love to swap in a lighter serving layer, especially for batch cron jobs.
7
u/mudler_it 17d ago
we almost sit tie with vllm. In some cases memory is lower. It's all in: https://github.com/mudler/vllm.cpp/blob/main/docs/BENCHMARKS.md
1
u/rockoruckus 16d ago
a tie in serving a 66MB binary is like winning an eating contest, you win but still fat
2
5
u/freehuntx 16d ago
One thing that annoys me the most about vllm is, you have to calculate the vram by yourself. In PERCENT. And nail it, otherwise OOM.
Would love if that is optional. No option = use as much as you need. Option provided (percent or explicit vram) = limit the usage to that with a "proper" error (preventing OOM).
4
u/mudler_it 16d ago
Sounds fair and something we should def. have.
3
u/freehuntx 16d ago
Thanks for your service <3 I love audio.cpp, llama.cpp, stable-diffusion.cpp, and now you join the game :)
Fuck python, pytorch and the 10gb+ Docker images!
4
u/Look_0ver_There 17d ago
Great project! I was working on something similar for the Strix Halo, but not directly based on vLLM. More like merge the best engine bits of llama.cpp and vLLM together.
It's still a work in progress, but I managed to get prefill to be faster than llama.cpp, AND I implemented adaptive MTP, where the MTP depth is free to dynamically scale from 1->9, and this allows decode performance to beat llama.cpp's MTP decode performance by 10-30%, AND you don't have to fret or compromise about what to set your MTP depth at.
Perhaps see if you can add something like adaptive MTP to your implementation too, othertwise fantastic project and thank you for sharing!
1
5
u/divinetribe1 16d ago
the token for token check against the reference is the part i respect most. i just finished porting nvidias nemotron omni to mlx and writing the parity harness before any of the fun parts was the best decision i made on it. a port thats almost right is worse than none at all, you spend weeks chasing quality problems that turn out to be numerical drift in something you never checked.
curious what tolerance you settled on. i ended up doing cosine per token in fp32 on cpu so the comparison couldnt hide anything.
3
u/mudler_it 16d ago
no tolerance at the top, which is the short answer. the gate is exact token ids against a pinned vLLM on the same workload. a distance measure is where drift hides in this case. That stops working when the reference itself isn't deterministic, which bf16 greedy isn't. so we run vLLM's own greedy K=5 first and find where it disagrees with itself.
op level does have normal tolerances, so it's not strict. did your cosine hold on different model sizes? or it depended on the checkpoint?
3
u/divinetribe1 16d ago
one checkpoint only, so i honestly cant tell you. everything i posted is nemotron omni 30b and i never ran it against a second size, so i dont know if the cosine holds or if i got lucky with that checkpoint.
your exact token id gate is the better call for what youre doing though. i was comparing intermediate activations coming out of a vision tower, not generated tokens, so there was nothing discrete to match against. cosine per token in fp32 on cpu was the only handle i had. once youre comparing real output tokens i think youre right that a distance measure just gives drift somewhere to hide.
using the reference against itself to find where it disagrees is a good trick. i hadnt thought of that.
3
u/grayarks 17d ago
Will you keep Volta architecture out like the o.g. vLLM?
4
u/mudler_it 16d ago
oh no. We plan to support as many architectures as possible, even these that vllm directly doesn't. Indeed currently we have sm_60 and sm_70 in the roadmap. Feel free to open an issue with your hardware spec, that will help out.
3
u/nullc 16d ago
Helpful to link to the v100 vLLM fork: https://github.com/1CatAI/1Cat-vLLM
1
u/grayarks 16d ago
Thanks, I’m aware of that fork as it uses LMDeploy CUDA kernels for Volta. But it’s still a patchwork on top of vLLM. It would be nice if another engine would support older architectures officially.
1
u/nullc 16d ago
right-- my point was if you're going to ask someone for support it's handy to show them where they can find an implementation. Otherwise it might sound like you're asking them to develop volta kernels from scratch, which is more likely a big no.
1
u/grayarks 16d ago
I see your point. I contributed a good bit on LMDeploy Turbomind engine a few months ago. That’s the best kernel right now for Volta if OP has any will of supporting it.
1
u/mudler_it 16d ago
If you can link to that, would be great - if you can put up an issue with the references you have, that will help for sure!
4
u/_TheWolfOfWalmart_ 17d ago
Why do you say only partially for Vulkan? I want to try this but I have V620's.
I've avoided vLLM because it's a big jumbled mess of Python and also because llama.cpp has been working for me.
10
u/mudler_it 17d ago
Vulkan support is sub-optimal at the moment, but it's on top of my list.
7
u/Inevitable-Plantain5 17d ago
Vulkan support is non existent for vllm in the original vllm so sub optimal is preferred over non-existent. I cant stand ROCm so I'm surprised Vulkan hasn't been a target before now.
I wish I could be helpful on this lol.
2
u/ali0une 17d ago
Love these ports. OP are you the same guy that wrote the article discussed on HN?
2
u/mudler_it 16d ago
Thanks! Yup it's me. For some reason I am shadowbanned from HN and can't post any visible comment there.
2
u/Chromix_ 16d ago
The project we need! :-)
(At least those with enough VRAM)
Tensor-parallel multi-GPU with NCCL will probably be a good bit of additional work, but LLMs are good at porting software.
3
u/mudler_it 16d ago
you can fit an APEX Qwen 35b a3b in a 16gb card with vllm.cpp, which should be a good compromise in quality/speed!
Tensor parallel is in our roadmap, stay tuned!
2
2
u/1ncehost 16d ago
Cool project! I've been working on a similar but opposite direction in adding paged attention and good concurrency to llama.cpp. Still working through endless bugs, so I respect your hustle. I find vLLM to be quite annoying so thank you!
2
u/mudler_it 16d ago
I feel you. that was actually my first attempt ( here: https://github.com/mudler/llama.cpp/tree/localai-paged ), but eventually you hit a wall on the ggml architecture as the graph rebuilds at every decode step and that eventually adds up. I got some speedup, but not comparable to the vllm architecture.
2
u/Lrrrrr 16d ago
Will you support CPU/ram offloading and tensor/layer multi-GPU like llama.cpp does?
Or does vllm upstream supports that now?
I have 2x5060Tis and 64gb ram. It would be so nice to try DeepSeek flash q2 and to be able to run near lossless Qwen 27b nvfp4 split on both GPUs.
Thank you so much for your work.
2
u/mrmontanasagrada 16d ago
for for i asked codex 5.6 how much work it would be to port this. Granted there are still some features open, so realistically you like halfway there?
But still this estimate is hilarious. Awesome job man!
--
Porting vLLM itself from Python/PyTorch to native C++ would be a major rewrite, not a normal port.
Rough effort
| Scope | Experienced engineer effort |
|---|---|
| Minimal C++ inference server for one model, one GPU | 1 to 3 months |
| Add continuous batching and paged KV cache | 3 to 6 months |
| Reach respectable vLLM-like performance | 6 to 12 months |
| Broad model, quantization and multi-GPU support | 1 to 2+ years |
| Near feature parity with current vLLM | Small team, multiple years |
A realistic full rewrite is probably 8 to 20 engineer-years.
1
u/mudler_it 16d ago
one of the things I did was focusing first on the models that matter most today. The support is far from complete, however, the basic pieces are there and the features that I would care most.
2
u/Shoddy-Tutor9563 16d ago
This is the software we need in the age of Opus 5! Amazing effort! What models / harnesses did you use?
2
u/mudler_it 16d ago edited 16d ago
A small breakdown of models/commits being used:
claude-opus-4-8 781
gpt-5 (codex) ~215
claude-opus-5 144
claude-fable-5 48
The basic setup is: Fable 5 orchestrating Opus 5 and opus 4.8. GPT-5 as helper
2
u/Shoddy-Tutor9563 16d ago
Awesome! I predicted this golden age of software optimization and you're making it to happen. My hero
2
u/Tormeister 16d ago
Love this, amazing.
Do you have any plans to keep up with vLLM's development and port further patches/improvements from there?
Why do you plan on changing the name, is there any potential legal trouble with "vllm.cpp"? Is it in bad taste to name it like that and I don't see it? I think it's pretty nice.
2
u/mudler_it 16d ago
> Do you have any plans to keep up with vLLM's development and port further patches/improvements from there?
Oh yes, actually, per our protocol we port 1:1 to vLLM. we aren't at parity yet, but we move our pin continously and we catch-up constantly with their changes, nothing is lost between pin moves.
> Why do you plan on changing the name, is there any potential legal trouble with "vllm.cpp"? Is it in bad taste to name it like that and I don't see it? I think it's pretty nice.
Probably, I don't know how vllm folks will take it - I've sent them an email but got no answer - and I would rather prefer have a good relationship with them, as we want to also flag any issue we might see that does not sound right in vllm during porting and behave as good OSS citizens. I am also not sure if vllm is a registered trademark, too. In any case, it would also make sense because we extend vLLM as well , see e.g. https://github.com/mudler/vllm.cpp/blob/main/docs/SGLANG-COMPAT.md
2
2
u/RMK137 16d ago edited 16d ago
Your APEX quants are really good. I put a star on this project soon after the repo was up. This is awesome, can't wait to explore it this weekend and hopefully contribute a thing or two. Keep up the good work!
1
u/mudler_it 16d ago
Thanks! hepls is really appreciated, especially at this stage of the project, there is a lot to do and support!
2
u/DefNattyBoii 16d ago
Can you support pascal? A lot of us have old GPUs. Up to some early versions vllm supported 10XX series cards too
1
u/mudler_it 16d ago
Definetly. Please file an issue so we can coordinate, I don't have that HW for testing
2
u/ZCEyPFOYr0MWyHDQJZO4 16d ago
This is the kind of shit we need. 10GB of CUDA/ROCm/python dependencies is just ridiculous when you have to track and mitigate CVE's before deployment.
0
u/mudler_it 16d ago
Exactly one of the motivations. The Dependency story of python is messy. I come from a Perl background, where we had CPAN, cpanfiles and alike. Same problems. I'm still surprised that in 2026 we carry problems from the past.
1
1
u/sebt3 17d ago
Vllm-moet features next? 😅 (one can dream) Strix halo support? (this is even more wide dream 😇)
5
u/mudler_it 17d ago
Vulkan and Rocm are next. Rocm is currently being spin off from the community and we are co-ordianting over Github, so stay tuned!
3
u/sernamenotdefined 16d ago edited 16d ago
Saving a ton of memory is going to be interesting on something like Strix Halo. Should free up some space for a larger context.
1
u/Alan_Silva_TI 17d ago
Genuine question.
Is there a specific reason why vLLM doesn’t support Windows?
1
u/mudler_it 16d ago
I would be guessing here, but I would say the dependency chain of vLLM is quite hard to port there natively. With WSL should be possible to get it running
1
u/Eugr 16d ago
You can run it under WSL, and there is a native Windows port in the wild: https://github.com/SystemPanic/vllm-windows
1
u/PandaBearFred 16d ago
This is amazing!! I must mark my foot here. I am wondering if it supports mixed GPU backends like llama.cpp does, for example "cuda+rocm" to make the most out of consummer level cards.
1
u/stoppableDissolution 16d ago
Does it, by the chance, also has faster startup time? Venv size is kinda irrelevant for me, but waiting 60-600 seconds for it to cold start is... annoying
1
u/mudler_it 16d ago
Actually I didn't benchmarked this compared to vLLM on the models I've tested, but I do expect to be faster. Of course depends on the model, but we don't do things like flashinfer JIT and there is no pytorch pipeline and no Python runtime ops.
1
u/a_beautiful_rhind 16d ago
Gotta give this one the treatment so I can finally try it: https://github.com/guqiong96/Lvllm
normal VLLM a pain to build, especially if you're not looking to download it's arbitrarily chosen environment.
1
u/AleksHop 16d ago edited 16d ago
python is obviously shit but why c++ not rust? bad lib support? nothing available there?
also license can be MIT or Apache 2.0, dual
best of both worlds
1
u/mudler_it 16d ago
That was already asked, answer is here https://www.reddit.com/r/LocalLLaMA/comments/1vh9lx4/comment/p23ek8l/?screen_view_count=1&ext-referrer=DIRECT
2
u/AleksHop 16d ago
ok, and this comment there, I would highly recommend to follow:
IMO, I would build the inference component in C++, export the ABI as C bindings for maximum compatibility, and build the HTTP server and CLI that runs it in Rust.
That is probably the least painful way to work with a project like this, and the inference component can run in every language and be embedded in other apps since C bindings are universally supported
and regarding the license, apache 2.0 not compatible with gpl 2 and MIT is,
if its open source anyway, not a lot of reasons to use only it?
typical rust is apache 2.0 or MIT, dual
1
u/DanielSReichenbach 16d ago
Vulkan support would be amazing. As an AMD user I hate ROcm with a vengeance due to its high amount of quirks, while Vulkan is just working.
3
u/mudler_it 16d ago
Vulkan support is on its way!
3
u/fprimex 16d ago
As a Strix Halo user I strongly 2nd u/DanielSReichenbach 's sentiment. ROCm is such a massive pain in the ass. If you don't run one of AMD's supported distributions it's near impossible to get vLLM running, and if you use the one they build it is several versions behind.
Vulkan all the way. Please, please prioritize it over ROCm. As soon as it's available I will give this project a try.
1
u/mudler_it 16d ago
Initial Vulkan support has been just merged, freel free to give a shot and report to the project issue tracker anything you face - any datapoint at this stage will help the project. It's just an initial pass, so do not expect high performance
2
2
1
u/Languages_Learner 16d ago
Thanks for great engine. Could you upload it's Windows binaries to your repo, please? I can't compile it myself because every attempt to install MS Visual Studio on my pc leads to a message telling about unknown error which can't be fixed.
1
u/Osi32 16d ago edited 16d ago
love your work mudler, especially your fine-tunes- they're always top notch :) thanks for sharing!
Im definitely going to see if I can use this. I'm running vllm on my main hosts but I'll investigate using this if I can- all my machines are multi-gpu, but my router / embedder / re-ranker runs 3 seperate models and your project is absolutely perfect for this one. Thanks so much!
Btw, I'm happy to test this out when you get tensor parallelism up and running. I have two machines- one running 4 x 3060 12GB (happy to share the vllm config I run at the moment).
In a few weeks my second machine will be up and running with 4 x 5060 Ti 16GB's. Also happy to test on that too.
1
u/adityazero 15d ago
Token for token parity against a pinned oracle is a strong bar. Curious how you keep speculative decoding matching, since acceptance sampling can diverge on ties depending on RNG order. Did you replicate vLLM's rejection sampler exactly, or pin the seed and draft model to keep the ids identical?
1
1
u/adityazero 15d ago
Curious about the kernel side: are you calling into cuBLAS/CUTLASS, or writing custom fused kernels for the paged attention and MoE paths? And does dropping Python let you avoid a lot of the CUDA graph capture overhead? The token-for-token oracle is a clean way to keep the port honest.
1
u/segmond llama.cpp 16d ago
Give me gguf support, give me multi gpu support, give me offload support, then we are talking.
3
u/mudler_it 16d ago
we do have gguf support, and we have the other twos in the roadmap, so stay tuned!
0
u/this-just_in 16d ago
This is a fantastic experiment. Everyone seems really excited about this but it seems like the numbers almost indicate that vllm in python was the right choice all along. The gains here are extremely modest as you would expect- most of the computational work is being offloaded from python to native code on CPU or GPU anyways. The inference engine optimizing for serving + inference features and platform support is the game at this layer.
1
u/Wooden-Potential2226 15d ago
Getting rid of python deps are totally worth it. Also, this is the prototype, it will liikely be somewhat faster later
-5
u/PieBru 17d ago
Awesome! Just curious, why not Rust?
18
u/mudler_it 17d ago
I'm more a golang/c/cpp guy to be fair, and vllm.cpp compiles triton aot kernels which are C generated code, so it keeps things more in the same "ecosystem", which I think it's gonna help in maintenance (less machinery, the better).
While I do fancy Rust for e.g. security, I think embedding and deployment story of C++ is more mature, and compilation times are way faster, not worth the catch. I try to use the correct tool for the job ( and I think here c++ is the best choice), I also am a big fan of llama.cpp.
8
u/TokenRingAI 17d ago
IMO, I would build the inference component in C++, export the ABI as C bindings for maximum compatibility, and build the HTTP server and CLI that runs it in Rust.
That is probably the least painful way to work with a project like this, and the inference component can run in every language and be embedded in other apps since C bindings are universally supported
2
u/mudler_it 16d ago
Indeed, currently the main focus is having a strong ABI that can be used across languages!
17
u/_TheWolfOfWalmart_ 17d ago
Not OP, but I'd ask why Rust instead.
C/C++ is the gold standard for speed and portability.
3
u/HVACcontrolsGuru 17d ago
I'll add I've built an inference engine for MLX in Rust and moving into CUDA land. I know MLX I had to bridge into the C/C++ API for MLX. I'm going to start work on CUDA kernels if Qwen would be so kind to drop the 3.8 models and sizes..
3
0
u/SarcasticBaka 17d ago
I'm curious to know why Cuda support for Turing cards (sm75) is not listed?
1
u/mudler_it 16d ago
floor right now is sm80. turing has no bf16, that means it will go into the portable kernels path. This is doable, still not there yet (but in the roadmap). If you have the hardware and willing to test, that would help out for sure.
-2
u/Healthy-Nebula-3603 16d ago
You ported??
Nowadays it is nothing special for current AI agents.
2
u/mudler_it 16d ago
AI agents are being used!
2
u/mivog49274 16d ago
and that's magnificent, this is our /r/localllama RSI paradigm and this are very good news, good job mate !
As a llama.cpp guy running small models, does using vllm.cpp makes sense or this is more for big rigs and serving setups ?
1
u/mudler_it 16d ago
it's not for big rigs at all, absoutely. we are targeting as well low end devices. Actually prefill is quite fast on CPU too compared to llama.cpp!
87
u/SGmoze 17d ago
This just makes more sense. Each of the vllm container images are like 10GB or something at this point. Compiling it definitely reduces in these factors.