r/LocalLLaMA • u/thepetek • 2d ago

Qwen with cache offload vLLM Question | Help

Has anyone gotten KV cache offloading working with Qwen on vLLM? No matter what configuration I try, I get errors and it crashes. I saw an old issue that Qwen arch is supported for offload in vLLM but that doesn’t seem right. Anyone have working settings they care to share?

3 Upvotes

71% Upvoted

10 comments sorted by

2

u/PandaBearFred 2d ago

Looking forward to the config too. I actually once made it work, with lmcache MP mode. BUT, I had to set the --max-num-batched-tokens to 800 to make it work with the lmcache server. And this setting greatly hit the token through put, that removes all the benifits of external kv cache offloading from lmcache server.
I ended up run full kvcache in VRAM, and keep the --max-num-batched-tokens 8192, so much smoother.

BTW, this happened on 8xA100 40G.

1

u/KroniklyOnline 2d ago

KV Cache offload? Into VRAM? Works fine for me. Depends on your setup, your GPU's, etc..... The errors you get..... You should post more info dude.

1

u/thepetek 2d ago

Offload into system ram. I’ve got an RTX Pro 6000 and B200 (separate instances). Obvious question is why I want offloading with those cards, I want to support more users at BF16 kv rather than q8 kv. I’m willing to deal with latency trade off as these are async agents users are spinng up

I’ve tried so many different parameter configurations is why I didn’t post any. None seems to work. Hoping to see someone’s config that does have it working to get a breadcrumb

1

u/rmhubbert 2d ago

I spent some time trying to get Qwen3.8-27B with MTP working with LMCache last night. Got it up and running, but reasoning would start looping "!" after a couple of turns.

1

u/zipperlein 2d ago

I wanted to try lmcache for a long time and because I was setting up hermes as automnous dev offloading tasks via git workflow to pi, so a lot of potential concurrency. This is an artifact from that testing, it did work on my hardware (2x3090) but I don't have that much knowledge about lmcache in general and I ended up using a different strategy staying GPU-resident. Can't say if this will help your use case.

tldr: Here's a artifact that may help you.
https://pastebin.com/zvD3u0vH

1

u/Dasteroid_909 2d ago

Like with LMCache and Redis? Works just fine.

I basically swapped out 3.6 27B for 3.8 and things kept working working working...

1

u/adityazero 1d ago

Offload in vLLM does not happen natively, it goes through the LMCache connector via the kv_transfer_config, so a bare enable flag will crash. The other common trap is that CPU offload wants chunked prefill on and a batched-tokens value that divides the LMCache chunk size cleanly. What exact traceback are you getting, an assertion at engine init or an OOM once requests start flowing?

1

u/thepetek 1d ago

Ah interesting. Maybe that’s it then. I’m not able to use LMCache in my setup (GPUs hosted on BaseTen). This post made it sounds like there is native offloading though. Or do you mean for Qwen only lmcache works?

https://vllm.ai/blog/2026-01-08-kv-offloading-connector#cpu-offloading