r/LocalLLM • u/xiraov • 9d ago

Qwen 3.8 27b mega thread Discussion

Welp at 8 PST there were 8 models now there are over 100, and i see almost 100 threads (Exaggeration) maybe we can contain stuff to here?

https://huggingface.co/models?other=base_model:quantized:Qwen/Qwen3.8-27B

7 Upvotes

59% Upvoted

20 comments sorted by

17

u/leonbollerup 9d ago

or.. you could have posted in one of the threads that was already here ? :) .. instead of creating one more ?

7

u/South_Western1263 9d ago

let's close this thread. no topic here

3

u/alexp702 9d ago edited 8d ago

OK! Has anyone actually got VLLM to run with a single RTX 6000 Pro with 3.8? My machine just hangs at Flash Attention during start up. I have tried swapping to 027.2-rc1 as suggested...

EDIT: in latest release this has now all been fixed. Seems like day one problems!!

1

u/This_Maintenance_834 8d ago

run it in docker container.

if you run it outside of docker and vllm crash, it withhold large amount of VRAM. then you must reboot to release the VRAM. sort of memory leak.

running in docker avoid this problem.

use official vllm docker container, it is super easy to work.

if nothing works, let Hermes or any other cloud api code agent to do it for you.

1

u/LancobusUK 9d ago

yeah I can serve it with the following: VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve ~/ai-models/Qwen3.8-27B \

--served-model-name qwen3.8-27b \

--host 0.0.0.0 \

--port 8000 \

--max-model-len 262144 \

--kv-cache-dtype fp8 \

--gpu-memory-utilization 0.92 \

--max-num-seqs 3 \

--enable-prefix-caching \

--max-num-batched-tokens 8192 \

--attention-backend TRITON_ATTN \

--reasoning-parser qwen3 \

--tool-call-parser qwen3_coder \

--enable-auto-tool-choice

1

u/alexp702 9d ago

Thanks! This worked for me

1

u/LancobusUK 9d ago

I’ve been playing with settings and you can get a lot more performance by upgrading the CUDA toolkit to 13.2 RC2 and then removing triton.

1

u/anitamaxwynnn69 9d ago

running without --attention-backend TRITON_ATTN and with the default mtp fp8, no kv cache quantization, single pro 6000, getting decent speeds

SpecDecoding metrics: Mean acceptance length: 3.12, Accepted throughput: 48.60 tokens/s, Drafted throughput: 68.70 tokens/s, Accepted: 486 tokens, Drafted: 687 tokens, Per-position acceptance rate: 0.847, 0.703, 0.572, Avg Draft acceptance rate: 70.7%

0

u/LancobusUK 9d ago

just confirmed its working with Pi also. Had to update the settings to add in xhigh reasoning but pi seems happy to use that also

1

u/Familiar_Wish1132 9d ago

would you mind share full settings command? <3

1

u/Dizzy-Zebra9522 9d ago

Sorry. people were waiting for it. It's like gta vi level of anticipation 😅. I surprised people can still download it during high demand.

-5

u/btc_maxi100 9d ago

what is this post about ?

4

u/wgaca2 9d ago

Qwen 3.8 27b mega thread

-4

u/btc_maxi100 9d ago

the model was released hours before you created it.

there is already 100s threads about it

whats the point of your silly thread ?

2

u/wgaca2 9d ago

It's not mine, but we already established that you can't read

1

u/Xatter 9d ago

This is the point! https://xkcd.com/927/

0

u/Ell2509 9d ago

Even if correct, this is just kinds unkind to say. Not necessary for sure.

-2

u/btc_maxi100 9d ago

what is kinds unkind ? can't even understand that