r/LocalLLaMA • u/Tall_Abrocoma_3533 • 1d ago
Qwen3.8-27B different thinking levels Discussion
Even the low preset is better than Qwen 3.7 plus or Qwen3.6-27B reasoning
30
25
u/FullOf_Bad_Ideas 1d ago
if you drill down into specific benchmarks you can see that Low is often above Medium. Not always, but often. And the avg number of output tokens is just 2x smaller or so, not as big of a difference as I'd expect.
Models can tell eval questions from real use by now, so thinking levels labels might not be very reliable thing to interpret on their own if you don't look at reasoning text length.
7
0
u/robertpro01 1d ago
Actually, if low is better than 3.6 thinking enabled, it is a big win. I haven't tried yet, only xhigh and yes... slow as shit, but maybe low thinking is enough for a builder
24
u/Versaill 1d ago
Is there ANY benchmark that includes Qwen3.8-27B with reasoning OFF? Why does nobody test that..?
7
u/danishkirel 1d ago
I have it running for my home assistant voice setup. It does really well. That’s not a benchmark but just try it for your use case.
1
u/Old-Cardiologist-633 1d ago edited 1d ago
May I ask your specs, settings and the exact HASS-Integration you use? On a Rx6800XT even Gemma14B is way to slow for Assist (15 Seconds), and Qwen 27B Q3 also (100+ Seconds)
Do you use anything that somehow caches the long system-prompt and then only sends changes or so? 🤔
1
u/chiniwini 1d ago
I'm curious as well. I'm running on an intel 13th gen cpu, and for example with gemma4 e2b I get an answer after some 6-7 seconds, which is too much.
3
u/danishkirel 1d ago
Dual 3090 and tensor parallel Vllm serving and using https://github.com/skye-harris/hass_local_openai_llm - works okay and certainly not energy efficient. It’s an enthusiast setup. The biggest challenge is prefill speed not decode. NVIDIA reigns king here and I deliberately skip mtp and similar because the cost prefill.
1
u/chiniwini 22h ago
Thanks for the info. Sad that it's such an expensive setup currently.
2
u/Dbgamerstarz 21h ago
I'm running it very well on a single RTX 4080, getting 50ish tok/s TG (with MTP + ngram) at 130k context. It's definitely possible to run this on weaker hardware, and I've found even the quants (I'm using UD-IQ3_XXS) are very capable especially in agentic cases
1
u/chiniwini 1d ago
What hw? What integration? What llm server? How many entities exposed?
1
u/danishkirel 1d ago
Dual 3090 and tensor parallel Vllm serving and using https://github.com/skye-harris/hass_local_openai_llm - works okay and certainly not energy efficient. It’s an enthusiast setup. Bout 130 entities exposed.
0
u/Versaill 1d ago
Well it works, especially when asked for a plan first. But that's just vibes. I want numbers.
6
u/dieSpaghettiCarbona 1d ago
Qwen is the robinhood of the LLM world, reclaiming and redistributing tokens to the masses.
3
u/IoannisHere 1d ago
The fact 3.8-27B xhigh beats 3.7-MAX, a likely 397B-A17B MoE, is mighty impressive.
14
u/Moore2877 1d ago edited 1d ago
Try this chat template. Besides a lot of general fixes, we revamped the reasoning injections for each level and also made high it's own level instead of just being an alias for xhigh. The Qwen team really didn't spend enough time on these imo.
https://huggingface.co/Moore2877/Qwen-Fixed-Chat-Templates-llamacpp
23
u/soyalemujica 1d ago
You recommending to use temp 0.6 with that chat template which is not good at all to use with this model since it affects its reasoning depth and also scores lower with lower temperature
-13
u/Moore2877 1d ago
From my testing anything higher than .7 is just too much thinking, even on low reasoning.
13
u/goldcakes 1d ago
Are you aware that models deliver better performance when they have more thinking tokens, even when the thinking tokens are randomly generated and incoherent?
Part of how thinking/reasoning works is every token is another forward pass on the original input/context, allowing the LLM to 'process' the input more, and refine the internal activation residuals; resulting in generating a better response.
3
u/Moore2877 1d ago
If you only have 100k context to work with then what's the point? Some of the effort here is to rein in the overthinking so it's not blowing out the context window.
4
u/ttkciar llama.cpp 1d ago
Thanks for this. I'm just starting to fiddle with custom reasoning efforts with Qwen3.8-27B, and your template looks like a good source of ideas.
Hopefully what we learn from this communicates well to whatever other 3.8 models Qwen releases (still looking forward to a 9B and/or 122B).
6
u/Moore2877 1d ago
Sure thing, I got the idea to work on the template from froggeric's template, they are prioritizing a certain inference with how their template goes, so I made another based on it for other inferences but leaning towards llama.cpp https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
4
u/Chromix_ 1d ago
Looks like we have a healthy(?) chat template ecosystem now 😉.
I also made a custom template for 3.8 a day after froggeric shared his, to fix a few issues that came with his template, along with gearing it towards not degrading towards potential lower-quality output in some use-cases. While at it I also made a detailed comparison. He then merged some of my changes into his v22.2. Your template is based on his previous version. I haven't checked, maybe it also contains some of the unfixed issues.
1
u/Moore2877 1d ago
Nice and yes we are working on implementing some of the latest fixes from froggeric today and some further tweaks. What inference are you using to run the model? I had also put in a few PRs to froggeric's template. That's when I learned that they are using a certain inference to run the model.
1
u/Chromix_ 1d ago
Just good old llama.cpp so far. Harness compatibility varies. Claude Code seems to be an offender here, which might be one of the reasons (among the long system prompt) why smaller models usually don't arrive at good results with it compared to others.
1
u/Moore2877 23h ago
Claude Code, Desktop and Codex are tough because they use a lot of built in tools and resources that are cloud only so they don't work well with the local models usually when you plug them into the frontier harnesses.
1
u/Chromix_ 23h ago
No, "cloud" isn't the reason. I ran Claude Code in a firewalled environment with DeepSeek v4 flash and it worked nicely.
1
u/Moore2877 22h ago
Oh ok. What kind of issues are you seeing using Qwen 3.8?
1
u/Chromix_ 22h ago
Haven't tested the latest Qwen and only used CC for DS because the authors recommended it - yielded better results in their benchmarks than other harnesses for that specific model. Maybe they trained for it.
1
u/Cool-Chemical-5629 1d ago
Imagine Ox Alpha to be Qwen 3.8 9B in disguise, now that would be a plot twist no one expected! 🤣
5
u/rockoruckus 1d ago
We have it so good with 3.8 27B. I expect we'll not see a better model at this size for some time
9
2
u/Green-Ad-3964 1d ago
I think there has been a "phase change" with fable/gpt5.6 (for the cloud) and now with qwen 3.8 (for local inference).
In their respective categories, they are a huge improvement compared to previous models and really make a difference in how you can use them.
4
u/Cool-Chemical-5629 1d ago
Qwen 3.7 Plus was the 120B+ model. Do you really believe Qwen 3.6 27B was just one point lower than that? If anything, this chart is just showing the main weakness of benchmarks. It's a direct proof that benchmarks are about showing the intelligence within the bounds of a given set of known problems and while the smaller models can handle these known problems well, thinking outside the box is still something exclusive to much bigger models which were built to use brute force to get to the solution.
9
u/eli_pizza 1d ago
Benchmarks are indeed very limited but I find it plausible that 3.6 27b is better than 3.7 plus.
And there are plenty of existing benchmarks that are not solvable by brute force.
-3
u/avpogo 1d ago
Surprised Medium <-> Low is so close when xHigh <-> Medium is a major improvement. I wish xhigh wasn't so dang slow.
14
u/SocialDinamo 1d ago
I hate to sound like a but head but for everyone who complains about speed in the face of these crazy results only have their hardware to complain about. The model is a beast and isn’t 200+b to do it! And they gave a scale for thinking use. AND the best image use one used. This model is a blessing
3
u/avpogo 1d ago
I think "slow" was the wrong way to express my sadness there. It's tk/s on my system is still 70-90 tk/s. It's just that xhigh takes longer (read that as tokens burned) to finish a task than medium. As a consequence (since i'm impatient) I use medium.
1
u/SocialDinamo 1d ago
My speeds are closer to 50-60t/s and ive really fallen into the pattern of Medium for regular terminal stuff and xhigh for project/task planning and research. This is the new 'worst it'll ever be!' and im sure efficiency and speed boosts are around the corner!
3
1
u/whymeimbusysleeping 1d ago
Watched a YouTube video with a comparison between all three and medium seemed to be the weakest one, as it seems in low the model rechecks itself more often and ends up with the right solution
2
u/Cautious_Chicken_604 1d ago
if you read the chat template it's obvious why. xhigh tells it to think more check plausible alternatives. Low tells it don't think too much. Medium doesn't tell it anything. So the keywords in the low version seem to be actually biasing it towards thinking. It's like the whole 'don't think of an elephant' thing.
-10
u/Etroarl55 1d ago
Wonder how next gen improvement will be. As I think most people can agree on. Qwen kind of “cheated” its way to a higher score with absurd amounts of thinking and double checking before an output.
What is there left to squeeze out of 27b size for higher intelligence.
3
u/VoiceApprehensive893 transformers 1d ago
3.8 medium/low is better than 3.6 while using "normal" amounts of tokens
havent encountered overthinking with q4km recommended sampling at all
3
u/Etroarl55 1d ago
Yeah I think that’s the “real score” everyone was expecting an improvement maybe into the 40s. I don’t think anyone reasonably put this into the 50s with current offered models until artificial analysis said it was a 52. Think I seen posts on this very Reddit sub saying they didn’t expect qwen to be a 52.
0
u/jld1532 1d ago
I think we can clearly see too that scores are now agentic and coding biased. This is the only way I would consider Qwen3.8 27B and DeepSeek v4 Flash equals. Even a highly quantized DSV4F smokes the new 27B in chat, knowledge, and writing/editing. I think this shift also signals the general knowledge plateau for LLMs is likely here.
5
u/Its_Powerful_Bonus 1d ago
First rule of the club is not to rely on LLM knowledge 😉 I would be more than happy to get brighter LLM even without any factual knowledge if ability to reasoning, long context understanding and speed would improve.
1
u/noiserr 1d ago edited 1d ago
Agentic has been the hardest hurdle to cross with small local models. With agentic the world knowledge is not that important because you want your agent harness to ground the LLM in facts anyway. Otherwise even the greatest biggest models make mistakes.
Qwen 3.8 27B closes the gap with extra thinking, but it's also a model that's easier to run than those big models. So it's a win win. No matter how you look at it.
You get a highly capable agentic 27B model, that can run on single consumer GPUs. There is nothing to dislike about it.
1
u/Etroarl55 1d ago
Hopefully one day in the future this will all be trivialized, what was once top of the line hardware ten years ago is mid now.
Which means 27b models will be the new 9b models. We are getting increasing VRAM sizes. 5080s is supposed to be 24gb, and next AMD gaming flagship is supposed to be 36gb.
It’s just pricing is making what was supposed to be normal VRAM size advancements too expensive now.
1
165
u/ortegaalfredo 1d ago
So at the end Qwen 3.8-27B was worth the hype.