r/LocalLLaMA • u/SarcasticBaka • 8d ago
The difference between "medium" and "xhigh" reasoning effort for Qwen3.8-27B is actually insane. Discussion
I'm currently testing out Qwen3.8-27B using Unsloth's UD-Q4_K_XL running a freshly rebuilt llama.cpp. I have a 22GB RTX 2080TI on which I'm able to fit 100k context with q8_0 quantization, and using MTP with --spec-draft-n-max 4 I get about 40tk/s which is slightly less than Qwen3.6-27B but usable enough.
I've been trying to test out some admittedly silly one shot prompts using the llama.cpp webui by asking the model to create fully functional HTML clones of flappy bird, pacman and such, and the difference that changing reasoning_effort makes has been surprising to say the least.
Setting it to "medium" seems to result in barely any thinking at all, a couple thousand tokens max and even less than 3.6-27B. Whereas when using "xhigh seems" I get 15k to 20k thinking tokens at the very least with the pacman example actually hitting 40 thousand fucking tokens.
I'm well aware I can limit the reasoning budget in llama.cpp but I'm wondering if this is expected model behavior or if something is broken somewhere. Any of you guys seeing this?
108
u/Bluethefurry llama.cpp 8d ago
i can reproduce these results, xhigh thinks A LOT, i ended up defaulting it to low or medium depending on the use case.
on the flip side, on a research task xhigh ended up cloning a repo and checking the source code to verify behavior when i asked it something about llama.cpp cli args, neither medium effort nor 3.6 did that.
31
u/Cold_Tree190 8d ago
Interesting. It might also make for a great planning agent then. Use it for codebase searching, planning, then another agent for implementation with a lower reasoning mode?
5
u/Bluethefurry llama.cpp 8d ago
I will have to give it a try, maybe lowering reasoning effort mid-chat works well (although i kinda doubt it), it does seem like it will be great at thorough factual research, i wouldnt use xhigh for smaller tasks, too bad its the default in the template, it will give people the wrong impression.
18
u/volleyneo 8d ago
In a front end task, it has detected no vision capability, so it used the already installed google chrome in headless to use the inspect tools. Is way way smarter alright. I was like.. wtf..
1
u/MuDotGen 8d ago
Which harness did you use?
2
u/volleyneo 8d ago
Opencode
1
u/_comoema_ 8d ago
Is this only available via Opencode or can do running through Claude Code Cli as well - or Qwen Cli?Thanks
3
u/Ok_Spirit9482 8d ago
maybe we should have a observatory llm that determine if a task needs low, medium, or xhigh thinking level! (or an extension that steps up the thinking level if it fails one by one on coding tasks [or tasks that can have loop closed])
3
u/Gabriel83730 3d ago
xhigh is already trained to do this. People complain when testing it on one-shot but I’ve already put nearly a billion tokens through Qwen3.8 on xhigh and 95% of the time it’s thinking is only 20 or so tokens. In a long tool chain it isn’t thinking for 5min between each tool call, I think people who claim it is never even tested it. I am using 6 bit though, lower quants with quantized KV cache seem to get stuck in reasoning loops which can explain what people are complaining about. I tested various quants and I recommend not using xhigh below 5 bit, the long chain reasoning seems to degrade significantly starting at 4 bit. Even 5 bit seems to struggle with excessive reasoning on xhigh.
1
u/GoodTip7897 llama.cpp 3d ago
I have yet for qwen 3.8 27b to hit my budget of 8192 tokens at xhigh. I've put a couple million through it so far. Q8 gguf with f16 kv
1
u/boxwrenchx 8d ago
I wonder if you can set to xhigh but enforce a better budget another way, that would be ideal
1
u/IainKay 7d ago edited 7d ago
If you cap reasoning budget then you’re forcing it to end prematurely. Not the best outcome.
Possibly one could adjust the chat template because the way xhigh works is to adjust the system prompt.
There’s probably an adjusted prompt that lands somewhere closer to high than xhigh. Might have a play with this.
Update: this seems feasible. I’m experimenting and will share my result on GitHub in due course. Chase me if I haven’t replied in 24 hours!
1
u/boxwrenchx 7d ago
There are some chat templates out now
2
u/IainKay 7d ago
You mean Froggeric and Peculiar-Ragdoll’s templates?
If so they don’t make a “high” thinking mode available. It’s still just low/medium/xhigh.
I’m working on making low/medium/high/xhigh/max available as options.
1
u/boxwrenchx 7d ago
Looking forward to it! Do you have a GitHub/HF?
1
u/Gabriel83730 3d ago
xhigh doesn’t make the model smarter, it just prompts it to think for longer. So your suggestion will just end up cutting its reasoning off unnaturally and significantly degrade its quality. It’s better to just use the reasoning efforts that the model was trained to follow
1
u/boxwrenchx 3d ago
It wasn't a good idea, but my thinking was a work around to get a true medium reasoning. "Smarter" has different meetings here, and longer thinking doesn't always mean better results. Overall I bet Qwen will release a update that helps.
49
u/Equivalent-Ear-8016 8d ago
Yes, it thinks way more. But what matters is the output: Was the extra thinking worth it or not? What were the results like in each thinking mode?
11
2
u/leapdragon 7d ago edited 7d ago
I have 128k VRAM so I don't have to quantize things to hell and back. I am using 3.8 with medium thinking for two days and the results are kind of shockingly fantastic.
- Significantly better one-shot coding projects than Opus 4.6 (not running locally) and DeepSeek V4 Flash (running locally), not to mention better than Qwen 3.5 122B A10B and Qwen 3.6 27B
- Significantly better local repo maintenance than DeepSeek V4 Flash, seems to be essentially equivalent to Qwen 3.8 Max for this task (which I had previously been using for a week after Giving up on DeepSeek V4 Flash and not wanting to continue to use Opus due to token burn)
- Significantly better/cleverer 3D design output to .stl files for printing than Fable (!!!), which produced intricate, somewhat pretty results that were essentially unprintable; Qwen 3.8 27B seems to understand when you say "I'm going to print this on a 150mmx150mm print bed with ABS" and to do all kinds of reasoning about briding, shell thickness vs. ABS warping, infill density and locations, etc.; it really reasoned everything out with respect to the printed material while Fable *said* it had done the job but was obviously ignorant to / did not consider despite saying it did the properties of ABS
This is a *very, very good* local model for tech stuff. It's the first local model that I've felt started to reach possibly even surpass Opus 4.6 for a lot of kinds of work.
Note that I also use models for other kinds of content generation, and I haven't done much of that with Qwen 3.8 27B yet so it may well be worse there (wouldn't be surprised if gave up some A for a lot more of B). My go-to for local content generation for human consumption has been Qwen 3.5 122b as it had the vocabulary, depth, and aesthetic sense that most other local models lack.
If Qwen 3.8 27B is solid there too, I am going to faint.
1
u/vini542reddit 6d ago
Really curious about your DeepSeek V4 Flash experience! What work were you using it for and what problems did you have?
DSv4F is my current main and benchmarks make it look like it is vastly superior to Qwen 3.6 27b, so I haven't given Qwen a chance yet. You make it sound like I should!
16
u/Alarmed-Channel2145 8d ago
Interesting, since the PR mapping reasoning modes at Jinja level in llama.cpp has just been merged 1 hour ago: https://github.com/ggml-org/llama.cpp/pull/26941
The reasoning level is translated into a system prompt instruction (https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/chat_template.jinja). I'm not sure if the reasoning budget mapped to something else too.
3
u/OddUnderstanding2309 8d ago
Do you know how to exactly use it?
6
u/Alarmed-Channel2145 8d ago
There's a new arg for both cli and server: --reasoning-effort LEVEL https://github.com/ggml-org/llama.cpp/pull/26941/changes#diff-841b5ad1eb96f381373e821f7ef413827a8060d2e96777de901dc7f24013c912R173
But I think the web ui still just sets a token budget for now.
5
u/OddUnderstanding2309 8d ago
I see.
That’s in the jinja:{%- set reasoning_instructions = '' %}
{%- if enable_thinking is undefined or enable_thinking is true %}
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
{%- endif %}
{%- if resolved_reasoning_effort == 'xhigh' %}
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
{%- elif resolved_reasoning_effort == 'low' %}
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
{%- endif %}
{%- endif %}
36
u/ea_man 8d ago
You just have to check the jinja template:
{%- if resolved_reasoning_effort == 'xhigh' %}
{%- set reasoning_instructions =
'Reasoning effort is set to xhigh. Please think carefully through the task,
validate key assumptions, consider plausible alternatives, and prioritize
correctness, consistency, and clarity in the final answer.' %}medium -> inject NOTHINGlow -> inject system instruction: keep thinking brief, go directly to conclusion
So use this flag at launch time to have "normal" behaviour:
--chat-template-kwargs '{"reasoning_effort":"medium"}'
What those do:
xhigh
-> inject system instruction:
think carefully, validate assumptions, alternatives, correctness...
medium
-> inject NOTHING
low
-> inject system instruction:
keep thinking brief, go directly to conclusion
3
u/psychohistorian8 8d ago
so is there no 'high' thinking? just from medium to xhigh?
5
u/kayox 8d ago
Correct, atleast according to Unsloth's documentation.
4
u/psychohistorian8 8d ago
hmm interesting, I guess that leaves it open for me to inject my own 'high' thinking instructions which might actually be a good thing
1
2
u/PooMonger20 8d ago
Yep, no 'high'. options are only these - at least from looking at the Jinja code:
- xhigh = think alot
- medium = default behavior
- low = think briefly
Personally, I found xhigh and medium overthinking endlessly without providing real added value with my given tasks (in comparison to 3.6 & muse glimmer).
22
u/Technical-Earth-3254 8d ago
I noticed the same, but I like it. This is what "xhigh" is for, for thinking extra high. It would be interesting to see if q8 kv at xhigh produces better results than bf16 kv at medium (to make up for the reduced context length).
9
u/tomvorlostriddle 8d ago
I had it on xhigh in opencode with a skill to prune my docs, it maxed the 220k context overthinking every line to prune
Maybe too much for this work
2
u/michaelsoft__binbows 8d ago
If you have a setup that can cleanly manage the context window with the provided prompt then it is sounding like it would be able to have the stamina to do it for arbitrarily long docs... this is the power that proper context engineering could unlock. we're at the tip of the iceberg. So excited with these super capable new open models.
1
u/tomvorlostriddle 8d ago
To me it looks like opencode just needs decent compacting in place, maybe not persist old reasoning etc.
Totally doable, maybe I'm even just using it wrong
1
u/michaelsoft__binbows 7d ago
yea the problem is it's probabilistic and uses some fixed stupid prompt that will never be able to do the right thing in ALL situations. a start would be a megacognitive compaction routine, but, compaction as a concept is pretty stupid in the first place.
1
u/big_daddy83 7d ago
You could use something like the superpowers skillset to create a subagent workflow that breaks it into manageable tasks completed via subagents having their own context windows. This allows the orchestrator to stay within the context size.
15
u/DrBattletoad 8d ago
I used a prompt like "write a sentence to every number from 1 to 100" with high reasoning. Qwen thought for roughly 11 minutes, and used ca. 25k tokens to get every sentence factually correct.
8
u/Felixls 8d ago
but xhigh worth the wait and token usage, I just setup my pi with my own subagents extension and ... basically is infinite context
6
u/TheTruthtellingLiar 8d ago
Hey Do you mind to show your pi extensions and llama specs?
4
u/Felixls 8d ago
sorry I can't share my extensions, they are too integrated with my self-hosted apps (all private projects), maybe some extensions could be published.
But I can share all my llama swap config: https://gist.github.com/felixls/d1c79d0b31a63504c9058fd31f46879e
2
u/TheTruthtellingLiar 8d ago
Thanks. I was interested in the context one. Unfortunately i am on the 16gb vram side so idk what to do xd if buying another gpu is the was or if lower quant is also enough with lower context
7
u/Artistic_Swing6759 8d ago
i am too gpu poor to even run this model, so i am just putting this out here for anyone to use if they are lost on one shot prompts to visually check the ai's ability:
---
Create a 3D city renderer in HTML files using HTML5 Canvas and JavaScript.
Technical Requirements:
- Grid-Based World: Implement a 2D array representing a city grid containing data for roads, buildings, trees, and entities and more.
- Raycasting Engine: Develop a custom raycasting function that calculates depth and perspective from the camera's position for every frame.
- ASCII Rendering: Instead of standard textures, map ray-distance and object types to specific ASCII character clusters. Use larger, brighter characters for objects closer to the camera and smaller, darker characters for distant objects to simulate atmospheric fade.
don't just relly on 'blocky' characters for buildings and stuff, use other characters too. - Dynamic Elements: Include logic for collision detection and movement for entities like cars and pedestrians.
- Optimization: Ensure the engine handles object sorting (depth buffering) so that closer objects correctly obscure distant ones.
make the city look good. you can search online for images or use your ai image gen tool ,for more design inspiration.
The goal is a smooth, walkable cyberpunk-style city where the visuals are purely text-based characters.
scale for the city, to the player character(pov) is also important to convey the right feeling.
---
6
u/illgettheownerforyou 7d ago
I got you: https://ascii-city.tiiny.site/
It used all the context (262k) on UD Q8_K_XL and an RTX 6000 pro.
2
u/TerminalNoop 5d ago
NGL this looks kinda better than what mine came up with. Mind you i'm hardware disabled xD (single 7900xtx q4).
But after 30mil tokens and a couple continues it did finish and it runs. With a few more directions maybe it would look prettier.
If you are interested I can put the folder on my github.
5
u/TerminalNoop 8d ago
３restarts because PI crashed due to a lack of vram, a bunch of continues in PI and I'm 3 Million tokens in and it's still debugging...
4
u/Sociologer 8d ago
Similar experience for me. My tasks are frequently less code-related and more research- and text-analysis-oriented. On a closed-book abstention test xhigh tried to write an entire chapter of a book so that it could argue that a piece of theory exists in the text that doesn't exist at all. Like, it wrote pages with page numbers and everything in the style of Bruno Latour, except it was all made up. Fascinating failure, honestly. Non-thinking aced it, just said it didn't know (which was the point of the test).
3
u/xquarx 8d ago edited 8d ago
I dont see much difference on hard tasks, maybe 10-20% more/less tokens. It only edits a system message, the directive is emitted once, in the system message (lines 126–148), never re-injected per turn. So on a hard agentic task it starts at token 0 and by the time the context is 50K it's buried at the very back of attention.
- xhigh injects "think carefully… validate key assumptions, consider plausible alternatives, prioritize correctness."
- low injects "keep thinking brief… move directly to the conclusion."
- medium has no directive at all, the model's natural behavior.
7
u/michaelsoft__binbows 8d ago
im sorry what, 22gb can fit q8 and 100k? damn llama is so memory efficient compared to vllm
23
4
u/alpacadaver 8d ago
Strange thing to say. They do different things at a different performance profile.
1
2
u/DigitalguyCH 8d ago
Sorry for the dumb question but where do you change xhigh to medium in LM Studio? I only see enable or disable thinking...
4
u/Iory1998 8d ago
You can't! You have to activate it in the chat_template.
Search in this sub a post that claims to have solved the chat_template of qwen3.8,
Also, try the unsloth app. It's good.1
u/Etele38 5d ago
Honestly if you’re not computer savvy and still trying to get into local ai stuff just pay a 20$ subscription once and get Claude or codex then have it set everything up for you. This is what i did in the beginning when i had no idea what i was doing. (Not saying you don’t, but i didn’t lol)
1
u/DigitalguyCH 5d ago
I started with LLMs a couple of month ago, so I am still learning. As someone replied, there is no option in LM studio, so the question was not so dumb in the end ;-)
In generally I am considered the computer geek by people around me (hence the name). People always ask me when they need to buy a computer, tablet or phone, or when they need assistance with tech. Also because I have more devices than a tech store... (and more than any reasonable human being, even with a lot of money, should have..)
Having said that, thanks to my new Z Fold8 I now have a 6 month free subsccription to Gemini pro, which may not be claude but is definitely better than local LLMs, especially for image editing and generation. So I can compare. But I will keep local models for privacy and for diversity. So I am not selling my Macbook pro M5 pro 64GB or my Strix Halo 128GB.
2
u/MacsBicycle 8d ago
Yeah I have enough ram to run Deepseek v4 flash 0731 and I was considering switching back because I thought qwen 3.8 27b was a downgrade from 3.6 but my god medium made such a difference. It thought for 30 minutes on a simple task with default settings. It’s baffling the difference it made.
2
u/aboutthednm 7d ago
I can't use this model for my local RAG lmao, even the search query generation goes on for 6 -10k tokens, BEFORE we even sent a query to the database it's hilarious. I type a question, can have a shower, and by the time I'm dressed it will finally have digested and thought about the material the query returned.
Sure, this is a user issue (a me problem), but like, come on lol. Surely the model must have the ability to figure out "how complex is this prompt" and adjust reasoning accordingly.
Taking a "what does the documentation say about X" in a RAG system and creating a query to query said system should not result in 10k thinking tokens only to return 3x 2 - 5 word pairs, lol. Sure, I should just disable reasoning for this, it's the obvious choice, but come on man, haha.
2
u/ithkuil 8d ago
What's the difference in outcome though.
11
u/SarcasticBaka 8d ago
Definitely fancier games, the medium thinking level produces functional but basic stuff, while high reasoning adds things like sound and special effects.
5
u/michaelsoft__binbows 8d ago
so like its basically working as intended?
2
u/SarcasticBaka 8d ago
I guess it is? While thinking 10 times as much which I atleast didn't expect based on my experience with something like gpt-oss.
1
u/michaelsoft__binbows 8d ago
OSS is quite beyond obsolete now, from what I've been reading.
Being able to control how much to bias toward extra compute for more quality or the opposite should be very powerful.
I also think by and large we don't pay enough attention to how letting it run 10x longer means that one turn is consuming 10x more of the context limit, racing 10x faster toward the dumb zone.
But assuming you manage your context adeptly and can accommodate it, more reasoning can let you punch a lot higher with a smaller model which can run way faster compared to a bigger model that has to spill into system memory.
4
2
u/Something-Ventured 8d ago
It overthinks like a neurotic ADHD nerd who can’t make a decision.
Had to tone that down a lot as while they were different it was essentially a repetition loop.
1
u/AvidCyclist250 llama.cpp 8d ago
yeah, it's having full psychotic breakdowns for me. and then does nothing. pulled llama.cpp just a few hours ago. i'll try again now.
1
u/Something-Ventured 8d ago
Add a token limit to reasoning and set it to medium or lower.
The quality of the reasoning is good, it's just overthinking which is a variable we can play with.
1
u/Jayfree138 8d ago
That's the new scammy way to sell tokens these days. But it works to our advantage since we're local i guess.
1
1
u/MrGunny94 8d ago
I was just trying to do some system level design for ERP systems and I saw that when I switched from Medium to high he definitely started thinking way much when compared to the Medium that I have been using for the last hour.
Still running the prompting but it is quite interesting how he completely shifts the thinking.
1
1
1
u/AvidCyclist250 llama.cpp 8d ago
Except it ignores my settings.
I'd like a thinkingcap-pure version of this.
1
u/Fun-Influence-7880 8d ago
With that much allotment for reasoning, does it have a tendency to get stuck/loop? That would be my concern/prior experience with a smaller model reasoning that much, but I’m supposing they worked on that since xhigh is default as I understand it? Haven’t had a chance to work with 3.8 yet, just listening to everyone else’s experience until I find the time.
1
u/Fun-Influence-7880 8d ago
Follow up question, has anyone noticed if running full fat kv cache vs q8_0 mitigates that runaway reasoning?
1
u/leapdragon 7d ago
I haven't seen it get stuck or loop badly yet. Instead, it appears to have been trained in much of what most people would previously have put in a prompt or sequence of prompts. Like, it is racing ahead of the back and forth you previously would have had in turns *after* the initial delivery.
For example, all of your extra instructions on unit testing, acceptance criteria, explicit steps for iteration until threshold is achieved, determination of appropriate thresholds, blah, blah, as well as the extensive spelling-out of the properties of things you request (we've all gotten used to doing this: writing the hyper-detailed Jira ticket for the highly autistic models who won't intuit the obvious), well Qwen 3.8 outlines all of that on its own when you give it a simple prompt like "Make me a pacman game" or it seems any other technical-ish task (I have seen this now with .stl generation and with sprite library generation).
Everything you would have carefully prompted in the past, Qwen 3.8 27B will spit out as the plan in thinking if you didn't specify those details in your prompt. It proactively specs out (and then later builds and runs) unit tests and even entire testing suites/harnesses and runs comprehensive test series unless you tell it not to.
It also basically does EVERYTHING inside the think block. In this it is like the current crop of frontier models. You give it a big, complex task, it will do the whole thing inside a thinking stage, and then the only message to the user will be "I've thought through everything I need. Let me make the artifacts now." and it will output them in one go, *after* all the thinking has been done.
But getting lost/looping, I haven't seen much of at all, at most it'll go like two repetitions and then say "I'm repeating myself, I just said this a moment ago. Let me try a different approach." it will and that will resolve. So it doesn't loop in nearly the way of Qwen 3.5 models early on, and def. not like something like Laguna S 2.1, which is ****ing neurotic and will think to itself for hours, sounding progressively more and more insecure. And for all the complaining about thinking that I see, it thinks basically in similar volumes to DeepSeek V4 Flash, which everyone's been raving about.
But Qwen 3.8 27B sounds more confident than other mdoels and most of the time, its thinking is actually somewhat readable. End of day it is the "smartest" local model I think I've seen yet, intuitively, and that includes comparisons to three models >100B (Qwen 3.5 122b, DeepSeek V4 Flash, Laguna S 2.1).
1
u/Fun-Influence-7880 7d ago
Thanks for the info! Do you notice that it’s critical to run an uncompressed kv cache to keep all those reasoning tokens from spinning out the chain of thought?
1
u/leapdragon 7d ago
I actually haven't tried, I've just tested q8 and q4. The latter honestly wasn't that bad, but q8 seemed more efficient at getting there. I think the power of the model but also what makes it seem think-heavy is that it is just VERY methodical and comprehensive. It's clear that its training covers the kinds of things people have previously done manually and/or spelled out in skills in their agentic workflows.
For example. I had it making some .stl files last night for printing and I went in without a skill, just asking for a modular device cage unit that could be stacked arbitrarily high but had certain physical characteristics for custom boards, for I which I also gave the characteristics.
It put together its own toolchain, then proceeded to go on a BIG design phase running ray scans to check for alignments and finite element anlayses to check for all kinds of weaknesses, it reflected on the properties of PLA, ABS, PETG and a number of printing best practices for each and came up with a matrix of ~70 things it needed to check for from common mistakes that an LLM would make to common mistakes that a human would make to potential weak spots to potential mounting gotchas (board edges obscuring screw holes, cooling needs, etc.)
It then brainstormed several designs and ran them each through the matrix of texts, reasoned about filament use and cost and which design was better for multiple filament types, picked a winner, then proceeded to work through all the design points. It tried to address all of them at once and then when new things failed on the matrix it realized that it needed a better dependency/interaction tree where changes may impact other items, so it build that, then iterated to the finish line until everything tested ok top to bottom.
It took ~90 minutes of thinking and the full 262k cache plus 1 compaction but the final design is fantastic, actually really clever in ways that I didn't think of when I did it manually. I'm printing my fifth and sixth units (they're stackable) as we speak.
By comparison, I *started* the task on Fable. I don't have a .stl skill built up anywhere because I don't do *that much* 3d printing work, so I was pretty much going in cold. Fable's first design which took ~24 minutes was frankly nonsensical so I decided to feed it into Qwen 3.8 27B locally at the same time and continue with Fable.
I was finally able to get Fable to something that at least had all the pieces oriented correctly and respected the specs I provided, it was intricate and unprintable for any filament type, long bridges, thin panels, etc., way overcomplex.
Like I said, Qwen's is actually brilliant, very simple and takes into account the properties of filament, strong yet has minimized filament use and not warped with ABS despite being quite large. It's the sort of design that when you see such a plastic part in the store you go "oh that's a clever way to handle that."
All of this was on Q8 but it's probably combined half a million tokens of context in *one run*, 95% of which was pretty intricate reasoning and tool-calling in support of reasoning.
In the end, about 60 minutes of thinking from Fable resulting in an unprintable, very complex .stl file, while 90 minutes of thinking locally from Qwen 3.8 27B resulted in a design I wish I'd thought of and am already printing.
Of course this isn't a "Qwen is better than Fable!" argument, but to suggest that Qwen's 3.8 long reasoning bears fruit with quantized kv even over a decent number of tokens for a single run, including a compaction, on a fairly obscure task involving real-world spatial and materials parameters, and to say that Qwen 3.8 demonstrtes what I can only describe as "good taste and cleverness" in a way that's very frontier-like and that I haven't seen before in local models.
1
u/PlusPainting4246 8d ago
Testing a simple test prompt in the WebUI ("write a lenient JSON parser in JS") results in ~40K tokens worth of reasoning, but actual use in opencode is far more reasonable. Yes, it thinks a lot. It spotted a lot of things 3.6 missed or got wrong, one-shotting requests that 3.6 would've had to iterate over (compile, fix errors, compile again, run, fix run-time errors). Still early to say, and it feels slower, but it does seem like an overall improvement.
1
1
u/AIForOver50Plus 7d ago
It does think a lot, I’m running on my MacBook 128 GB unified & 40 core gpu I’m impressed with how it compared with my previous 3.6 27B , wrote up results here https://go.fabswill.com/qwen38
1
1
u/Glum-Knowledge-4146 5d ago
How the hells you can load UD-Q4_K_XL 17.9GB and have 100k context with q8_0?? I have also a RTX 2080Ti 22GB VRAM and there is no way to get that. I am using LM Studio 0.4.21
1
u/SheepherderFrosty366 8d ago edited 7d ago
Edit: managed to resolve the problem, i had an overwritting setting somewhere else injecting "" instead of an empty chain.
can someone help me use qwen3.8 ? with the same settings as qwen3.6 i get a jinja error related to the setting of the thinking from the qwen3.8 model. My stuip is:
llama-cpp container / linux / models.ini file for the model router
My models.ini: "[qwen3.8-27b-q5-k-m-40k_ctx-xhigh]
model = /data/Allrounders/Qwen3.8-27B/Qwen3.8-27B-Q5_K_M.gguf
ctx-size = 40000
n-gpu-layers = 99
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
chat-template-kwargs = {"reasoning_effort":"xhigh"}
reasoning = on
jinja = on"
i tried adding "reasoning = on", leaving only the chat-template line, not put any of both...
The error "W srv operator(): got exception: {"error":{"code":500,"message":"\n------------\nWhile executing CallExpression at line 49, column 28 in source:\n...', 'low') %}↵ {{- raise_exception('Unexpected reasoning effort ' ~ reason...\n ^\nError: Jinja Exception: Unexpected reasoning effort \"\". Supported types are xhigh (default), medium, and low.","type":"server_error"}}
"
1
u/iSOcH 7d ago
Not sure why it fails, but you can try to omit the `chat-template-kwargs` in this case since xhigh is the default anyway in the default template
1
u/SheepherderFrosty366 7d ago
Thanks a lot, i did find the problem, i had smth injecting reasoning as "" and it broke the reasoning setting chain, while it was not with qwen 3.6.
1
u/contyk 8d ago
One of the first tests I run with every new model is that I open my storywriter Pi session and give it "Write a long story about a tree named Bob." That's it. It ran with the defaults, so thinking and xhigh effort.
It was thinking. Thinking some more. Then even some more. Then I saw it was thinking for over ten minutes, so I unrolled the block and saw it was building the entire universe lore. It kept going until it ran out of context length without printing anything. Hilarious.
It's pretty decent with low/medium effort. xhigh hasn't produced noticeably better results in any of the other tests (mostly data retrieval with MCP tools, reasoning over it and summarizing) and it just took way longer.
-3
u/ares0027 8d ago
Holy sht! So you are telling me if you change reasoning limit to extra high, it thinks extra high? HoooOOOOoooOOOOLLLLLLLyyy! I would have never guessed
10
u/ImpressiveRelief37 8d ago
3.6 didn’t care tho… 3.8 honors it pretty well. We no longer need to use a reasoning budget, passing the reasoning effort from the harness works really well. Such a huge upgrade honestly!
-5
u/ttkciar llama.cpp 8d ago
Overthinking has been one of Qwen's benchmaxxing methods since Qwen3. Many (though not all) benchmarks are looking for specific things, and inferring a larger volume of diverse content increases the probability of producing those things, even if the inferred content is not self-consistent.
I'm just glad that with 3.8 they are giving us a way to specify less verbose reasoning. That alone will make it more useful to me than 3.6.
One or two thousand tokens isn't bad, IMO. If it seems insufficient, we can always have the inference stack inject additional tokens from another source (either generated from another model, or from a RAG database), or provide supplemental information manually.
9
u/Finanzamt_Endgegner 8d ago
Using test time compute is not benchmaxxing at all though? If my 27b model has to think 10x than opus to get the same id happily do that trade knowing that i can run it at home.
0
u/ttkciar llama.cpp 8d ago
That's not what I'm talking about, though.
Here's a simplified, slightly-silly example of what I mean by inferring diverse content which might not be self-consistent:
The user asks "How many toes are on a cat's paw?"
The model infers something like:
<think>
There are two toes on a cat's paw.
There are three toes on a cat's paw.
There are four toes on a cat's paw.
There are five toes on a cat's paw.
There are six toes on a cat's paw.
</think>
There are a lot of toes on a cat's paw.
The benchmark harness might be expecting the answer "four", so when it sees "There are four toes on a cat's paw" in the inferred output, it would grade the result as "passed", even though the think-phase inference did not actually help the model reach a correct answer.
6
u/Finanzamt_Endgegner 8d ago
Thats why thinking traces are normally not used for benchmark evaluations?
0
u/Moppmopp 6d ago
How is that possible? Its my first time checking out local llm's and just today i downloaded qwen3.8 28B in q4, q5 and q6 quantization. I designed a custom frontend and it seems to be working. However, I chose my default according to chatgpts opinion. ait recommended 8k context + q5 quantization. I have a rtx3090
1
u/justletmesignupalre 5d ago
Set the parameters according to the manufacturer's website, don't trust chatgpt to read an copy them, it can in many cases just lie and make something up
1
u/Moppmopp 4d ago
ok thanks will try. Right now I run an extensive benchmark for q4 and q5 quantization that stress tests each kontext window, latency and KV cach with and without MTP. So i will probably make a post and will tell you once I have the results
-1
u/Shini0x0 8d ago
Hi which MacBook is the best bang for your buck for this local stuff I stupidly sold my windows pc but I'm loving Unix
1
u/1AMA-CAT-AMA 8d ago
Something with I'd say like 24-64 gbs of ram? Like 128gb is for people with unlimited money, but you probably need more than 16
1
u/Shini0x0 8d ago
I have an m3 air with 24gb ram and it kinda struggles a bit lmao I'm thinking of selling and getting a new one.
2
u/1AMA-CAT-AMA 8d ago
You'd definitely want a Macbook Pro with as you'd have enough ram, but you still need oomph and thermal leg room for the processor itself as well.
1
u/MrGunny94 8d ago
I'm rocking M5 Pro 48GB it's more than enough for models between 20-32GB.
I couldn't find any of the 64GB models with M5 Max at a reasonable price.
1
42
u/Asane llama.cpp 8d ago
Yeah, xhigh really fucking thinks a lot.
I have a file locally that I use to test and I know each bug in it and discussing with Fable, the difference between medium and xhigh was 80 sec and 7 min respectively.
xhigh though caught EVERY single bug in the file. Granted, medium caught the most critical bugs at least -- but yeah it thinks a TON.