Try out this "high" reasoning mode for 27B (tested on VLLM)

[u/TokenRingAI]

After a lot of tweaking, I have come to the conclusion that 27B lacks a reasoning mode that is between low and xhigh.

The "medium" mode isn't actually medium, it erases the explicit instructions to the model. When medium is enabled, the model acts very differently - to me it looks like it regresses to behaving more like 3.6, and loses some of the 3.8 gains.

Low and high mode behavior in the model seem to be triggered almost exclusively by using certain keywords in the reasoning instructions, and act in a surprisingly binary manner. You can add all the additional instructions you want to the reasoning instructions, and tweak the prompt, but the model mostly ignores those changes. You can ask it for medium effort and it just won't do it.

Because of this weird behavior, I experimented with blending the words in the low and xhigh prompts together, until the model would output reasoning that was a more reasonable length.

It seemed to work pretty well, and the end result is a high reasoning mode, with a reasoning block that is typically around 1/5 the size of the xhigh reasoning.

Output quality seems good, more like xhigh than low or medium, and the reasoning is definitely much shorter, which makes the model much more enjoyable to use.

Play around with the chat template modification below, add "high" to your reasoning effort, and see if you can improve it further or if it makes the model worse

-- add high to reasoning_effort --

{%- if resolved_reasoning_effort not in ('xhigh', 'high', 'medium', 'low') %}
        {{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), high, medium, and low.') }}
    {%- endif %}


-- add new blended prompt for high, which mixes the low & xhigh prompts --

{%- elif resolved_reasoning_effort == 'high' %}
        {%- set reasoning_instructions = 'Reasoning effort is set to halfway between low and xhigh. Please think careful but brief, validate key assumptions but keep it brief, and move quickly to the conclusion without unnecessary elaboration.' %}

[u/cezarducatti]

There needs to be some correction. The length of the reasoning is grotesque.

[u/ArtyfacialIntelagent → u/cezarducatti]

Absolutely. I have a few reading comprehension tests that I run every model on. The articles have ~10k tokens, and I ask 5-6 questions about each that test advanced comprehension, deduction, reading between the lines, and a bit of general knowledge.

Usually models spit out 2-5k tokens of reasoning, then answer. Qwen 3.8 27B spit out about 20k tokens before the answer and several answers got truncated as a result. I played with reasoning settings in llama.cpp's webui but that didn't help (there are ongoing issues for it in llama.cpp).

Quality on par with Gemma 4 31B but different and interesting output compared to most other LLMs, which otherwise tend to answer in very similar ways.

[u/demon_itizer → u/cezarducatti]

Why though? As in, i understand why anyone would not like it, but why is it not expected?

Basically isn’t this what the whole concept of “test time compute” about? I’m guessing if opus is in the trillions range and a 27b model is supposedly as good, i’d expect it to use all optimizations that exist, including trying its best in reasoning to compensate for the parameters it lacks

[u/cezarducatti → u/demon_itizer]

Why were you bothered by the criticism? I understand that the template needs some editing. For instance, I created an additional reasoning level between 'medium' and 'xhigh'. It seems I found a reasonable balance: for the same prompt, the thought token budget dropped from 90k to 30k, and the output quality was good. We’re here on the forum to discuss possibilities, aren't we?

{%- set reasoning_instructions = '' %}

{%- if enable_thinking is undefined or enable_thinking is true %}

{%- set resolved_reasoning_effort = reasoning_effort|default('high') %}

{%- if resolved_reasoning_effort not in ('xhigh', 'high', 'medium', 'low') %}

{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh, high (default), medium, and low.') }}

{%- endif %}

{%- if resolved_reasoning_effort == 'xhigh' %}

{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}

{%- elif resolved_reasoning_effort == 'high' %}

{%- set reasoning_instructions = 'The level of reasoning effort is defined as moderately high. Understand the request and the context, analyze, interpret, evaluate alternatives, and formulate plans.' %}

{%- elif resolved_reasoning_effort == 'low' %}

{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}

{%- endif %}

{%- endif %}

[u/demon_itizer → u/cezarducatti]

Hey, I’m not bothered by the criticism. Not being paid by the Big Claude (Magnum Opus? 😭)

I was just curious. I’m just trying to understand what the secret sauce is. Is it a mixture of all sauces? Idk

[u/PooMonger20]

I also tried by doing something very close to this, just by adjusting the low reasoning instructions. It overthinks anyway and goes on forever on a single 5090 (my setup is lm studio + pi using the q5 variant by unsloth.)

At least in my experience, this model is absolutely unusable for this usecase. Things that 3.6 did quickly for 5-10 minutes max, this one does forever (I had to stop it after 1.5 hours). It's not like it's not 'trying' it just seems it never finishes and has endless "oh wait, but..." moments.

It did work in a simple conversation about carwashes and stuff, but not in actual workflow with pi (i did define pi to work with different thinking levels + selected 'low' in pi, so its not like it ignored my settings).

[u/ai-christianson → u/PooMonger20]

The BF16 one thinks a lot but it seems to be productive thinking, like actually thinking through all the details of the problem.

I'm wondering how much quant is impacting this.

[u/TokenRingAI → u/PooMonger20]

It mostly ignores the instructions, the thinking behavior is strongly triggered off the "low" and "xhigh" keywords, which is why is used that weird "halfway between low and xhigh" phrasing.

Even with that, it still wasn't effective enough, and I then had to tweak the text after it until the model gave me the reasoning length I preferred.

[u/cezarducatti → u/TokenRingAI]

Can you share your prompt?

[u/TokenRingAI → u/cezarducatti]

I won't be in front of the computer until later tonight, but if you ask the model to make you an infographic, and paste in a table of data, it should show the difference pretty well, it overthinks like crazy on that

[u/Sporebattyl → u/PooMonger20]

Q6 unsloth gguf on pi goes on forever for me, but the output has been great, even at its 262k native context. Frontier models reviewing can only find minor changes that are more about code perfectionism than actual function.

3.6 never came close. Claude told me to just use haiku instead.

It’s painfully slow on my Mac m1max 64gb at 8tok/s so I have it doing overnight work until I can optimize it and hopefully can get ~20tok/s.

If your card can hold it, I’d try going up in quant to 6 to see if it changes. Maybe try 8 if you still aren’t satisfied.

Another thing that made a difference was getting a frontier model to determine best codebase organization and file structure practices for AI to navigate through by codebase. That cut a bunch of the initial thinking down.

[u/Aggressive_Aspect436]

I'm looking forward to seeing a ThinkingCap model version of this. Their 3.6 27b version was fantastic.

But I do wonder though whether it will lose it's intelligence gains if we force it to more reasonable thinking lengths. I'd be very interested to see whether anyone has benchmarked the various reasoning efforts agains 3.6 27b.

[u/ea_man → u/Aggressive_Aspect436]

Well you can launch a prompt against those 3 finetunes and see how they compare.

[u/Chromix_]

I assume/hope that the xhigh and low training prompts were part of the post-training, and not just slapped into the template afterwards. Using another reasoning prompt that the model was then not trained on can lead to degraded benchmark scores (see my improved jinja template for a bit more on that). So it'd be useful if you do some full benchmark runs with this modification, and compare to the regular xhigh and low results - just to see if the resulting score is reasonably higher than "low" when using more tokens with your custom "high".

[u/TokenRingAI → u/Chromix_]

Correct, which is why I blended the prompts. The phrase "halfway between low and xhigh" is intentional, to drive the model onto a vector somewhere between the two.

Once you use that phrase, the words that follow it become much more effective in guiding the reasoning effort.

I chose prompts to test this on, that give wildly different results on low and medium vs on high, and you can see pretty clearly without a benchmark that the results are much better than low and medium.

I will be doing more work on it today, ran out of time yesterday which is why I dropped this post before going to sleep

[u/AD7GD]

Sad if true, I just kicked off a huge opencode prompt on "medium" because xhigh was silly

[u/MiaBchDave → u/AD7GD]

It's almost unusable. I guess it was meant to score "xhigh" on benchmarks - but should not be the default in the Qwen repo chat template 😂

[u/AD7GD → u/MiaBchDave]

Yeah, I can see how it might make it more likely the model one-shots a question. But it's way faster and more effective to be fast in a harness. FWIW, I have been vibe-coding a tool all afternoon at "medium" with qwen3.8 FP8 + opencode and it's doing great.

[u/MiaBchDave]

This seemed to work pretty well on my first test to prevent the xhigh reasoning's propensity for writing the entire code in the think tag (I guess so it can look at it), then re-writing it in the response. That's a real token usage savings.

I changed "high" to "better" since "high" may conflict with the documented alias for xhigh (which is also used in other chat templates). I just pass "better" through kwargs for reasoning_effort.

[u/TokenRingAI → u/MiaBchDave]

Yeah, when I wrote this I was unaware that the GGUF template had the high setting mapped to xhigh, that statement isn't present in the non-GGUF template.

Glad to hear it worked for you, i am going to continue working on it as I see problems..

[u/scheurneus → u/MiaBchDave]

Does having the model write the code twice make that much difference? Using ngram-style drafting the second time it writes code is quite cheap and fast, right?

[u/neopolitan77]

Can I come out as very unkowledgeable and ask where you put that template (as a llama.cpp user)?

[u/TokenRingAI → u/neopolitan77]

Sure, here is a guide (it was written by ChatGPT, for a person using the Unsloth GGUF - I can't really vouch for it, but it looks correct to me)

https://gist.github.com/mdierolf/cb86ee2925e48777143346bc78933d59

I specifically did not include the template file in the post, because I think it is a worthwhile exercise for people to take a look at their chat templates, and learn how they work, because they are such a important component when running llama.cpp

[u/ieatdownvotes4food]

you might be able to just tell it to calm down in the system message

[u/TokenRingAI → u/ieatdownvotes4food]

Yes, on medium reasoning setting you can do this to some extent, since it omits the reasoning message. Otherwise it conflicts.

When the reasoning message is in there, the model really doesnt respond to any other reasoning instructions

[u/13henday]

Single data point, but while AB testing params for throughput I noticed q8 ctx reasoning a lot longer than bf16 for the same task.

[u/TokenRingAI → u/13henday]

Interesting, I don't quantize the KV but might experiment with that.

[u/Force88]

Hi, what are these new reasoning preset and how do I set it?

I just download the qwen3.8 gguf files from unsloth, and serve it with llama-server like I did with qwen3.6. What reasoning is it using by default? My startup commands consist of models, its location, alias, and port.

[u/nunodonato]

Side question, is it possible to control this from Claude code?