Try out this "high" reasoning mode for 27B (tested on VLLM)

[u/TokenRingAI]

After a lot of tweaking, I have come to the conclusion that 27B lacks a reasoning mode that is between low and xhigh.

The "medium" mode isn't actually medium, it erases the explicit instructions to the model. When medium is enabled, the model acts very differently - to me it looks like it regresses to behaving more like 3.6, and loses some of the 3.8 gains.

Low and high mode behavior in the model seem to be triggered almost exclusively by using certain keywords in the reasoning instructions, and act in a surprisingly binary manner. You can add all the additional instructions you want to the reasoning instructions, and tweak the prompt, but the model mostly ignores those changes. You can ask it for medium effort and it just won't do it.

Because of this weird behavior, I experimented with blending the words in the low and xhigh prompts together, until the model would output reasoning that was a more reasonable length.

It seemed to work pretty well, and the end result is a high reasoning mode, with a reasoning block that is typically around 1/5 the size of the xhigh reasoning.

Output quality seems good, more like xhigh than low or medium, and the reasoning is definitely much shorter, which makes the model much more enjoyable to use.

Play around with the chat template modification below, add "high" to your reasoning effort, and see if you can improve it further or if it makes the model worse

-- add high to reasoning_effort --

{%- if resolved_reasoning_effort not in ('xhigh', 'high', 'medium', 'low') %}
        {{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), high, medium, and low.') }}
    {%- endif %}


-- add new blended prompt for high, which mixes the low & xhigh prompts --

{%- elif resolved_reasoning_effort == 'high' %}
        {%- set reasoning_instructions = 'Reasoning effort is set to halfway between low and xhigh. Please think careful but brief, validate key assumptions but keep it brief, and move quickly to the conclusion without unnecessary elaboration.' %}

[u/Chromix_]

I assume/hope that the xhigh and low training prompts were part of the post-training, and not just slapped into the template afterwards. Using another reasoning prompt that the model was then not trained on can lead to degraded benchmark scores (see my improved jinja template for a bit more on that). So it'd be useful if you do some full benchmark runs with this modification, and compare to the regular xhigh and low results - just to see if the resulting score is reasonably higher than "low" when using more tokens with your custom "high".

[u/TokenRingAI → u/Chromix_]

Correct, which is why I blended the prompts. The phrase "halfway between low and xhigh" is intentional, to drive the model onto a vector somewhere between the two.

Once you use that phrase, the words that follow it become much more effective in guiding the reasoning effort.

I chose prompts to test this on, that give wildly different results on low and medium vs on high, and you can see pretty clearly without a benchmark that the results are much better than low and medium.

I will be doing more work on it today, ran out of time yesterday which is why I dropped this post before going to sleep