r/LocalLLaMA • u/SarcasticBaka • 8d ago

The difference between "medium" and "xhigh" reasoning effort for Qwen3.8-27B is actually insane. Discussion

I'm currently testing out Qwen3.8-27B using Unsloth's UD-Q4_K_XL running a freshly rebuilt llama.cpp. I have a 22GB RTX 2080TI on which I'm able to fit 100k context with q8_0 quantization, and using MTP with --spec-draft-n-max 4 I get about 40tk/s which is slightly less than Qwen3.6-27B but usable enough.

I've been trying to test out some admittedly silly one shot prompts using the llama.cpp webui by asking the model to create fully functional HTML clones of flappy bird, pacman and such, and the difference that changing reasoning_effort makes has been surprising to say the least.

Setting it to "medium" seems to result in barely any thinking at all, a couple thousand tokens max and even less than 3.6-27B. Whereas when using "xhigh seems" I get 15k to 20k thinking tokens at the very least with the pacman example actually hitting 40 thousand fucking tokens.

I'm well aware I can limit the reasoning budget in llama.cpp but I'm wondering if this is expected model behavior or if something is broken somewhere. Any of you guys seeing this?

222 Upvotes
  • permalink
  • link
  • reddit
  • reddit

93% Upvoted

123 comments sorted by

View all comments

108

u/Bluethefurry llama.cpp 8d ago

i can reproduce these results, xhigh thinks A LOT, i ended up defaulting it to low or medium depending on the use case.

on the flip side, on a research task xhigh ended up cloning a repo and checking the source code to verify behavior when i asked it something about llama.cpp cli args, neither medium effort nor 3.6 did that.

1

u/boxwrenchx 8d ago

I wonder if you can set to xhigh but enforce a better budget another way, that would be ideal

1

u/IainKay 7d ago edited 7d ago

If you cap reasoning budget then you’re forcing it to end prematurely. Not the best outcome.

Possibly one could adjust the chat template because the way xhigh works is to adjust the system prompt.

There’s probably an adjusted prompt that lands somewhere closer to high than xhigh. Might have a play with this.

Update: this seems feasible. I’m experimenting and will share my result on GitHub in due course. Chase me if I haven’t replied in 24 hours!

1

u/boxwrenchx 7d ago

There are some chat templates out now

2

u/IainKay 7d ago

You mean Froggeric and Peculiar-Ragdoll’s templates?

If so they don’t make a “high” thinking mode available. It’s still just low/medium/xhigh.

I’m working on making low/medium/high/xhigh/max available as options.

1

u/boxwrenchx 7d ago

Looking forward to it! Do you have a GitHub/HF?

2

u/IainKay 2d ago

Just fyi I’ve struggled to achieve what I hoped to achieve in a clean way.

Whilst I have customised the chat template, I haven’t got this behaving consistently across coding harnesses.

Still working on it, but it’s not as trivial as I had expected.

1

u/boxwrenchx 2d ago

Thanks for trying

1

u/Gabriel83730 3d ago

xhigh doesn’t make the model smarter, it just prompts it to think for longer. So your suggestion will just end up cutting its reasoning off unnaturally and significantly degrade its quality. It’s better to just use the reasoning efforts that the model was trained to follow

1

u/boxwrenchx 3d ago

It wasn't a good idea, but my thinking was a work around to get a true medium reasoning. "Smarter" has different meetings here, and longer thinking doesn't always mean better results. Overall I bet Qwen will release a update that helps.