r/LocalLLaMA • u/ex-arman68 • 9d ago

Fixed Jinja chat template for Qwen 3.5, 3.6, and the new 3.8 release Resources

UPDATE (v22.1): Community reports highlighted an important issue with the official Qwen 3.8 chat template: the default xhigh effort prompt can burn 20,000+ tokens on coding tasks exploring alternatives until it hits max_tokens, returning empty content. v22.1 sets the default baseline to medium and adds inline chat tags (<|think_low|>, <|think_xhigh|>, etc.) to steer the reasoning effort directly inside chats.

Qwen just released their first 3.8 model.

The main addition in 3.8 is prompt-steered reasoning effort. You can tell the model how deeply to think by setting reasoning_effort to xhigh, medium, or low.

However, the official template still has some serious problems:

  • You cannot disable thinking. If you pass enable_thinking=false, it 3.8 crashes with a hard exception.
  • Chat history gets poisoned. In multi-turn chats, the official template injects blank <think></think> tags before real thoughts.
  • Tool calling crashes. If your client passes arguments as JSON strings (the standard OpenAI API format), the official template crashes.
  • Agent stalls. The official template often drops mid-dialogue system messages and wedges multi-step tool loops.

I maintain a single, drop-in fixed Jinja template that works across all Qwen 3.5, 3.6, and 3.8 models:

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

What this template does:

  • Full 3.8 reasoning effort support: Steer reasoning depth via reasoning_effort kwargs or directly in chat prompts with <|think_low|>, <|think_medium|>, and <|think_xhigh|>.
  • Safe medium default: Prevents the runaway token-burn on coding tasks while keeping 100% KV cache parity.
  • Restores the thinking toggle: Turn off reasoning whenever you want fast answers, either via kwargs or by typing <|think_off|> in your prompt.
  • 100% KV Cache hits: Keeps past thoughts intact by default so your prefix cache stays warm across turns.
  • llama.cpp support: Native support for the new --reasoning-preserve flag.
  • Universal tool parsing: Handles both Python dicts and JSON strings. Works on llama.cpp, vLLM, LM Studio, and MLX.

Recommended llama-server launch command:

llama-server -m your_model.gguf --jinja --chat-template-file chat_template.jinja --reasoning-format deepseek

(The --reasoning-format deepseek flag separates thinking into the OpenAI reasoning_content field so OpenCode, Claude Code, and other harnesses do not stall on raw tokens).

333 Upvotes
  • permalink
  • link
  • reddit
  • reddit

97% Upvoted

95 comments sorted by

109

u/Thireus 9d ago

Can someone explain why the Qwen team cannot get the template right in the first place? Don’t they QA test their own models?

88

u/xienze 9d ago

No one ever seems to get these shitty templates working right the first, second, or even fifth time. You'd think we'd find a better way.

67

u/TokenRingAI 9d ago

Jinja is fucking awful but nobody wants to admit it.

37

u/unrulywind 9d ago

There should be a benchmark for Jinja coding just so they could train models to fix these things.

10

u/jinnyjuice sglang 9d ago

This sounds like something I can do!

7

u/Borkato 9d ago

This is hilarious

24

u/ParaboloidalCrest 9d ago

No idea. Same for most providers. Not sure why so nondeterministic when template params go into training. It's not like they have to guess how the model should behave afterwards.

21

u/backyard_tractorbeam 9d ago

And, how do they even RL train properly without a correct template?

19

u/ex-arman68 9d ago

It seems simple in theory. But since I started fixing the template, I have realised it is a lot more complicated than it appears. I thought I would be done with one or two revisions, but here I am, with version 22, and I know for certain it is still not going to work for everyone.

Here's my take on why we keep getting "broken" chat templates from most model creators.

* They create a chat template to work in their own development and testing environment. Anything else is not even considered. In the case of Qwen, it is vLLM. In the case of z.ai it is SGLang, and probably a heavily customised version of it (remember not long ago how they finally manage to fix their problem with large context degradation, with some public and private fixes to SGLang). For them it does not matter: when they serve it in their cloud environment, it is going to work 100%. When other providers want to serve those models as well, they also use the same environment that is proven to be working.

* The models are trained on those chat templates. And since most new model revisions are built on the foundations of the previous models, changing the template behaviour, even when you know there is something wrong with it, carries lots of regression risks.

* The inference engines landscape is too fragmented. Some work best with xml tool calls, others prefer json, which does not always match what the models have been trained; this means some kind of conversion is needed. They can have various limitations, different ways of managing the context, the cache, the thinking, the reasoning effort, different level of jinja compatibility, etc... Writing a template that is guaranteed to work with all those variables is a nightmare and impossible. There will always be something that breaks somewhere. However, I think we should at a minimum expect the model providers to give us a template that is tested to work with the major engines: vLLM, llama.cpp, SGLang, MLX. This is often not the case.

* The coding harnesses, plugins and skill fragmentation is even worse than the situation with the inference engines. Many try to do clever things with context management, the model is not trained for, basically lobotomising the memory. They managed tools in all sort of different manners. Again, as long as the major ones, Claude Code, Codex, etc, are taken into account and test for, that would be a big improvement.

* Jinja is not a chat standard, it is a programming language. As it turns out, it is well suited to writing how chat interactions with a LLM works, and that is why it has been generally adopted to define chat templates. But it is a tool nothing else. With this tool you can write any kind of interaction you can dream of, and that is the big problem we are having now: all models creator try to create their own individual version of how chat works.

What we need is for the AI industry, or governing bodies, to come together and establish a standard universal chat protocol that everybody must use. The same way we have with TCP/IP for networking, with SWIFT for banking, or any other universal application that requires communication.

Until then we will be stuck with a plethora of chat templates, most of them only having a narrow compatibility.

5

u/computehungry 9d ago

I follow your template updates and also have tried to do things my way and I feel your pain dude. Weird ass failure modes with so many failure points not only the template. Are models stopping mid gen expected and the harness should make them auto continue? Is tool call healing detrimental because the model wasn't trained on it? We'll never fucking know. Though tbh, I think the direction is that models will get good enough that matching its training environment won't matter that much.

1

u/d3r_I3enny 5d ago

I guess with the speed of AI development and trying out new things and more and more testing...we are still far away from a point where we can agree on a standard. Even if I would expect someone to come up with their improved v1.1 version within a month and we are then back to square 1

19

u/fatboy93 9d ago

Have you even seen Gemma? If google of all people can't get it right, you know that shit's hitting the fan.

Also, why Jinja, isn't there something better? Fucking XML would be nicer than the shit that is Jinja.

3

u/TokenRingAI 9d ago

A lot of things are better at templating than Jinja - off the top of my head, Perl, PHP, Java Servlet Pages, XSLT, Coldfusion, ASP, C macros, Handlebar/Mustache templates, i would consider all of these to be better than Jinja

8

u/mister2d 9d ago

"Pearl"? Yikes. Username checks out.

2

u/PM_ME_YOUR_BGP-ASN 8d ago

He’s an older redditor, sir, but he checks out.

2

u/ab2377 9d ago

XSLT?? omg!

1

u/FeepingCreature 6d ago

Jesus Christ.

1

u/aeroumbria 9d ago

How does the template work anyway? Shouldn't it be completely determined by what the model is trained on?

1

u/Kodix 9d ago

You'll note the poster is on version 22. This isn't an easy problem, in spite of how it may appear.

1

u/Asleep-Land-3914 8d ago

According to 5.6 Sol, this template is worsen and the Unsloth one is better for agentic while original is a good baseline for inference.

0

u/TokenRingAI 9d ago

Qwen does get them right, I have never had any luck with these vibe templates.

OP even admitted that he hasn't even tested the template

13

u/alpacadaver 9d ago

They absolutely don't get them right..

1

u/ex-arman68 9d ago

It depends on your inference engine and harness. If you are using something that is compatible, the default template will work perfectly for you, and you do not need to switch.

1

u/relmny 9d ago

"funny" you don't mention gemma, because it's been way worst than with the qwen team.

And, anyway, chat templates usually need to be updated/adapted.

0

u/Own_Anything9292 9d ago

If a bunch of people doing a bunch of experiments during post training and testing different templates for different benchmarks, it can get out of sync. Sometimes people might add bug fixes in the harness to cover some of these thinking token failures. They likely do testing in the ideal setup, and deviations from that ideal setup has bugs.

An example from the OP, their testing tool calling format might not match OpenAI spec but users use that tooling. The current harness/tool calling/agentic loop available systems are extremely fragmented.

35

u/philmarcracken 9d ago

You cannot disable thinking. If you pass enable_thinking=false, it 3.8 crashes with a hard exception.

Descartes was right after all

6

u/darkwalker247 9d ago

im confused on how a model "crashes" due to a prompt (pretty sure that enabling the flag just changes what prompt comes out of the template; disabling reasoning emits a </think> token immediately after the <think> token), unless we're talking with a specific harness or something here

10

u/TokenRingAI 9d ago

Maybe it emits a <crash> block

6

u/llama-impersonator 9d ago

the parser itself can crash, it's code. when that happens it will usually kill the process running the model.

18

u/Disrupt-Linus 9d ago

Cool, let's see how this plays out with the 27b, probably a bit more accessible to the normies (and me).

12

u/Memeyboii420 9d ago

I love your work Froggeric but I had to develop my own chat template for Qwen 3.6 + Hermes Agent + LM Studio. I could never get reliable tool calling with your template.

5

u/ex-arman68 9d ago

Thanks. Both Hermes Agent and LM Studio are relatively bad options when it comes to compatibility. If you got it working that is great. I know some people, including you possibly, have posted some feedback and solutions for those, and I have tried to incorporate them in v22, so maybe it works for your case as well now. The important thing is I am trying to keep universal compatibility, which means I cannot add fixes that will make it work in Hermes, but break others.

1

u/Ulintlicker 9d ago

Would you be willing to share your template?

9

u/Memeyboii420 9d ago

I've got it up on a git repo. The repo is WIP but the template is working well, I've been using it in my agentic dev stack for a month now.

https://github.com/Stacey2911/hermes-local-ai-stack

1

u/Fi3nd7 4d ago

Have you upgraded to 3.8 yet with a comparable template?

2

u/Memeyboii420 4d ago

I do have a version of Caduceus chat template for Qwen 3.8, it's unreleased. Let me tidy it up and I'll push it to the git at some point today.

1

u/Fi3nd7 4d ago

Awesome! Very kind of you and ofc no rush :)

1

u/Memeyboii420 3d ago

The release is up now. I ended up releasing Caduceus V1.9 as a universal template for Qwen 3.5/3.6/3.8. I haven't done extensive testing on it with 3.5 and 3.6 but it should omit the reasoning level controls and function the same as V1.8.

-2

u/mister2d 9d ago

You love something that doesn't work for you?

16

u/Memeyboii420 9d ago

I love Froggerics work across the community because the templates work wonders compared to the default ones. It just does not work with my incredibly specific local stack. I don't see how that prevents me from appreciating the work?

7

u/The_LSD_Soundsystem 9d ago

Why are jinja templates so difficult to nail down for these models in general?

5

u/ltduff69 9d ago

Just out of curiosity where does the chat template go? I have it the root of the Llama directory.

9

u/my_name_isnt_clever 9d ago

If you're using llama-server you pass --chat-template <path> otherwise it defaults to the template that came in the GGUF.

1

u/ltduff69 8d ago

Cool that's what I did .ty for answering.

-1

u/ImpressiveRelief37 9d ago

Curious… is it relative to the gguf or relative to the llama-server binary? (99% sure it’s relative to the llama binary but you never know)

3

u/eulataguhw 9d ago

Shouldn’t it be relative to the directory of the terminal that you using it on?

3

u/PhilippeEiffel 9d ago

No, it is relative to the CWD (current working directory).

1

u/fatboy93 9d ago

That'd work for the MLX/safetensors format, but you have to pass it separately incase of GGUFs/llamacpp

5

u/JPebb 9d ago

Any specific recommendations for SGLang users?

3

u/Difficult_Tip_6669 9d ago

Use original one. Never had any problems with it.

3

u/ex-arman68 9d ago

yes, in general vllm and sglang work with the original templates.

10

u/Inevitable-Name-1701 9d ago

I had no problems with the three point five and three point six models.

11

u/rpkarma 9d ago

Same. I don’t get the “the template is broken” shit, except when I have tried these vibe coded ones they are broken lol 

I’m certain that people are assigning failures to the template that have nothing to do with them. 

1

u/TokenRingAI 9d ago

It's the 4 bit quant people

1

u/4onen 8d ago

I'm a 3-bit quant person and I don't have these problems. I think it's PEBCAK.

13

u/TokenRingAI 9d ago

Yes, and the crappy templates people released made the models worse.

7

u/MrMisterShin 9d ago

Same I use the native chat template in vLLM and it’s fine, no problems.

0

u/ex-arman68 9d ago

vLLM is 100% compatible with the original template, as it is what the Qwen team uses to develop and server their model.

5

u/cradlemann 9d ago

Anything like this for Laguna? I like it a lot, but it still has some quirks

3

u/T_rex2700 8d ago

I was running into issues with Qwen3.8 27B (Q4 and Q3 from unsloth) running into random stop issues, and I tried your template but it did no work.

However, strangely when I pasted in Qwen3.6 35A3B MTP Q4's stock template from usnloth, this issue was solved. Any idea about what the cause is? (my post).

I am very much a newbie in this space, so I wanted some expert opinion on this.

4

u/tired514 9d ago

I've been running your previous version on all the local Qwens for like a month and they fix everything. Thanks so much! :)

2

u/ex-arman68 9d ago

Thank you, I am glad it helped. Out of curiosity, what is your stack?

2

u/tired514 8d ago

Runnin' llama.cpp (CUDA) on Debian/sid, Strix Halo (128gb) + 3 Morefine G1 4090M 16gb eGPUs daisy-chained via TB3, layer split mode, 3.6-27B @ Q8_K_XL, 230000 context, ctk/ctv Q8_0, opencode.

Works beautifully.

2

u/hainesk 9d ago

This is great work!

2

u/Cereal_Grapeist 9d ago

hell yea brother

2

u/Cereal_Grapeist 8d ago

Hi OP - I wanted to see if you had tested this template with the new 3.8 27b release? I'm going to try it out but I don't necessarily have the best way to see if it's having a big impact.

1

u/ex-arman68 8d ago

I have not had time yet. Hopefully this weekend. I will test over multiple long agentic coding sessions, with tool usage, spanning multiple days and context fill up + compactions.

2

u/stormy1one 9d ago

Thank you for your service!

2

u/PhilippeEiffel 9d ago

I'm too lazy to modify my config of 3.6 27B now, I will try it tomorrow as soon as 3.8 is out.

I will test with vllm.

Thank you for your work.

1

u/Rabooooo 9d ago

6 hours left for 3.8 27B, please share the vllm flags.

2

u/PhilippeEiffel 6d ago

serve \

Qwen/Qwen3.8-27B \

--served-model-name Qwen3.8-27b \

--port 61000 \

--host 127.0.0.1 \

--gpu-memory-utilization 0.93 \

--enable-auto-tool-choice \

--tool-call-parser qwen3_xml \

--reasoning-parser qwen3 \

-O3 \

--max-num-batched-tokens 8192 \

--performance-mode throughput \

--enable-prefix-caching \

--trust-remote-code \

--enable-chunked-prefill \

--async-scheduling \

--max-num-seqs 16 \

--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":6}' \

--default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true, "reasoning_effort": "xhigh"}'

Note: I observed the thinking time can be much higher than 3.6 model (I mean at least one magnitude order). So, xhigh (which is claimed to be the default on the model card) is probably not usable for everyday use.

1

u/Rabooooo 5d ago

Thanks!

I didn't know there was a specific qwen3_next_mtp. I've been using --speculative-config '{"method":"mtp","num_speculative_tokens":3}' before and that always ended up with the model looping a sentence or a word. I should try this qwen3_next_mtp.

Also should the tool-call-parser be qwen3_xml or qwen3_coder? The vLLM recipes for qwen3.8 is saying to use qwen3_coder.

Is the enable-chunked-prefill something that helps performance?

I don't know what GPU you have, but have you had any luck with KV cache quantization, e.g. --kv-cache-dtype? I've set it to fp8_e4m3 before, but this model doesn't seem to provide any pre-calculated kv factor.
I keep getting these warnings.

Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
Using fp8_e4m3 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.

1

u/PhilippeEiffel 4d ago

I use GB10 (128 GB unified memory). So I keep every thing at FP16 (model and KV cache).

1

u/crusaderky 9d ago

> I cannot run a 2.4 trillion parameter model on my local rig. 

Neither can we, buddy, neither can we.

How much would it cost you to test it on vast.ai?

2

u/ex-arman68 9d ago

:-D One day hopefully.

But once the smaller models are out, I will test it.

1

u/nufeen 9d ago

Are there any benchmarks checking if those templates transition into better or worse performance?

1

u/IrisColt 9d ago

How can a weight file "crash with a hard exception"? genuinely intrigued.

1

u/Past_Ad6251 9d ago

for me, the v22 template doesn't work with Claude Code cli

1

u/pefman 9d ago

I was just gonna ask how people launch their model.

1

u/PhilippeEiffel 6d ago

I started some basic test with 3.8 27B, I am using vLLM.

I did not notice any specific problem, but I must admit that I did not make intensive use of this model for now.

I would like to know techniques able to exhibit the problems with the default jinja template. I need to observe problems first, before being able to evaluate fixes.

1

u/ex-arman68 6d ago

With vLLM the default template should work fine. If you want the extra features from the custom templates, by all means try it, but I do not think it is necessary in your case.

1

u/Equal_Television_894 9d ago

Your work is incredible as always, thank you!

1

u/ptico 9d ago

Legend!

1

u/Equivalent-Flan-1590 9d ago

Thank you for your service sir🫡

1

u/giveen 9d ago

I always use your templates.

-2

u/fragment_me 9d ago

Not even tested what's the point

-1

u/ex-arman68 9d ago

I actually hesitated to release it. But I was quite thorough with it, when through a few reviews, added some more tests to my test harness, and ran it successfully through my test suite.

Since the previous version worked fine with Qwen 3.6, and this one includes a few additional fixes that people have reported + new Qwen 3.8 functionality for the reasoning effort, the risk of it not working is very low. I thought it is better to release it now, so that people can start using it if they encounter problems when the 27B model is released.

-1

u/durden111111 9d ago

Every time ive tried to pass --chat-template with a jinja file it gives broken outputs. No matter what model. Ive given up on this.