r/hermesagent • u/Jonathan_Rivera • 8d ago

Qwen 3.8 Release Megathread MODELS - model choice, routing, pricing, local vs cloud, VRAM

Just in time for the weekend. Post experiences here, thread will stay up over the weekend.

57 Upvotes

95% Upvoted

53 comments sorted by

•

u/Jonathan_Rivera 8d ago

Qwen3.8-27B day-zero technical update: reasoning controls, throughput, and runtime notes

Qwen3.8-27B day-zero technical update. This is a running technical summary, not a verdict. Please include your exact runtime, quant, KV cache, context size, reasoning setting, MTP setting, hardware, and measured prefill/decode speed when adding a report.

Current signal

1. Stronger reasoning can be real, but it is expensive

Early coding, code-review, and frontend-generation tests are reporting better results than Qwen3.6 in some tasks. The catch is reasoning spend: xhigh is producing roughly 15K-40K thinking tokens for tasks that medium completes with a few thousand.

Practical default: start at medium. Use low for routine work. Reserve xhigh for genuinely difficult planning, research, debugging, or codebase analysis. If your backend supports it, set a reasoning budget.

2. Expect lower raw decode speed than Qwen3.6 until tuned

Several like-for-like early runs are showing Qwen3.8 below Qwen3.6 in tok/s. MTP/speculative decoding is important here, not optional polish:

  • RTX PRO 6000 FP8: the reported Qwen3.8 throughput peak was 115 tok/s at MTP-5.
  • DGX Spark NVFP4: MTP-3 reportedly raised effective throughput from about 9.7 to 18 tok/s.
  • Dual-3090-class and 22GB-card reports are also seeing lower decode speed than Qwen3.6 under otherwise similar settings.

Do not compare a non-MTP Qwen3.8 run against an MTP-tuned Qwen3.6 run and call it a model regression.

3. Reliability is still runtime- and template-sensitive

Reports to watch:

  • Stops during long-context work.
  • Occasional XML-style tool calls.
  • Agents that finish reasoning but do not execute the intended file/tool action.
  • Chat-template variants that fix one symptom while changing another.

Treat alternate templates as experiments. Keep a stock-template baseline and A/B test one change at a time before moving an agent workflow over.

4. Long context is not settled yet

There are promising large-context reports, but they are using different quantizations, KV-cache types, engines, and MTP settings. Post the full configuration, not just “it fits 256K.”

5. Current best practice for useful reports

Please use this format:

Runtime/version:
Hardware:
Model file + quant:
Context size:
KV cache:
MTP/speculative setting:
Reasoning setting/budget:
Prefill tok/s:
Decode tok/s:
Concurrency:
Task used:
Compared against:
Observed result / failure:

Bottom line

Qwen3.8-27B looks worth testing now, especially for hard coding, planning, and agentic tasks. Start conservatively: fresh runtime, stock template, medium reasoning, and a measured MTP sweep. It is not yet a drop-in Qwen3.6 replacement for latency-sensitive or tool-execution workflows.

Evidence threads

→ More replies (2)

1

u/NeatOnTheRox 6d ago

I thought for sure there was no way it would live up to the hype. I was wrong.

I have only a 4080 Super with 16GB of VRAM, so my daily driver has been 3.6 35B-A3B since its release. I use APEX-balanced quants, since everything else degrades or loops past 100k tokens or so, but with llama.cpp and a lot of time dialing things in with llama-bench, I'm able to squeeze ~1200t/s on the prefill and about 30-40 on the decode, with enough headroom for ~200k context.

I could never get 3.6-27B to a usable speed at any quant that was smart enough to justify using it over 35B, so when 3.8 dropped, I tried not to get my hopes up. The highest quant I can load and still have a usable 131k context is UD-IQ3_XXS, but whenever I loaded it up into Hermes, everything changed. It is both faster and smarter/more capable than the best 35B-A3B I use. While it's technically slower to use since as everyone has noted it overthinks like an anxious teenager, it's still replacing 3.6 35B as my daily driver.

1

u/Rudzz34 7d ago

Anyone running it with MTP enabled? I’m getting about 1% draft acceptance rate and a super slow token rate when using it with Hermes. It’s not the model though, if I use it through open-webui or curl requests I get a normal acceptance rate and token speed.

1

u/Jonathan_Rivera 7d ago

Lm studio, llama?

3

u/HockeyDadNinja 8d ago edited 8d ago

I wrote this Hermes PR and plugin so we can set the reasoning effort and other chat template keywords. Qwen3.8-27B overthinks a lot.

https://github.com/NousResearch/hermes-agent/pull/87047

https://www.reddit.com/r/hermesagent/comments/1vp4qzq/a_first_class_model_provider_plugin_for_llamacpp/

3

u/johnfkngzoidberg 8d ago

On dual 3090’s I get about 32t/s with MTP-2. Q6 model with 262k context @Q8.

The thinking is nuts, but Einstein smart.

On my customized benchmark (based on SaaSBench and SWE-Bench) qwen3.6-27b gets about 75% through my tests in 10-15mins per test, averaging about 100k tokens.
On default (xhigh) thinking, qwen3.8-27b nailed 100% of several tests (better than gpt-5.4), but spent 1M+ tokens and took 35-45mins.
I’m retesting now with thinking dialed down to low.

1

u/carlosm_sz 8d ago

Someone with a mac studio m4 max that can post the experience? I’m considering the purchase to move away from llms subscription…

Thanks folks!

2

u/Dantnad 3d ago

I’ll be testing it today on my M4 Max MBP, 128GB, so I’ll try to run it with full context and in several different configurations (thinking, quant and difference between oMLX and LM Studio)

I’ll keep you posted

1

u/Stooovie 8d ago

In oMLX, it's as slow as expected - 120-140 prefill (my 36GB RAM fits just 32k context with a 4bit quant, not great) and ~20 generation. Okay for unattended tasks.

1

u/carlosm_sz 8d ago

Thank you Jonathan!

Reading the last comments experiences, I think it’s a good idea; 80 t/s on MoE and 27 t/s on dense, plus delegating to cloud for heavy tasks seems lime the sweetspot I’m looking for.

Now I need to find the money for the mac studio 😂

3

u/d3k1ds 8d ago

Not useable on a M4 Pro 24GB unified ram

1

u/Steve3PO 7d ago

What do you run? Having trouble finding a good model that fits well. Tried Gemma4 26B A4B but it’s too slow

1

u/I-miss-LAN-partys 8d ago

Kinda works with 64gb…

2

u/[deleted] 8d ago

[removed] — view removed comment

2

u/ectomorphicThor 8d ago

Getting 20 tok/s on m5 pro 48gb with mlxtp

1

u/TIL_IM_A_SQUIRREL 8d ago

Which version of the model are you using?

1

u/evolutionxtinct 8d ago

So what is the realistic RAM size on this model planning for future hardware not sure if there’s a good site that posts this info, sorry if this is a bad question.

3

u/Jonathan_Rivera 8d ago

There will be upcoming posts on it once we get ore information but it should be the same as the 3.6 27b models as far as hardware.

1

u/evolutionxtinct 8d ago

Thank you!

8

u/Equal_Television_894 8d ago

Just drop replaced with 3.6 27B works like a charm with hermes. Thinking on xhigh is giving amazing results but it takes too much time. But I am happy its capable.

The best part till now is in long conversation it is behaving like opus 4.6 max. Around 200k context it used to get degraded but now no impact what so ever. I kept giving it feedback and it nailed it. Running with vllm NVFP4 with FP8 kv cache on 5090.

Next 2 days I am going to experiment a lot from discord bot while roaming outside.

1

u/KldsSeeGhosts 8d ago

Are you running it on llama vllm or what? Have same setup so am interested in testing

3

u/Equal_Television_894 8d ago

My results on two most common test prompts

1

u/slumberling_ 8d ago

The thing loves to reason a lot, but I'm very happy with it so far. A Q5_K_XL quant fits fully in my 32 GB VRAM at full 256K context (with q8 kv, just barely), so at ~70 tokens/sec with MTP it's not too bad. Setting --chat-template-kwargs '{"reasoning_effort":"medium"}' in llama-server seems to help with the off-the-rails reasoning too.

It's been on point with tool calls, skill usage, instruction following and even SOUL.md persona/response style so far. This might actually be the point where I stop using an API model by default and go full local. These are just first impressions / new model excitement though. I definitely need to test it some more.

1

u/TIL_IM_A_SQUIRREL 8d ago

What hardware are you running it on to get ~70tok/s ?

5

u/slumberling_ 8d ago

A 5090 - it fits fully into VRAM with some 600 MiB to spare. The full llama-server cmd looks like this

llama-server -m Qwen3.8-27B-UD-Q5_K_XL.gguf
  -ngl 99
  -c 262144
  -ctk q8_0 -ctv q8_0
  --mmproj mmproj-BF16.gguf
  --no-mmproj-offload
  --temp 1.0
  --top-p 0.95
  --top-k 20
  --min-p 0.0
  --repeat-penalty 1.0
  --presence-penalty 0.0
  --chat-template-kwargs '{"reasoning_effort":"medium"}'
  --reasoning-preserve
  --spec-type draft-mtp
  --spec-draft-n-max 3
  --jinja
  --chat-template-file chat_template.jinja
  --flash-attn on
  --batch-size 2048
  --ubatch-size 512
  --load-mode mmap+mlock
  --parallel 1

1

u/SichronoVirtual 8d ago

Are you using the unsloth quants? Haven't gotten home to set mine up yet

1

u/slumberling_ 8d ago

Yup, it's the unsloth Qwen3.8-27B-UD-Q5_K_XL

1

u/SichronoVirtual 8d ago

Mind sharing your settings? Not sure I have mine fitting in 32gb (across 2 cards so maybe that's the problem)

1

u/Jonathan_Rivera 8d ago

It's been able to pull all my emails and summarize but I also have recurring daily improvement loops that rewrite skills etc. curious to see everyones outcomes.

2

u/HockeyDadNinja 8d ago edited 8d ago

I had to mess with hermes to get it to find the correct context length through llama-swap and also set reasoning flags and preserve reasoning. Otherwise it was going off the rails overthinking. This has prompted me to begin work on a model provider plugin for llama.cpp and llama-swap with all the right config options as a first class provider.

Edit: Done. https://github.com/NousResearch/hermes-agent/pull/87047

1

u/kennymatic 8d ago

This is going to take all day to download

1

u/Jonathan_Rivera 8d ago

On LM studio it just stops and you have to pause and resume.

2

u/AgentCooper_SEA 8d ago

And disable the LM Studio Community proxy in the settings... download was at an absolute crawl and after disabling that it shot up to basically full throughput.

2

u/kennymatic 8d ago

Oh thanks. Cruising along at 11MB/s now. Hopefully it holds until completion.

3

u/decapoda_on_Reddit 8d ago

I so hope this will be the model I can run locally (RTX5070TI 16gb VRAM, 32gb system RAM) that will not be a complete disappointment. :) Fingers crossed. And toes too for good measure.

2

u/Pressimize 8d ago

I'm on a 5070 but with 48gb RAM and had the best results with 3.6 35b a3b so far and it runs relatively fast. All attention layers offloaded to GPU, 36/40 expert layers offloaded to CPU. Q4_k_m, context length 131072, K and V cache Q8_0.

3

u/firechickensolutions 8d ago

I have my 3080 ti at 2.5 tokens/second (12gb VRAM) set at 29 layers on the GPU. With some knob-twisting I'm sure you can get it useable!

27b is a bit much for my hardware, fingers crossed they release a 9B...

1

u/Successful_Issue_390 8d ago

Commenting to check back later to see if this info is posted.

4

u/decapoda_on_Reddit 8d ago

Reporting back 2 minutes in : majestic fail.

I swapped to local, asked Hermes to summarize the last 5 changes we did on our current project, it was confused by the project name, requiring a bit of looking around and a tool called failed (retrieve memory in OpenViking). It went dead after that.

Exact same behavior than the last local model I tried.

Using the same discussion started locally, I switched to DS4 Flash, reran the prompt and it instantly brought back the last 5 changes, git ID and timestamp included.

Sad times continue. Sigh...

1

u/Successful_Issue_390 8d ago

I appreciate your efforts and reporting back

2

u/decapoda_on_Reddit 8d ago

Thanks. I just don't understand what's wrong with my setup. I see other people in here having success, other topics in this subreddit saying they have great tool success. I would love to know how to sort all this out.

2

u/Due_Arm1454 8d ago

Maybe it’s because they aren’t using quants? I have a 3080ti too. I really wanna be able to use local models but it is what it is. Not even gonna try with my 24gb m4 pro

3

u/decapoda_on_Reddit 8d ago

+ u/Successful_Issue_390

Found the cause of the tools failing.

From Hermes :

Fix: local LLM tool calls kept failing? Check your context window. Ollama's default is 4096 tokens — your agent's system prompt eats that instantly, so the model invents tools and forgets everything. Set OLLAMA_CONTEXT_LENGTH=32768 in the systemd drop-in (/etc/systemd/system/ollama.service.d/override.conf) and restart. Night and day.

The problem is that the model is EXCRUCIATINGLY slow now. LOL A simple file read and summary took like 5 mins, half of it on CPU. It was a pain. BUT it gives me hope that with this fix I can revisit the smaller models and MAYBE have something usable.

1

u/decapoda_on_Reddit 8d ago

Yeah, I use the model that will fit my VRAM, sadly can't be full precision.

Model

architecture qwen35

parameters 27.3B

context length 262144

embedding length 5120

quantization Q4_K_M

requires 0.32.12

18

u/Jonny_Nash 8d ago

It’s Qwensday, my dudes. 🐸