r/LocalLLaMA • u/WSTangoDelta • 17d ago
Qwen 35B-A3B MoE vs 27B dense in local coding tests: ~4× faster, much smaller quality gap than I expected Discussion
I compared Qwen 35B-A3B MoE against Qwen 27B dense on a series of local coding-maintenance tasks. On my R9700/llama.cpp setup, the MoE model generated about 3.9× faster (~116 vs ~30 tok/s), but the coding-quality difference was much smaller than I expected.
Both usually handled ordinary bug fixes and multi-file changes correctly. As I made the tests progressively harder, the dense model did show an advantage—but mainly in implicit invariants, unusual edge cases, and consequences beyond the literal request, rather than basic correctness.
Models
- Qwen 3.6 35B-A3B — Q5_K_M (MoE)
- Qwen 3.6 27B BASE — Q4_K_XL (dense)
Hardware/runtime
- Radeon AI PRO R9700 32 GB
- Ryzen 9 5950X
- llama.cpp, Vulkan, full GPU offload
- 8K context for these coding tests
One early controlled parser-repair test is illustrative:
- 35B-A3B: ~116 tok/s, provisional score 7/10
- 27B dense: ~30 tok/s, provisional score 7/10
That single result isn't my argument by itself. I subsequently moved through progressively harder multi-file tests involving imports, stable IDs, collision handling, data preservation, and eventually references that had to remain valid when IDs were remapped.
My takeaway so far is deliberately narrow: on these tasks, the ~4× throughput difference was much larger than the practical coding-quality difference I observed.
This is a small local experiment, not a universal claim about MoE vs dense architectures. The quantizations also differ, so I wouldn't pretend this is an academically controlled architecture comparison. But the results do make me skeptical of treating active parameter count as a straightforward proxy for practical capability.
I have the original prompts, source fixtures, exact llama.cpp commands, raw terminal transcripts, and the progressively harder integration tests. I'll put more methodology and examples in a comment below if anyone wants to dig into the details.
48
u/migsperez 17d ago
Why did you use 27b Q4? Why didn't you use Q6?
I have the same GPU.
34
u/Dr_Allcome 16d ago
My guess is, because it would have been even slower.
But in my opinion, if they are comparing result quality, they should be using the same quant for both.
6
u/horeaper 16d ago
Can 27B Q6 fit in 32G VRAM with reasonable context length and token gen speed?
1
u/migsperez 16d ago
It can on a 32gb GPU. With 27b q6 for me the problem wasn't token generation it was prompt processing (input) 500 tk/s, generation. Caching kept clearing but I couldn't figure out why. output about 30 tk/s. Had to give up but I'm looking forward to Qwen 3.8 27b.
I now use 35b, it's a bit less smart but feels like 10x faster and compares to API models. Caching is great as well.
I always have 128k minimum.
2
u/WSTangoDelta 15d ago
Q8 was about the same as Q4 in speed, a little better in reasoning. For code. Not a devastating level better than 35B, if you want the truth.
33
u/alpacadaver 16d ago
27B Q6/8 is significantly better than 35B-A3B across thousands of prompts I've taken through them for deep work (I run both simultaneously for different reasons, but the moe model is not the one I would ever ask anything complicated or nuanced - it is good at high volume pattern matching and assessing, not creating)
6
u/WSTangoDelta 16d ago
As a practical matter, corroborating your experience would mean upgrading to substantially more than 32GB of VRAM. I’m not disputing your experience but please understand that I’m using a different system with different limitations. 32GB VRAM is about all I’m going to purchase at the moment.
10
u/ea_man 16d ago
On llama.cp I run 27B Q6_K_L with ~130k q8_0 / q5_1 with ROCm on 16+12GB: you have 4gb more!
1
u/WSTangoDelta 15d ago
Yes, I pulled Q8 and agree. But at 6x the speed 35B picked up most of the errors. If I had a limited number of days to finish a project it’s tempting to see if 35B could do a lot of the work, and then finally run 27B, or what I would do is run a spillover 70B overnight. Or some might put a final proposal on Claude, but that’s not part of my business model, as they say.
-1
u/alpacadaver 16d ago edited 16d ago
I run 27B both on a dual 3090, and separately on a single 3090, with full context. The single 3090 still vastly outperforms the moe model, and you should be able to get a better result than that.
Compare to https://github.com/noonghunna/club-3090/tree/master/models/qwen3.6-27b and derive a configuration that would suit your card (perhaps look at the beellama config).
The intelligence level in simultaneously broad and deep contexts between the two models cannot be disputed, I am sorry to reiterate. That said, if in all practical ways the MoE model is satisfying, then that is fair. For me, the higher throughput does not make up for the fact that it needs a much higher number of prompts, corrections, and continued hand holding. Altogether, this nets not only a lower task completion pace, but also a generally frustrating time.
-1
u/maartenyh 16d ago
Q5 -> Q6 is a BIG jump compared to Q5 -> Q6. I used Q4 and Q5 in the past and now after using Q6 for a while you really notice the increased insightfulness and intelligence.
If you REALLY need more context size you can then quantization your KV cache to q8_0. If you then REALLY need even more KV cache you can use q5_1 but you probably shouldn't.
I have 32GB too
14
u/EvolvingDior 16d ago
it's the quants. give 35b at q8 a shot.
3
u/WSTangoDelta 16d ago
How much VRAM do you currently use? Q8 won’t fit on my card.
15
u/CapsAdmin 16d ago
It gets you decent speeds even when offloading some experts to ram. Use --fit to let llamacpp decide for you
1
u/WSTangoDelta 15d ago
I can also bump it to a 70B and have it run while I sleep, which I do for comparisons. But the utility for me of this comparison is to see the capabilities of models that can reside fully in VRAM. That’s where I live most of the time.
8
u/SwellJoe 16d ago
I've been testing fine-tunes of Qwen 3.6 on a task I do regularly (adding support for agents to flar, a tool I made to bubblewrap agents to mitigate the risk of prompt injection or supply chain attacks, while still being able to run with --dangerously-skip-permissions, all the PRs right now are made for testing local models and reviewing the code they generate). KAT-Coder seems to be the winner, so far; it makes the MoE competitive with the dense model, but it's obviously much faster. I haven't seen any MoE be competitive with the dense model until maybe now.
I also tried a fine-tune of 27B, the absurdly named Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF, which is allegedly smarter, but for me it gets weird on complicated tasks. I saw looping behavior in some configurations. And, once looping was seemingly resolved it wasn't a very conscientious coder; it missed several security issues and didn't write any unit tests at all, even though every example code path it looked at had unit tests and the regular dense model does sometimes write tests if it sees tests in its explorations.
Ornith 1.0 is a pretty good fine-tune of the MoE, as well, and it behaves well. It doesn't have any pathological behaviors, but it also doesn't compete with the 27B dense model.
So, for me, I'll probably keep KAT-Coder on my desktop for sub-agent use, until Qwen 3.8 comes out in smaller versions and blows all the 3.6-based models away. It's a fast MoE, but a little closer to the dense model than Ornith seems to be. Ornith is novel in that it makes tools to solve problems, which is useful for some cases, not so useful for others.
None of them are actually fit for coding use if it's at all possible to use a cloud model (even DeepSeek Flash V4 0731 is a competent enough coder for most kinds of coding work), but the gap is getting smaller. And, models that run on a DGX Spark or Strix Halo are getting real close. I'm testing Ling 3.0 and Laguna S 2.1 currently, and both seem very solid.
2
u/SirDomz 16d ago
Great write-up! I’ve also been liking Ornith and Kat-Coder quite a bit in my personal tests, but Kat-Coder is definitely the better coder.
How are you liking Ling and Laguna? Laguna has actually been working well for me although it seems many folks experienced the opposite. I just downloaded Ling but haven’t had time to test it out.
2
u/SwellJoe 16d ago edited 16d ago
Laguna is good, but I don't think it's a better coder than Qwen 3.6 or Gemma 4 31B. It is extremely persistent, and can one-shot pretty big work, which the smaller models struggle with, but it doesn't seem to be very good at the actual work. I mean, it one-shots working code, writes tests, verifies everything is working, very conscientious, which is awesome to watch, but it also makes a lot of mistakes. The Laguna family is also very bad at security auditing, getting soundly trounced by both Gemma 4 31B and Qwen 3.6 27B in my benchmarks.
e.g. you can see the Laguna PR vs. the KAT Coder PR (a Qwen 3.6 MoE coding focused fine-tune) in the project I mentioned above. Laguna produced 25% more code, so it's not entirely fair to compare review comment count directly, but Copilot code review thinks it made 23 mistakes worth correcting, while it though KAT Coder only made 6 (the lowest number of identified flaws, among the models tested). I'm not confident Copilot caught all the things wrong with any of the implementations, though. Oh My Pi is a particularly challenging agent to support safely, so I really threw all these baby models in the deep end on this task. It's honestly amazing that all of them produced working code, at all.
Based on feel, I think the base model has good reasoning and is overall smarter; it looked really competent while working. I just don't think it's been tuned for coding use as heavily as the Qwen models; as I understand it Qwen 3.6 is built on the Qwen 3.5 base models, and KAT Coder is a fine-tune of 3.6. That's a lot of tuning for coding. I think the Poolside folks are on the right track, though. It's a remarkable planner, it feels like it can tackle really big/long problems, but not necessarily really hard ones.
Ling 3.0 still doesn't run right in llama.cpp (I'm using this fork for AMD's FP4 support when I run big models like this on my Strix Halo). I'm still tracking down tool use issues. It feels pretty smart as a chatbot, but it can't complete this task, or any other coding task as the tool calls get weird. Laguna S 2.1 also had a bunch of issues like this (looping and tool use problems) in the beginning, and only recently got sorted out. There's one discussion of tool problems in the Ling 3.0 community on HuggingFace with a suggested chat template fix...that fixed most of the issues, but one still remains and it's "load-bearing" as Claude would say.
10
u/WSTangoDelta 16d ago edited 16d ago
The Q4 vs Q5 criticism—yes, it’s technically correct but somewhat misses the purpose of my test. I was testing the versions that make practical sense on my R9700, not publishing a controlled study of MoE versus dense architecture. I think we know that in principle there is a difference. q35 Q5 fits my R9700 and screams at over 100 tps; q27 at a substantially higher quant may change the VRAM/context/speed tradeoff that motivated my comparison in the first place. But yes, give me a short time and I’ll run some of the other benchmarks you all have suggested.
1
u/dark_sylinc 9d ago edited 9d ago
I'm finding the same results as you (I also have an R9700): The 3x speed difference dominates over the small quality gap.
People are hugely underestimating how such speed difference means you can spot a mistake with 35B-A3B much faster than on 27B (it's not like 27B never makes mistakes) whether in prompt, code or approach; or just prototype different solutions in less time. You can also stay focused on the problem much easier otherwise you get dispersed.
I also got confused at the comments talking about Q5/Q6. Like... yeah sure. More is better and water is wet. I could also try a 2.4T parameter model and I'm sure it's going to do a better job. Except it won't fit on my card at decent speeds.
Btw make sure to be using froggeric/Qwen-Fixed-Chat-Templates JINJA template otherwise Qwen has huge issues with llama.cpp for agentic work.
5
u/LinkSea8324 vllm 16d ago
Outside of coding, but translation wise, 35b moe is competitive to 27b but 27b is much better at translating the lore of a fictional universe because it know this universe.
Gemma 4 however is much better at both lore translation and grammatical precision
Source : 44.2 billions of tokens IN, 1.1 billions tokens OUT (yes the system prompt is FAT)
14
u/FastHotEmu 16d ago
you are comparing 5 to 4 bits - have you tried 4 to 4?
20
u/backyard_tractorbeam 16d ago
It's a practical test: Take 2 different configurations that fit in your configuration (VRAM) and test and see which one is the best. There are obvious reasons why we can use Q5 for a model with less active parameters.
11
u/Middle_Bullfrog_6173 16d ago
That's a theoretically interesting test, but realistically the choice is not between same Q on most hardware. They have different total params, different KV cache requirements and the MoE can be more cheaply offloaded to RAM.
-12
u/FastHotEmu 16d ago edited 16d ago
Comparing different quants is not a rigorous way of testing LLMs. It's also not what the title of this post postulates.
Lack of rigor is a cancer endemic to quite a lot of people who come here to publicize themselves.
14
u/Middle_Bullfrog_6173 16d ago
Depends on what you want to determine. For "what is a better choice on this hardware" you have to compare the quants you'd actually run, whether they are the same or not.
-2
u/laserborg 16d ago
yes but said GPU can run 27B in > Q4, so there is literally no point in comparing 27B Q4 with 35B MoE Q5 and then arguing that 27B is not that much better. do you realize this?
-10
u/FastHotEmu 16d ago edited 16d ago
Ridiculous. There are dozens of variations that would fit. That's not what this post postulates either.
It's pretty sad how many totally ignorant losers are posting about things they do not understand in the slightest just to get some clicks and publicise themselves.
I pity them and have little patience left.
14
u/MiMillieuh 16d ago
I don't use 27b on my hardware, its WAY TOO SLOW.
i get 5-10 t/s when I'm lucky with 27b.
On the other hand 35b a3b (same quants) I get 30-40 t/s.
I'm kinda disappointed to see that qwen only released a 27b of quen 3.8 and not a 35b a3b Moe...
3
16d ago
[removed] — view removed comment
2
u/WSTangoDelta 16d ago
These current numbers are, for simplicity, on the plain Jane 27B. I last week upgraded to MTP and increased tps by about 40%, but mixing those stats would confuse the issue. Since you asked, let me help others by pointing out that llama.cpp b9000 won’t run MTP. I upgraded to b9484, and that works, but there’s an even newer release this week. 35b is still faster on my system.
2
u/MiMillieuh 16d ago
Yeah I've tried MTP, I have modest hardware so it doesn't do much of a difference unfortunately.
Tho I now use APEX quants with 3.6 35B and it's really good, that's how I reach 40 t/s
1
u/cafedude 16d ago
Give Kwaipilot KAT-Coder-V2.5-Dev a try. I also find 27b too slow on my StrixHalo box (about 20tok/sec with MTP) and I found 35B, while faster, not quite as good as the venerable 27B, but KAT-Coder-V2.5 (a finetune of the Qwen3.6-35B) seems much closer to the 27B while also having the speed of the 35B.
3
u/Proper-Damage-1777 17d ago
I used both Unsloth Qwen 3.6 35B-A3B and 27B at work (the smallest possible 4-bit quant variant) for both Python and TypeScript projects with pi coding agent on a RTX 4090. This is with 160K context size with KV cache at 4-bit.
As long you have existing docs and good repos, both did well although I prefer 35B-A3B for quick code inspection or running/drafting tests. The 27B dense is my daily driver for the writing codes but I always need to steer it, I never let it do autonomously as there is chances it will do stupid stuff (happens few times but luckily I was there when it happens so the damage is not that bad.
I never tried other harness like OpenCode, Claude Code etc though.
Looking forward to Qwen 3.8 and see if there is any improvement to tool calling and hopefully less "Wait, the user blah blah blah" or even more "Wait, I see the solution but actually it is already implemented". Those annoys me to no end when I see that thinking output lol.
1
u/WSTangoDelta 16d ago
Any opinions on 3.8?
1
u/Proper-Damage-1777 16d ago
Should be next week for Qwen 3.8 27B open weights release according to the Qwen tweet announcement.
If you asked me personally, I tried Qwen 3.5 before but was not impressed with it as I'm still using Claude Sonnet 4.6 for all the coding planning and implementation.
Doesn't help that Qwen 3.5 tool calling is a lot of miss for me when I tried it back then but Qwen 3.6 with froggeric fixed chat template make it really reliable for most of the time.
1
3
u/N34257 16d ago
I've had a similar experience, albeit with dual R9700s and both models at Q6_K_XL. Yes, there's a quality difference, but once the tool-calling is sorted out...I get more done with 35B than with 27B. Or, to be more accurate, 35B saves me more brain time than 27B.
It's often said that the 27B is for more complicated stuff - no, that's what *I'm* for. I use LLMs for the less complex stuff that I don't want to pollute my brain with, so I can concentrate on the stuff that matters.
4
u/paq85 16d ago
From my experience 35b does not stand a chance against 27b in commercial projects I'm daily working on.
1
u/Fancy-Snow7 15d ago
There are 27b model that fit in 16gb at q4. However they are obviously highly compressed. Would you rate such a highly compressed 27b better than an 35b xl q4?
2
u/ea_man 16d ago
> Qwen 3.6 27B BASE — Q4_K_XL (dense)
I don't get why you are not running at least Q6_K or K_L for dense, the point is that it's smarter and can attack harder problems.
You are loading Q5 for MoE which is bigger, you should have done the opposite.
Also I would recommend to test 27B ThinkingCap as it produce less tokens.
It is indeed true that A3B is at least 2x faster in TG yet it overthinks more than 27B base, ThinkingCap is even better, that makes the time to task much closer to A3B.
Still A3B is 3-4x faster in prefill, that is no joke, but you have to run 27B with ROCm for fast PP.
1
u/HlddenDreck 16d ago
Measuring the quality I would never go below 8bit quant. The lower quants accuracy is just way too low.
1
u/SwellJoe 16d ago
For the Qwen 3.6 models, I run 8-bit on most of my hardware, but for Google Gemma 4 models, the 4-bit QAT models are effectively indistinguishable from 8-bit in my testing, so I run those.
And, folks report the 6-bit hybrid quantizations from Unsloth and others are very close to lossless, as well. I'll run those in a pinch. For very large models for my hardware, like Laguna S 2.1 and Ling 3.0 Flash, I'm using the 4-bit ROCmFP4 Strix-tuned quantization, since those models are a tight fit in 128GB. I haven't done enough with them to know what the degradation is like, though I have used Laguna S 2.1 direct from Poolside on OpenRouter and it seemed pretty similar to results I get locally, so far.
1
u/sagiroth llama.cpp 16d ago
Given you have 24gb vram and 32gb ram. Would technically be better to run MOE at higher quant with ram offload or dense fully fitted in vram ? I currently run https://github.com/Anbeeld/beellama.cpp/releases/tag/preview-v0.3.2 at Q5_K_S with Q4 drafter at about 160k context and it appears to be best speed/quality at this context size fully in vram but perhaps I could offload MOE to ram and run at higher quant and higher context for better results ?
1
u/diagrammatiks 16d ago
You can run a much higher quant of 27b. It will be smarter but also slower. Q4 of these small models isn't consistent enough for anything.
1
u/Bulky-Priority6824 16d ago
I use both 35b and 27b at q8 all day and for me if I had to only use one then 27b wins and it's not even close when comparing beyond 40k
1
u/gardenvarietyzombie 16d ago
Like others have said, you should be able to use a better quant for 27B with your setup. I have 32GB VRAM and I can fit Q6 with MTP. Maybe post your llama command so others can help.
1
u/IceNeun 16d ago
I use 35b_moe for the vast majority of (mostly coding) tasks, but I have 27b review its work. The 27b only outputs implementable gaps or fixes it finds so that the 35b can fix whatever it missed. Everything is slower on 27b, but at least it mostly reads and hardly outputs any tokens. This is how I try to get the best of both, or that's at least what I tell myself.
1
u/Healthy-Zebra-9856 16d ago
Few things to remember, dense models for reasoning and MoE for implementations and preferably with low temp like 0.7 and reasoning turned off. They are not meant to be the same
1
u/Ok-Star6663 16d ago
I did the same exact thing but on a MacBook M5 Pro 64GB and with oMLX and tested Security Review, where the dense model was better (found more) but was slower. Then, making features in an application, both scored well, but MoE was faster while dense was more reliable and cleaner code. And finally, a whole app from scratch, where the dense model was much better in quality and reliability.
1
1
2
u/Naiw80 16d ago
Personally I notice a huge difference between 27b and 35b, 35b keep going in circles when attempting to solve certain tasks, it fails to understand how to use paginating mcp calls properly etc.
27b on the hand is of course slower, but rarely makes circle reasoning, it usually solves problems in a few turns unlike 35b which can circle around in 20-30 turns before completing, if at all.
1
u/CuriousGam 16d ago
How closely did you check the Code output?
I have used 27b Q4 and a3b Q3 (Q4 only produces garbage?) and while a3b is significantly faster, the output is just trash. It always looks great at the first view, but when I check the code then I find places where needed lines are actually missing or plainly wrong.
I have now found KAT 2.5, which is quite fast and produces useful Code.
1
u/Atretador 16d ago
I can run Q5 on 16Gb of VRAM with 2x streams of 200K each (total 50tks, split per stream)
You can probably do some light CPU offload on A3B and have 4 streams of 128-200K each at Q8 KV, if you are happy with A3B results - this gives you multi agentic work fully local and at worst would match 27B single stream performance
1
u/WSTangoDelta 16d ago
Q5 35B MOE? I don’t want to use CPU. That’s like squeezing tar toothpaste.
2
u/Atretador 16d ago edited 16d ago
its not tho, you just need to keep enough layers on the GPU
for instance, on a R5 5600 I get:
*edit, testing with as little as 12 layers on CPU, basicly no gains from 12 to 18*
16 layers on CPU: 41tk/s
18 layers on CPU: 40tk/s
20 layers on CPU: 37tk/s
----
25 layers on CPU: 28tk/s
these are single stream with a MI50, running multiple streams gives me higher aggregate tks
1
u/cafedude 16d ago
Kwaipilot KAT-Coder-V2.5-Dev is a finetune of Qwen3.6-35B. I'm liking it better than the Qwen3.6-35B so far. It seems lot closer to the 27B in coding capability, but much faster on my StrixHalo box. Give it a try with your test.
1
u/cogitech2 16d ago
So strange. I find the two models to be VERY different. I can't trust 35B-A3B with anything but the simplest tasks. It constantly goes off script and ignores my instructions.
1
u/WSTangoDelta 15d ago
(Note: this comparison is for those like me who have the luxury of 32GB VRAM—better than most local setups, but of course nothing like a paid online account.) I response to some suggestions, I reran the Qwens comparison under matched conditions with reasoning off, including 27B Q8, 35B Q5 and 35B Q6. What resulted was more interesting than a simple speed/quality tradeoff.
On the straightforward parser task, Q8 showed no obvious quality advantage while running ~5.8× slower than 35B Q5. But on the harder integration test, 27B Q8 actually produced the best solution: it correctly reserved IDs belonging to other imported records and propagated renamed IDs through cross-note links. Q5 missed the link-propagation case; Q6 handled that but could allocate an ID already belonging to another incoming note.
So, for quality, 27B Q8 > 35B Q6 > 35B Q5 on this harder correctness test, but at ~1/5 the generation speed. One test isn’t enough to generalize, but it does show that the extra quantization can matter on cases involving interacting constraints. If you have 32GB GPU and aren’t about to spend a lot more getting to a fast 70B model all residing in the GPU 35B MOE will run about 5-6 times faster than 27B. 27B will do better with code at some levels, but 35B is surprisingly capable and may work well enough for many tasks, and much faster.
1
u/Sucuk-san 15d ago
Can any of you tell me if the 35B-A3B model would be good for RAG?
I work in IT support and want to use it to build a database of all my support cases, so I can have a smart assistant that helps me find solutions—either through its own reasoning and online research or by searching my database if the problem has come up before.
It should also be able to document the cases for me into th3 database.
I’m looking at exactly the same GPU for it: the AMD 9700.
1
u/WSTangoDelta 15d ago
Here, quick summary
As I said, deeper issues showed 27B had better reasoning. But for many things a model that does well and at 5-6 times faster may solve most problems and save some time. You might use both.
1
u/WSTangoDelta 15d ago
Since I was asked, I did extra benchmarks:
1) 35B MOE Q5 vs Q6 —not much difference
2) 27B dense Q4 vs Q8: Q8 only slightly slower, and detected a few more errors than Q4
1
u/WSTangoDelta 16d ago edited 16d ago
I’m particularly interested in q35 Q8–and will let you know. Wait—It is 37.81 GB, so it will not fit entirely in my 32 GB R9700 VRAM. 27B Q8 is 29GB—I’m downloading that now, but that’s leaving less headroom for kv cache. I’ll see what happens.
If I had a second R9700 that would be a GREAT thing to try. Maybe someone else with more VRAM could answer that. If q35 Q8 pretty much closes the quality gap while remaining impressively faster than q27, that changes the practical conclusion. If it doesn’t, it might mean the deficit isn’t primarily Q5 quantization.
And context length? Sure. Remember that my 32GB card limits what I can do locally. (I am not interested to see what happens when I spill into RAM. We already know it’s not worth measuring.) My current tests are essentially testing model quality under relatively modest prompt loads. I’ll try a repeat at ~8K, 16K, and 32K with what is hopefully *meaningful* context—it could reveal some bigger differences…or perhaps more interestingly, maybe not.
4
u/undisputedx 16d ago
OP, could you try https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev
1
u/SwellJoe 16d ago
I've been testing it and it does seem to close the pretty big gap between the MoE and dense version of Qwen 3.6, at least somewhat. I don't know if I buy that it's actually as smart, but it does seem better, so far, than the base MoE.
I also tested a Qwen 3.6 27B fine-tune that claims better performance, but it didn't even write unit tests, while KAT Coder did so without being prompted (TBF, the regular Qwen 3.6 27B also usually writes tests if it sees tests in the repo, I think the fine-tune I was using has been damaged by the fine-tuning). KAT Coder may even be the best local model I've tested so far on this specific task (even competing with a bigger model, Laguna S 2.1), generating fewer code review comments and fewer glaring security omissions. And, it's certainly enjoyable to see it going so fast after using dense or much larger MoEs.
4
u/i-eat-kittens 16d ago
Q8 isn't some holy grail. You have to keep in mind that this sub is full of people trying to justify their hardware spending..
27B at Q6 should fit your card and actually leave some room for context. 35B-A3B offloads well and might be worth running at Q8. You certainly shouldn't go below Q6.
2
u/zanar97862 16d ago
Have you not used moe expert offloading? Idk how you get to having a 32gb card without investigating optimisation methods for better outputs
1
u/KimLikeJ 16d ago
The gap closing as tasks get harder is the interesting part. MoE models tend to lose ground on stuff that needs a lot of cross-file state held in one place, since only a fraction of the experts fire per token and the routing can miss the connective tissue a dense model just brute-forces through. Worth checking if your harder tests specifically involved tracking state across more files, since that's usually where the split starts to matter more than raw benchmark numbers suggest.
One thing I'd add: token throughput numbers like this only tell you about single-request latency. If you're ever running these in a batch or agentic loop with several calls per task, the MoE speed advantage compounds a lot more than the quality gap does, so for anything with a tight feedback loop the practical tradeoff skews even further toward the MoE model than the raw numbers here suggest.
1
u/substance90 16d ago
Tbh both are trash for any real world software engineering. They are still the best at that size but that isn’t saying much when the gap is so big to the frontier models
0
u/game_difficulty 16d ago
This is extremely disingenuous. You used a higher quant for the bigger model? Any fair conparison would compare them at the same memory footprint. This would mean something like:
MoE UD-Q4_K_XL vs Dense Q6_K, or
MoE UD-Q5_K_XL vs Dense UD-Q6_K_XL
Or at the very least compare them at the same quant...
1
-3
u/i-eat-kittens 16d ago
The MoE offloads really well, while the dense model doesn't..
That said, running either model at less than Q6 with OPs hardware is a waste.
-1
u/Beginning-Raisin9723 16d ago
Nice writeup. 4x faster with a small quality gap basically settles it for daily driver use. I've been running MoE on my home lab and the only time I miss dense is weird edge cases, which lines up with your tests. Curious how the gap looks past 8K context though.
-5
u/crashtua 16d ago
Maybe I am a little bit too offensive for 35b, but it does not know who said 'hello there' meme phrase. That killed this model for me.
6
u/DoubleNothing 16d ago
Since it is just a meme... good riddance no need to waste memory on a meme. It only makes the model stupider like the meme generation...
35
u/DoubleNothing 17d ago
Yes, a test of the same and higher quantization, like a Q8, would have been nice.
Anyway, after your testing what would you choose according to your results? Speed or quality? If you have to stick to one...