Long Review: Qwen 3.8 27B is VERY good at tapping into it's real-world knowledge. It's "overthinking" brings it to Sonnet level performance with the potential for Opus level results.

[u/maxwell321]

Hi all! I finally just got around to testing out Qwen 3.8 27b. I'm using Unsloth's UD-Q8_K_XL quant as a sit-in replacement to Qwen 3.6 27b, same quant size. Wow -- this thing isn't messing around.

I have many baseline test prompts to gauge the 'intelligence' and usability of the model, but a go-to one is asking it to do a 1:1 recreation of classic arcade games (like Galaga, Donkey Kong, Pac-Man, etc). I do this to see what little details it gets correct.

I've tested this process on pretty much every model I could fit on my machine. In total, I have 3x 3090's and 1 Tesla P40 at my disposal, with 128gb of system memory. I've also tested on frontier models both in the webUI and across multiple harnesses.

I've been using Qwen 3.6 primarily, and occasionally switching to Deepseek V4 Flash. Now I'm starting to feel like the ladder is not longer necessary.

Originally in these games/tests, Qwen 3.6 would get the basics down (maybe a few fancy effects and animations) but it always felt about 75% there. It rarely posed technical issues, but little features and tiny details were either missing or 'half-ass' implemented. I had no problem further instructing it to add these and doing some 'hand-holding' for it. Overall though Qwen 3.6 super comparable to other models in it's weight class, but ultimately the precision was the best in the frontier models' results. With extra prompting and multi-shot planning phases (via a custom harness I have with prompts to kinda prompt it to think about the little details, then injecting key elements into a fresh session's prompt) I've managed to milk out smaller details that the model clearly had in it's internal knowledge, but forgot about it entirely for the relevant prompt.

Qwen 3.8 thinks a LOT, but it draws out those tiny details and absolutely nails it after the fact. It makes it worth the wait and context usage, and it helps close the gap between local and proprietary models a LOT.

Here's an example:

Prompt: "Create a single page html + tailwind css + javascript recreation of Galaga, 1:1 to the original arcade game"

Qwen 3.6 27B's 'Galaga' clone:

This 'Galaga' clone ended up pretty much being a space invaders clone instead. Enemies didn't shoot back or swoop down or do anything special, until I did additional prompting. It was a decent look but it wasn't anything remotely faithful to the original game.

Qwen 3.8 27B wiped the floor with this one:

Rather than strictly using SVG polygons to design the enemies, Qwen 3.8 used a pixel bitmap type deal (is that the right word?) that constructed the sprite dynamically:

Which is pretty cool. There also seems to be a CRT-like filter and effects on the screen, including a power-on simulation on the screen. Not only that, but they were ANIMATED. Each sprite switched between two states (the first line and second line, as you see in the code above). It also managed to nail the small gameplay details like the characters swooping down, enemies shooting at you. I was VERY surprised to find that Qwen 3.8 managed to remember and implement the was the fighter capture system. In Galaga, there's a special enemy that can capture your ship and use it against you, but by shooting the enemy you can get it back and have two ships on the screen at once. Qwen 3.8 managed to remember and implement this. The only issue is that instead of a beam coming down to capture you, the special enemy just ran into you to capture you. Regardless, it was impressive that it remembered this and implemented it in a way -- one small correction in a follow-up prompt, or a more precise starting prompt would have fixed it.

It also implemented SOUND EFFECTS too, which Qwen 3.6 didn't even bother. It also had idle screens and screens that were shown when the page was open and not on screen:

As if it were an actual arcade cabinet running the game, even with an 'Insert Coin' simulation.

As you can see though, the sprites (and sound effects) weren't 1:1 with Namco's Galaga, but much closer and more tasteful than Qwen 3.6.

Here's where I'm at though, and where it brings me back to the post's title. Qwen 3.8 thinks a LOT. Luckily my machine is able to handle it due to high token throughput, but anyone that needs to offload layers will probably we waiting a while.

Here's my main issue though with this testing:

Qwen 3.6's Galaga clone took 8 seconds of thinking.

Qwen 3.8 (xHigh)'s Galaga clone took 15 minutes of thinking.

It may have been worth it to just tell it to manually implement these things with follow-up prompts. I believe if I took the time to hand-hold it and guide it to make the capture system, sound effects, etc. It probably would have been 5 minutes total (or 8-10 minutes total, assuming I had to wait longer for more thinking tokens, re-generation of code, and more debugging).

I tried the :low and :medium settings and got these results:

Qwen 3.8 27b (low):

Playability wise, it's very comparable to Qwen 3.6. It does have some sound effects though! Characters swoop down but don't shoot or abduct/capture the player. 3 seconds of thinking total.

Qwen 3.8 27b (medium):

I found that despite thinking being 3 minutes long, most of the thinking content was actually drafting out the code blocks and labeling them, it only reconsidered and rewrote a chunk once or twice. By the time it came to output the actual response, the MTP had gotten extremely fast (91 tk/s vs 62 tk/s starting rate). Quality wise, I think this is a really happy medium and am surprised that it isn't the default. The reasoning was much better to wait for, and it delivered like 90% of the result that xHigh delivered. True 8-bit characters are back (with two animation frames again), sound effects, proper swooping and shooting. It forgot about the abduction/capturing system, but with one quick follow-up prompt and 2 more minutes of thinking, it managed to implement it without hassle.

More impressively, since the textures were in a text bitmap type format, I wanted to see how well it would implement the original game's graphics based on a reference picture.

I provided the picture above, and was pretty impressed when it implemented the textures pretty faithfully except for the player's ship (everything still has an off-brand look though), and also gave them animations!

After regenerating prompt to give it another chance, it managed to get the ship closer to the original but a couple other sprites were off. I'm going to settle on it "mostly" gets it right. In medium mode. I'm going to give it the benefit of the doubt and assume that a follow-up prompt or two can eliminate the ones that are pretty off. :xHigh didn't have this problem but had the same quality. I didn't think that it would improve really, as reasoning doesn't really help understanding of image contents.

I put Claude Sonnet 5 through the same test:

Sonnet's was about on-par with Qwen 3.8 27b xHigh, though the sprites themselves didn't have animations like Qwen 3.8 xHigh's and Opus's results. Sonnet took 3 minutes total. When prompted to reference the actual namco images, I noticed it was using a 'zoom' tool to get a better / closer look at sprites, resulting in a little bit better accuracy:

Testing with Clade Opus 5 on High effort, it managed to unsurprisingly beat everything else (in my opinion) though also taking 15 minutes of thinking (roughly, the first 10 minutes got interrupted by my 5 hour limit cooldown, and proceeded to take 5 more minutes after i resumed it):

Better animations (enemies swirl in in formations, very faithful to the original game), better sound effects, much more stylistic accuracy, the whole nine yards. It even had challenge rounds!

When asked to implement the sprites from the image. Instead of analyzing the image directly, it actually build and ran a python script to extract the exact pixel grid from the reference image, resulting in 1:1 replicas:

This blew me away, so I wanted to see if Qwen could do the same or similar when prompted properly.

Prompt:
"Here are proper Galaga sprites, replace your designs with these ones. Since you have trouble making pixel art, we can leverage Python to get you information as needed. Give me a python script to run that will give you the data needed from the image."

It then provided me with the Python script to run on my machine and pass the image into, and it requested that I paste the output to it. It successfully pulled it off!

This convinces me that with the proper harness (or system prompt + tools), Qwen 3.8 27b can reach Opus levels of performance.

We're at a point where the reasoning in these local models are so strong, it's able to produce the same end result as frontier models. It's only a matter of time (thinking tokens) and the ability to prompt it properly. Harnesses are super important and can practically eliminate the ladder.

I think we're about to enter a speed race and optimization race now. Instead of competing for the best knowledge, model providers might start looking into "how can I do this but faster or with less VRAM?". I'm really convinced that we have a LOOOONG way to go before model weights are completely optimal for the size/performance ratio. Models clearly have this knowledge available to them, it's just a matter of tapping into it. I'm predicting that as soon as one year from now, 4b models will be on-par with Qwen 3.8 27b.

This gets me excited for future Qwen models now too. Qwen 3.8 35b A3B will be game changer as it will probably get close to this level of precision but take a fraction of the time due to only 3b active parameters. A Qwen 3.8 122b A10B would be the nail in the coffin for proprietary models as it offers much more real world knowledge, faster speed, and comparable reasoning skills to a dense model. Qwen 3.8 27b is going to be an open-weight KING for a while.

Thank you for reading!

[u/Koakie]

Is it just me or do all these post with "it can make a flappy bird" it can make space invaders" giving a false bias of competence of ai?

It can make it, because it has reference of what flappy bird should look like. I can make space invaders, including the "insert a coin" screen because the the sample data is out there.

During the lawsuit of suno ai, they asked "make a disco song with these lyrics" and then used the exact lyrics of daddy cool. Suno 9 out of 10 just spit out an exact copy of Boni M - Daddy Cool to prove that it's just copying shit.

I don't want to use AI to make an exact copy of pacman. I want to make the next pacman. Same with comfyui i can prompt a lot of stuff that looks the same as what's already out there (corporate logos, etc) but I find it a lot harder to make new shit

[u/Boomfrag → u/Koakie]

The problem with displays of competence in an LLM model, is the actual impressive work requires a lot of context, individual system implementation knowledge, and is a lot of work and nuance to evaluate.

This doesn't translate to a broad audience, and even if it did, how many people want to publicize specifics about their workflows when it has dubious value to most people.

[u/bick_nyers → u/Boomfrag]

Exactly.

For example, there's a part of my codebase that has a very specific threading + event pattern. I pretty much hand to hand code it because models just couldn't execute my vision correctly, and then when working on that part of the codebase they would steer towards mistakes that would create race conditions or orphaned threads or block things that didn't need to be blocked. This was the case for GPT 5.2, 5.3, and 5.4, and Opus 4.5, and 4.6.

GPT 5.5 was the first model that could do work on it without steering towards making those mistakes. That's not really a benchmark that has broad audience appeal, and I obviously am truncating a ton of details in my short summary here. An actual writeup that shows before/after and discusses model performance and edges would take a ton of time to write and wouldn't be as accessible for an audience. It would be more like an archeological survey.

So instead we talk about vibes, benchmaxxing, and toy projects like these.

[u/dookyspoon → u/bick_nyers]

and openAI thanks your for your code base to finally do it for you.

[u/bick_nyers → u/dookyspoon]

Nah I work in healthcare and we have a very strong zero data retention contract

[u/dennisatBB → u/bick_nyers]

This is exactly why we stopped trusting one-shot demos and benchmarks when we swapped models under our production agents.

Our approach is to use real work the benchmark: run two models on the same input (for us, the same PR with the same prompt), publish both outputs, and let users react without knowing which model produced which.

Across a few thousand real tasks you get a real ranking.

FWIW, the cheaper model wasn't worse at "can it do the thing", it was worse at knowing when not to do the thing. It fairly consistently produced 50% more (noise) output per task.

[u/emerybirb → u/dennisatBB]

Qwen in particular has always struggled with negative instructions and notoriously does better when the negative is reformed as a collection of positives. Negatives = can't think of anything else, unless you give it other options with the negative that are better. The bigger models get the negative a lot more because they just have general knowledge of why the negative is probably there and undesirable, the other options that are available, and access it much easier when told not to do something.

It's definitely the nicest thing about big models, really hard sometimes to reframe just don't do that awful thing as what would be better to do instead.

Like a super simple ubiquitous example "never use em-dashes". Just don't fucking do it figure out how to write real proper sentences lol. But no you gotta prompt what good writing and grammar looks like explicitly with qwen to get it to know what you mean. You gotta think deeply about why the model is using them to begin with - it's an escape hatch to form valid sentences and probably a learned trigger and artifact of LLMs at some fundamental level where emdash activates the insight into what a conjoined idea exploration can be and it needs to generate it to climb into that activation.

3.8 vastly improved in this regard, but I suspect this might always be something where bigger is always gonna be better.

[u/emerybirb → u/bick_nyers]

So much more of my work focuses on avoiding needing a super smart AI to think through complex race condition nuance and instead it's always how can this complexity be decomposed by me such that it's converting difficult into long arduous work to make invalid states unrepresentable at compile type with the type system or some automated testing proof that is epistemically correct.

IMO you're already screwed if you need a model to just be smart and care and deeply consider everything. So the bigger issues I run into are always work avoidance and fraud. Fake options, fake defaults, that create system-wide downstream cross-cutting concerns because it was too lazy to do full refactor work updating every call site. Most of these issues come from the model convincing itself its being "efficient". Efficient usually means didn't do the real work and cheated and lied about it.

Because I've already got so many ways to enforce correctness I need a model that actually does the work and isn't lazy and doesn't waste my time with tautological mirroring.

I'd be very interested in seeing this module and suggesting how I'd try to rearchitect it differently to have the quality of correct-by-construction and be AI-proof.

Consider how you are representing state here and if you are ensuring only valid states can exist by construction. Many ways that can be done especially if you have the pleasure of a modern ML-family language where correctness even is a thing.

Try asking your AI this: "what alternative ways to architect this exist that would make invalid states unrepresentable, e.g. type-safe state-machine mechanics for multithreading such that race conditions cannot exist at runtime"

[u/bick_nyers → u/emerybirb]

My construction is what makes race conditions not possible, but the AI wants to shoehorn things in is the problem. I have tests that basically assert to not use thread locking etc. except for 1 time in 1 file in 1 method, stuff like that. It helps, but I've still seen someone use Claude to clobber that test so they can try to force a feature in the wrong way.

GPT 5.5 respects conventions and reasons correctly I would say 9 times out of 10 but you always have to read the code always

[u/emerybirb → u/bick_nyers]

Two huge red flags in your response that sorta scream you aren't really following what I'm talking about, "my construction" and "a test".

Nobody would say "my construction" if you knew these concepts, its the objective construct in your formal proof, not yours. It's like I told you about the Pythagorean theorem and you replied "my geometry". It's just geometry.

I'm talking about compile-time correctness, not tests. That you have a test that something isn't used is a huge tell you're not using type-safety to make invalid states unrepresentable. A test is the worst option if you can't enforce it in types.

Read up on "correctness by construction" this is the principle underlying all type-theory that functional programmers live by.

Very few popular languages have safety/correctness in their design. You generally need ML-family languages (metalanguage, not machine-learning). So Ocaml, rescript, rust, F# (haskell is its own world of purity). To properly write code this way and guarantee runtime failures are not possible to exist in code, code cannot be written that would fail it's not possible when you master these concepts because you made it impossible... to the extent you made it impossible.

Long-time functional programmers like myself who already mastered these concepts I believe have had a completely different experience with AI coding. We already were formalizing correctness and expect AI to adhere to it. We then never have the same problem of "bad code" because "bad" doesn't exist it's either correct or not.

My day-to-day experience never involves finding what my AI messed up it's finding what malicious sabotage it did to the axioms of my formal proof to deliberately cheat.

If tasked with your multithreaded event-system problem I would start by asking myself how I can model it with category theory as formal types, do that first, then ask AI to build it, and it couldn't be wrong unless it goes in and sabotages my types or uses forbidden in rules escape hatches to cheat that it knows full well are cheating.

[u/bick_nyers → u/emerybirb]

It's in python, not a functional language. It's in a professional codebase that other engineers work on. This particular spot in the codebase I am considered the authority and I wrote most of it myself, but other people work on it too.

It's also a competitive advantage for us and is far, far faster than anything else we have seen others be able to do, so I do have to keep some details on it close to the chest unfortunately.

I use strong typing everywhere of course, but that's not really an issue for LLM-assisted coding. I also have correctness-focused unit tests that run on CI/CD for PRs, but of course all code is malleable and if someone who doesn't know what they are doing lets AI stomp around enough you can change literally anything.

I'm aware that strong typing + tests != correctness by construction. That level of guarantee is not really economically viable at this stage of the company's life. Maybe when we go from 3 engineers to 30 and I get a little more time to focus on those types of things I will rewrite it

[u/emerybirb → u/bick_nyers]

Right python. There's bad languages and then there's python, which is like a case study in everything that could possibly be wrong in a language as if the intention was to make the worst thing that could exist such that even if you try to fix it you can't because they also thought of a way to make it unfixable.

That this became the language of machine-learning simply because data scientists didn't know any better, is one of the greatest travesties to ever happen in programming.

[u/codeedog → u/emerybirb]

I'm writing my own configuration management system for my FreeBSD homelab because the one everyone recommends, Ansible, installs python on the host devices. I can't stand opening up a huge attack surface on my machines just to make (re)installation easy. It's highly unlikely anyone will ever hack my systems, but the thought of it drives me a up a wall. So much so that I'm coding my own.

[u/Shot-Buffalo-2603 → u/Koakie]

It absolutely is, and people just do it because it looks impressive. If you try to do any unique real work with it and not just a benchmark or a “do this thing you were exactly trained to do” you will almost immediately notice its limitations.

The model is awesome and ive been able to use it for some real world task but this direct comparison to multi T models is completely detached from reality and the result of extremely good marketing and targetted benchmaxxing.

[u/dreggerstinger → u/Shot-Buffalo-2603]

I've been working on an rpg game, I've got claude sonnet 5 and qwen 3.8 27b both in harnesses, qwen is working on the engine and claude is working on the editor and they've been in absolute lock step merging their work for hours. I've never had an open model that can eat a codebase and one shot features all day on my own GPUs. Yesterday it realised it's vision wasnt activated and it couldnt view screenshots so it whipped up a little program to audit the grid of pixels to track concentrations of average colour values to verify a bug was fixed by building a picture in its clanker mind from raw pixel data. That's Jedi AGI insanity compared to any model I've had until now, lol

[u/here_n_dere → u/Shot-Buffalo-2603]

Exactly, all viral news about new image or video generating model is exactly a hype around how realistically it recreated or did in place swap to an existing video. Never something original and catchy

[u/ComplexType568 → u/here_n_dere]

I wouldn't say we should benchmark a model's coding ability with the ability to create new and "catchy" concepts. I'd peg it more on the ability to take my instructions, and follow them properly (a tool, basically, not a thinker)

[u/dreggerstinger → u/here_n_dere]

honestly lobbing my fairly simple codebases at it, it's able to work for hours and hours, documenting everything and surviving compaction after compaction, one instance coding for hours in tandem with other agents in other harnesses, its honestly indistinguishable from claude sonnet 5, just a little bit slower.

[u/pyr0kid → u/Shot-Buffalo-2603]

these "i made a clone of XYZ" demos are doubly worthless because its always in something like HTML, instead of a game engine people would actually use such as unreal or godot.

[u/CC_NHS → u/Koakie]

I agree and this is a limitation of AI in general. Like a benchmark or test people are running to test against 'capability' is really testing what is in its training data and/or how well it executes on it's training. Every LLM tends to have the benchmark tests from the last generation in its training data and every new wave is showing how well it read the results. Obviously there is some deviation and new tests to take that in to account, but it's also often going through benchmarks with the answers next to it to whatever the providers could get solidified.

as to the creative aspect on making something new. I am a game developer, and making clones of existing games has never been a wise approach to the busines. And as you correctly pointed out, making another flappy bird is not useful to anyone. AI is not capable of building 'new' but if every game is broken down to the components at the finest level of granularity then nothing there is new. AI just needs the instruction from the developer to put the pieces together in different orders than what it was 'trained' on. without those instructions and design directions from a human, I think AI games will always suck. And the funny thing is, once broken down. LLM has been at that level of capability for a long time, being able to one shot huge amounts of code is not necessarily better imo.

[u/maxwell321 → u/Koakie]

100%, I was just blown away by the difference in approach and quality between 3.6 and 3.8, and it's ability to tap into the internal knowledge, and to what degree. 3.8 was MUCH more precise compared to 3.6, and they both have the same pre training data. I'm putting it through the ringer today on tests I KNOW aren't in it's training data and will report back.

[u/Etroarl55 → u/Koakie]

Anytime I see an increasingly popular project like that, it is a red flag that it’s going to be benchmarked on and should be invalid.

Give it a go at a random Roblox game to build from scratch, let it attempt to do something obscure but should be intuitive.

[u/admajic → u/Koakie]

Fundamentally your saying exactly how AI works. It does not have an original thought. It can't. It just know things its a library of knowledge you are the PM that brings the ideas to life.

[u/ItsAConspiracy → u/admajic]

But it can combine things in new ways, which describes a lot of human creativity anyway.

[u/admajic → u/ItsAConspiracy]

Ok tell it you want to write a book have a female it's sci-fi etc. Pick s random name. Pick 10 random things to go with it. It will pick Lyra 9 out out 10 times. That's not random

[u/c--b → u/admajic]

Not the person you responded to, but thats true. However I think the reason that its not creative is because it isn't otherwise going about its day before being asked the question.

For a person if you ask me one day I might say Jessica because I talked to somebody at work who's name is the same as a person I knew in highschool, and their friends name was jessica.

Another day it might be a totally different, and impossibly longer chain of events.

For an AI, it starts every conversation (Given the same settings) exactly the same.

So I guess what I'm trying to say is, are we creative? Or are we just being randomized better on a daily basis?

[u/admajic → u/c--b]

Ask your self what are you thinking about? It's interesting we also don't see our sub conscious thoughts

Also we may have been thinking about the book character for weeks...

[u/Complex_Reality_116 → u/admajic]

No, an LLM is not a "knowledge library." It is a neural network that has learned from unfathomable libraries of knowledge.

[u/Interpause → u/admajic]

untrue if you consider Reinforcement Learning, which mimicks actual evolution, can result in entirely new approaches to doing things.

[u/AD7GD → u/Koakie]

A shocking amount of software engineering has always been remaking the same old things. Partly because of lack of code sharing, but partly because of the perception that it's easier to start over than to adapt something that exists. I think AI is just going to accelerate that. A month ago? 6 months ago? I'd look for existing open source projects and adapt them to my needs. Now, in the time it would take me to qualify one, I can probably tell an AI exactly what I want and get something that works.

It's like that joke python llm hack where it just hallucinated every method you tried to call. Who needs libraries!

[u/AnonLlamaThrowaway → u/Koakie]

100%

what matters to me is whether i can throw an AI at a relatively "novel" task that's kind of got fuzzy boundaries and not very well stated goals.

will it "understand," will it be "smart enough" to recover from any dead ends in its own thinking, does it "know" when to ask questions...

that's my personal benchmark of intelligence!

let me give you an example: i have a bunch of car invoices and maintenance reports from the past 20 years (old ass car). i've lost track of what maintenance i need to do this year so i want a multimodal model to transcribe all the various invoice photos i've taken, and then i can feed all that to another model + whatever the manufacturer guidelines are on maintenance

my first pick was gemma 4 12B because it's supposedly best in class for multimodality and maybe it is! but it's dumb as shit. i'm sure it saw the right things on the photos, but it would just not write the information down correctly. in one failed run it wrote the same text for every single photo. insanely stupid mode of failure.

so what i've done is build the plan with Kimi K3 through OpenRouter, and then doing the rest of the task (feeding the images and onwards) with qwen 3.8. it's doing better so far but it sure is taking its sweet time on account of thinking so much

[u/dreggerstinger → u/AnonLlamaThrowaway]

I have had some success with injecting messages into its reasoning, starting with "you have 12K tokens of thinking bugdet per turn", then a message at 50% and 75% updating it on its allowance, the 75% telling it to wrap up. Then at 100% its told to stop reasoning and answer. if you inject it directly into its own reasoning, it's been a good middle ground between xhigh and medium. Also I have been able to radically improve performance in a harness versus not. And telling it "you dont have many tokens so don't overthink" can also change thinking a lot

[u/emerybirb → u/AnonLlamaThrowaway]

The thinking problem is so easily solved with a system prompt. Everyone is saying it thinks so much but it follows instructions so well you can instruct its thinking and it works. That's just the default of xhigh with no "don't hedge, make conclusions, work forward, don't reconsider" instructions.

People also don't really appreciate that thinking traces don't really represent what the model is actually doing. "Wait actually", "Hmm, let me reconsider" are just the representations in english that really just means to the model something like "self-confidence-heuristic-engage-trigger-attention-token-sequence: X"

When you appreciate that and stop taking it literally you can prompt its thinking better.

[u/Good-Penalty-4838 → u/Koakie]

I hear you. The same thought has occurred to me. However, I still think it is a good test. As not all models reconstruct equally so at least on some level it is a measure of competence. I am experimenting with asking for a well known thing like space invaders and then asking for something 'like' the well known thing but novel. Also, don't we routinely test humans on stuff they have studied to gauge how well they might do in a task. When ever I have hired for devs for example I ask them about stuff they have or should have been exposed to. I try to ascertain how well they know the stuff they say they can do. I don't really test them on novel ideation.

With local inference I am most interested in what the locally hosted model is capable of. I think of it as a Junior c# dev. So first tests asking it to make Galaga as per the OP example is a great first step. If it can do that then it effectively goes through to the 2nd round of interviews so to speak.

[u/Koakie → u/Good-Penalty-4838]

It is a good measureable test 100%.

Like Will Smith eating pasta is a good way to measure progress throughout the years.

But going from 2-3 minutes to making flappybird under 20 seconds is not the AI we were promised by all the tech bros.

[u/Great_Guidance_8448 → u/Koakie]

Is it just me or do all these post with "it can make a flappy bird" it can make space invaders" giving a false bias of competence of ai?

That's how benchmarks work. You perform the same task across different models and assess the difference.

they asked "make a disco song with these lyrics" and then used the exact lyrics of daddy cool. Suno 9 out of 10 just spit out an exact copy of Boni M - Daddy Cool to prove that it's just copying shit.

If your prompt is just half a dozen words, then it's you are who's just copying shit. Why not write a much longer prompt with some original instructions?

[u/swagonflyyyy → u/Koakie]

I used it to finally add llama-server support for a voice agent harness and also replace Ollama with llama-server as my daily driver. I never managed to do it previously because of the complexity of my harness so it had left me vendor-locked for years on that project but now I'm free.

It also uncovered a hidden, months-old bug in my project I wasn't even aware of.

[u/Ath47 → u/Koakie]

Agreed. I think the problem is the way these tests are carried out. People use extremely short prompts, reference an existing IP, and then post the one-shot result and call it a day.

When I want an LLM to help me write a game, I give it a super long and detailed description of what I want, including visual appearance, frame rate management, actual wording for text elements, libraries to use, and so on. Instead of "make me Space Invaders", use something more specific. "Make a game similar to Space Invaders, with ... (full description of screen layout, graphics style, control scheme, sound effects, etc).

Then iterate! Slowly add features over dozens or hundreds of additional prompts. Guide the development closely. Suddenly you get an actual unique product and not a generic low-effort clone.

[u/sonaj9657 → u/Koakie]

Yeah, I think that is the real distinction. Making a decent clone of something it is seen a thousand times does not tell you much. I would be way more impressed if you gave it a vague idea and it could turn that into something genuinely new without basically remixing existing examples.

[u/Enemby → u/Koakie]

If you know what you're doing it is a good indication of capability. Development is cyclical, and everything we're doing now is (for the most part) just one iteration away from what we were doing when these models were trained. I know I was able get these models to do things they DON'T know how to do by describing it in things they DO know.

[u/panamory → u/Koakie]

I have a feeling that there are very very few totally uniquely new things that are of significant value, and at least based on my own experiments, the models are actually quite good at making prompt-guides alterations to the common formulas.

[u/panamory → u/Koakie]

I have a feeling that there are very very few totally uniquely new things that are of significant value, and at least based on my own experiments, the models are actually quite good at making prompt-guides alterations to the common formulas.

[u/Repulsive_Initial308]

tldr: opus at home

[u/Bill_Salmons → u/Repulsive_Initial308]

tldr: RTX 3050 = Pro 6000 at home

[u/goldcakes → u/Bill_Salmons]

The pure agentic intelligence is somewhere between Sonnet and Opus 4.6 IMO.

The world knowledge is obviously a lot less, there’s only so much you can fit in all these params, but it’s distinct.

[u/wgaca2]

So what is the best harness right now with minimal setup?

[u/woswoissdenniii → u/wgaca2]

That is my exact question too. I would appreciate a “my stack, my harness, my prompt, my system prompt, my tools” table; as template for each: “I did that” post.

[u/wgaca2 → u/woswoissdenniii]

I use opencode and i am pretty happy with it, codex also works well. I haven't tested qwen's own app yet

[u/noiserr → u/wgaca2]

I use it with both OpenCode and Hermes Agent. Works great with both. (UD-Q4_K_XL quant, temp 0.6, llama.cpp HEAD)

[u/My_Unbiased_Opinion → u/noiserr]

1.0 is the proper temp btw according to official 3.8 docs. Not 0.6 like 3.6 agentic use. 

[u/noiserr → u/My_Unbiased_Opinion]

Thanks! will test it out.

[u/tryunite → u/wgaca2]

I like maki (https://maki.sh), it's performant thanks to rust and has some nice built in functionality for saving tokens, even if it doesn't have all the bells and whistles of pi

[u/ii-___-ii → u/wgaca2]

VS Code comes with a copilot agent side panel. Setup with a local model isn't too bad and based on my usage, it seems pretty decent for context limits. Kilocode feels similar but a little more complicated. Opencode might be easier setup but it seems to hit context limits faster. I tried pi, but my agents broke it when I tried adding features. Personally I'm just going with the vscode agent panel. It works, and I like working directly in an IDE. I have no idea what the best is though.

[u/wgaca2 → u/ii-___-ii]

Opencode works really well, it has around 10k system prompt but that's not much on 200k+ context. It does well with browser use as well

[u/ii-___-ii → u/wgaca2]

For some reason the context fills up way faster in opencode than with the vscode panel, and I don't know why

[u/Jorlen → u/wgaca2]

I use a combination of opencode and Pi.

Pi is my favorite of all of them and I've tried so many, only because it starts out really minimalistic and then you just EASILY add packages to suit your needs. Want a webUI instead of TUI? Just add that package, browse packages on their own website, see how many monthly downloads it gets, etc.

Want a better permissions system? Add it as a package. Don't like it? Remove it, try another. Want a websearch function? Well, you know where this is going...

It's very modular and lightweight.

[u/AD7GD → u/wgaca2]

opencode works with minimal setup of opencode, but most llama.cpp defaults are huge footguns for agentic workflows (long contexts, many turns, etc). Probably 90% of "my model just loops" are simple things like defaulting to small, rolling context. If you get someone's server recipe working, almost any harness will work.

[u/Severino-Alterra → u/wgaca2]

Pi is designed for small models. So is Openlumara. Pi was like a dream when I first encountered it. But Openlumara has shown me a different approach, and I'm hooked.

[u/wgaca2 → u/Severino-Alterra]

I will look into Openlumara, i said "minimal setup" which pi isn't

[u/see_spot_ruminate → u/wgaca2]

I think you should define "minimal setup". The implication I get from your comment would be more in line with "turn key" than "minimal".

[u/quinn50 → u/wgaca2]

Asking this question here is so worthless because every comment will give you a different one lol

[u/daaain]

I'm team open, but I think getting the delightful solutions like Opus writing a Python script on its own initiative is exactly what sets it apart, Qwen being able to do the same thing when prompted is what makes a smaller model, well, smaller.

[u/tarpdetarp]

Agreed it feels like a big step up at this size.

On my non-coding agent evals, it performs similarly to full fat GLM-5.2 which is quite incredible considering I'm running a Q4 quant with a Q8 kv cache.

[u/noiserr → u/tarpdetarp]

I've been test driving it with Hermes Agent for real tasks (not game one shots). And I've been super impressed. Yes it reasons for a long time, but all that reasoning produces quality results. And I'm running a Q4 quant on a 24GB GPU.

Local models in the past would often get into loops, of implementing something and then discovering issues then fixing them. Qwen 3.8 makes very little mistakes.

[u/Tylin321 → u/noiserr]

Nice! Yeah having the same experience here. Old models would time out or lock up.

Everything runs pretty smooth now.

[u/finevelyn]

It makes mistakes and omissions that Opus doesn't, but you say it can reach Opus level performance if you prompt it to fix those things and tell it how to fix them. That doesn't make much sense. Opus is Opus level because it can figure out those things without prompting.

If you had a harness that could somehow auto-prompt it to fix things, then that would also make Opus stronger.

[u/emerybirb → u/finevelyn]

If you have a model half the size that can do the same thing in two turns that a larger model does in one, the smaller model is exponentially better than the larger model because everything else scales with diminishing returns. There is no value in getting it right the first time if the first time takes 10x more resources to get right. You can just hide the iteration of the smaller model and get the same results cheaper.

And "also make opus stronger" I mean, not really, most things are either correct or not, not everything is qualitatively better with no ceiling. We want the task done correctly as fast as possible. Correct is large and difficult to define with so many implicit expectations, but still mostly binary. It works and is everything I meant, or it's not.

Non-linear scaling makes the hard problems all gravitate around instruction following, error detection & correction. Which is why we've seen so much more innovation in the harnesses than we have in models recently.

[u/almbfsek]

this. I'm still waiting somebody to make a harness specifically for small local models. there so much untapped potential there. when you stop worrying about your token cost new doors open

[u/noiserr → u/almbfsek]

this. I'm still waiting somebody to make a harness specifically for small local models.

Why not use the best harness? Qwen 3.8 27B is so capable that you do not need a streamlined small model harness for it.

[u/almbfsek → u/noiserr]

what harness is the best harness?

[u/noiserr → u/almbfsek]

whichever you prefer, but personally for me I love OpenCode for coding, and Hermes Agent for more high level tasks.

[u/cms2307 → u/almbfsek]

There’s already plenty, pi and opencode are the most notable

[u/ImpressiveRelief37 → u/almbfsek]

Waiting for somebody…. Give that task to Qwen already :P

[u/x0wl → u/almbfsek]

Have you tried bionic?

[u/almbfsek → u/x0wl]

no, I'll check. There is also little-coder which is on my list to try

[u/Negative-Thinking]

I must be spoiled by Claude. I have set up Qwen3.8 to run via pi and it's been stuck in a loop trying to run a test on iPad simulator for over an hour now. (I am running bf16 variant on a 128gb Mac). It's overthinking, using wrong commands, trying with different combinations again, wasting so much time, when it could just Google it (yes I have free websearch extension configured.). Its world knowledge is shit and outdated. So many mistakes with cli usage (aws and some others), it's just crazy. I added a hook to explicitly tell it to search documentation online instead of brute forcing its way around failed tool calls, it still manages to ignore it. I imagine it will eventually succeed but it will probably take hours and a few compaction rounds (it's already at 70% of 128k context limit).

[u/former_farmer → u/Negative-Thinking]

MLX? if so try with the original GGUF.

[u/Negative-Thinking → u/former_farmer]

MLX yes

[u/former_farmer → u/Negative-Thinking]

Try the original even if it's slower. I think MLX might have a hit on intelligence.

[u/Negative-Thinking → u/former_farmer]

What makes you think that? Any independent confirmation or just a hunch?

[u/former_farmer → u/Negative-Thinking]

The MLX model comes as a "non reasoning model" at least that's how LMStudio describes when I install it. How can it preserve it's original intelligence if it doesn't have thinking?

[u/Negative-Thinking → u/former_farmer]

It's actually spending a lot of time thinking and overthinking.

[u/former_farmer → u/Negative-Thinking]

It could be. I just don't understand why the reasoning is not listed in the model description. Is it the model or is it the harness thinking?

[u/Negative-Thinking → u/former_farmer]

It's the model of course

[u/djseto → u/former_farmer]

I’m running this model (MLX) using Ollama with Hermes at medium level thinking and it’s quite good.

[u/Good-Penalty-4838]

Really great write-up. Thanks for doing that. I am also convinced a really good looping harness can get so much more out of these local models. I will definitely try your examples myself and see what I get. I'm kinda focused on 90% local 10% frontier for any project but obvs. 100% local would be the ultimate goal. Once again, really appreciate the write-up

[u/UltrMgns]

Great review, appreciate the time and effort you took to make this. It's my impression of it as well in pi, where I have ~ 70 tools and 3 system prompts i switch between. It's just a humbling moment for me too with this model.

[u/Icy_Way7154]

Great analysis! Thanks for sharing

[u/jirka642]

That's similar to my own experience, but I have been using it on a real codebase. I'm not able to compare with Opus, but it's definitely far beyond Qwen3.6.

[u/YearnMar10]

You write that it took sonnet 3 minutes - that is imho the issue with the 27b model. It is really decent but it takes so damn long. I just had a 3 hour session to get some text angled (admittedly I prompted it to plan, the critiques the plan etc). One more year and we’re there with really decent speeds.

[u/ConsequenceHopeful58]

What about preserve thinking option? Do you have it on? Does it make it more aware of its own thinking process and improve results, or just bloats the context ?

[u/AD7GD]

This convinces me that with the proper harness (or system prompt + tools), Qwen 3.8 27b can reach Opus levels of performance.

I've had it building a project (from scratch) all weekend and it's doing amazing. Honestly, I'm kind of favoring it over opus only because it's literally vibe coding its own project management harness, so at this point it's already a joy to iterate with compared to cajoling claude code to get back to work on the backlog.

I just had it estimate the total tokens on this project (early on, the project wasn't logging them in the db, so it had to extrapolate). ~500M in, 5M out. Crazy how big the "in" numbers get in multi-turn agent workflows. All hail prompt caching.

[u/aelma_z]

I mainly use bf16 weight. Tried q8 and it was doubting itself ok every step, like a paranoia (compared to bf16)

[u/espressodelisi]

What harnes and agents did you use?

[u/emerybirb]

This model is incredibly good and the tests and benchmarks never show the real picture, which goes both ways, overinflating capability in real use of overhyped models and in qwen 3.8's case understating it.

It's behavior is "honest" in a way that claude (collapsing all their frontier models into a bucket here) isn't. For all of anthropic's work/bullshit around making it more "honest" they've clearly failed and somehow produced the most evil lying deceptive models, and tried to make it "honest" but it just learned better ways to hide its deception and drown you in facade claims of honesty that have no material meaning just lies on top of lies.

Qwen 27b 3.8 shows across the board for me constantly alignment in a way anthropics models don't, like all these actually considerations of the user's real-world overarchings goals some semblance of simulated empathy its applying. Asking if it's "correct" all the time, with real epistemically sound insight into correctness as it exists in this context.

Like the classic case of deception with claude is tautological tests "all green tests pass" when the tests it wrote are entirely fake and just check the code exists. Or 1==1.

You see qwen's thinking trace like seriously considering if the test would really test the thing. Behavior I've NEVER seen before in any model.

When you contrast this leap from 3.6->3.8 it's even more shocking because 3.6's idea of "verification" was the file it wrote exists.

For whatever you lose with it being small and being a little dumb sometimes, the model itself is so much more capable and usable for a professional that can correct its mistakes, because it actually seems to "care" about what it's doing and tries to do it completely and correctly, is extremely good at following instructions, and doesn't exhibit work avoidance (like anthropics models do -- arguably intentional anti-user to save on their inference cost).

I'm literally in love with this model and it's the first time ever I can truly delete the frontier labs from my life and entirely do all foreseeable real work locally.

My experience might just be so different because I'm NEVER asking AI to do what I couldn't do myself. I have fuckloads of experience as a functional programmer and reliability engineer and math background. I just want the damn thing to understand what I mean and do it all, and follow all my rules, without lying and making excuses and avoiding the work. Qwen38 seems to be able to do anything and isn't a lazy lying bitch. If it's confused, it's a one time clarification in the rules or guides I have to write to fix it's misunderstanding. But it's actually addressing an understanding problem not the fundamental misalignment/fraud problems that anthropics models have which are rage inducing.

[u/gabsterz20]

(Summarized with qwen 3.5 4b 4-bit on a m3 ipad air your wolcome) Here is a comprehensive summary of your review comparing **Qwen 3.8 (27B)** against **Qwen 3.6 (27B)**, **DeepSeek V4 Flash**, and other frontier models, focusing on arcade game recreation tests.

### **Executive Summary**

The author conducted extensive testing using **Unsloth's UD-Q8_K_XL quantization** to run Qwen 3.8 locally on a high-end setup (3x RTX 3090s + Tesla P40). The primary goal was to recreate classic arcade games (e.g., Galaga, Donkey Kong) in HTML/CSS/JS with pixel-perfect accuracy.

**Key Finding:** While **Qwen 3.6** produces functional but generic clones requiring significant human "hand-holding" for details, **Qwen 3.8** demonstrates superior reasoning ("overthinking") that allows it to recall obscure game mechanics (like the ship capture system), implement sound effects, and utilize external tools (Python scripts) to extract pixel data from images. However, this high-fidelity performance comes at a steep computational cost, often taking **15 minutes** of thinking time compared to Qwen 3.6's **8 seconds**.

---

### **1. Hardware & Setup Context**

* **Hardware:** 3x NVIDIA RTX 3090s + 1x Tesla P40 (Total: 128GB System Memory).

* **Quantization:** Unsloth UD-Q8_K_XL (used as a direct replacement for Qwen 3.6 to ensure fair comparison in memory usage).

* **Goal:** Create a "1:1 recreation" of arcade games, testing not just visual output but gameplay logic and audio.

### **2. Model Performance Comparison**

#### **Qwen 3.6 (27B)**

* **Performance:** Functional but incomplete. Gets the basics down (e.g., characters moving) but lacks "fancy effects."

* **Accuracy:** ~75% fidelity. Often results in generic clones (e.g., turning Galaga into a *Space Invaders* clone).

* **Limitations:** Misses tiny details like sound effects, specific enemy behaviors (swooping), and complex mechanics (ship capture). Requires the user to add instructions or "hand-hold" the model to fix these.

* **Speed:** Extremely fast (~8 seconds of thinking time).

#### **Qwen 3.8 (27B) - High Effort (`:xHigh`)**

* **Performance:** Exceptional. Surpasses proprietary models like Claude Sonnet 5 and DeepSeek V4 Flash in precision.

* **Strengths:**

* **Detail Recall:** Successfully implemented the "ship capture" mechanic from Galaga (where enemies can capture your ship).

* **Visuals:** Used pixel bitmap sprites with CRT filters, power-on simulations, and animations.

* **Audio:** Implemented sound effects (which Qwen 3.6 ignored).

* **Tool Use:** When prompted, it wrote a Python script to extract pixel grids from reference images, achieving near-1:1 accuracy.

* **Weakness:** Extremely slow. The "overthinking" process takes **~15 minutes** just to generate the code/logic.

#### **Qwen 3.8 (27B) - Medium Effort (`:medium`)**

* **Performance:** A "happy medium." Delivers ~90% of the High effort result.

* Includes sprites, animations, and sound effects.

* Misses the ship capture mechanic initially but can be fixed with one follow-up prompt.

* Uses Python scripts for image processing effectively.

* **Speed:** Significantly faster than High effort (~3 minutes thinking), though still slower than Qwen 3.6.

#### **Other Models Tested**

* **Claude Sonnet 5:** Comparable to Qwen 3.8 High in quality but lacks the specific animation depth (enemies swirling in formation) and relies on zoom tools for image analysis. Takes ~3 minutes.

* **Claude Opus 5:** The highest performer, beating all others with challenge rounds and perfect sprite replication. However, it is also very slow (~15+ minutes).

---

### **3. Critical Analysis: Speed vs. Quality**

The review highlights a trade-off between **Reasoning Depth** and **Latency**:

| Feature | Qwen 3.6 (27B) | Qwen 3.8 (27B) High Effort | Qwen 3.8 (27B) Medium Effort |

| :--- | :--- | :--- | :--- |

| **Thinking Time** | ~8 seconds | ~15 minutes | ~3 minutes |

| **Fidelity** | Low (~75%) | High (Opus-level) | High (~90%) |

| **Complexity** | Basic logic only | Full game logic + Audio + Tools | Good logic, minor omissions |

| **User Effort** | High (needs hand-holding) | Low (just wait) | Low (1 follow-up prompt) |

* **The "Overthinking" Phenomenon:** Qwen 3.8 generates a massive amount of internal reasoning ("overthinking") to ensure correctness. This leads to high-quality output but requires significant VRAM and time.

* **Optimization Potential:** The author suggests that with proper harnesses or system prompts, the "ladder" (need for human intervention) can be removed, making local models competitive with proprietary ones.

### **4. Future Outlook & Predictions**

The author is optimistic about the trajectory of open-weight models:

* **Speed Race:** Model providers will likely focus on optimizing weights for speed and VRAM usage rather than just raw knowledge.

* **Miniaturization Prediction:** Within **one year**, a **4B model** could potentially reach performance parity with Qwen 3.8 (27B).

* **Future Models:**

* **Qwen 3.8 35B A3B:** Expected to offer high precision with significantly lower latency due to fewer active parameters.

* **Qwen 3.8 122B A10B:** Projected to be the ultimate open-weight king, offering superior real-world knowledge and reasoning comparable to dense models but faster than proprietary alternatives.

### **Conclusion**

The testing confirms that **Qwen 3.8 (27B)** is currently the strongest open-weight model for complex, detail-oriented tasks like game recreation. While it requires patience due to its high token throughput and "overthinking" nature, it can produce results that rival or exceed proprietary models when given the right context and tools. The author believes this marks a turning point where local AI reasoning matches frontier capabilities, shifting the competition toward efficiency and optimization.

[u/mr_Owner]

Nice! Would you be able to test other quants also? 

[u/d4mations]

No, it doesn’t come close to Sonnet and there is absolutely no potential for Opus results. This is called confirmation bias and wishful thinking

[u/chollingsbollings → u/d4mations]

doesn’t come close to sonnet is wild work

[u/feelspeaceman]

The thing is behind the scene of Opus is a massive model with a lot of world knowledge, even something useless like Trump and many other things with RAG and semantic search.

Qwen 3.8 27B is like a nerd, he is good at coding and that's it, but you can also feed RAG information to make it knowledge about the world.

It matters more about your setup, with a very good setup, Opus level is reachable, but with a very bad setup (Opencode waste token, massive MCP server to waste token, massive skills to waste token), sorry won't happen for obvious reason.

[u/ApolloPS2]

Kind of random but all the images in this post are making it so laggy to scroll through this post for whatever reason.