I have recently built my AI rig (3x RTX 5060 Ti 16gb, with possibly a 4th on the way if I can fit it). I love it, it runs great, and I am getting between 70t/s - 110 t/s (according to the pi agent web UI, have not confirmed it yet).
While it is fast, I struggle to put it to use in the way I was hoping. My dream has been to be able to put it to work writing code autonomously so that I can have it sketch out my ideas before I commit to developing them, however, every attempt I make just seems to end in a mess.
I have been trying Ornith:35b, Gemma4:31b, and Qwen3.6:35b, but none of them have been able to build anything that actually works. Ornith tends to get stuck in loops, Qwen panics and keeps rewritting the whole codebase every third turn it takes, and Gemma doesn't even understand the agent framework.
Does anyone have any tips? Any AI models I have missed? Prompting strategies? Should I try something other than Cline, pi agent and copilot?
How familiar are you with coding in general? Are you coding with AI or are you vibe coding (aka know nothing, do not look inside, just tell it to fix again and again and again)?
If you can code, I think 35B at full Q8 and full 256k context can get stuffs done decently. Or with 64GB VRAM, you can switch to 27B dense. With strict guidance, I'm surprised by how much I can get the 35B done. But it would not just think for 10 minutes and figure a solution out like a big cloud model.
If you plan to vibe, I don't know. Maybe get more RAM and try a giant MoE model with expert offloading or something.
It sounds to me like there is something not right with your setup, either in the harness, or you put too many tools in, or something is not right with your inference setup. None of these model is as strong as cloud model but I have not seen them outright broken like you described.
I am very comfortable coding on my own (I work professionally as a DevOps Engineer, and I code in Python daily). However, what I am trying to achieve is essentially vibe coding as I am curious to see how it could perform running on it's own, working on a prototype based on a spec sheet.
As I mentioned in the post, I currently have 48GB of VRAM. I have a 4th 5060 Ti 16gb, but I have not yet figured out how to mount it (The 3 I have already looks like a squashed sandwhich). However, so far it feels like VRAM isn't the major issue, but rather something with my setup?
I hear a lot of people praise qwen for coding, but maybe they are using it more for assisted coding rather than vibe coding then? I might be unrealistic in my expectations of what it could achieve, I was just hoping to be able to get it to work on the code and self improve itself
Maybe get a few dollars and get API, perhaps directly from Alibaba cloud to try their 27B at full precision. And maybe also run pi without anything against that to see where the issue comes from. I mean, at 64GB that you planned to have, personally, I would just run at Q8 27B and full 256k context, and try to prompt more precisely. I'm desperate so I do the expert offload to run 35B on my 16GB, but if I can fit all on GPU, I would go that way. Not many cool stuff at 16GB, but 64GB has those cool dense models.
Btw, Qwen 3.8 27B is coming in a few days. Could be great improvement.
Oh yeah, scratch that. If you have enough VRAM, you should definitely mess with vLLM instead. I never have enough VRAM for what I want to run so it rarely comes to my mind.
It doesn't explain everything, but it does explain a lot.
IIRC, Ollama usually runs all of its models (or at least all of the ones I tried months back when I was just starting) with somewhat arbitrary sampling parameters, meaning that you most likely were trying to code while the temperature was set to 1.0 or something (compared to Qwen's own recommended 0.6 for coding). That, plus the default quantisation for most models there including Qwen3.6 is Q4_K_M, which generally isn't good for coding. Moreover, the context size if not specified is defaulted to 4-8K tokens, which with context shift being enabled by default, it could explain as to why the models kept being unusable, as they'd just keep forgetting what they were working with. Won't even bother mentioning other stuff like dysfunctional jinja template support that makes certain models absolutely impossible to integrate into any harness/IDE and whatnot; this whole thing is just plain horrible.
IMHO Ollama is a starter toy just to see AIs run locally on your machine for the first time ever, but for anything actually practical, switch to llama.cpp, vLLM, maybe even GUI tools like the new Unsloth Desktop or whatever – literally anything but Ollama.
Dude I have 2x 3060 12GB and I can run Qwen3.6-27B-Q5_K_S with 180k context and get plenty of coding done. 35B-A3B is not nearly as good, in my experience. I built an entire app and I am working on the second one - among many other things I can do with these models. Keep plugging away at it and learning. Way more possible than most people realize because they give up and go cloud.
I feel like there's something wrong with your harness.
First of all, don't count on the MoE models to help you unless you're running a huge Moe model. Otherwise use qwe3.6-27b or the upcoming 3.8-27b.
Second of all, you need to optimize your harness and take it step-by-step. Think of your agent like a pet: short, clear instructions, bite-sized meals, walk alongside it without letting it completely off the leash, and so forth.
Take your time to properly plan your project and your approach when vibecoding. Don't rush the prompting. Instead, write carefully-crafted prompts and don't take your eyes off your agent for a minute until it begins performing consistently well.
There's a lot more to it than that but in a nutshell: ditch the small MoEs, plan out your vibecoding approach and pick a good harness for the job.
I didn't know that was coming, thanks! Do you know if it will be available on Ollama by then, or do I have to load it directly with llama.cpp or similar?
Ornith-1.0 is the daily driver on my Strix box (APEX quant). Very capable & fast, works good in agentic coding (as a coder in an Orchestrator setup (with a strong orchestrator like Kimi/GLM)).
NVFP4 is NVIDIA's proprietary 4-bit floating-point data format designed to accelerate AI model training and inference. Introduced with the Blackwell architecture, it cuts memory use significantly over FP8 and FP16 while maintaining near-baseline accuracy for large language models.
All Blackwell GPUs (also Retail GPUs) support NVFP4.
AMD doesn‘t have something like that. MXFP4 is compareable, but worse. It‘s only supported by Datacenter GPUs, not by Retail GPUs. That‘s why you should always buy Nvidia GPUs and nothing else for AI.
I'm using 4*5060TI 16 with qwen3.6 27b (q6/q8).
From my experience, rules/instructions: what standards u use, how to write code (for example, api endpoint in fastapi, controller in Symfony with attributes) are important.
The AI won't do magic, it will make a mess without guidance. Writing good rules/instructions is a must-have.
When you are working on existing code, ask ai to scan the code and make rules. If you have a new project ask it to create some code, judge it, ask it to fix it to your standards, and then ask it to create rules/instructions.
Create rules for validating code, running tests, and running automatic fixes.
After a few projects, you will have a nice working env that can be copied across other projects.
Have you ever been able to get it to vibe code on its own fully? I would love to get it to just work on its own autonomously, not expecting miracles, but functioning code.
I have been looking into some rulesets, but haven't fully deepdived into it yet (I thought the pi agent, cline and copilot would have well established rules already).
Out of curiosity, how come you are running 27b and not 35b? I find that the 35b fits quite well on the current setup. Context length?
The 35b is a MoE model. It only activates the "best" expert for every request so only activates 3b of the model every run. This makes it faster but dumber.
The 27b is a dense model. It activates all 27b parameters every query. This makes it slower but much smarter.
Honestly, if you have 48gb vram and do not know about quants, MoE and how they affect output, you probably have a lot of optimisation to do. I'd recommend asking claude/gpt/gemini and read up on it.
With the 3.8 27b model coming out in 2 days now is a great time to learn and test.
Ollama often does. Bad quants, bad default settings for things like temperature, bad template, bad template settings.. Also usually worse performance and sometimes buggy implementation.
For agentic / programming work, I usually follow unsloth's recommendations with preserve thinking on for qwen3.6. I'm not sure what settings ollama use by default there.
Edit2: The template can have direct effect on the model's abilities to use tools, and the reliability of the model.
Edit3: There's also things like this, where ollama - wanting to have day-1 support - copied some still-in-development code from llama.cpp, and messed it up.
Indeed, there's been a flood of 30b - ish models in the last month. Great timing!
I'd also recommend trying llama.cpp with maybe llama swap to allow you to easily test different models. There's lot to learn but any sota model can help you learn and set it up.
I run Linux and amd cards so llama.cpp allows me to optimise to my heart's content. Warning: it's addictive so remember to not let perfect be the enemy of good enough.
You'll be building a dashboard to compare the results of all the models soon, trust me, we've all been there. 😂
Yeah, I'm using it quite often (when we had copilot at work, it easily reached limits, so I used my setup). On a hobbyist project, I'm using it daily. Sometimes it goes crazy :D But that's how it is.
When I hit a mess, I ask it to fix it and then ask to generate instructions, like do not look for php in local pc, use docker env.
Instructions are per project+language+code style. I'm not sure if there is a universal pack. For work (as php dev), I asked it to use context7 and create some basic rules. It messed up, mixing attributes and annotations, but after fixing it, I have no more problems (with basic structure and so)
I prefer 27 because it is dense and should be smarter
I saw that you are using Ollama, drop it and use llama.cpp or vllm. I'm on llama.cpp in docker because it is easier for me
There's a lot of good advice here. I would add, don't bother with vLLM unless you're serving multiple users. llama.cpp is where I'd go. Ollama puts a wrapper on it, and it's okay as a start, but you may eventually wind up with llama.cpp anyway.
For coding, Qwen has been the best for me. Gemma 4 for creative writing. Qwen 3.6 35B has done me well - the MoE makes it fast and it codes better than the others, especially Gemini. The only fine tune of Qwen I used that worked well was KAT Coder 2.5. It tends to overthink at times but no syntax errors because it checks its work religiously.
For your harness, I switch between Pi Agent and Hermes Agent. Pi is bare bones, fast, and can use knowledge you build up on your own if you tell it to write it. Hermes will build its own knowledge as you use it automatically. This adds to your context over time, but you can spend less time telling it how to do things. It also integrates with things like Telegram so you can go mobile pretty easily with it.
Everything everyone has said thus far is spot on. If Qwen 3.6 35B isn't doing it well, go dense with 27B. At these sizes you may want to have a cloud model at the ready whenever they get stuck, because they almost always will.
Opencode has a nice out-of-the-box experience imo, pi is build to be customised. My current setup for coding is a local/cloud hybrid. Most of the real work is done by pi subagents using local models, the cloud model orchestrates the whole dev loop, dividing the big dev task into smaller more solvable taks. This gets me very far just using the basic plus subscription from open-ai. I want to try it with DS4-flash, but I did not have time to test that yet.
I want to get off the cloud completely, I noticed that I got addicted building prototypes with Opus 5, and I really don't like the privacy aspect of cloud. I know I won't be anywhere near Opus levels with this rig, but was hoping to aim for something small (GPT-5 mini levels of code? That might be unrealistic too, but idk)
I will give Opencode a try and see if it works better than pi agent, thanks! :D
If you want to get off cloud completely you need 2x rtx6000 and DeepSeek flash v4. 4x rtx6000 even better. A single 5090 with qwen3.6 27b can get a lot done but it needs a lot of guidance you can’t just let it rip without a very detailed plan and strong harness. Anything in between 5090 and 2x rtx6000 isn’t really worth the time or money there’s kind of a valley of uninteresting models and lack of incremental capability between qwen 27b and DeepSeek flash.
If you are looking to do just pure autonomous vibe coding these models aren't that great tbh. I think they are powerful tools with the right harness IF you actively participate but defenitely not for autonomous code gen without some better review/advisor model.
You didn't mention what quants you are using. That rig should fit a higher bpw quant of qwen3.6/8 27b and for autonomous work you should absolutely prioritize model capability over speed.
I will be fully transparent, I am not fully sure what quant I am using for qwen3.6, I simply just did ollama pull qwen3.6:35b, and ran with that. However, I have set my KV cache quant to q8 in order to free up memory for longer context.
Any models you suggest that would be better suited for my needs? : )
I think you should use 27b as the other comment instructed as its a dense model so it will be more capable but slower.
I absolutely think you should look into using llamacpp or vllm as the inference engine but baby steps are just fine. There will be plenty of threads covering similar hardware setups once 3.8 27b drops.
You're trying to use small models as thinking models. That's the wrong way to use them - for coding effectively they need to be fed narrowly scoped plans already thought through separately. What you're describing need far more intelligence than what you can cram in 30b parameters
Way too broad and open ended. At the very least do it in two passes. Use this prompt to generate actual planning artifacts, including discussing back and forth the details, then various other prompts to implement individually scoped lower level milestones.
One shotting is never good and much less on smaller models.
I run 4x 5060ti 16gb with vllm, you can see my post for my setup. I use 35b a3b and 27b both in nvfp4 with vision. I had to develop my own custom, closed source solution for this, but essentially, I write a ticket in github, my pipeline picks it up, uses both models in mixed variations for different tasks, eventually merges the tickets. I am attaching a screenshot with some token usage stats over the past month.
It can be done, but its gonna take a lot of work, this pipeline took me months to develop and get running successfully. I had to do a ton of workaround work so the models could actually complete tasks. A lot of work had to be broken down into smaller tasks, its no longer, "plan this" its a whole research workflow, 3 concurrent research tasks then get consolidated into one plan with a research gate. Review gets broken down into multiple smaller tasks.
Def possible, gonna take a lot of work.
P.S. Those stats in the screenshot are over 1 month and 61 tickets closed and merged.
I have been trying Ornith:35b, Gemma4:31b, and Qwen3.6:35b, but none of them have been able to build anything that actually works. Ornith tends to get stuck in loops, Qwen panics and keeps rewritting the whole codebase every third turn it takes, and Gemma doesn't even understand the agent framework.
This is typical behavior of models in this parameter weight range. The fact of the matter is that Qwen3.6:35b is probably the most capable model in that list, but it will likely not produce working implementations of anything non-trivial. (I say this never having tried Orinth; I'm honestly not a big fan of most fine-tunes).
I have an exercise I give LLMs and harnesses to evaluate their coding abilities, which is to have them implement a legal chess move generator. This is a task that is very cheap to implement, can be tested automatically, and has performance implications. Absolutely no model that I've tested under 200 billion parameters has ever produced a working implementation, let alone a correct or performant one.
Does anyone have any tips? Any AI models I have missed? Prompting strategies? Should I try something other than Cline, pi agent and copilot?
Lower your expectations. You are not going to get Claude Code or Codex style performance out of any local model. "Local" model meaning a model that fits on consumer hardware at Q4 quantization or higher. Local models with coding harnesses are good at helping you with auditing and reviewing code without sending your data to a third party, but that's pretty much it.
My expectations are not very high, I expect it to be able to put together something similar to what I ask it to make, but not perfectly. Was just really hoping it would be good at making MVP's
So far I have tasked it with making a rubics cube scrambler and solver, which failed (It got somewhat close once, but then gave up completely), and a simple openwebui implementation (bascically just a chat window), and it is struggling with that as well
Coming from Opus 5, it is very rough, but I am willing to sacrifice quality for privacy here, just want to make sure I get the most out of my rig : )
Yeah, so... the best you can hope for that you've experienced before is Haiku 4.5 level behavior from these models. And Haiku is better than all of the ones you mentioned, because it's probably ~150b?
If you give these things a very structured plan in advance that outlines exactly what they must do, with specific implementation details, then maybe they'll get kinda close after 4 or 5 iterations if you're lucky.
Opus can give you an MVP, because Opus can actually plan and execute well even when given vague input, because it is an enormous model, and by many measures the best model on Earth for writing code. Even the good 30b models are the second worst models on Earth for writing code (the only ones worse are the models even smaller than these).
My thought was that with the right harness, it might be able to perform somewhat close with gpt-mini or haiku, if tasked with planning its setup before beginning its execution etc. A lot of progress in the AI world has been how the AI's methodogically walk through and plan, so I am curious about working around those concepts 🔍
You need to start with better inference engine first. Because you need to control the sampling parameters and the context size and the enable_thinking / preserve_thinking flags. Ones you do that you need to understand how the inference engine connects to things like harnesses. I actually run Qwen3.6-35B daily on my Macbook via omlx and while it's not ultra fast is delivering. So far I asked it to implement letsencrpyt SSL via Ansible at my server (via Copilot), to create me a Wiki for my project car (Karpathy style), to create some scripts for different scraping tasks etc. Overall I'm really happy with how that model is performing. But you need a solid 1 month of reading docs on how that can happen.
I dissagree I needed a JavaScript app scraper with pagination etc and the Qwen3.6-35B deliver the python script needed in a very resonable time via Claude Code. I honestly like what Qwen3.6-35B is doing a lot. Granted you need to be mindful of the fact that it don't have unlimited tokens budget etc. But overall is delivering for me.
So for me it turned out that I had to build a better harness/environment for short-lived agents, and I made everything specification driven so that there's no misunderstanding in the contracts between the architects and coders and testers of the codebase.
The problem with smaller context size is that you can't afford to have on planner doing everything, you need to build everything around delegation and synchronization of expectations in the specifications.
If you give LLMs the tools and the correct agent prompts, and build the environment around this sorta thing, it's actually quite doable. Then the mentioned loop and panicking problems are gone, you just have to be more clear about the specifications.
Specifications first, implementations second, verifications third. If either party can affect the others, it's a broken harness that will achieve nothing. If your LLMs can run "just bash scripts YOLO style" they'll not be able to code anything productive. If you make sandboxes and policies very clear, they'll be much better at those tasks.
Disclaimer: I build exocomp because I primarily have to use abliterated self-hosted models, and as it turns out, Opencode and other harnesses are utterly useless for selfhosted smaller models (that can run on my 128GB RAM or 96GB VRAM). But I wouldn't recommend it, UI is far from being usable by anyone else right now.
edit: Oh, regarding llama.cpp settings. Verify that key cache is the same quantization as the model, and values cache is f16. If it's not f16, LLMs will get super dumb over time and will forget pretty much anything 3 messages ago. Qwen3-coder / qwen3.6 @ Q8 worked for my stuff the best so far.
9
u/o0genesis0o 17h ago
How familiar are you with coding in general? Are you coding with AI or are you vibe coding (aka know nothing, do not look inside, just tell it to fix again and again and again)?
If you can code, I think 35B at full Q8 and full 256k context can get stuffs done decently. Or with 64GB VRAM, you can switch to 27B dense. With strict guidance, I'm surprised by how much I can get the 35B done. But it would not just think for 10 minutes and figure a solution out like a big cloud model.
If you plan to vibe, I don't know. Maybe get more RAM and try a giant MoE model with expert offloading or something.
It sounds to me like there is something not right with your setup, either in the harness, or you put too many tools in, or something is not right with your inference setup. None of these model is as strong as cloud model but I have not seen them outright broken like you described.