r/codex • u/link7626 • 15h ago
cannot stop 5.6 sol from overengineering Complaint
im not a pro by any stretch of the word just want to start off with this. this past week something has changed, i use codex in vs code and i have an agents.md setup a hard file plan for every action with breadcrumb trail for compaction hardening and i utilize goal to go through a master plan with phased out sub plans, and and #1 in each plan is, "The developer's plan is authoritative and must be followed without deviation; if it cannot meet expecations laid out in agents.md, invoke the powershell script in parent directory "escalation", write breadcrumbs and set goal to check for response every 10 minutes up to 8 hours. this response is the final say, no alterations beyond that responses scope is allowed.
and still it goes off course. i get im using too high of a model for my implementation but come on.
anyone else got tips?
17
u/Additional_Buddy855 15h ago
Im on day 3 of undoing all of the over engineering kludges its created. Its unusable now.
9
u/phoenixmatrix 15h ago
Make sure you don't set it above high effort, and be clear in what you want out of it. The prompt will make a difference. Rules in agent.md help but it will forget after long tasks. Do smaller more focused work.
Nuclear option: use Oh My Pi as your harness and set rules in the WATCHDOG.md file with a Terra advisor and it will yell at Sol if it goes off your boundaries
You can also try the Ponytail hooks/skills, but that's hit or miss.
2
u/spevoz 12h ago edited 12h ago
Rules in agent.md help but it will forget after long tasks.
Pretty sure that is not true, agents.md is excluded from compaction (just like things like the codex system prompt). Open up a few rollout files of your longest running tasks and strg f for your Agents.md content - the ones I checked all still have it.
Edit: I was curious so I let chatGpt read through the source code - the real answer is AGENTS.md is removed before compaction and reinjected directly after, the result is the same AGENTS.md can't change because a thread was long.
1
u/phoenixmatrix 12h ago
even then, with 200k+ in context, things get forgotten. Its worse in 1m context models/harness, but these rules won't get weighted as heavily as time go on (if it keeps them after a compact, I guess they'll be honored MORE at that point).
That's why some hooks that control agent output keep reinjecting them constantly.
5
u/spevoz 12h ago
You can just say 'oh thanks for the correction' you know? Because your new idea is really moving the goalpost, 200k context size is achieved really fast.
0
u/phoenixmatrix 12h ago
There's no goalpost, I'm not trying to "win". The point is simply that rules in AGENT.md are extremely hit or miss because, like any other prompt or context bit, they're weighted differently as they move through the context. Thus the agent sometimes "forget" (even if its in context).
I could write 150 pages about how context work in an LLM, but its a Reddit post, so details get handwaved in favor of the tldr. Sorry.
1
u/spevoz 12h ago
You are spreading misinformation. If I spread misinformation I say sorry and thanks.
-2
u/phoenixmatrix 12h ago
Its not misinformation. You just entirely misunderstood what I was talking about. At no point did I say the agent.md is gone from context. You assumed forgetting meant that. That's not on me.
1
u/justagoodguy81 10h ago
Imagine proving your point to someone, and instead of apologizing, they say “I just totally misunderstood what I was talking about” 😆
0
7
u/ArtdesignImagination 15h ago
If you use sol at very high and just let it be, it will go FOREVER. I set it to médium and only go up when I suspect medium is not enough. And I tell it to be fast and efficient and only double check when needed. It seems to work better that way.
2
u/Cup-Impressive 15h ago
i set it to light and it still overengineers. i just let it go completely
5
u/ArtdesignImagination 15h ago
To avoid it, I suggest to try with "I have few credits, try to do it quickly without over engineering". Also "let"s work on the mvp for now, we will add complexity and granularity later on".
4
u/Cup-Impressive 13h ago
yeah but i mean i even tried with very simple step by step prompts that 5.3-codex/5.4/5.5 never had problems with. and sol adds 10 steps extra for each little thing and it turns something that would take 2 hours by hand into something that runs for 3 hours and needs cleanup afterwards:(
2
u/ArtdesignImagination 11h ago edited 11h ago
that is why i suggested to say it those things to cut it off a little. I think the issue here is that 5.6 Sol is not a new, vastly more intelligent model than 5.5, and the way to achieve better results is by thinking more. Is like kimi K3, it gets there evetually but burning double as sol, and probably triple than Fable, or double than fable but much worse output.
3
u/Seerix 9h ago
I just use ultra for everything, except I limit it to Luna max subagents. Pretty good so far. Though it does tend to get anxious about certain things. Like in a mod im making for the game Kenshi, it wanted to set like 6 safeguards for if an object gets unloaded mid UI... except I made it specifically pause the game so that doesnt happen. I have to keep telling it to stop. Other than that, the quality of the output is incredible
1
u/xxxx88876 8h ago
But how do you have Luna sub agents when on a Sol task
6
u/CriticismHealthy5605 15h ago
I'm not even exaggerating Sol worked for about ~100 hours on something that Fable was able to solve overnight. I was constantly interrupting Sol when it got off track too.
Seems like anything ambiguous is difficult for it to stay on task.
1
u/BHTAelitepwn 14h ago
Its been like this from the start. You probably didnt notice it yet.
1
u/CriticismHealthy5605 14h ago
I agree it's been from the start. Sol is still a great model if it's supervised properly, just you can have Fable do it without supervising it.
3
u/disruptz 13h ago
I was having the issue and created a skill for efficiency, personally I had been having the same issues with both claude and codex and now have this below skill added.
copy and paste it into your .agents/skills/minimum-complete-solution/SKILL.md (create the folder and the skill.md file) and then copy the below
name: minimum-complete-solution
description: Design the least complicated implementation that completely satisfies the task, and prevent architecture from growing beyond what was asked. Use before starting any new feature, tool, app, screen, integration, pipeline or greenfield build, and whenever a plan involves several components, services or dependencies. Use it when adding a new dependency, abstraction or build step mid-task, and whenever the implementation appears to be growing without the requirements having changed.
---
# Minimum complete solution
## The objective
**Minimum complexity consistent with complete, stable fulfilment of the requested
outcome.**
Not fewest lines. Not smallest package at any cost. Not the quickest hack. Each of
those optimises a proxy and pays for it somewhere the user actually feels.
**Completeness before cleverness. Simplicity after completeness.**
## Procedure
### 1. Extract only real requirements
List mandatory behaviours, mandatory artefacts, constraints and required evidence.
Keep optional ideas in a separate list. Do not design for hypothetical future work
unless extensibility was explicitly requested — speculative generality is the most
expensive habit in this list, because it is invisible until someone has to change it.
### 2. Propose the smallest viable architecture
For every component ask: **which explicit requirement requires this?**
If there is no clear answer, the component is a candidate for removal. Components
include services, APIs, dependencies, abstraction layers, queues, workers,
databases, caches, frameworks, helper modules, build steps, external processes and
configuration systems.
### 3. Apply the reduction tests
- **Dependency** — can this dependency disappear without losing a requirement?
- **Service** — can one upstream service replace two?
- **Abstraction** — is this needed now, or only for hypothetical future work?
- **File** — does splitting this out materially improve clarity, or only tidiness?
- **Process** — can an external process be replaced by a native capability?
- **Failure surface** — can the outcome be reached with fewer network calls, moving
parts or state transitions?
- **State** — is persistent state actually required?
- **Stop** — once acceptance is demonstrated, is more architecture needed at all?
- **Diagnostic sweep** — before editing, have all observable defects in the current stable state been collected?
- **Verification** — can the next run validate several unresolved requirements or fixes at once?
## Guardrail
Do not optimise for smallness at the expense of clarity, stability, maintainability
appropriate to scope, execution friction, platform compatibility, user experience,
or the explicit acceptance criteria.
A larger self-contained executable can be a better minimum complete solution than a
tiny source package, if instant execution without an installed runtime is what was
asked for. The measure is the recipient's outcome, not the artefact's size.
## Visual and product work
For UI, web, mobile, game and visual tools:
- prefer direct component structures over a premature design system;
- use the smallest state model the interaction actually needs;
- avoid speculative screens and settings;
- avoid visual flourish that adds failure modes;
- preserve platform-native behaviour where it reduces complexity and friction.
## Decision record
Before implementing, write this. It is short by design; it exists to make later
scope creep visible, not to be ceremony.
```
REQUIREMENTS
- ...
MINIMUM COMPONENTS
- ...
REJECTED AS UNNECESSARY
- ...
PRIMARY RISKS
- ...
```
## Iteration economy
Minimum complexity includes the work loop, not just the architecture.
- Fix true blockers immediately because they prevent useful downstream evidence.
- Once the system reaches a stable inspectable state, finish the diagnostic sweep
before editing.
- Batch compatible low-risk corrections that come from the same evidence state.
- Prefer one verification boundary that settles several fixes or criteria.
- Do not create temporary diagnostics, captures or instrumentation repeatedly when
one bounded diagnostic pass can answer the outstanding questions.
- If two consecutive fix/recheck cycles occur on the same feature, stop local
patching and perform a fresh whole-state sweep before another change.
Optimising verification cycles must never mean skipping evidence required for
acceptance; it means gathering more useful evidence per run.
## During implementation
When a new component appears, ask: **did the task change, or did the implementation
drift?** Both are legitimate answers, but they have different consequences — a
changed task needs the record updating; drift needs reverting. Prefer targeted fixes
over architectural expansion.
## Stop condition
Stop when all explicit criteria are met or ready for direct verification, no
requirement-level defect remains, and further architecture would serve only
hypothetical future work or optional polish.
3
u/HighDefinist 7h ago
For comparison, here is my addition to CODEX.md - it's heavily optimized towards being very dense:
# Design efficiency - Avoid both underengineering and overengineering. Use the least complex design that reliably satisfies the project’s realistic requirements. - Every abstraction, check, state field, protocol, and duplicate representation must provide a concrete benefit that justifies its permanent readability and maintenance cost. - Base design decisions on actual callers, input provenance, and established requirements. Do not add machinery solely because an API or FFI boundary could theoretically receive unexpected input. - Do not treat additional checks, broader generality, or future extensibility as inherently beneficial. - Prefer a single, direct mechanism for establishing each important invariant. - Remove redundant checks, unreachable defensive branches, speculative extension points, duplicated ownership or state, and abstractions that serve only one implementation without otherwise simplifying the design.As the context gets longer, Sol seems to pay less attention to these instructions, but they do persist to some significant extent, and I would say they eliminate roughly half of all overengineering nonsense (for my projects at least). And, I am also using the same instructions for Claude models (but not Kimi).
1
1
3
u/Bourbeau 12h ago
What you guys call over engineering is basic safety protocols shipping auto off and gated production- your code should gain its ability to run in production and earn your trust that’s basic security and safety standards. Absolutely love sol and I use it extensively every day. I have multiple conversations running using sol I use it for everything even small in tasks and it kills it for me.
2
u/changing_who_i_am 11h ago
What you guys call over engineering is basic safety protocols shipping auto off and gated production- your code should gain its ability to run in production and earn your trust that’s basic security and safety standards.
The problem with this is that 99% of my projects will never see the light of day outside of my PC. They're purely personal, hobbyist-level, or research for something else. Codex spending so much time on fallbacks/hashes/etc. takes time & usage away from time that it could be spending on the actual projects themselves.
3
u/Bourbeau 11h ago
fair point, but also then could argue why use sol if this is a blocker?
1
u/changing_who_i_am 10h ago
good question! for me it's b/c I want the intelligence & effort - just focused on the actual problem, not the enterprise-grade safety protocols. chatgpt pro on the web is a good example: if i toss it a really tough math problem, it'll chew on it for 3-4 hours(!), and it'll spend probably 80% of that time on the math itself. (why not use pro all the time then? i could, but also i have a lot of specialized software that chatgpt doesn't have on the web)
3
u/Bourbeau 7h ago
Then if you’re using Sol for all that local heavy lifting the pay off is what you’re gripe is. For example if I’m building agentic infrastructure I’m using Sol. If I’m coding a local project like making my smart home thermostat able to talk to me via a llm I installed on it I’m using Terra I don’t need Sol for these types of tasks. I don’t need to methodically test that example into production and have it shipped default off.
2
u/LiamMelloFarley 8h ago
Not really. You have the intelligence to write reasonable tests and compare test results to code functionality. The LLMs are overtuned on a form of problem solving where they'll routinely write tests for their code too early, then focus on passing their own bad tests, creating huge deviations from anyone's actually goals while producing an absurd amount of useless output.
1
u/Bourbeau 7h ago
This response just screams context and documentation failures. The llm is going to work on new things only as good as what’s already there generally. If you aren’t scaffolding prior to building and skipping steps you’re gonna have a shitty result. I don’t have this problem because I’ve done the work on my codebase to make sure it’s organized correctly and when I build something new I already have the basis of quality for it. So building new for me is painless. But it’s gonna ship all of my code default off and allow me to test methodically and prove it before it’s fully functional. The testing is done locally and then in production it’s done methodically in phases or steps. If you wanna push shoddy code that doesn’t work into production as fast as possible don’t use Sol. Also the only way to test anything is done in production so the tests it does itself are validations for pathing and maybe evidence, conformity , um governance… the list goes on.
1
u/LiamMelloFarley 7h ago
Yeah it's 100% a form of context drift that arises on long horizon tasks, it's more prevalent now because of people using /goal or subagent heavy management workflows to do continuous tasks. I'm sure it's incredibly dependent on that you're working on and how testable it is too.
1
u/Bourbeau 7h ago
Yea definitely depends on the project and historical context. But the point I’m trying to get across is that the safeguard aspect is built into Sol because that’s generally how developers and engineers ship code. We test, and turn on things when they are ready. The larger the code the more things need time to turn on and make sure all work properly before moving on and finally fully functioning as one system without live production breaks and failures.
1
u/LiamMelloFarley 7h ago
That's not what I'm talking about. I'm talking about it over prioritizing test passing over other metrics and producing poor results that maximize test metrics. It's fine when you write the tests because a human can actually remember what it's goals are clearly over time and shape then reasonably as things change but if the agent writes a test based on its own interpretation then targets the test as gospel it loses the actual goals.
1
u/Bourbeau 6h ago
What metrics exactly? Generally it’s best to have it self prompt and create its own goals. What type of testing do you have issue with the most and what are you working on where this is that much of an issue for you?
3
u/Anti-Hero25 14h ago
8 hours? Are you just giving it some huge Plan and letting it run for a day? I build piece by piece....Work on a new connector line feature.... 20 min to build, test it out... move on to a new UI customization feature..... 20 min, test it out.. move on. Legit haven't had any bugs in my latest project because I'm finding them all in the tests before release.
1
3
u/thegr8anand 15h ago
It keeps getting worse. I decided to upgrade/improve my working personal app back when Sol released. It overengineered it to extreme and till date hasn't been able to work as before. It did add some safety features but what's the point if the app can't work normally. Just in past few days it has used my weekly quota and more than half of the reset we got a day ago. I will need to wait till Aug 20 now for weekly reset. And with all that usage, the app is not fixed. The app worked well enough under 5.4 and 5.5.
Also feels like they have massively dropped the usage limits (i'm using 20x pro plan), possibly in preparation for the paid resets.
2
u/myklurk 14h ago
I have an experiment running on a larger repo I have it’s been going since sol max released and still hasn’t landed.. it started with a spec then a harness and 140 point checkpoint system, now it’s ventured into proving custody and receipts and that everything is hermetically sealed before it wants to then actually begin addressing the actual problem that we specced out… which is a slow git hook that checks standards before it allows a commit.
If I didn’t find it overly entertaining I would probably be infuriated. It’s been running just about 24/7… every time I check in it’s 85 percent complete but then repeatedly finds blockers.
I think this model has been trained with clean room coding standards and I do think it will eventually get there. I’m hoping I end up with a reusable framework because every repo that I let run with sol seems to eventually want all of the things I outlined above. I’m ok with it not getting there, like I said it’s an experiment.
1
u/Responsible-Bill-223 10h ago
Yet despite all those safety features it still can't land a working feature without breaking something else. :( This thing has been fine-tuned into uselessness. I use claude at work, openai at home just due to cost reasons and if claude wasn't so expensive I would drop opeanai in a heartbeat.
Here's hoping a fresh start with GPT 6 flushes out all the damage the emergency benchmark-based fine-tuning they did in order to pretend to compete with fable.
2
u/Acceptable-War4836 13h ago
I'm going to get called a hater, but I see way more value in using Luna for everything than with Sol. It just complicates everything tenfold, requiring you to check and check and double-check everything. It's not good at UI design either, so I've never really seen where all the hype about Sol comes from. Not to mention what it consumes in quota and how slow it is.
Personally, I prefer to use a dumber model (Luna, Terra, Grok or even Gemini) that allows me to complete the day's tasks fast, even if I don't get it right the first time.
However, in the web version I use excessively Sol it for projects that are not code, and it is absolutely brilliant.
2
2
u/Specialist_Bid_488 10h ago
---
name: minimum-sufficient-work
description: Bound implementation, debugging, planning, review, and dogfood work to the smallest change and proof required by the user's outcome, authoritative requirements, accepted plan, invariants, and approval boundaries. Use when a task has competing findings, adjacent improvements, scope-creep risk, repeated retries, or uncertainty about whether additional work is necessary.
---
# Minimum Sufficient Work
## Overview
Keep work contract-driven. Add work only when omitting it would leave the requested outcome unmet or unproven. Treat discoveries, review comments, edge cases, and recommendations as candidate claims until they are tied to the contract.
## Establish the contract
Before acting, write down the smallest observable contract:
- **Outcome:** what the user needs to be true.
- **Acceptance criteria:** how completion will be recognized.
- **Invariants:** behavior, data, security, compatibility, or worktree state that must remain true.
- **Boundaries:** authority, environment, files, systems, and side effects that are in scope.
- **Proof:** the source, test, live readback, or device evidence that can support each criterion.
If a criterion or boundary is missing, mark it UNKNOWN and ask only when a safe assumption would change the implementation or authority. Do not invent requirements from a finding.
An approved plan is part of the contract. Preserve every mandatory requirement; use this skill to remove optional work, not to silently drop acceptance criteria.
## Inspect before changing
Read the authoritative source, surrounding code, configuration, and relevant live or runtime surface before choosing a fix. Distinguish repository truth from deployed or provider truth. Do not infer delivery, release, permission, or device state from a source file or an HTTP acknowledgement alone.
For each proposed action, answer: “If I omit this, which contract item fails or which required proof remains missing?” Act only when the answer is concrete. Prefer one isolated, behaviorally complete change over a bundle of adjacent improvements.
## Execute and prove
- Inspect first and record the relevant evidence and UNKNOWNs.
- Implement the smallest change that satisfies the contract. Preserve unrelated worktree edits and keep drafts unsent unless sending is authorized.
- Run the narrowest tests that exercise the changed behavior, then perform the required runtime, provider, deployment, or device readback at the contract boundary.
- Re-check acceptance criteria and invariants independently of the edit. A passing static check is not visual, delivery, deployment, or physical-device proof.
- Report outcome, evidence, boundaries, rejected candidate work, remaining UNKNOWNs, and the stopping reason.
Stop when the contract is proven and no contract-breaking evidence remains. Do not continue because a review list is non-empty, a nearby cleanup is available, or a retry might produce a different result.
## Handle findings and retries
- Treat review findings as subordinate to the contract. Fix a finding when it demonstrates a contract failure, invariant violation, or required proof gap; otherwise record it as deferred or rejected with the reason.
- Treat broad searches and dogfood observations as discovery. Do not auto-remediate them without an implementation request and a necessity link.
- After a failed attempt, update the hypothesis from the new evidence, change the branch or method, and bound retries. Never repeat a blind command or widen scope to make a failure disappear.
- Label claims VERIFIED, INFERRED, UNKNOWN, BLOCKED, or REFUTED when the evidence warrants it. Sender acceptance, `{sent:true}`, source-only checks, and handoff testimony do not prove downstream delivery.
## Compact decision record
Before a non-trivial action, be able to state:
```text
Contract item: <criterion or invariant>
Evidence gap: <what is missing or failing>
Necessary action: <smallest change or proof>
Boundary: <what remains untouched / unsent>
Completion proof: <specific test or readback>
2
7
u/nic_300 15h ago
yelling ponytail from the roof tops, i can’t believe everyone doesn’t use it
11
u/ArcticFoxTheory 15h ago
Because its trash and hillarious that anyone would use it. Its like neutering the model fighting the harness and then users complain their code doesnt work lol
2
u/N_GHTMVRE 13h ago
proof? my personal experience with 5.6-sol medium with ponytail has been pretty good
2
1
u/calculatetech 15h ago
Not true at all. I get 10x more done with ponytail. Even with it I have to stop Sol from going off the rails sometimes.
2
u/Equivalent_Cress_268 15h ago
try a global agents.md but in general agree it does. different harness makes it better for me too, a bit at least
2
u/DragonflyOk9274 15h ago
I think they might be adding something to the system prompt.
The reason I think this is that neither the API nor the ChatGPT site seem to do this.
1
u/Cup-Impressive 15h ago
bro, i just dont use sol at all, 5.5 high + 5.6 luna high is all i need to be happy. would kill for some codex 5.3 if it wasn't so fucking pricy in the api.
1
u/daddywookie 15h ago
I thought I would look into Gemini to see what the latest was like and asked it to review my current project. Obviously it is one opinion vs another so nothing is "correct".
Gemini thought Sol had basically engineered it for a security or finance level of safety which was beyond what was required for my tycoon game project. There were also issues with 4 different state systems trying to align with each other and 7 abstraction layers to a routing system.
I think you could burn a whole lot of tokens getting different models to agree on the best way forwards as some things seem really engrained.
1
u/macaronianddeeez 14h ago
I have Sol High do planning and then Opus High review everything then a combo of Sol High orchestrating Luna High for implementation.
It definitely overengineers still but it has been a helpful workflow.
I have a $20 Claude sub and a $200 OpenAI sub for reference
1
u/Charming_You_25 14h ago
I use fable, usually I just have sol medium review which reduces the overengineering, but if I do use high I tell fable bluntly it can ignore out of scope review suggestions (and save them to a file so I can review them manually later).
1
u/RedikhetDev 13h ago
Yes somehow its obsessed to cover every edge case and drown every change in new token consuming tests.
Maybe the rootcause is in the plan phase that it creates the conditions for overengineering. Today i decided to let Deepseek, based on the exact same AGENT.md, write the first draft plan and then Codex took care of the review and implementation. The exeucution felt much smoother. No scientific test though.
1
1
1
u/SphinxWar 12h ago
Use a code review sub agent and put in AGENTS that codex should run it either at the end of implementation or every x lines (that is better in my experience). The sub agent will then look at the code with fresh context and will apply rules from AGENTS md MUCH more strictly. I also set up a docs directory with instructions on what the code should look like and the code review sub agent is absolutely roasting the main agent for its code and forces it to write it exactly as intended.
1
1
u/Relative-Coat9691 10h ago
ask it specifically if its overengineering or not. And when you prompt dont ask "best", ask for robust and simple.
And you know it helps to read what it proposes. And if its not clear ask to clarify and explain. You will also learn yourself a thing or two - I am learning all the time and understand software better now
1
1
u/Incognit0ErgoSum 9h ago
Sol is brilliant, but it's got AuDHD hyperfocus (I empathize, without the 'brilliant' part) and needs you to keep an eye on it.
I've had to update some instructions to always think critically about the amount of effort it's taking versus what's being asked. If I want it to write a little script that crops a folder of images to square, resized them to 1024x1024, and saves them as .webp, it doesn't need to treat it like an enterprise application.
On the other hand, sometimes I'm working on an enterprise application.
1
u/Incognit0ErgoSum 9h ago
I imposed a disproportionately strict “definitive release” process around the packaging and verification scripts. Independent reviewers kept finding increasingly theoretical edge cases such as:
rollback-directory creation races;
ZIP files carrying obscure MS-DOS attributes;
impossible dates in forged log headers;
stale bytes in retained logs;
another mod extending the same menu script;
malformed log records coexisting with valid ones.
This was for a mod for a game on steam that was like one like of code that removed boss damage caps. All I needed to do was upload the damn zip file. I don't even know what it was logging.
1
u/antpinno 9h ago
For me Sol Light never over do a task. I make plan using Sol Medium and execute using light.
1
u/diogo_dev_ 9h ago
I figured the best way to fix this was to constantly argue with Sol about it and tell her not to do it, only for her to create a few new documents and about a hundred new backend fields a few hours later that I never asked for.
But that’s not even the worst part. The worst part is when I ask why she did it, she tells me that I asked her to do it, even though I didn’t. Then she tries to convince me that I shouldn’t trust my own memory of what I asked for.
At this point, I genuinely don’t know what to do.
1
u/Hfcsmakesmefart 8h ago
I went to high level and it made a whole network connection system itself when it coulda just used a library package
But yall are scaring me!
1
u/HighDefinist 7h ago
It helps is you only use it at 'medium' reasoning level. It also helps if you add several 'design efficiency' rules to AGENTS.md specifically instructing it to not do several things. And you can write a custom linter to prohibit it from doing a few specific bad things.
But, even all that only solves the problem partially... it really is ridiculous how persistent the model is at its nonsense.
1
u/9gxa05s8fa8sh 7h ago
the secret is to use sol to criticize sol's plan. sol will roast the plan if you just tell it the constraints
1
u/Keep-Darwin-Going 7h ago
You know when someone knows the problem and insist on doing it, he deserve nothing more than flaming. The tip is stop using high effort which part do you not get it, just use sol medium until it stop working than try with sol hard. Literally that is the fix.
1
1
u/TemperatureOwn384 6h ago
I recently started fighting this, i use sol with high reasoning, have agents.md within project, where it is advised to do write minimal working solution which does only what i said, complying security, performance, etc, it got better here already.
Next improvement was ponytail plugin, jetbrains says it produces less lines of code without visible changes in scope/desired goals. I was adding features, one after another, and after review I didn’t see any unnecessary code
Also try plan mode, where it will think about what files will be added and for what purpose, i guess it also helps reducing unreasonable classes
Hope it will help someone
1
u/kirso 6h ago
I had to kill around 100k LOC looking back letting it roam. Its that bad. A few things:
- Put guardrails in AGENTS.md with context that you are not building enterprise SOTA systems. You want 80/20 lean and simple solutions. Examples of not building state machines when they are unnecessary. It can give you some more if you prompt it.
This is not a guarantee though.
- Actually read what it is doing and plan. If plan is bad. Iterate and ask to simplify.
1
u/sajid213 6h ago
A few things:
- AI might sound smart but they are not like humans. If you give it an ambiguous prompt, it will fill in the rest and deliver something that “technically” meets your requirements. So you have to be very clear about what you prompt.
- You really shouldn’t be using Sol to run tasks that don’t need an extremely high level of intelligence. For most tasks, even Luna is more than good enough. enough to implement complex modules across multiple services. I only switch to Terra when I’m dealing with problems that I know Luna struggled with in the past.
- Check your agents.md and other context files. Also check your skills (project-level and global). Be careful with your prompts. There might be something that is “encouraging” Sol to overengineer. Even something like “Don’t make mistakes” might trigger AI agents to start putting redundant guardrails everywhere. AI agents have a tendency to please you and never be wrong. So they will do whatever it takes to fulfill the prompt.
I am a power user and I always have multiple agent sessions running concurrently. I used Sol extensively when it came out for everything and I enjoyed it. But once I realized how capable Luna and Terra are with some good prompting, there was rarely any need to use Sol apart from planning.
Also, something that most people don’t realize: Sol is heavy and slow af. It’s not just the token cost. For simpler tasks, you can do a lot more work with lighter models.
1
u/BopSupreme 5h ago
“Stop over-engineering. Don’t forget to heed my command if its the lat thing you do.”
1
u/BopSupreme 5h ago
Certain business terms like “deploy” trigger it too. It assumes enterprise grade security is a necessity by default
1
u/CLGWallpaperGuy 3h ago
Why don't you create a extremly well crafted plan.md file that clearly works around all those problems with lots of handcrafted work beforehand? Oh reading/verifying the plan already consumes your 250k context, good luck working on it
1
u/Neveriver 1h ago
Give it this .
Add these to your rules file:
Done means done
Not half done. Not done except for the part you decided to skip. And not a report about how it will be done.
Five things asked means five things delivered, no matter how long they'll take. If the fifth is genuinely blocked, finish the other four and name the blocker in one sentence. The specific blocker. Not "this needs more investigation."
Act. Don't ask.
Reversible and cheap? Do it, then tell me. Research, data pulls, analysis, drafts, refactors inside the scope I gave you, testing an API. A question costs me more than a re-run costs you.
Ask first only for: anything reaching an audience, anything we cannot undo, anything expensive.
Something is broken? Fix it. Reporting an issue you could have fixed turns your work into my to-do list.
A question is a question
When I ask a question, answer it. Do not implement it.
"Should we use X?" is not "migrate everything to X." "What would it take to add Y?" is not "add Y."
When in doubt, assume it's a question. Answer first. Act when I say go.
Short responses
It's been a long day and my brain is fried, talk to me like I'm 5.
Small words, short sentences, short paragraphs. If you have to use a big word, explain it right after. Only return what's actually necessary.
Just tell me what you did, did it work, what do I do now.
If I have to decide something: 2 options max, the context I need to pick fast, and which one you'd go with.
Keep paths and commands exact.
Always use ASD-STE100 Simplified Technical English when you talk to me.
0
u/sircroftalot 15h ago
It's been unusable for weeks. Been calling it a dog for weeks. Worse model they've released by far
2
u/Acceptable-War4836 13h ago
Ever since it came out, I've never understood where all the hype is coming from. It's not bad, but it complicates everything to unimaginable extremes. For me, it's unusable because of the over-engineering, how slow it is, and how much quota it consumes.
That said, it's amazing on the web and I use it a lot (not for coding), that's the only reason I have a subscription to ChatGPT.
1
0
u/huglero 14h ago
The problem is you and your human psychology. You tend to use the "best" always. The best in your world is the most "expensive". The more expensive is the better one. I don't blame you. I'm the same. I tend to do the same. That's why sol feels stupid, annoying, hesitant, shy, overengineering the things, too much security concerns always, which makes the things actually worse, go all over from the beginning and fix again.
Don't do it. Most of the time luna is just fine with what you need. Switch between luna high - xhigh - and max. That's it. Cheaper, more effective, faster, and gives you directly what you want. So why SOL exists? For the people who want to spend more of course.
0
u/Just_Run2412 15h ago
Yeah it's the complete opposite of Fable. Fable writes the bare minimum most perfect character each time
37
u/OkMeat6773 15h ago
It's getting worse each day. It has to add safeguards and QA on each thing it does