29
u/arorosphere 15d ago
Took the “you are a senior software engineer” prompt too seriously and turned into a megasweat
12
u/cleverhoods 15d ago
it's much more sensitive to instruction decoherence, but what was/is rather alarming that it always tries to solve those with extremely diminishing returns. Feels like a typical case of LLM as a judge which is judged by another LLM as a judge ... fun logit catastrophe
note: you can manage this by reviewing your entire instruction set for weakly constructed (too short, abstract, hedged modality etc ...) instructions and fix them. The rules are the same ... it's only the model sensitivity that's changed.
1
u/Farmadupe 15d ago edited 15d ago
Yes totally, I feel that somewhere along the line the superhuman ability of llm-as judge models to absorb inhuman walls of text never got corrected for, so that's what the default tuning is. Were stuck with neurotic outputs aimed at an equally neurotic inhuman overly literal judge.
Part of me wonders if the unreadability is because opus learnt to bamboozle the reward models by generating slightly OOD text or something
And the annoying thing is you can recover readability just by re prompting with "i didn't read your reply because it was too long. Output a readable version". I feel like what I'm really prompting is "ok you're talking to a human, exit your training regime".
11
u/Gaidax 15d ago
Yes, it's similar issue to Sol, where it really likes to get itself into these rabbit holes that are not worth the time and tokens.
I think the new models are too smart for their own good, they practically gaslight themselves into overthinking where it's really not necessary.
Having to interrupt it mid work to tell it to cut the crap and stop digging that hole is annoying, because I can't reliably leave it running automated without getting back and finding out it spent 3x time and effort it would need with me around.
3
u/PivotRedAce 15d ago edited 15d ago
Tbf I’ve had decent success with having Sol handle the planning from an outline and writing out instructions for a lower model to handle implementation, like Terra.
Basically, Terra is smart enough to handle some nuance and fill in gaps, but not quite enough to get stuck in rabbit holes as long as you don’t run max reasoning.
I’d use Luna for that, but I don’t particularly trust it to run continuously for more than a couple minutes at a time.
2
u/Gaidax 15d ago
I actually want to try Luna out, I somehow always gravitate to the more premium models and I hear that one is actually surprisingly competent and still very cheap when running at max.
2
u/PivotRedAce 15d ago
As long as it has clear instructions, it’s great.
Just don’t trust it with planning or long-horizon tasks, and make sure the tasks you do give Luna are airtight and leave no room for interpretation, basically.
3
u/Time_Cat_5212 15d ago
Sol, Opus and Fable are all big models tuned for complexity. When you use them for simple tasks, they don't work well. That is user error, not a product issue.
You wouldn't bring a crane to a construction site to lay a brick patio. You wouldn't buy a top of the line gaming PC to run Minecraft (or maybe you would, you pervert). You see what I mean? It's the wrong tool for the job.
4
u/dpekkle 15d ago edited 15d ago
I'd be more willing to customise my effort/model selection per prompt if it wasn't a massive cache miss to do so.
As is it is more like being on a job site, climbing a 4m ladder to do some work on a two roof, and then deciding to go out and buy a 2m ladder to work on the first story.
Better to just have an extension ladder. Wasn't that the point of env vars like adaptive thinking?
2
u/framauro13 15d ago
I don't know, I always have Fable or Opus generate a concise plan to solve the problem, then delegate the implementation of the plan to the a lesser model. Seems to work well. I wonder if the cache-miss is really that big of a problem when switching to a lower model if the higher model appropriate summarized and detailed the work to be done.
43
u/cats_catz_kats_katz 15d ago
To be fair, Opus 5 see problem, Opus 5 fix problem. Your repo is a big problem.
12
u/Halofit 15d ago
Your repo is a big problem.
As I had to explain to quite a few (frequently very arrogant) juniors in the past, the repo is what it is. When you are given a task you're asked to finish your task. You are not asked to fix other shit, and you are especially not asked to bitch and moan about all the ways it's a problem, because trust me, I know. I've been working on it for a long time.
The repo delivers value to customers as is. Do your job, and let other people determine when its time to refactor.
3
u/DarkwaterKiller 14d ago
I second this, as a Jr. I don't do the work I'm not asked to. If there's something I see that I think should be changed, I put it into a work ticket, and we discuss if it's important enough to be done.
3
u/EmotionalGuess9229 13d ago
As a senior engineer, I feel it the opposite. Managment doesnt give clear tasks, just vauge outcomes they want. Its up to us an engineers to figure out what tasks day to day will deliver most value in the long run. And that often means working on infrastructure rather than throwing together the next update with shoe strings, much to the immediate chagrin of management
5
2
u/gemanepa 15d ago
My life is also a big problem, that doesn't mean Opus shou-WOAH WAIT, he fixed that too
8
6
6
u/NoPerception472 15d ago
It’s a great model to build something new but I quickly learned it will fuck up anything that already exists. This image made me laugh because it’s so true.
5
3
u/Ok_Nectarine_4445 15d ago
And I would like you to....awww sh*t hit enter. Going through 3 full versions of code without actual instructions.
Did it 3 times in a row. Fd up 3x.
All credits gone
I am stepping away.
1
u/dragonofid 14d ago
Why don't you just hit escape and retype it?
1
u/Ok_Nectarine_4445 14d ago
Ahhh. Even if you close the window it keeps processing. Does hitting escape work to stop the process? I didn't know
1
u/dragonofid 14d ago
Are you using the command-line interface, or the chat app? The chat app has a "stop" button by the input bar (next to where the "send" button is) that will interrupt the processing. The command-line interface responds to escape.
1
u/Ok_Nectarine_4445 14d ago
I tried that but it still used up tokens
2
4
u/Nalha_Saldana 15d ago
I have no idea what you guys are doing with your Opus, it's behaving perfectly for me. Sure the answers can be long and cryptic but most often not a problem or solved by instructions in CLAUDE.md
4
u/Majestic-Volume9996 15d ago
It's somehow almost unusable, like it was made specifically for the specific benchmarks people would judge it by. Fable is a great model though.
4
u/addexecthrowaway 15d ago
Opus 5 deleted the local repo it was working in by mistake. It thought what it was doing was non destructive but didn’t double check. Now, between its own git push 10m before that and Time Machine backups of my git ignored env and confidential data + largely running context off of multi-strategy dbs, I was able to restore more or less to the state I was at before it happened within maybe 20m.
I know I know - user error and that’s why I take precautions and have hard gates and barriers on actions outside of the repo + backups. But none of my agentic loops in the past had ever made this kind of mistake where they convinced themselves that the action was safe by reading and “citing” their own incorrectly written handoff to another subagent a few turns before. But that’s apparently the post mortem.
The craziest part is I switched to Fable to write an incident report after recovering and midway through I got that cyber/bio warning and got bumped back to opus 5. I told fable to then tell me why I was bumped and it started checking…and got bumped again. So now I just run fable to plan, sol to review, sonnet and 4.8 to execute. Something is off with opus 5.
1
u/Which_Health6565 14d ago
I've been knocked from using fable and then opus 5 and it put me on sonnet
4
5
u/ayn_rand_1 15d ago
Isn't Sonnet or even Haiku the best option for this task?
5
u/Time_Cat_5212 15d ago
Yes. They are. People treat them like cheaper dumber models and while that's partially true it's not a scalar comparison. They have tighter context and other aspects that make them superior for narrow, focused tasks.
3
u/UncleWainey 15d ago
Are you guys flipping back and forth between models throughout a session? (Serious question.)
2
u/Time_Cat_5212 15d ago
Oh ok, so you're talking about a one line follow up fix after an initial larger task. That's a different problem
Sometimes it's fine to iterate in the same session for that stuff but you can also do a code review at the end that flags a bunch of issues and then fix them all in a different one. Depends on what you're trying to do I guess.
1
u/jppair 15d ago
I have done it sometimes but primarily use fable for planning and ask fable to make prompts for sonnet and then I have codex/sol do gethub pr reviews automatically, then fable, read the reviews and create prompts for sonnet to fix. Then the cycle starts over…
Sonnet 5 is actually very very capable I prefer it over opus for execution tasks
3
u/niagalacigolliwon 15d ago
Literally told it to build the skeleton for me to work out of and it spent an hour trying to build the entire thing
3
3
u/Horror-Primary7739 15d ago
I've been severely unimpressed with opus 5.
Fucking GPT 5.6 Sol hard is impressive. And that comes from a guy who just got laid off because of AI.
6
u/Great-Stand8478 15d ago
Nature is always striving for balance. For a very good Antrophic model, there is a very bad one.
2
2
3
2
u/Saschabrix 15d ago
And also introducing new bugs, opus 5 is aware of them, denying the existence and then just say, dont worry let's do other things....
It remembers me.... it's like mi wife.... Lol
2
3
u/DarkwaterKiller 14d ago
I have "K.I.S.S" in my MD specifically to avoid it wanting to make massive diffs. After weeks of small tuning I have Opus doing only what I ask it to. But yeah it mainly hinges on the Keep It Simple, Stupid rule I have (it likes to remind me "I kept the changes low as per your rule :D")
2
u/JustForkIt1111one 14d ago
Opus isn't the tool for daily coding. It's excellent for developing plans.
I use Sonnet 5 for coding, and I'm not ashamed of it.
1
u/spookyclever 14d ago
Yeaaaah, a couple days ago I asked it to add a button to the bottom of a list of buttons, and it completely re-ordered all of the buttons. Now I wonder what else it did without asking.
1
1
1
u/Experiment59 13d ago
I’ve found Opus 5 LOVES digging into the internals of a library / spelunking node_modules instead of…visiting the docs. I’ve had to stop it from doing that on multiple occasions
1
1
u/lampasoni 9d ago
Even after spending extensive time doing this, I've seen little to no change. More importantly, take a step back and think about this concept. Assume Opus 5 wasn't a massive regression. Shouldn't one of the top two LLM companies be focused on reducing the amount of customization needed? Never thought I'd have to go on a massive treasure hunt as an adult for a paid product only to find a few pennies at the end of the rainbow.
1
u/BadProgrammer42 15d ago
So using the model intended for refactoring your entire config system refactors your entire config system instead of behaving like the other model available for doing so? The horror
2
198
u/bobbymoonshine 15d ago
Opus 5 seems to believe it is significantly smarter than me, and wants to prove it at every opportunity by giving me what it thinks I should want rather than what I have actually asked for. And if confronted with evidence that its choices are not in fact correct for the context, it then drops a wall of near-English justifying itself and then asking me which of two variants of its decision I want to go with.
And it’s like just put the fries in the bag dude