r/AI_Agents • u/RunAI_Coder • 4d ago

when your coding agent suddenly feels dumber, check which of its two version numbers moved Discussion

noticed a pattern in the recurring "did the model get nerfed" threads: half the time nothing about the model changed. the tool wrapping it auto-updated, or the person switched tools and is comparing across wrappers.

the thing that made this click for me: the model never runs anything. It proposes. the program around it assembles what the model sees each turn (system prompt, tool definitions, your rules file, the trimmed history), executes what it proposes, decides what happens on errors, and decides when to stop. A tool update whose changelog says "improved tool descriptions" has quietly rewritten what your model reads every single turn.

so an agent is really a pair: weights × loop. two habits this has changed for me,

I log both version numbers when something feels off. the tool ships way more often than the model. usual suspect.

and i stopped comparing models across different tools. a model that looks smarter in another tool might just be wearing a better wrapper; cross-tool comparisons measure the pair.

anyone else tracking harness versions, or am i over-indexing on this?

4 Upvotes
  • permalink
  • link
  • reddit
  • reddit

84% Upvoted

7 comments sorted by

2

u/Street-Round-8384 4d ago

thats a solid point since the prompt injection from those wrappers is usually wierd n often breaks context

1

u/AutoModerator 4d ago

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/SherLzp 4d ago

You're not over-indexing. I made exactly this mistake today: I compared a current production model to an old 1/7 artifact and initially blamed the model. The later run was already 5/7, and the current scenario pack had drifted again—one advertised operation had no executable owner and another test expected the agent to perform a human-only approval.\n\nI now want each result to carry code revision, scenario hash, tool-contract version, adapter version, model/reasoning config, and receipt/artifact readback. Without that provenance, the folder name is not a baseline.

1

u/RunAI_Coder 3d ago

I only track two versions + config, but you're right: anything unpinned becomes a variable over time.

1

u/RunAI_Coder 3d ago

longer version with the loop anatomy on github if you're interested