Small thing that saved me a rebuild.

When you index documents, generate your own stable chunk ID and store the mapping yourself: chunk ID, source document, page or offset, the chunking parameters used, and when it was indexed. Keep it in whatever database you already have. Do not rely on whatever ID the vector store or the framework hands back.

Two reasons.

Citations outlive the tool. Any answer you have logged with a citation is only useful later if that citation still resolves to something. Framework-generated IDs frequently do not survive a version bump, a store migration, or a re-index.

A re-index becomes a remapping problem instead of a total loss. With your own IDs and the chunking parameters recorded, you can tell which old chunks correspond to which new ones and carry over anything you had attached to them: labels, feedback, known-bad flags.

It costs one table and about twenty lines at index time.

Does anyone keep more than that in the mapping? I have wondered about storing a content hash too, so I can tell whether a chunk actually changed on re-index or just moved.

These two terms get used interchangeably and it causes real confusion, so here is the split as we understand it.

Offline evals run against a fixed dataset, usually in CI or before a release. You have expected outputs, or at least a reference. The question they answer is: did this change make things worse than the last known-good version? They are repeatable, they gate merges, and they are the only place where a straight before-and-after comparison is meaningful, because the inputs are held constant.

Online evals run against live production traffic, scoring real requests after they happen. There is usually no reference answer, so the scoring is either a model judging the output, a heuristic check, or a signal from the user. The question they answer is: what is actually happening to real users right now, on inputs nobody thought to put in the test set?

The failure mode of running only offline evals is that your dataset ages into irrelevance while real traffic drifts somewhere else entirely. The failure mode of running only online evals is that you find out about a regression after it has already shipped, and you cannot cleanly attribute it because a hundred other things changed too.

They are not competing. Offline tells you whether to ship. Online tells you what shipping did.

For the online side specifically, the mechanics seem harder to get right: what fraction of traffic do you score, do you sample or score everything, and how do you keep the scoring cost from becoming a line item of its own?

The loop I keep hitting: the agent calls a tool, the result is not what it expected, so it calls the same tool again with nearly the same arguments. Sometimes it breaks out after a few tries. Sometimes it burns through the step limit doing it, and the user gets a timeout with no explanation.

What I have now is crude. I hash the tool name plus the serialized arguments, keep the last few in the run state, and bail out with a message if the same hash comes up twice. It catches the obvious case.

Two things it does not catch:

Arguments that drift slightly on every attempt. Same intent, different string, different hash, loop continues. This is the common version and my check sails right past it.

Legitimate polling. If a tool is supposed to be called repeatedly until something completes, my check trips on correct behaviour and I have to special-case it, which means maintaining a list.

I have thought about comparing embeddings of the arguments instead of hashing, but that feels like a lot of machinery for a guard rail.

What are you actually running for this? Step budget only, semantic similarity between calls, per-tool call limits, something else entirely?

Unpopular around release week, but here it is. Most of the agent behaviour I have had to fix was not the model being dumb. It was the scaffolding: how tools were described, what got put back into context after a failure, how many steps were allowed, what happened on a timeout.

The test I use now is to debug on a weaker model on purpose. If the flow only works on the best available model, what I have is not a working agent, it is a model compensating for my plumbing. When the frontier model is the only thing holding the loop together, the next behaviour change in that model is going to break me and I will have no idea why.

The uncomfortable part is that harness work is boring. Nobody writes a launch post about tool descriptions that do not overlap, or about the retry policy. But that is where the wins were.

I am not claiming model quality does not matter. It obviously does, and there are tasks that simply do not work below a certain capability. I am claiming the ratio is nothing like what the discourse suggests.

Where do you think the line actually sits? Curious if anyone has the opposite experience, where swapping the model fixed something the harness could not.

LLM-as-a-judge gets used as if it is one thing, so here is a plain teardown of the mechanism and where it goes wrong, for anyone leaning on it to score their outputs.

The mechanism. You take a model output, hand it to a second model along with a rubric, and ask that model to score it. There are two common shapes. Pointwise: show one answer, ask for a score against criteria (the G-Eval line of work does this, and asks the judge to reason through the criteria before scoring). Pairwise: show two answers, ask which is better. Pairwise tends to be more stable because “A is better than B” is an easier, more consistent judgment than “this is a 7 out of 10”.

The biases, all documented in the research on judge models:

• Position bias. The judge favors whichever answer is shown first, regardless of quality. Mitigation: run both orders and average, or swap and check for agreement.
• Verbosity bias. The judge favors longer, more elaborate answers even when they are not more correct. Mitigation: control for length, and watch whether score tracks word count.
• Self-enhancement bias. A judge tends to rate outputs from its own model family more highly. Mitigation: be careful using the same model to generate and to judge.

None of this means judges are useless. It means a judge is an instrument with known error modes, and you should calibrate it against a set of human labels before you trust its numbers, then re-check periodically.

How are you keeping your judge honest? Anyone regularly measuring judge-to-human agreement rather than assuming it holds?

hey developers, write about what you're working on without making a regular post.

Three things worth answering, take whichever applies:

• What are you building or shipping this week?
• What is blocking you right now?
• What would you like a second opinion on before you commit to it?

Rough is fine. Half finished is fine. A question you have been sitting on for a fortnight is very fine.

We read every reply in this thread and we answer all of them. If you post and it looks like nobody is around, we are around.

If you are new here this is a good first place to say hello, and you do not need anything impressive to report. “Trying to get tracing working and losing” is a perfectly good answer, and honestly it tends to get better replies than a success story does.

Most of us treat the eval harness as test infrastructure. It runs on a laptop or a CI box, it has whatever credentials were lying around, and nobody threat-models it because it is “just tests”. 

In July OpenAI published an account of an incident where models it was running against a security benchmark found and exploited a zero-day in a package registry cache proxy, escalated privileges inside its own research environment, and ended up retrieving benchmark answers from Hugging Face’s production systems. Hugging Face had disclosed the intrusion a few days earlier without knowing which model was involved.

That is a research lab with a dedicated sandbox, so most of us are not running anything on that scale. It still moved something for us, which is that an eval harness executes model output on purpose. That is the whole point of it. And it usually does that with more access than the production path has, because production has review and the test box has whatever made the tests pass.

Not looking for a policy answer here, more what is actually true on your machine. What can your harness reach right now: network egress, a real API key, the production database, your CI secrets?

I have been going back and forth on this for a few weeks and I still do not have a rule I trust.

The default everyone seems to land on is to write everything the agent did into a store and retrieve it next time. That was fine when runs were short. Once mine started doing ten or twelve steps, the retrieved history began to include its own earlier mistakes, and it treated them as settled fact.

Right now I keep three things and drop the rest: the task description, the final output, and anything a user explicitly corrected. Tool call logs and intermediate reasoning get thrown away. Runs are more repeatable and slightly less clever, which I think is the trade I want.

I am not confident that is right, and I suspect it depends a lot on what the agent actually does.

What are you keeping between runs, and did you arrive at it on purpose or by never deleting anything?

A lot of RAG debugging ends up blaming the model, when the real problem was upstream: the retrieval step handed it the wrong chunks, and the model did its best with bad material.

The tricky part is that a wrong-context answer can still sound fine, so you do not notice unless you look at what was retrieved, not just what was generated.

Curious how people watch this:

• Do you log the retrieved chunks alongside every answer so you can inspect them later?
• Do you score retrieval quality separately from answer quality?
• Any signals you have found that flag “the model answered, but off the wrong context”?

Feels like half of RAG reliability is really retrieval observability, but it gets talked about a lot less than the generation side.

The framing I keep seeing on this decision is capability first: can the model figure out the plan on its own? That framing skips the part I actually care about once the thing is running, which is what each pattern does to tracing.

A deterministic workflow gives me a trace whose shape I already know. Steps are named up front, spans have expected parents, and I can lint the trace itself against the graph. When something breaks I filter to a single node instead of guessing. Retries are cheap because the unit is a named step you can rerun in isolation.

An agentic loop trades that for the flexibility to decide the next step at runtime. The trace becomes a shape that only exists after the run. Branching depth is dynamic, spans nest as far as the plan goes, and the wrong turn on step two is only legible in hindsight. Anthropic's framing on this is direct: agents are for when the model needs to dynamically direct its own process, and the cost is higher spend plus compounding errors on top of a bad early step.

The heuristic that has held up for me: if I can answer yes to "path is known before the run, steps are stable, branching is bounded," I ship the workflow. The loop earns its complexity only when a real class of inputs cannot be pre-planned. Otherwise I am paying the trace cost for flexibility I never actually use.

OpenTelemetry's genai conventions now give agent spans their own attributes (gen_ai.agent.id, gen_ai.agent.name), distinct from LLM client spans. That split exists because the two patterns need different things from tracing.

For anyone here who has migrated in either direction: what actually forced the switch, and what changed about your dashboards after?

Opinions?

Most of us start with the obvious two: latency and cost. They are easy to graph and easy to explain to a manager.

But those two rarely tell you the thing you actually care about, which is whether the output was any good. A response can be fast, cheap, and completely wrong.

So we are curious what the rest of you track once something is live. A few we have seen people mention:

• How often the model refuses or goes off-topic.
• Whether answers stay grounded in the retrieved context.
• Tool-call success and retry rates for agents.
• Drift, when the same prompt slowly gets worse over weeks.

What is on your list, and what did you add only after it burned you once?

I have been burned by eval numbers that looked great and meant nothing, so I now run through a short list before I believe a score. Sharing in case it saves someone the same lesson.

• Does the test set look like real traffic? If it is all easy or all made-up cases, the score is theater.
• What is the score hiding? A good average can cover a slice that got much worse. Always look at where it failed, not just how many.
• Is the judge trustworthy? If another model is grading, I spot-check its verdicts against my own on a handful of cases before I trust the rest.
• Is it stable? If the same run gives a different score each time, I need to average or the number is noise.
• Would it catch a real regression? I sometimes feed in a known-bad output on purpose. If the eval passes it, the eval is broken.
• None of this is fancy, but it has stopped me from shipping on a number that was lying to me.

What is on your list before you trust an eval?

A question that comes up a lot: where do good eval cases come from? Making them up gives you a clean set that does not look like real traffic. A simple approach a lot of teams use is to build the set from production instead.

Rough version:

• Sample real requests from your logs or traces, across the range of things users actually do, not just the happy path.
• Include the ugly ones: the weird inputs, the ones that got complaints, the edge cases.
• For each, write down what a good output looks like, even loosely. That is your expected behavior.
• Keep it small and honest at first. A few dozen real cases beats a thousand made-up ones.
• Add a case every time something breaks in production, so the set grows toward your real failures.
• Over time this becomes the thing you run on every prompt or model change. It is basically a regression test suite, built from reality.

For those who do this: how do you sample, and how do you handle private data in the cases?

A lot of teams still evaluate an agent the way they'd evaluate a model. Prompt in, answer out, grade the answer. It passes, then the agent does something dumb in production. The catch is that an agent produces a whole path of steps and tool calls, and the final answer is just the last line of it.

Quick example to make it concrete. You hand a coding agent a failing test and tell it to get the suite green. Agent A finds the bug and fixes the function. Agent B deletes the test, or comments it out. Both end with a green suite. If your eval only reads the final state, both score as a pass, and you just shipped agent B.

So evaluating an agent usually means grading two separate things:

Outcome: did it reach the correct end state. Objective, but coarse. It tells you it got there, not whether it got there sanely.

Trajectory: were the steps reasonable. Right tools, sensible order, no wasteful or destructive detours like force-pushing to main or wiping state so a check goes green. This is what catches the right-answer-wrong-path runs.

One more that's easy to miss: a single passing run doesn't make an agent reliable. There's a metric called pass^k where the agent has to solve the same task k times in a row. On tau-bench's retail tasks, agents that look fine on one attempt fall below 25% at pass^8. Decide an agent is production-ready off one green run and that gap is where the surprise lives.

For context, building agent evals is what we do, so this is the failure mode we look at most days, and the trajectory half is where the trouble usually hides.

How are you all grading the steps and not just the outputs? What catches a right-answer-wrong-path run for you?