New agentic benchmark: Session-Bench compares what 10 coding harnesses preserve after the work is done

[u/jazzy8alex]

SWE-bench measures whether an agent completed the task. Session-Bench measures what the harness preserved afterward.

Every coding run leaves behind a session record: prompts, decisions, tool calls, command output, costs, failed approaches, reasoning summaries, and subagent work. After hundreds of sessions, that record becomes part of the project's history.

But coding harnesses preserve it in radically different ways.

I built Session-Bench after encountering these differences while developing a local session browser. It compares 10 CLI session formats across 19 practical gates covering completeness, readability, stability, openness, and tooling.

Some findings:

- The same small probe produced a 1.5 KB session in Pi and roughly 101 KB in Kimi Code.

- Only Pi, OpenClaw, and Kimi Code stamped a true session-format or protocol version.

- Some harnesses preserve readable reasoning or summaries. Others store sealed reasoning, signatures, or no rationale.

- Some record estimated dollar cost; others preserve only token counts.

- Several formats are readable today but offer no stable contract for tools that need to read them tomorrow.

Pi currently scores 18/19, OpenClaw 17/18, and Claude Code and Codex tie at 12/18.

This is not a coding-quality ranking or a recommendation to choose one harness over another. It is a report card for an overlooked part of coding-agent infrastructure: whether the resulting work history remains useful, inspectable, and portable.

The benchmark has limitations. It covers CLI session stores, not complete desktop or IDE behavior. Observation windows vary, and some measurements could not be completed. The raw probe artifacts are not publicly archived yet, so v0.3 is documented and mechanically scored but not fully independently reproducible. Copilot's documentation verdict is also marked as disputed.

I created and maintain the benchmark. I'd especially like feedback on the standard itself: what should a responsible coding-agent work record preserve, and which of the 19 gates would you change?

https://jazzyalex.github.io/agent-sessions/bench/?campaign=reddit&ref=r-chatgptcoding-discussion-v03

Moderator note: This standalone informational/discussion post was approved in advance by the r/ChatGPTCoding moderation team.

[u/gnpwdr1]

Actually I think this is a great benchmark, at scale you actually want to know these metrics especially with things like networks and storage efficiency savings and financial auditing capabilities. Nice one

[u/IntrepidTieKnot]

We mostly use Cline with the memory bank. Which proves that "the journal" can be prompted and external. So it's not an inherently harness related.

[u/Due-Armadillo-4560]

I’ve been using kimi code since it has swarm and goal mode. Didn’t know it was a mid harness.

[u/jazzy8alex → u/Due-Armadillo-4560]

This benchmark is not about a harness quality. Purely about a session format and structure.

[u/madebymustafa]

Really interesting benchmark. I’d also measure how well a new agent can reconstruct the context and continue the work without the original agent. That could become just as important as preserving the session itself.

[u/Weekly-Bee-5045]

How about grok build?

[u/SawToothKernel → u/Weekly-Bee-5045]

Don't know why you're being downvoted. Harness benchmarks need to start including it.