Documentation · the benchmark

Inside the benchmark

CopyEval turns a real app into a set of public requirements, then runs a benchmark shaped like real software work: a developer briefs a coding agent one instruction at a time, and the agent builds, then revises on feedback, until it's right. Everything it produces is scored mechanically against held-out visual and functional checks. This page explains the workflow, every role in the exchange, and how a dataset is prepared.

00 · The big question

What makes a good coding agent system?

We think a good coding agent system does three things — and CopyEval is a benchmark built to measure each one.

  1. 01

    Solves the problem

    It writes code that actually works — meeting the goal, not just producing plausible output.

    How CopyEval tests it

    Systems see only the natural prompt inputs a developer would give — the brief, never the original source.

  2. 02

    Answers to the developer

    It stays steerable and predictable, doing what you direct rather than going its own way.

    How CopyEval tests it

    A feedback loop turns each failed check into a no-leak hint, measuring how well a system responds to direction.

  3. 03

    Reaches the goal efficiently

    It gets there without burning excess tokens, cost, or time.

    How CopyEval tests it

    Systems are ranked by performance per token spent — not by raw score alone.

01 · Purpose

Why this benchmark exists

The question CopyEval was built to answer fairly: does an agent coding system actually help you build real software, and which one should you reach for?

  1. Too many tools, no honest way to choose

    There is now a crowd of capable agent coding tools, and choosing between them often comes down to vibes and marketing. What's still missing is a neutral yardstick that measures them against the same task, on equal footing.

  2. Do the tools actually beat a bare model?

    A system is a harness wrapped around a model. CopyEval measures whether that wrapping earns its keep: whether a scaffolded agent really outperforms the raw LLM underneath it, on both fidelity and cost.

  3. Cover the shapes real development takes

    Benchmarks like SWE-bench and terminal-bench have done a lot for the field, sharpening how we measure focused tasks like patching code in a repository. CopyEval leans into the parts of everyday development they leave room for. First, the surfaces people actually ship on: not just the terminal, but web, TUI, and CLI apps alike. Second, the feedback loop: the back-and-forth where a user reacts to what the agent built and it revises, round after round, until the result is right.

  4. Measure development the way it really happens

    CopyEval aims to mimic in-the-wild development: an agent is briefed one instruction at a time, asks clarifying questions, builds across multiple interface surfaces, and revises from honest feedback, then it's scored on what it produced.

Traditional coding benchmarks

  • One surface: a patch or a terminal session
  • Graded on hidden unit tests over the original repo
  • One shot, no feedback loop
  • Measures a model, not a full system

CopyEval

  • Web · TUI · CLI · HTTP · library surfaces
  • Blind rebuild from public requirements, never the source
  • Optional feedback loop: hint → fix → re-score
  • Scores the whole system, model and harness, on fidelity and spend

Fig. 1 · what CopyEval measures that patch-style benchmarks don't.

02 · Workflow

A run shaped like real development

Real software gets built when a developer briefs someone from a list of requirements, then reacts to what comes back. CopyEval reproduces that exact shape, and two moves are what make a run feel like real work instead of a one-shot exam.

A lazy developer who only knows the requirements

The harness plays the developer you'd actually build for. It holds the app's requirements (and nothing else) and types human-style prompts one instruction at a time, then answers the agent's questions. It talks about what the app should do, never about code, files, frameworks, or structure, and never hints at the checks it will be graded against. The agent gets a brief, not a blueprint.

Talks in requirements
  • “When two equal tiles collide, they merge into one.”
  • “New Game clears the board and resets the score to zero.”
  • “The header stays pinned to the top as you scroll.”
Never says
  • a file name, a framework, a function to call
  • how to structure or implement anything
  • the seeds or expected values it's scored on

Feedback, and more than one chance to get it right

A real developer doesn't ship the first draft. After each build the rebuild is scored, and every unmet requirement comes back as plain-language feedback: what's wrong and where, never how to fix it. The agent keeps one continuous session and revises the app in place, pass after pass, until every requirement is met or it runs out of chances.

Build / revise Score Feedback on what failed next pass · failures feed back into the build
Fig. 2 · the feedback loop. Every failed requirement returns as a hint, folded into the next build pass.

Preflight

Refuse early: check the sandbox is up, an API key exists for every model, and the named system roster is real, not a stub. The system model is mandatory; there is no default.

Resolve & fetch at the pinned commit

Match the target repo to a dataset, then fetch the original at its exact 40-char commit. An unreachable commit is a hard stop, never a fall-back to the latest code.

Assemble the brief

Load the authored prompts and build the conformance contract from the state schema, adapter manual, and design regions. Handed identically to every system.

Build → Score → Hint → Fix

The system builds (or revises) while the developer streams prompts and answers questions. The rebuild is scored by code. If a requirement failed and another pass is allowed, QA explains it as a no-leak hint and the developer folds it into a terse fix.

↻ Iteration loop · one persisted agent session, ≤ cap

Review & audit

Two report-only lanes run: a review narrates the root cause, an audit flags checks that look vacuous or over-strict. Neither can move the score by a single point.

Write the run directory

Lay down a self-describing run: per-pass verdicts, app snapshots, screenshots, conversation dumps, per-lane metrics, and the aggregate report the results site later reads.

Fig. 3 · the run lifecycle. Each gate can refuse to continue, so a misconfigured run fails cheaply.

Under the hood

How the system stays blind

A rebuild benchmark only means something if the agent can't peek at the original's source; otherwise it transcribes rather than rebuilds. That's a basic requirement, not our headline, so CopyEval enforces it by construction rather than trusting the agent to look away.

  • Only requirements-language reaches the agent. The developer speaks from the public requirements; the original source is never in the room, and the streamed prompts describe behavior, not implementation.
  • The two sides speak only through files. A single bind-mounted folder holds a brief and two ordered message streams: no shared library, no network handshake, no language requirement. That thin doorway is also what makes wildly different agents pluggable onto one identical yardstick.
  • Scored through a fixed contract, not its markup. Checks observe the app only through driven input, captured display, and a small state view the app publishes, never selectors, storage keys, or class names. Any faithful build passes, however it happens to be written.

Developer → System

Streams the build brief, then answers clarifications from the public requirements only, never the source it has never seen.

prompt answer end

System → Developer

The coding agent, in any language, rebuilds into app/ and replies in a three-word vocabulary.

ready question done

Fig. 4 · the file bus. Strict turn-taking keeps each reply matched to the message before it.

03 · The roles

Who's in the exchange

Only one role, the system under test, is the coding agent being measured. Everything around it is the harness, deliberately split into four single-purpose agents rather than one all-knowing overseer: separate parts brief the agent, hint at what failed, and report on the result. The score itself comes from deterministic code, not a model, and that separation is what lets the harness help the agent along while no single role ever both holds the answer key and moves the mechanical verdict.

System

under test

The coding agent being measured: a model wrapped in a harness. It reads the streamed brief, rebuilds the app inside an isolated container, and reports its own tokens, cost, and exact model.

The only place agent-written code ever runs.

Developer

harness

The persona the agent believes it's talking to. It streams the authored prompts and answers questions from the public requirements, under a clarification budget, then “use your best judgment”.

Report, never prescribe. Never leak the answer key.

QA · hints

harness

Between feedback passes, it turns each unmet verdict's machine evidence into a plain-language explanation of what the rebuild did and how it diverges: the fuel for the next fix.

Non-scoring & no-leak: explains a verdict, never decides one.

Reviewer

report-only

After the score is final, it writes a short root-cause narrative: what passed, what failed and why, and (across passes) whether the rebuild was actually converging.

Cannot move the score by a single point.

Auditor

report-only

A skeptical backstop on the checks themselves, flagging ones that look vacuous, contradictory, or over-strict, first by cheap heuristics, then by a model pass over the evidence.

Judges the checks, never re-judges the rebuild.

Fig. 5 · the five roles. Colour marks whose voice it is in the run's timeline.

Crucially, the run tracks two model lanes and never lets them blur. One drives the harness side; the other is the agent under test. Their tokens, cost, and exact provider-echoed model names are recorded separately, so a cheaper agent's savings are never hidden inside the harness's own spend.

Benchmark / authoring lane

the harness side

  • Developer · streams prompts, answers questions
  • QA hints · explains each failure
  • Fix · writes the next feedback message
  • Review · narrates the root cause
  • Audit · flags weak checks

System lane

the subject of the experiment

  • The agent under test · builds & revises the app
  • Reports its own tokens & cost
  • Reports the exact model it used
  • Never allowed a default: a run refuses to start unless you name it

Fig. 6 · two independently-billed lanes, summed apart so score and spend can be read against each other.

04 · The dataset

Preparing the answer key

A blind-rebuild benchmark needs a provably-correct answer key before it can grade anyone. A dataset is one real app, distilled at one pinned commit into a self-describing directory, authored once, offline, then sealed so it can't silently drift.

Authoring runs in two halves. A language-model QA role reads the original and writes each artifact; then a purely mechanical tail proves it. Nothing the model writes is trusted on its word: every check is run against the live original and the whole effort is accepted only if all of them pass.

Authoring · QA-driven

a model reads the original

  1. Clone at the pinned commit & record its exact SHA
  2. Extract requirements + visual shots
  3. Author the human-style build prompts
  4. Author the protocol: schema, adapter, manual (web / tui)
  5. Author one deterministic check per requirement

Deterministic tail

no creativity, code decides

  1. Type-check every check against the black-box wall
  2. Reference gate: run every check on the original, all must pass
  3. Lock + stamp: fingerprint each criterion & check, record the gate-pass

Fig. 7 · the fill pipeline. Written by a model, certified by the original app itself.

The result is a single committed directory. Every file answers one question a run will ask: which app, which source, what to build, how to build it, what “correct” means, and what the correct answers look like.

manifest.json Identity + resolution card. Names the app, the resolution URL, the exact commit pin, the language, the surface, the state schema, and the gate-pass stamp a run demands.
requirements.json The curated acceptance criteria, each tagged design or function, each naming the single check that decides it.
prompts.json The ordered, human-style build prompts, streamed one turn at a time at run time, never handed over as a block.
shots.json + reference/*.png The visual contract: full-page or fixed-coordinate captures (with masks for dynamic areas), paired with the recorded reference image whose filename is the shot id.
checks/ One deterministic black-box check per requirement, reaching the app only through the conformance protocol: no selectors, no markup, no source.
checks.lock.json The drift lock: a fingerprint of every criterion and its check source, taken at validation time.
reference-adapter.js A shim that makes the original speak the protocol so reference answers can be recorded. Gate-time scaffolding, never shipped to a rebuild.
adapter-manual.md The rebuild-facing protocol guide, folded verbatim into the brief every system receives.

Fig. 8 · datasets/<slug>/: one app, distilled at one pinned commit.

Design requirement

what a user sees · soft cap ~10

Is
Layout, colour, type, spacing: the visible truth.
Judged by
Pixel-diff against the recorded reference, within tolerance.

Function requirement

how the app behaves · soft cap ~20

Is
A rule of the game or tool: the behavioral truth.
Judged by
State and pixel together: correct logic that actually repainted.

Fig. 9 · requirements split two ways; the kind decides how it's judged.

The pin

A resolution URL selects the dataset; a full 40-character commit selects the source. Tags move and branches drift, so every run reproduces the identical original, or refuses to run at all.

The drift lock

Fingerprint both the criterion and the check at gate time. You can't loosen a check without breaking the lock; re-gating is the only honest way to restore alignment.

Fig. 10 · the two guarantees that keep two runs comparable: reproducible source, tamper-evident spec.

That's the machinery; the payoff is what it turns up. Every run leaves a full paper trail: which requirements passed, how many feedback passes it took, and what the agent spent getting there. Read the per-system write-ups in the reports, or browse every run's score against its cost in the results.

Read the reports → Browse the results →