Is my coding agent good enough?
CopyEval evaluates coding agent systems on multi-turn problem solving and cost-effectiveness.
The Ranking
Every system (a roster paired with a model) ranked by mean accuracy (score ÷ max) across all datasets it has run. Best on top.
- Loading ranking…
Accuracy vs. tokens spent
Each point is one system (roster × model), averaged across every run. Tokens are the agent's total spend — input (including cache reads/writes) plus output — so a system that reads less cache sits further right. The x axis is log-scaled (tokens increase to the left), so equal spacing means an equal multiple in spend. The shaded ellipse spans ±1 standard error of the mean on each axis. The mean of a wider ellipse is less certain, reflecting more run-to-run variance.
Loading chart…
Accuracy vs. cost
The same systems, plotted against the agent's self-reported USD spend instead of tokens. The x axis is log-scaled (cost increases to the left), so equal spacing means an equal multiple in spend. The shaded ellipse again spans ±1 standard error of the mean on each axis.
Loading chart…
Limitations
Reading back the per-system run reports surfaces where the benchmark measures its own machinery as much as the agent. These are the load-bearing caveats to keep in mind before trusting a number on this page.
-
Feedback channel
Design feedback has no target to aim at
Design requirements are graded by pixel-diffing the render against a reference the agent never sees. When the logic is right but the pixels drift, the only signal back is “this region differs by
0.55” — no colour, size, or position to correct toward. The loop converges on behaviour it can read and stalls on appearance it can't. -
Feedback channel
Hints name the symptom, not the cause
Checks observe the app only through a small state view, so a critique can only echo what those fields show. When the real bug is invisible to the checker — a word missing from a list, a path joined against the wrong directory — the agent gets a correct but unlocalisable hint and often edits the wrong thing.
-
Environment
The sandbox is half of what's tested
On compiled datasets the outcome is often decided by whether the agent can reverse-engineer a constrained sandbox — no network for dependency downloads, a pinned toolchain, host quirks — not by whether it can build the app. The feedback reports the build error but can never say “you are offline,” so environment-reasoning gets scored as coding skill.
-
Scoring
Bring-up is all-or-nothing
A single build failure zeroes every requirement in a run at once, so one upstream gate can dominate a score. Because the prompt describes behaviour but not the packaging contract, nearly every run scores zero on the first pass — the benchmark front-loads a packaging guess before it can measure anything else.
-
Measurement noise
Scores wobble between identical runs
Some checks race the app — a post-exit terminal snapshot, a timing-sensitive state capture — so byte-identical builds can score differently across iterations, manufacturing phantom recoveries and regressions. Recoveries also cluster at the final allowed iteration, so the 3-iteration cap moves rankings, and each system×dataset cell is only three runs.
Future work
Where the benchmark goes next — closing the gaps above, widening what it covers, and keeping the numbers meaningful as models improve.
- Fix the limitations Work through the caveats above: give design feedback a target to aim at, localise hints to the real cause, surface the sandbox contract, and soften all-or-nothing bring-up and timing-sensitive checks so a score reflects the agent, not the machinery.
- Add more LLM models Extend the field beyond today's Sonnet and Haiku runs — Opus, the GPT family, and open-weight models — so the leaderboard compares systems across a broader, cross-vendor set instead of one provider.
- Add more agent systems Bring in more harnesses and agent loops beyond the rosters here, so the benchmark measures how much the scaffolding around a model matters, not just the model it wraps.
- Build a private dataset Author a held-out set that never ships publicly, so future models can't be trained or tuned against the tasks. Keeping part of the benchmark hidden preserves an honest signal as it becomes a target worth gaming.
Notice
Two open invitations — one to help keep this research going, one to bring it to your own team. If either interests you, please contact the maintainer.
- Support this research CopyEval is independent, open research — and it runs on tokens. Every run spends real model credits across many agent systems, models, and datasets, and that cost is the main thing capping how far it can go. If you're an organisation or an individual who believes in open, vendor-neutral evaluation of coding agents, I'm looking for sponsors to help keep it running and widen its coverage.
- Evaluate your system Run an internal coding agent? I'm open to evaluating it with the same method CopyEval uses in public — multi-turn tasks, real builds, cost-per-result — as a focused project with your team. If you'd like your system measured this precisely, let's talk.