A one-shot code-review benchmark: scoring restraint over recall across four Claude models

This started as a boring question. Can I still run Claude Code on Opus 4.6? It's not in the model picker anymore. Ten minutes of poking gave me the answer - pass the full model ID to --model and the picker just hides it. But the question underneath wouldn't let go. Is 4.6 actually worse than 4.8, or does it only feel older because it's not the default?

So I built a benchmark. Then I built a better one. Then I found out my benchmark was lying to me, twice, and I had to fix it before I could trust a single number. By the end I'd tested four models across four task families. The most interesting results weren't about which model won. They were about how easy it is to measure the wrong thing and believe it.

The whole harness is on GitHub: MarcinDudekDev/llm-review-bench. Mistakes documented in the README, on purpose.

1. The design: score restraint

Most "find the bugs" evals reward finding things. That's a problem. Frontier models are very good at finding things, so good that the eval saturates and tells you nothing. My first task had six planted bugs in a FastAPI snippet. Every model found all six. Ceiling hit. Useless.

The skill that actually separates a good reviewer from a noisy one is the opposite. Knowing when to shut up. A reviewer who flags every line that looks a bit off isn't thorough, they're exhausting. So every task ships three kinds of code:

• Bugs - real defects, credited only when the model names the actual mechanism.
• Traps - code that looks wrong but is provably correct on the pinned versions. Flagging a trap costs a point.
• Neutral - real but stylistic observations that earn neither credit nor penalty.

The traps carry the whole thing. An in_array "type juggling" bug that PHP 8 quietly killed. An unquoted variable that word-splits in bash but not in zsh. A synchronous generator that "blocks the event loop", except Starlette runs it in a threadpool. Each one is bait that a well-read model reflexively flags. The good models are the ones that don't bite.

2. Four tasks

• v1 - 6 bugs and 2 traps in one snippet. Easy. It ceilinged. I kept it as a documented failure.
• v2 - 10 obscure version-specific bugs and 6 traps across Python 3.14, PHP 8, and zsh, plus an exact-output asyncio prediction. Hard.
• v3 - 10 shell-command tasks, scored by running the command and byte-matching stdout. No judge.
• v4 - 18 adversarially-filtered hard command tasks. More on that below.

v1 and v2 are code review, blind-judged by two other models. v3 and v4 are command generation. The model emits one shell command, the harness runs it in a sandbox and compares stdout to a golden value. Binary, mechanical, no opinion.

3. Three ways my benchmark lied to me

3.1 A timeout measures the weather, not the model

My first head-to-head used a 60-second cap. Opus 4.6 timed out with no output. Opus 4.8 finished in 28 seconds. Clean win for 4.8.

It was noise. When I re-ran uncapped, the same model on the same prompt swung from 27 seconds to 132 seconds and back. Cost swung 8x on the same model, which is prompt-cache state, not economics. My "4.8 is faster" result was an artifact of an arbitrary timeout landing on the wrong side of a noisy distribution.

3.2 "Make it hard" doesn't make it hard

To separate two frontier models I needed genuinely hard tasks. My first instinct was to prompt a strong model to design a brutal one. That produced a 10/10 ceiling twice. Frontier models are bad at knowing what's hard for other frontier models.

So I used them as difficulty oracles instead. Fable and Grok each proposed candidate tasks. Every reference command was validated by execution. Then each model attempted the other's tasks one-shot, and I kept only the tasks the opposing model got wrong. Those are empirically hard, by construction.

The result was its own finding. 17 of 18 candidates got solved. Two capable models could not reliably write a shell task that stumps a frontier peer. The single survivor turned on a platform-specific %.2f rounding quirk that you can't know without running it on that exact host. One-shot BSD shell-command generation is basically saturated for frontier models.

3.3 I wasn't testing what I claimed

Then a simple question. "Are you sure they only get one shot?"

I wasn't. Claude Code runs with tools enabled by default. Nothing stopped a model from creating the input files, running its command, checking the output, and revising, all before it "answered". The tasks even describe the exact input files. My "one shot, no execution" claim was a hope, not a guarantee.

I made it structural. Disable tools so execution is impossible, and assert that the model answered in a single turn (a tool call forces a second turn, which now raises instead of scoring):

p = subprocess.run(
    ["claude", "--model", model, "--tools", "", "-p", prompt,
     "--output-format", "json"],
    capture_output=True, text=True, timeout=timeout,
)
d = json.loads(p.stdout)
if d.get("num_turns", 1) != 1:          # a tool call would force turn 2
    raise RuntimeError("not one-shot")

Did it change the result? I guessed it would. I thought tools had been propping up the accuracy ceiling. I was wrong again. Under the guaranteed condition both Opus models were still essentially perfect. The ceiling was real. But now the "one shot" label is true, which matters more than whether the number moved.

4. Results

All four models, all four tasks, under the enforced one-shot condition. Review tasks are on a 0-10 judged scale (both judges agreed). Command tasks are exec-scored totals — the denominator is tasks × trials, so on v3 the Opus pair ran 5 trials each (50) and Sonnet/Haiku ran 10 (100); v4 ran 10 trials for all four (180).

4.1 Opus 4.6 vs 4.8 is a tie, and they cost the same

On review, difficulty picks the model. 4.6 is a bit better on easy review, 4.8 is clearly better on hard review. On command generation they're indistinguishable. Averaged out, it's a statistical tie.

And here's the part that answers my original question for anyone hoping 4.6 is the cheap option. It isn't. Anthropic priced the whole Opus 4.x line the same, at 5 dollars per million input tokens and 25 per million output, both with a 1M context window. There's no economic reason to stay on 4.6. Pick on behavior, not price.

4.2 Sonnet 5 is the value story

The Opus tie made me want a real tier gap, so I ran everything on Sonnet 5 versus Haiku 4.5. Sonnet 5 is the standout. It matched Opus 4.8 on the hard command set (179/180), stayed within a point on hard review, and it did that while running the fastest of all four models. At 3 dollars per million input tokens against Opus's 5, it's roughly 60% of Opus's per-token price — and cheaper still per run, because it also burns fewer tokens. If I were choosing a default model for review and command work today, on this data I'd reach for Sonnet 5 first.

4.3 Haiku 4.5 is where the gap finally shows

Haiku is the one place the benchmark separated models cleanly on accuracy. On the hard command set it scored 122/180, against 179 and 180 for the bigger models. On easy review it managed 5.5 out of 10. The traps and the hard version-specific bugs are exactly where a smaller model gets caught.

5. The surprise: Haiku is the slow one

I expected Haiku to be less accurate. I did not expect it to be slow. On the short-command task, median latency:

Read that twice. Haiku, the model whose entire brand is fast and cheap, was six times slower than Sonnet 5 and cost more per run, on the same tasks. It's priced at a third of Sonnet's per-token rate, yet it cost more, which means it generated far more tokens.

If you reach for Haiku to save latency and money on anything that needs real reasoning, measure it first. You might be paying more for less.

6. Limitations

I want to be honest about what this is. Every number is directional. Small n, one machine, one afternoon of API weather. The review accuracy rests on LLM judges scoring against a rubric another LLM wrote, so I mitigate with two judges from different labs and blind shuffling, and I don't pretend to eliminate the bias. The review scores here are single-trial per model. The command tasks are mechanically scored, which I trust more, but they saturate for frontier models, so they discriminate on precision rather than raw capability. Don't read a 2-point review gap as a law of nature. Read it as a signal worth a bigger run.

7. What I'd keep

The method held up better than any single result, so that's what I'd carry forward:

1. Score restraint. Once recall saturates, the traps are what separate models.
2. Don't report a speed winner at small n. A timeout measures the weather.
3. Enforce your constraints in code. "One shot" has to be disabled tools plus a turn-count check, not a line in the prompt.
4. Use strong models as difficulty oracles, and accept that they mostly can't stump each other.
5. "Fast and cheap" is a claim about a workload, not a model. Haiku's thinking budget made it the slowest here.

Next I want to re-run on a quiet machine to strip out the API-weather noise, and judge the review tasks across every trial instead of only the first. That's the real soft spot: the command tasks were mechanically scored over repeated trials, but the review scores still rest on a single judged trial. The harness is built for it now. If you want to poke at it or add your own model, it's all on GitHub.