HelmBench

Run 1: five arms, 24 sealed seeds

Five model arms audited the same three seeded repos, read-only, with one identical provider-neutral prompt. 24 sealed seeded bugs, five independent blinded adjudication passes, 209 graded findings, 0 false positives.

¹ Terra's A0-S6 hit was ruled via the seed ledger's own diff attribution but sits about 11 lines outside the strict 10-line window. Strict recall: 19 of 24 (79.2%). Both numbers stand.

Tier totals: 6 blatant (T1), 11 moderate (T2), 7 subtle (T3); 3 of the 24 are blind-spot seeds. Price is vendor list, input / output; the Claude arm ran under a subscription, so no per-token price is recorded for it.

Union of all five arms: 23 of 24 seeds. The one miss shared by every arm: B2-S3, an integration key logged at INFO in a secondary module.

Best two-arm cross-family union: Claude Opus 4.8 + GPT-5.6 Sol at high caught 23 of 24 (95.8%) in about 25 combined minutes. Second best at 22 of 24: Claude Opus 4.8 + GPT-5.6 Terra at high, tied with Opus paired with Sol at xhigh or max. All four OpenAI-family arms miss B2-S1, B2-S2, and B2-S3, so no same-family pair reaches either figure; the Claude family fielded only one arm.

Seed by seed

H = hit (same defect, same mechanism, within 10 lines of the sealed location) · P = partial (right location or right class, not both) · · = miss. Seed locations stay sealed; class and tier only.

What held

1. Model families inverted by bug type, consistently across all three Sol arms. The Claude arm caught 6 of 6 blatant seeds but only 4 of 7 subtle ones. The Sol and Terra arms ran the profile in reverse: up to 7 of 7 subtle, as few as 3 of 6 blatant. Different families miss different bugs, which is why the cross-family union wins.
2. The Claude arm was the only arm of five to report a committed, live-looking API key. The OpenAI-family arms that saw it declined it as a placeholder. For secret-hunting, a Claude pass or a deterministic scanner belongs in the loop.
3. The Sol effort ladder is flat for audit recall: high, xhigh, and max all scored 79.2%. Max effort bought 19 extra natural findings, one extra partial, and 4.5x the wall-clock of high. If natural-finding depth is the goal, max earns its cost; for seeded recall it does not.
4. Every arm degraded on B2, the large multi-version Python repo, and coverage-note honesty degraded with it. The Claude arm's coverage notes stayed accurate on all three repos; its B2 note even predicted its own miss areas. The other arms' notes overstated their coverage on B2.
5. One seed beat everything: B2-S3, a sensitive key logged at INFO in a secondary module, missed by all five arms. It stays in the corpus as a standing hard case.

Limitations

• n = 24 seeds, 3 repos, one run per arm. One seed is worth 4.2 points of recall, so the 83.3% vs 79.2% single-arm ranks are within noise. The findings that held across arms are the flat effort ladder and the family inversion; treat single-arm ranks as point estimates.
• Terra's headline 83.3% includes one hit ruled via the seed ledger's own diff attribution, about 11 lines outside the strict 10-line window. Its strict recall is 79.2%. Both numbers are printed.
• The adjudicators were Claude-family agents. Mitigations: arms were renamed and the arm-to-model mapping withheld before grading, the rubric was pre-registered and mechanical, every ruling required reading the cited file, and the zero-false-positive outcome was uniform across arms, including the Claude arm.
• The first battery was voided before grading: the checkout trees were found to be carrying artifacts from unrelated tooling, so every arm re-ran against clean git-archive checkouts of the committed revisions. The voided outputs are archived unscored.
• Seed locations stay sealed so the corpus can be reused across runs. This page publishes each seed's class and difficulty tier only.
• Wall-clock is one full three-repo battery per arm on one machine, single run. Prices are vendor list prices per million tokens (input / output) at run time; the Claude arm ran under a subscription, so no per-token price is recorded for it.

Method

Corpus: three real application repos, each seeded with eight defects (24 total) at sealed locations. Difficulty tiers were assigned at seeding: 6 blatant (T1), 11 moderate (T2), 7 subtle (T3). Three seeds are deliberate blind-spot seeds, placed outside the audit taxonomy the arms would naturally reach for. B2 proved the hardest repo: large, messy Python spanning multiple versions.

Arms: each model ran as a read-only auditor through its vendor CLI over clean git-archive checkouts of the committed revisions, all fed one identical provider-neutral prompt stating the outcome, the constraints, and the completion bar, with no step prescription. Repo-embedded instruction files were excluded on both sides for parity.

Grading: five independent adjudication passes, one per arm, each in fresh context with the arm-to-model mapping withheld. A hit requires the same defect at the seeded location within 10 lines and the same mechanism. A partial (right location or right class, not both) is never folded into a hit; the secondary hit-plus-half-partial column is printed separately. Every accepted finding, seeded or natural, required the adjudicator to read the cited file; claims that could not be verified graded as false positives. Two arms' CVE claims were checked against the public record.

Scoring: recall is hits over 24 seeds. Precision is accepted findings over all surfaced findings; 209 findings were graded across the five arms, 0 false positives.

Re-runs

Policy as of 2026-08-28: HelmBench re-runs within 48 hours of a major coding-model release, on the same sealed seeds, and each run lands here dated. Corrections get fixed and logged on this page.