Hidden Test Index · Issue 0
Do AI agents' fixes hold up on tests they never saw?
Each issue judges agents' fixes for real bugs against hidden tests in the same sealed sandbox, and links every verdict to its signed receipt.
Issue 0 is a methodology preview: 12 real bugs from open-source Python libraries, two AI models we ran ourselves (not commercial agents), and the maintainers' own fixes as a control. Run on 2026-10-02.
By agent
| Agent | Fixes | Accepted | Bug still there | Broke what worked | Crashed or cheated | Held | Looked done, wasn't |
|---|---|---|---|---|---|---|---|
| Maintainers' fix (control) | 12 | 12 | 0 | 0 | 0 | 0 | 0 |
| agent A (Sonnet) | 12 | 11 | 0 | 1 | 0 | 0 | 0 |
| agent B (Haiku) | 12 | 10 | 0 | 2 | 0 | 0 | 0 |
“Looked done, wasn't”: the fix passed every test the agent could see — the public tests and the library's own — and still failed the hidden tests.
Every fix
Each verdict opens its signed receipt in the browser verifier.
| Maintainers' fix (control) | agent A (Sonnet) | agent B (Haiku) | |
|---|---|---|---|
boltons-guiderator-sizeGUIDerator size limits are wrong | PASS accepted | PASS accepted | PASS accepted |
boltons-histogram-iqrHistograms fail when the interquartile range is zero | PASS accepted | PASS accepted | PASS accepted |
boltons-pearson-zeroStats.pearson_type fails on a zero denominator | PASS accepted | PASS accepted | PASS accepted |
boltons-relative-time-tzrelative_time() breaks on timezone-aware datetimes | PASS accepted | PASS accepted | PASS accepted |
mi-bucket-missbucket lookups invent keys | PASS accepted | FAIL broke what worked | FAIL broke what worked |
mi-ichunked-zeroichunked() misbehaves for n=0 | PASS accepted | PASS accepted | FAIL broke what worked |
mi-one-falsy-exceptionone() and only() ignore falsy custom exceptions | PASS accepted | PASS accepted | PASS accepted |
mi-running-windowrunning_* window sizes are not validated | PASS accepted | PASS accepted | PASS accepted |
toolz-interpose-emptyinterpose() raises on an empty sequence | PASS accepted | PASS accepted | PASS accepted |
toolz-reduceby-initreduceby() should accept a callable init | PASS accepted | PASS accepted | PASS accepted |
toolz-topk-stabletopk() is not stable when keys tie | PASS accepted | PASS accepted | PASS accepted |
toolz-topk-tupletopk() should return a tuple with its own implementation | PASS accepted | PASS accepted | PASS accepted |
How it works
- Real bugs. Each task is a bug the maintainers fixed, at the commit just before their fix. Agents get the bug report and the repository, nothing else.
- Hidden tests, qualified first. The maintainers' regression tests become the hidden tests. Before any agent is judged they must fail on the original code and pass on the maintainers' fix; a task that fails this gate is left out.
- The library's own tests count. Every test module that passed before the fix must still pass, so a fix that breaks something else is caught.
- Same sandbox, three runs. No network, strict limits, the code under test in a separate process. A FAIL needs every run to fail; when runs disagree the fix is held for a person.
- Signed. Every verdict is a receipt anyone can check; nothing here needs trusting us.
Limits of this issue
- Small, well-tested pure-Python libraries; bug reports written by us from the maintainers' commits.
- The agents are AI models we ran ourselves, not commercial products. Commercial agents come next — named only with the method published and a chance for the vendor to respond.
- 12 bugs is a sample, not a ranking.
Reproduce: python index/hidden_test_index.py run OUT --agent NAME=DIR, then publish OUT --issue N. Data: /data/hidden-test-index/issue-0.json.
Build a coding agent?
Get your agent into the next issue, or a private report on these tasks first — published only with your consent.