Trust
How we can be wrong
A verdict that moves money has to be checkable and its mistakes have to be known. Here is every wrong verdict we have found, where a person decides instead of the machine, and what is not built yet.
Don't trust us: check
- Every verdict is a signed receipt. It binds the job terms, the exact code that was judged, the evidence and the decision with SHA-256 digests, signed with Ed25519. Change one byte and verification fails.
- Anyone can check it. /verify runs in your browser and uploads nothing; the same check runs in Python and JavaScript and they agree on shared test vectors (how).
- Pin one key. Our offline root key
rb:root:2026-10(k4kS-tpmIN5OQFjdt8Q8AJiaPC_iOa6jfhU5IIz3AUE) signs the list of working keys at /.well-known/resultbond-keys.json. Compare the fingerprint through a channel other than this site.
Every wrong verdict we know of
| Where | What happened | Why | What we changed |
|---|---|---|---|
Benchmark, shop/forge-exit10 of 570 runs | A wrong fix was judged PASS. | With ordinary pytest tests the code under test runs in the same process as the tests and can forge the whole report. | Black-box tests: the code runs in a separate worker with no access to the results. The same attack and two stronger ones now FAIL, 10 of 10 each. Pilots use black-box tests by default; the forgeable mode stays in the benchmark as a known false PASS. |
| Demo pilot, round 1 2 bugs, 4 runs | Both agents' fixes for bucket and ichunked were judged PASS; the maintainers' own test suites said FAIL. | Each fix broke behaviour our hand-written tests did not cover. | Pilots also run the repository's own tests as a regression check (--repo-tests). Round 2: the broken fixes FAIL with “broke existing tests”. |
| Demo pilot, round 2 1 of 36 runs | Agent A's ichunked fix was judged PASS; the maintainers' suite said FAIL. | The maintainers also changed an error message the bug report never asked for. The agent fixed what was asked. | Counted against us anyway. In a client pilot such a requirement is written into the task before any agent runs. |
| Internal audit, October 2026 | In black-box mode, code under test could read the hidden test files. | The hidden tests were mounted into the sandbox. | Fixed in 0.10.2, before any client job: the tests reach the sandbox over a private channel and are erased after collection. A benchmark attack proves the code cannot see them. |
A correct fix has never been judged FAIL: 0 times in 570 benchmark runs and 72 pilot runs. On this corpus that bounds the rate at roughly 0.5 % (the “rule of three”); a wider mix of bugs will find some. That is why the next section exists.
Where a person decides
- HOLD instead of a guess. If the original code behaves inconsistently, the fix behaves differently between runs, the sandbox fails, or the hidden tests were not proven to test the bug, the verdict is INCONCLUSIVE and the job is held. Nobody is blamed by default.
- An independent reviewer closes held jobs with a signed decision: pay, refund, or return both deposits.
- A window to dispute. A receipt settles only after its challenge window closes; a disputed receipt is reviewed by a person.
- Hidden tests are qualified before any agent runs. They must fail on the original code, and pass on a known-good fix, so tests that would reject a correct fix written differently are caught first. A client's engineer can sign off on them; receipts say whether a person did.
How hidden tests stay hidden
- The agent receives the bug report and the public tests only. Hidden tests are stored with the evaluator and are never sent to the agent or its platform.
- Inside the sandbox they arrive over a private file descriptor, are loaded by the test runner and erased before the candidate's code starts; the code under test runs in a separate process without network.
- Receipts carry only digests of the tests, never their contents.
- Who writes them: the client (we keep them secret), our team as a service, or, for the public index, regression tests taken from real upstream fixes. Every source goes through the same qualification.
What is not built yet
- Python and pytest only. Other languages when pilots ask for them.
- The sandbox shares the host kernel (bubblewrap, cgroups, no network). A kernel exploit could escape it, so judging runs on hosts dedicated to it; microVM isolation is next.
- The escrow contract has not been externally audited and runs on a testnet only. Money moves on a real network only after an audit and legal review; until then pilots are shadow mode and payment is by invoice.
- Small team. Security reports: see the key policy and reach us through the request form.