Failures & catches

What the build caught, and what it could not

This page is where the design gets tested against real runs. It records five failures: the Patcher refusing a fix instead of faking one, two Reviewer verdicts that were wrong under closer scrutiny, a model's own success report that was false, a verifier bug in the pipeline's own sandbox, and a gap found in the signing chain by auditing it. Every claim below links to a cert run, a commit or a bundle in the repository.

The agent was wrong, and refused to fake it.

The upstream fix for CVE-2025-6020 in linux-pam is a patch with 34 separate changes against pam_namespace.c alone. Fifteen of those changes do not apply to the pinned source, plus one more against the module's header file. The Patcher refused outright and named the specific changes that do not apply, instead of writing a patch that applies cleanly but leaves most of the fix out. A patch that compiles and looks finished is more dangerous than a visible refusal, so the contract treats refusal as a first-class output. The model used it.

The reviewer reached the right verdict for a fabricated reason.

On a held-out zlib canary, a fine-tuned Reviewer reached the correct reject verdict, but its written comparison claimed the authored patch was missing a bounds check that was actually present: a correct verdict with a fabricated reason behind it. The first fix tried was a prompt change, and it did not close the gap on its own; the case was eventually closed by a deterministic grounding check, which verifies that text a model cites as evidence actually appears in the file it claims to quote. A verdict is not enough on its own; the citations behind it have to check out too.

The reviewer would have rejected the correct fix.

One bundle in the set has ground truth: a case where the upstream nixpkgs maintainer's own fix was the thing being judged, not an agent's attempt at one. A frontier Reviewer scored that maxima adaptation (Maxima is a symbolic math engine, written largely in Common Lisp) at 0.0, not equivalent. It had applied its own rules as if every file were a separate C translation unit, and flagged a Common Lisp helper function as an unresolved symbol because it is defined in a sibling file of the same package. The upstream maintainer's own fix uses that exact cross-file pattern, so the Reviewer's rule, applied consistently, rejects the fix it was supposed to validate against. This surfaced only because this one bundle happened to carry independent ground truth. On an ordinary bundle, that Reviewer ships and silently rejects good patches forever. The mirror failure happened too. On a libxmp backport (a library for playing module-music tracker formats), with no ground truth to compare against, the same Reviewer rejected a correct patch and invented a different one in its place.

The pipeline's own sandbox hid a crash.

CVE-2024-8176 in expat is a stack-overflow bug that needs the ordinary 8-megabyte process stack limit to actually crash. Nix's build sandbox inherits whatever stack limit the build daemon runs under, and on the CI runner that daemon runs with no limit at all, so the unpatched build processed the malicious input cleanly instead of crashing. This did not pass silently: the cert run failed loudly, with the build log stating plainly that the process never crashed when it was supposed to. The fix was one line, an explicit stack limit set before the test runs. A verifier built on top of a shared build system inherits that system's own assumptions. This time the mismatch was visible. It will not always be.

A model's own claim of success was false, twice.

The Reproducer role writes a fixture and a log describing what it saw when that fixture ran. Twice, the log described a result the build never actually produced. On CVE-2026-78136 in chirp (an eval() code-injection flaw in a ham radio tool, later carried into a fork review PR), a signal-substring check (already part of the contract, not added in response to this) caught a drafted fixture that was wrong in four separate ways: an invented driver class name, a guessed file format the real parser does not use, a missing build dependency, and a marker file written to a path two parallel runs of the same build cannot both use safely. The same class of check caught a second case on an assimp canary, where the model's log claimed a specific crash signal that the unpatched build never actually produced; the real build showed it exiting cleanly on a rejected input, not crashing at all. In both cases the build decided the bundle, and the model's own account of what happened counted for nothing. A reproducer log can only claim what happened; the build, run once without the fix and once with it, is what actually gets checked.

Retired, in the open.

The eval harness exists to kill overconfident claims, this project's best ones included. A fine-tuned 14B model scored 100% against a 90.9% frontier baseline. That number came from eleven examples, and one of those wins turned out to be a parsing-format artifact. A larger set of 34 examples put the fine-tune at 0.794 with seven false accepts, against 0.971 for a frontier reviewer. A training technique applied on top of that, direct preference optimization on the edit shape, was measured and found strictly harmful. Both results stay in the record, and the first buggy run of the project's first signed bundle is still in the repository.

The signing chain itself had a gap, found by auditing it.

Every approved bundle is meant to carry a cosign signature over a fixed manifest of its own file hashes, so a reviewer can check that the bytes in front of them are the bytes that were actually checked. An audit of the tool that opens pull requests against the fork found that it trusted each bundle's own mutable, self-reported hashes instead of the signed manifest, so an edited file paired with an edited hash record would have passed unnoticed. The check was fixed, and the gap written down in the same commit, before a second pass ran the new check against the whole signed set. That second pass found 27 of 63 real signed bundles reconcile cleanly. Most of the remaining gap was drift in bookkeeping files the signature covers but nothing downstream reads. Five bundles had their actual patch or test fixture replaced after signing with no new signature, and two of those already had a public pull request citing the now-stale evidence. One affected bundle has since been re-signed, with the old signature's coordinates kept on record rather than discarded. The affected pull requests were told directly what had drifted, and the rest of the set is still being worked through. This project asks its models to read the actual evidence instead of trusting a claim. Its own signing chain gets the same treatment.