Architecture

Put the model in the smallest box

Of the eight roles in the pipeline, three touch a model: the Patcher drafts, the Reviewer critiques, the Reproducer writes a test fixture. Eligibility, provenance, hashing, building, routing and signing are all deterministic code that a human can read start to finish. A model's output is always executed or checked by something else before it counts for anything.

Why a separate reviewer

The Patcher drafts a fix; the Reviewer critiques it with a separate model invocation and a separate prompt, never the Patcher's own context. Today both roles run on Anthropic models, Opus for the Patcher and Sonnet for the Reviewer, so the review is not literally independent in the strongest sense: a genuinely different vendor's model, immune to the same training-time blind spots, is a stronger bar the project has not yet met. What the current setup does buy is a model that never sees its own draft's reasoning and grades the diff with no view of how it was drafted, which is distinct from a model proofreading its own work in the same context.

Built twice, with and without the fix

The Reproducer writes a small test that triggers the specific bug the advisory describes. trace then builds the package twice with nix-build, once without the fix and once with it, keeping every other input identical, and runs the test against each build. The build without the fix has to crash or misbehave in the way the advisory names. The build with the fix has to survive the same input. Neither result comes from the model's own account of what happened. Both come from how the builds exit, inside an isolated sandbox with no network access, in CI. See Failures for two cases where a model's own log claimed a result the build never produced.

A policy decides the route

Before a candidate ever reaches a model, a versioned routing policy assigns it one of four states: auto-merge-eligible, human-review-required, advisory-only, or reject, the default. Once a bundle is finished, a second, separate policy checks it against the evidence it actually produced and either allows or denies signing, with every deny carrying a stable reason code. Auto-merge means a branch on trace's own nixpkgs fork, never upstream; nothing this pipeline produces is pushed directly at NixOS/nixpkgs. The fork review PRs that exist are accuracy-comparison artifacts against the human fix, not merge candidates waiting on maintainer attention.

What a signature proves

A bundle the Steward signing gate allows is signed with Sigstore keyless OIDC and anchored in the Rekor transparency log, bound to the exact trace workflow identity locked in policy/signing-identity.json. Not every bundle in the set is signed; signing has been run against a subset so far, covered on the findings page. A signature confirms who produced a bundle and what exactly they claimed. Whether the claim is true is a separate question, answered by the build rather than the signature: cosign verify-blob only checks the signed summary of a bundle, not that the payload files beside it match what was actually signed. See Failures for what checking that the harder way actually found.

Inside a single bundle

A single bundle is a stack of claims, each with its own evidence, that a reviewer can audit link by link without trusting any other link: the advisory Scout found, the resolver's grounded upstream reference, the Patcher's diff and its written rationale, the cross-family review, the reproducer that crashes the unpatched build and passes the patched one, a replay record pinning the exact nixpkgs revision, prompt hash, model id and policy version so the whole run can be independently replayed, and finally the signature over all of it.