How to verify AI coding-agent changes without trusting the worker report

An AI coding worker can say that it changed only the requested files and ran the right tests. That report is useful context, but it is not acceptance evidence. The repository—not the report—must answer what changed, and the driver must choose and run the checks that decide whether the candidate is acceptable.

This guide shows the evidence boundary implemented by codex-agy-worker. It is a bounded engineering workflow, not a security certification or a claim that every semantic defect can be detected automatically.

1. Freeze the base before dispatch

Work in a branch-backed disposable Git worktree and record its immutable base commit before the worker starts:

BASE="$(git -C "$WT" rev-parse HEAD)"

The base gives the driver an independent comparison point. A worker-provided list of changed files cannot replace it.

2. Keep the worker report as untrusted input

Dispatch only after approving the exact repository/path content that may be sent through agy to Google/Gemini. Exclude credentials and denied paths.

echo "$TASK" | AGY_WORKER_MODE=accept-edits ./agy-worker.sh \
  --workdir "$WT" --add-dir "$WT" > envelope.json

The envelope may contain files_changed, commands_run, and tests_run. The gate parses those fields as claims; it never executes worker-supplied shell text.

3. Derive scope from Git

Declare the paths that the task is allowed to change and compare the actual candidate against both that policy and the envelope:

./qa-gate.sh \
  --envelope envelope.json \
  --repo "$WT" \
  --base "$BASE" \
  --only 'tests/**' \
  --expect-edits \
  --verify "cd $WT && python3 -m pytest -q tests/test_parser.py"

The maintained gate rejects undeclared or missing paths, outside-policy edits, malformed envelopes, an unexpected no-op, mutable base evidence, and verifier-created mutations. It also accounts for nontracked paths, including ignored files, within its documented trust boundary.

4. Let the driver own verification

--verify is mandatory for gate acceptance. Choose it from the target repository’s source, configuration, and documented test surface—not from the worker report. A passing check proves only the exercised command against the exact candidate. Review the diff for semantic correctness and run additional checks when the change warrants them.

The useful distinction is simple:

5. Preserve honest outcomes

A failed check is normally a bounded repair signal, not permission to erase a useful candidate or start an unbounded retry loop. codex-agy-worker reports verified only after its strict evidence policy passes; otherwise it preserves distinctions such as partially_verified, rejected, or blocked.

Try the boundary without a provider

From a reviewed clone:

./proof-demo.sh

The demo creates two private synthetic repositories, exercises one exact passing edit and one rejected scope mismatch, and then removes both. It invokes no provider or network and changes neither the checkout nor credentials. These two fixtures are a fast introduction, not proof about a real coding task.

Continue with the project README, the bounded gate contract, or the privacy disclosure.