Backtest & calibration

What this is

Every time a configured gate blocker fires, LoopOver records it. Every time a human later overrides that call (or confirms it), LoopOver records that too. Paired together, the two histories become a labeled corpus: this rule fired against this PR, and a person later said it was right ("confirmed") or wrong ("reversed").

That corpus makes a rule change testable before it merges. A PR that tunes a confidence threshold or rewrites detection logic gets scored against the real recorded history — the same targets, the same raw inputs — instead of being eyeballed.

The corpus

• Fired events — recorded when a configured blocker actually carries gate authority, with bounded raw context: the issue text, PR title/body, and diff the rule evaluated, and (for the AI-judgment rule linked_issue_scope_mismatch) the model's own raw response text.
• Override events — recorded when a human reverses an automated call, or confirms one (for example, an owner closing a PR that was held for low AI confidence confirms the hold).
• secret_leak is permanently excluded from raw-context capture. Storing the diff that triggered a leaked-credential finding would store the credential itself in the audit trail. This rule can therefore never be logic-backtested — by design, not omission.

Two backtests, two mechanisms

Threshold changes run inside LoopOver's own review pass. When a PR's diff changes a known confidence-threshold constant, the old and new values are replayed as classifiers over the rule's recorded history, and the comparison is rendered directly into the unified review comment. No code from the PR is ever executed — a threshold is just a number.

Logic/regex changes run in a dedicated CI workflow instead, because honestly verifying a rewritten detection function requires executing the PR's own code against history — something the production review service must never do. The workflow checks out both the PR's head and its base, replays each recorded case's captured raw context through both versions of the detection function, and posts its own clearly-labeled "Logic backtest" PR comment, separate from the unified review comment. It triggers only for PRs touching the watched detection-logic paths, skips drafts and forks (fork runs get no secrets), and fails open: an infrastructure problem produces a skip notice, never a red check.

Reading the comparison

Both backtests score the same way:

• "Reversed" is the positive class. A classifier predicting "reversed" is saying the rule's original firing was wrong. Precision and recall are computed against the human labels, and stay N/A when there is not enough decided data — unknown is never coerced to zero.
• The Pareto floor decides the verdict. A candidate that regresses on any axis is REGRESSED, even if the other axis improved. Trading precision for recall is a regression, not a net win. Otherwise the verdict is improved (some axis moved up) or unchanged.
• Cases recorded before raw-context capture existed are skipped and counted in the comment — replaying them against empty inputs would bias both sides, so they are never scored.

The track record

Every backtest run persists its comparison as structured audit data. The accumulated record — how often runs regress, per rule — is the evidence base for the pending merge-gating decision, readable at any time with the maintainer CLI:

npx tsx scripts/backtest-track-record.ts --db loopover --remote

The corpus itself can be exported as a versioned, checksummed snapshot:

npx tsx scripts/backtest-corpus-export.ts --rule-id <ruleId> --output corpus.json --remote

Both CLIs are strictly read-only against the database.

Override precedence (per-repo autonomy)

Earned knob overrides resolve through one seam with one precedence order — most specific wins, and an explicit configuration choice always beats anything the system earned on its own:

Priority	Source	Where it lives
1	Explicit per-repo .loopover.yml setting (e.g. gate.aiReview.closeConfidence)	config-as-code, resolved into settings
2	Per-repo earned override	system_flags key <knob>:repo:<owner/repo>
3	Global earned override	system_flags key <knob>
4	Shipped default	the constant beside the rule

Every earned scope applies the identical loosening-only validation (strictly below shipped, at or above the hard minimum), and the knob's autotune flag gates all earned scopes at once — turning the flag off restores shipped behavior everywhere instantly, no cleanup. The knobs status endpoint lists each knob's per-repo overrides so a lingering row is always visible.

Config-drift sentinel

Stale configuration — rules that were right when set and wrong after the repo's reality moved — is the largest empirically-measured wrongness source in the historical ledger. The drift sentinel closes that gap: on the calibration tick (behind CONFIG_DRIFT_SENTINEL_ENABLED, default off) it replays each live knob's current value against the trailing corpus and alerts when a tighter alternative (or a revert to the shipped default) Pareto-dominates what is live — same splits, same floors, same never-on-noise minimums as every other calibration verdict. Looser-dominating findings are deliberately suppressed (the loosening loop owns that direction). One alert per drift episode: a standing unchanged drift never re-alerts, a changed one does, and recovery clears the episode. The sentinel is alert-only — it never writes a knob value — and the knobs status endpoint shows each knob's current drift report regardless of the flag, so a standing drift is always visible.

Tightening under the same evidence standard

Historically the two directions ran on different evidence regimes: loosening moved only on Pareto-floored backtests, while tightening lived in a legacy loop (precision circuit-breakers plus a shadow-soaked advisor). The tightening unification brings threshold raises under the same backtest standard — with the axes orientation made explicit instead of reused blind. A deliberate tightening exists to move one axis at a bounded cost to the other; which axis is which depends on corpus polarity (for the confidence-threshold classifier, the positive class is "predicted reversed", so raising a bar helps recall and risks precision — the inverse of the rule-firing frame). Each knob's optional tightening ladder therefore declares its own orientation: the axis a raise must strictly improve, and the maximum sacrifice the other axis may suffer per comparison slice before the trade is a regression.

Everything else is the loosening discipline transposed verbatim: the same split seed and fraction (held-out membership is identical in both directions), the same never-on-noise sample floors, smallest step first, a hard maximum no evidence may cross, and per-direction double gating — the tighten loop and the above-shipped override read are both gated by the ladder's own default-off flag, separate from the loosening flag, so each direction's autonomy is opted into independently and flipping either flag off instantly restores the shipped default for that direction. A declared ladder also joins the drift sentinel's candidate pool, so tighter-alternative findings and the tighten apply path judge the same values under the same floors.

The legacy tightening triggers that are not threshold moves stay bespoke on purpose: the merge/close precision circuit-breakers are boolean capability breakers judged on realized outcomes (the strongest evidence regime there is), and re-basing them on fired-signal backtests would weaken, not strengthen, their grounds. Their behavior is pinned byte-stable by their own suites.

Reliability curves

Beside the candidate-ladder machinery, each live knob's status now carries its rule's claimed-confidence reliability curve (per-bucket empirical precision over the trailing corpus, null below the sample floor) and the floor that falls out of it at the 0.9 precision bar. The advisor shows the derived suggestion next to the ladder proposal whenever they differ. Curve-derived suggestions are surfacing-only: the bounded ladder still owns any movement, and replacing it is a recorded soak decision (#8227), not a side effect of the curves existing.

Counterfactual replay (design)

Deterministic rule changes replay against history before they land; the same discipline is being extended to the judgment layer — reviewer prompts and model choices — under the counterfactual-replay contract (@loopover/engine's counterfactual-contract module). Three decisions define it:

• Fixtures. A historical case is replayable only when it carries the bounded raw context the live capture recorded (the PR diff) and a human verdict. Fixtures keep their capture-era provenance (live-captured vs backfilled), so a variant that only improves on one era is visible as such. When a run cannot afford every eligible fixture, the subset is chosen by a seeded deterministic sample — recorded in the run artifact so any re-run reproduces it exactly — never "the first N".
• Spend. Replay budgets use the same neuron-estimate units as live AI budgeting, with a hard per-run cap defaulting well under one day's live-review spend. A run that exhausts its budget persists partial scores and a resume cursor. Local models are the smoke path; paid providers sit behind explicit flags. The harness is offline-only: it never posts to GitHub and never touches live reviews.
• Scoring. A replayed variant is just another classifier over the same labeled cases, so the standard scorer and Pareto-floored comparator apply unchanged. Unparseable variant output is an abstention — excluded from the confusion matrix and counted separately, never coerced to a verdict, so a degenerate prompt cannot farm precision by failing to answer on hard cases.

The replay harness

The contract is implemented by scripts/counterfactual-replay.ts (pure core in counterfactual-replay-core.ts). A worked local example against an ollama model:

# 1. Export the labeled corpus (raw-context diffs included) to a manifest:
tsx scripts/backtest-corpus-export.ts --rule-id ai_consensus_defect --output corpus.json --remote

# 2. Score the current prompt as the baseline:
tsx scripts/counterfactual-replay.ts --fixtures corpus.json \
  --variant review-prompt-v1@llama3.1:8b --out baseline.json

# 3. Score a candidate prompt against the SAME deterministic sample and print the Pareto comparison:
tsx scripts/counterfactual-replay.ts --fixtures corpus.json \
  --variant candidate@llama3.1:8b --prompt-file candidate-prompt.txt \
  --baseline baseline.json

Raw model outputs cache under .counterfactual-artifacts/ (never committed), so re-scoring after a parser or mapping change costs nothing. Exit codes reflect operational success only — never a verdict.

The prompt-change workflow

A PR that touches the reviewer judge-prompt surface (src/services/ai-review.ts — bump REVIEW_PROMPT_VERSION whenever a change shapes the judge's verdict) triggers the counterfactual-replay advisory workflow: it extracts the canonical judge prompt from both the base and head checkouts, replays both over the identical seeded sample (seed = head SHA, so re-runs per push are deterministic), posts the comparison as an update-in-place PR comment, and persists a calibration.counterfactual_backtest_run event feeding the same REGRESSED-verdict track record as the threshold and logic backtests. The check spends nothing unless the deployment explicitly configures a replay budget (repo variables COUNTERFACTUAL_REPLAY_BUDGET/COUNTERFACTUAL_REPLAY_MODEL plus the provider endpoint secret) — otherwise it posts a visible "skipped: no replay budget configured" notice. Advisory forever until a separate, explicit authority decision, exactly like every other backtest gate.

Self-hosting

On a self-host deployment the same calibration system runs against your Postgres instead of D1 — the capture writers, the persisted backtest runs, the trend and satisfaction-floor endpoints, and the loosening loop all go through the storage shim, so the corpus lives in your own audit_events table and never leaves your infrastructure. Parity is pinned by a real-Postgres integration suite that runs in CI.

All three maintainer CLIs accept --pg to run against a self-host Postgres instead of D1. An explicit connection string wins; a bare --pg falls back to DATABASE_URL, the same variable the self-host stack itself boots from:

npx tsx scripts/backtest-corpus-export.ts --rule-id <ruleId> --output corpus.json --pg "$DATABASE_URL"
npx tsx scripts/backtest-track-record.ts --pg
npx tsx scripts/backfill-calibration-corpus.ts --pg   # dry-run; add --apply to write

The pg path reuses the deployment's own dialect adapter, so the backfill keeps its INSERT OR IGNORE idempotency contract, and exported corpus manifests name only the database — never the credentials in your connection string. Each deployment's corpus is its own: there is no cross-store sync between a self-host Postgres and any other deployment.