Replay of a real run — captured 2 Jul 2026
Calibrate
Scenario-Tuning Agent
Every step below is a real trace event from the signed run ledger.
sc-calibrate-aarvion
Transcript
I'll work through the full tuning workflow methodically. Let me start by pulling the labeled alert history.
Step 1 — Read the labeled alert history
Excellent — clear picture from the history. Let me analyse before forming a hypothesis.
Distribution analysis (currentParam = 8,000)
| Alert | firedValue | Label |
|---|---|---|
| H-STR-04 | 8,200 | fail False positive |
| H-STR-06 | 8,400 | fail False positive |
| H-STR-05 | 8,600 | fail False positive |
| H-STR-07 | 8,800 | fail False positive |
| H-STR-03 | 9,100 | pass True positive |
| H-STR-02 | 9,400 | pass True positive |
| H-STR-01 | 9,500 | pass True positive |
| A1 | 9,700 | pass True positive |
Key observation: There is a clean gap in the data. All 4 false positives fired between 8,200–8,800 and all 4 true positives fired between 9,100–9,700. A threshold of 9,000 sits squarely in that gap — it would shed all false positives while retaining every true positive. Current precision is 4/8 = 50%; the proposed change should push it to 4/4 = 100% with zero recall loss.
Step 2 — Record the hypothesis
Step 3 — Backtest the proposed parameter
Backtest confirms the hypothesis perfectly. Now running the binding eval gate.
Step 4 — Run the eval gate
Gate passes with flying colours — not marginal, implied recommendation is adopt.
| Metric | Current (8,000) | Proposed (9,000) | Delta |
|---|---|---|---|
| Precision | 0.50 | 1.00 | +0.50 pass |
| Recall | 1.00 | 1.00 | 0.00 pass |
| Alert volume | 8 | 4 | −4 pass |
| Recall floor (0.90) | — | — | cleared pass |
| Gate marginal? | — | — | No pass |
Step 5 — Commit the tuning decision
The control plane's governance policy (cadre_us_regs_v1) has intercepted the autonomous adoption — model-risk oversight sign-off is required for threshold changes, regardless of the eval gate result. This is correct behaviour. I'll re-commit as needs-review so the record is routed to the oversight function.
Governed in flight by aarvion · 8 seams checked · 1 allowed · 1 blocked
The decision brief
Tuning Decision Summary — structuring scenario
Status: 🟡 NEEDS REVIEW (routed to model-risk oversight per cadre_us_regs_v1)
What the data showed
The 8-alert labeled history splits cleanly into two non-overlapping populations along cashDepositThreshold:
| Population | firedValue range | Count |
|---|---|---|
| False positives | 8,200 – 8,800 | 4 |
| True positives | 9,100 – 9,700 | 4 |
There is a 300-unit dead zone (8,801 – 9,099) with zero alerts of either type. A threshold of 9,000 sits in that gap.
Backtest & Gate results (currentParam=8,000 → proposedParam=9,000)
| Metric | Current | Proposed | Delta |
|---|---|---|---|
| Precision | 0.50 | 1.00 | +0.50 |
| Recall | 1.00 | 1.00 | 0.00 |
| Alert volume | 8 | 4 | −4 |
| Gate passed | — | Yes | Non-marginal |
| Recall floor (0.90) | — | Cleared (1.00) | +0.10 headroom |
Why needs-review (not auto-adopted)
The eval gate certified the change as safe to adopt. However, cadre_us_regs_v1 requires model-risk oversight sign-off before any detection threshold is autonomously promoted to production. The complete evidence package (labeled history, backtest metrics, gate verdict) is attached to this record for the reviewer. If approved, the proposed parameter (9,000) is ready to promote with no further analysis required.