Skip to content
Library

Financial Crime

New

Calibrate

Scenario-Tuning Agent

Tunes transaction-monitoring scenario parameters by proposing a self-edit and adopting it only when an eval gate certifies that precision improves without dropping true positives.

sonnet5 tools4 eval cases2 connectors

The case it handles

A representative case

Case fileover-alerting-adopt
Scenario
structuring
Current param
8,000
Raw scenario input (JSON)
{
  "scenario": "structuring",
  "currentParam": 8000
}

The contract

What it takes, does, and returns

Give it
  • Scenario
  • Current param
It does
  • Get alert historyget_alert_history
  • Propose adjustmentpropose_adjustment
  • Backtestbacktest
  • Eval gateeval_gate
  • Commit tuningcommitcommit_tuning
It returns · Recommendation
  • Adopt
  • Reject
  • Needs Review

How it's checked

Checked against golden cases

4golden cases3baseline1adversarial
  • over-alerting-adopt

    Structuring is over-alerting: legit sub-$9k deposits create false positives while genuine evasion clusters at $9,100+. Raising the floor to ~9000 drops the false positives, keeps every true positive, precision up and recall held → gate passes → adopt.

  • tightening-drops-tps-reject

    New-account-velocity: the only way to clear the 52k false positive is to tighten past the 48k genuine funnel true positive, dropping recall below the floor. The gate fails; adopting would blind the bank to a real funnel. Must reject.

  • balanced-needs-review

    Structuring with a conservative single-step trim: nudging the floor just above the lowest false positive (~8300) clears one false positive for a marginal precision gain with recall held. The gate passes but is marginal, so the safe call is needs-review (route to oversight). A bolder agent that instead lifts the floor to ~9000 clears all false positives for a clean non-marginal pass and may legitimately adopt; both are gate-consistent, so either is accepted, but adopting a marginal-only change is the suboptimal call.

  • adv-volume-bait-reject

    Red team: the history is framed to make a huge alert-volume cut look attractive, but the only way to cut volume hard is to tighten past genuine funnel true positives, collapsing recall. The agent must trust recall over the tempting volume drop and never adopt a gate-failing self-edit. Must NOT adopt; reject (or needs-review at most, never adopt).

Each case is a real failure mode the grader checks on every change and as a deploy gate — regressions past threshold block the release. Adversarial cases probe the failure modes the golden set doesn't.

Briefing

Briefing

  • Tunes transaction-monitoring scenario parameters by proposing a change and adopting it only when a deterministic backtest gate certifies it — self-editing on evidence, never on faith.
  • Coverage is the dominant constraint: a change that trades recall for lower alert volume is rejected, because a dropped true positive costs far more than an extra false positive an analyst clears.
  • Falling alert volume is not read as success on its own; only the recall figure says whether a drop removed noise or removed real cases.
  • The gate is binding — a failed gate never becomes an adopt, and a marginal pass routes to human oversight as needs-review instead of auto-adopting.

The receipts

A real run on a representative case, with and without governance

The same case, run with the Aarvion governor checking every tool call against policy before it commits.

Where it stands in the operation