What it is
The Dual Run is an engineering protocol for launching an AI process. It is the operational protocol of the Embed phase, and it applies again at every increase in autonomy. Without it, moving from level A2 to A3 becomes an experiment rather than a controlled shift. At scale, the difference in the cost of a mistake between those two modes easily runs into double-digit multiples.
The protocol solves three problems at once. First, it validates the quality of the model on real traffic rather than on test sets. Second, it builds the team's trust: people watch the AI work before its output is applied to anything. Third, it collects training data — every disagreement between the AI and a person is captured in the Decision Capture pipeline and feeds the Data Flywheel. The political benefit, that the board sees numbers instead of slides, follows from the first three.
In Oper8 this is the governing rule for moving between autonomy levels. Any movement from A1 Assistant to A3 Co-pilot and on to the more autonomous levels (A4–A5) goes through the four phases. Skipping a phase is a failure mode: the saving looks cheap up front and gets expensive at scale.
How to run it
Four phases of decreasing protection. Each has a fixed threshold to advance, a threshold to roll back, and a key metric.
Phase 1. Shadow (1–2 weeks)
The AI processes 100% of incoming work and its results are not applied. People work as they always have. Key metric: the share of disagreements between the AI and the expert. Threshold to move to Canary: under 30% disagreement for ten consecutive days. Threshold to stop the project: over 50%, or one systematic error repeating. The artefact of the phase is a log of disagreements, classified — AI was better, AI was worse, context was incomplete, the rule was never written down.
Phase 2. Canary (2–4 weeks)
The AI processes 5–10% of real traffic and its results are applied. A person checks every one. Key metric: the override rate — the share of cases where a person reversed the AI's decision. Threshold to move to Gradual: under 15% overrides and zero critical failures over two weeks. Threshold to roll back to Shadow: over 25% overrides, or any critical failure. The main work of this phase is classifying disagreements into the three branches below.
Phase 3. Gradual (4–8 weeks)
The AI takes a steadily growing share: 10% → 25% → 50% → 75% → 100%, one to two weeks per step. Key metric: whether the eval scores hold as the share grows, alongside the business numbers — cycle time, cost, quality. Threshold to advance: eval scores do not degrade and the business numbers hold or improve. Threshold to roll back: eval scores fall below the baseline, or the override rate passes 25% on any segment.
Phase 4. Full
The AI handles the main flow. People move to monitoring, escalations and improving the system. Key metric: the business numbers — Cost-per-Outcome, cycle time, customer satisfaction. Rhythm: a weekly look at the metrics, a monthly review of the eval set, a quarterly architectural review.
The three-way fork: disagreements during Canary
This is where the Dual Run usually breaks. Canary has been running for two weeks or more, and the override rate sits steadily at 10–20%. Formally you are passing — under 25% — but above the target of 15%. What to do depends on where the disagreement is and what is happening to the business numbers.
| What is disagreeing | Business metrics | Action | Review date |
|---|
| Safety (the agent broke a regulatory rule or failed a safety test) | any | Roll back to the previous version of the Rulebook. Review within 24 hours | Immediate |
| Accuracy, business steady or improving | steady / improving | Keep the Rulebook. Update the eval set: the traffic changed, not the agent | Three weeks, then measure again |
| Accuracy, business dropping | falling | Narrow the operational design domain: keep the AI only on the segments where it works | Three weeks to fix, then repeat |
The most common failure is a single reflex response to every disagreement, usually a quick edit to the prompt. Three months later the Rulebook and the model's actual configuration have drifted apart, nobody remembers what the original was, and there is nothing to roll back to. The three-way fork is cheap insurance against that ending.
When it applies
The Dual Run starts at every increase in a process's autonomy, not once per project. Launching a new AI process aimed at A2: run all four phases. Raising an existing A3 to A4 because you widened the domain: run Canary and Gradual again on the widened segment; Shadow can be skipped. Switching from one model to another: Canary and Gradual are mandatory with the existing Rulebook, otherwise you are changing two variables at once and will not know which one worked.
The shape changes by segment. In SMB and mid-market, a Dual Run on one process takes seven to fourteen weeks of calendar time, and the process owner runs it inside their weekly window. Reviewing the Canary disagreements needs an uninterrupted block of thirty minutes or more; without protected time the protocol stretches to three or four months and loses momentum. In enterprise there is usually an AI governance board that signs off phase transitions for processes at A4 and above. The known risk is that the board turns into a rubber-stamp committee that blocks transitions on formalities. The counterweight is to put the time taken to clear phases into the board's own targets, not just the absence of failures.
There are two cases where it does not apply. First, a deterministic process with no variability — bulk data entry into a fixed form; a Dual Run tells you nothing there, so go straight to quality monitoring. Second, a critical process where regulation forbids automated decisions: diagnosing serious illness, approving large loans. There the AI is not an actor but an assistant to the expert, capped at A2.
Who is accountable
The process owner. In every phase they owe three artefacts: signed exit criteria, a dashboard of the metrics, and the classified log of disagreements. A named person reviews the disagreements daily inside their owner's window — a dashboard with nobody's name next to it becomes wallpaper. Each disagreement is decided on the spot: one goes into the evals, the next into the Rulebook, the third into narrowing the autonomy domain for a segment. At the organisational level, the person leading the AI transformation consolidates the process dashboards and watches for any Dual Run sitting still longer than the norm for its segment.
Common mistakes
The endless Shadow. Shadow has run six weeks or more, the formal threshold was cleared, and the move to Canary never happens. Asked why, the answer is "we are still collecting data." In reality nobody wants to own the first time the AI touches real customers. The cure: the exit criteria of every phase are signed before the phase starts, not after. "Under 30% disagreement for ten consecutive days" is an automatic trigger to advance, not an invitation to discuss.
One response to every Canary disagreement. Three months in, the last fifteen changes to the code all carry the comment "fixed the prompt for the disagreement we found," and not one went through the safety checks. The cure: keep the three-way fork next to the dashboard, and classify the disagreement before the prompt change is committed, not after.
Rollback treated as an event rather than a routine. The Dual Run plan contains no word for rollback, and the message to everyone is "if all goes well we will reach 100%." At 50% in Gradual, the metrics dip on the VIP segment, the team reads it as a failure and freezes the whole process for six months. The cure: write rollback into every phase of the plan as a normal outcome. Rolling one segment back to Canary with a review date three weeks out is as routine an operation as moving between phases. In our experience, teams that plan two or three rollbacks during Gradual reach Full noticeably faster than teams that go without.
