Definition. The Dual Run is a staged trial in which an AI-supported process is compared with current work, then introduced to a limited share of real cases under agreed controls. The name refers to running the old and new ways of working side by side long enough to learn what each does well.
A convincing demo is not enough to change a live process. The difficult questions appear in ordinary work: which cases the AI misunderstands, how people correct it, what happens to customers, and whether the team can monitor it after launch. A staged trial brings those questions into view while the team can still narrow the scope or stop.
Start with the decisions, not a traffic percentage
Choose the decision types the AI might support. For each one, name the process owner, the person who reviews the AI's output, the outcome that matters, and the cases that must stay with a person. Build a small set of representative tests, including difficult and unusual cases. Decide in advance what evidence would justify a wider trial and what would cause a rollback.
The thresholds should come from the risk and economics of the particular process. A mistake in an internal draft has a different cost from an incorrect decision that reaches a customer. One fixed disagreement rate cannot serve both.
Four useful stages
Shadow. The AI works on real inputs, but its output is not used. People complete the work as before. Compare the two results and classify disagreements: missing context, an unclear rule, a model error, or a case outside the agreed scope. The Decision Log should capture enough information to revisit each finding.
Canary. The AI handles a small, agreed set of live cases. A person checks the result at the point defined in the trial plan. The team watches quality, exceptions, review time and the business outcome. A critical error, a repeated failure pattern or an unexplained decline should pause the trial for the affected case type while the cause is investigated.
Gradual expansion. Add case types or volume only when the previous step meets its criteria. Keep the old route available. Expansion does not have to mean more autonomous decisions: the team may retain approval on high-impact cases while widening AI assistance on routine ones.
Steady operation. The process owner accepts the ongoing review, escalation and maintenance work. Recent cases continue to enter the test set, and the team checks whether quality holds as the work changes. The right end state may be human-approved assistance rather than independent action.
The stages are a planning pattern, not a fixed timetable. A simple internal process may move quickly; a complex or regulated process may need a longer trial or a different route altogether.
What to do when results disagree
Separate three questions before editing a prompt or changing a rule.
| Finding | Immediate response | What to review |
|---|
| A safety boundary was crossed | Stop the affected AI action and return to the previous controlled route | The boundary, test coverage and who can approve a change |
| Quality is mixed but the business result is stable | Hold the current scope while reviewing the disputed cases | Whether the issue is new traffic, missing context or an unclear rule |
| The business result is worsening | Narrow or pause the affected case type | Quality, human review cost, customer impact and the baseline |
Record the decision and set a review date. Rollback is part of the operating plan, not a failure to conceal. Avoid turning every disagreement into an immediate prompt edit: that makes it hard to know which change helped and whether a safety rule still holds.
When to repeat the trial
Run a fresh comparison when the scope changes materially: a new decision type, a new data source, a different model, or a proposed increase in autonomy. The trial can be shorter when much of the process is already understood, but its controls should still match the new risk. If the process is deterministic and a conventional rule handles it reliably, an AI Dual Run may add little value.
The useful output is a decision the team can defend: expand, revise, keep human approval, or stop. Each is better than letting a pilot drift into production without an owner or a way back.
