The Oper8 method · Concept
User data feeds the process; the process gets smarter.
Definition. The Data Flywheel is a feedback loop in which every decision the AI makes, and every human confirmation of it, improve the system itself.
TL;DR. Without a flywheel, an AI project stops at the pilot: the model does not learn from your data, and six months later its accuracy is no higher. With one, it is. Below: what the flywheel is made of, and why it cannot be bought in a box — only built for your own process.
The Data Flywheel is an architectural feedback loop inside an AI system. The core is simple: the AI proposes something, a person confirms or rejects it, and the result becomes a test case against which the next version of the system is checked. The longer the system runs, the more material it has to learn from — and the harder it becomes for a competitor starting a year from now to catch up.
The flywheel has three levels of maturity, from logging cases to fine-tuning models. They are built in order and you cannot skip one. Without a flywheel, an AI project looks like a one-off integration: you connect a model, you get an effect at the start, six months later quality has stopped improving, and the competition catches up.
The most accessible level: log every AI decision along with the human response. The system proposes something — a classification, a ranking, a piece of text — and a person approves or rejects it. An approval becomes a positive example in the eval set; a rejection becomes a negative one.
What it needs: full logging of the context (the request, the parameters, the time), a pipeline that turns logs into eval format, and versioning of the set alongside versions of the system.
What it gives: over six months an eval set grows from 200 to 4,500 cases — a typical pace for a process handling around fifty decisions a day. This is operational discipline, not magic. Confidence that a new version has not broken the old cases stops being a hope.
The best records from the eval set are placed into the context of the next request. The model sees how you did this before and adjusts to your context. Answer quality improves without retraining anything.
What it needs: semantic search or embeddings over the examples, control of the context window size, and caching for the examples used most.
What it gives: better quality with no delay. Few-shot updates in real time — there is no waiting days or weeks for a retrain.
Once the eval set holds 10,000 or more labelled examples, the model can be fine-tuned on your domain. It learns from your company's real cases rather than a universal dataset.
What it needs: MLOps infrastructure (a retraining pipeline, monitoring), validation on a held-out sample, and a way to roll back if quality drops.
What it gives: the largest return, at the largest complexity. It is worth going here only once levels 1 and 2 are stable.
What follows are two composite cases: the numbers are drawn from two or three real projects in each area and combined. Treat them as orders of magnitude, not as a report on one company.
In a bank's anti-fraud system handling around 50,000 transactions a day, reacting to a new fraud pattern used to take three to four weeks — manual analysis, then a rules update. After three months of growing the eval set (2,000 cases), reaction time fell to two or three days and the model's confidence rose from 60% to 78%. After six months with few-shot, around 85%. After twelve months with fine-tuning, around 91%, with false positives down from roughly 8% to 3%.
In legal contract classification at a consultancy, two months of growing the eval set produced a benchmark for comparing new models. The argument about whether a new model was better or worse stopped: every claim now had a number attached. Few-shot added 12 percentage points of accuracy with no infrastructure spend. Fine-tuning, eight months in, added another 8 and cut manual review time by roughly a third.
The effect compounds: every month of operation adds value a competitor cannot copy off the shelf. That is the Cost-per-Outcome falling over time — while a competitor without a flywheel keeps paying the old one.
The first step is not a model and not a platform. The first step is the eval set. By the end of week one you should have 50 to 100 labelled examples of "the right decision for our process." That is less than it sounds: two or three hours from a domain expert, plus an engineer to put the examples into JSON or CSV.
The process owner leads this — the person accountable for the business result, not for the AI stack. An engineer sets up the infrastructure, but the owner defines what counts as right and wrong. Without that pairing the flywheel never starts: you get handsome logs with no business meaning.
In a mid-market company, level 1 is often enough for twelve to eighteen months — a simple eval set plus running new versions against the old set gives steady gains in accuracy without expensive infrastructure. In enterprise you usually need level 2 within four to six months: the volume of decisions is such that without few-shot the model cannot keep up with changes in the data. Go to level 3 only when levels 1 and 2 are operationally stable and there is budget for an MLOps team.
Dirty data amplifies the error. The flywheel improves whatever it sees most. If the source data is skewed — say 80% of enquiries in one language and 20% in another — the flywheel will entrench the skew rather than correct it. The labeller unconsciously approves more examples in the dominant language, the model gets better there and worse elsewhere, and then everyone is surprised that accuracy is falling for one group of customers. The cure: once a month, check which groups are thin in the set — by language, region, industry, period — and deliberately top those groups up in labelling, even if they are a small share of live traffic.
Labelling quality decays. The labeller tires, gets careless, the examples get noisy. Three or four months on, the model is learning not from correct decisions but from a tired person in a hurry. The cure: once a month give two different labellers the same fifty examples and compare. If they disagree on more than 20%, stop and retrain the labellers.
Jumping levels. The most expensive mistake, and the most common in enterprise with a large budget: we have the money, let us go straight to fine-tuning. The model looks better because the architecture is new, but there is no benchmark to compare against, so there is no way to prove the improvement was not luck. The cure: always start at level 1 as your reference point, even when it looks too simple. Without it, every later step is faith rather than engineering.
The flywheel entrenches yesterday's world. The most insidious failure. The flywheel turns honestly: the eval set grows every month, the scores climb three to five points, each new version beats the last. All of it against an outdated slice of reality. The distribution of customer requests shifted four months ago — a new segment, new kinds of enquiry, new requirements — and the flywheel is busy tuning the model to the old distribution. The tell: the eval score rises while the business number (satisfaction, conversion, share of cases resolved without escalation) stays flat or falls. The cure: once a quarter take thirty fresh cases from production and check whether they land in any cluster of the eval set. If more than 30% fall outside, reweight the set to the traffic you actually have. It is the same trap described in evals before the system — it just shows up faster here, because the loop is shorter.
The labeller copies the AI. When a labeller sees the AI's proposal and decides whether they agree, they drift towards the model's answer. Three or four months on, the labelling has lost its independence and the flywheel is entrenching its own mistakes. The cure: label 10–20% of cases blind — the labeller does not see the AI's answer and writes their own from scratch. If the gap between blind and sighted labelling is over 15%, raise the blind share to 30–40%. For risk-critical processes such as anti-fraud or medicine, blind labelling is mandatory throughout.
Related articles
We’ll look at your problem, pick the right format, and show where AI actually strengthens your operations.
50+
companies have gone through AI transformation with us — from diagnostics to working solutions.
© 2026 Kiberry — the European practice of ScrumTrek
