Drift is not an outage
An outage is loud and unmistakable: the system stops responding, an error message appears, someone reports it. Drift is the opposite – quality declines in small steps, each individual answer still looks plausible on its own, and there's no single moment where something obviously "breaks".
Three sources of drift
"Why the launch is only the beginning" described three changes that happen after launch, and all three can cause drift: if the data distribution shifts, more and more requests arrive that the system was never designed for. If the provider updates the model in the background, response behavior can shift slightly without anyone announcing it. And if user expectations rise, unchanged quality suddenly feels like a step backward, even though nothing about the system itself has changed.
Why drift is so dangerous
Because each individual step is so small, people get used to it. Anyone who doesn't measure quality regularly against a fixed baseline, but instead relies on the impression that "it still works", often notices the decline only once it's already clearly felt – and usually through a customer complaint rather than their own observation.
How to catch drift early
The metrics from "The metrics that actually matter after launch" exist for exactly this: error rate, escalation rate, cost per request, and usage rate only work as an early-warning system if they're measured regularly against the same baseline – not against the last measurement, but against the value from launch.
What to do when drift shows up
Drift is not a reason for panic or an immediate restart of the whole project. It's a signal to repeat the structured test from "How to really check if an AI solution works" – with current, real requests instead of the original test data – and to make targeted adjustments based on the result.