Why a one-time test score isn't enough
A test before launch produces a single number at a single point in time. Whether that number still holds three months later can only be answered by repeating the same measurement regularly – against the same benchmark, not by gut feeling.
Metric 1: error rate over time
The same check that ran once at launch can be repeated at fixed intervals – for example monthly, against a sample of real requests. A declining success rate is a direct signal of the data-distribution or model changes from the previous module.
Metric 2: human escalation rate
How often does the system hand a request off to a human because it can't proceed on its own or needs approval? A rise in this rate is often an early warning sign that becomes visible long before a customer complaint does.
Metric 3: cost per request over time
"Token economics: how AI costs actually arise" explained the four cost levers. In ongoing operation, what also matters is how these develop over time: if cost per request rises even though the model and provider haven't changed, that points to a shifted usage pattern – for example, more complex requests than assumed at launch.
Metric 4: actual usage rate
Is the system actually being used, or are employees quietly reverting to the old, manual way of doing things? A declining usage rate often signals a trust or adoption problem before it shows up in other metrics like error rate.
Why this matters to you as a decision-maker
None of these four metrics is sufficient on its own – only together do they show whether an AI solution still does what it was introduced to do. Building them in from the start, rather than reacting only once customers complain, means catching problems while they're still small and cheap to fix.