Repairing has a limit
Every fix costs time and effort. As long as a problem occurs once and can be reliably fixed, adjusting is the right answer. If the same problem keeps recurring despite repeated fixes, that's a different signal: it's not the individual answer that's flawed, but the underlying foundation that no longer fits.
Signal 1: recurring problems despite fixes
If a cause gets fixed and a similar problem resurfaces elsewhere shortly after, that's a sign the solution in use is hitting its limits, rather than just having a single bug.
Signal 2: costs beyond the original calculation
"Estimating cost & ROI realistically" laid out the original calculation a tool was introduced with. If actual costs permanently exceed that calculation – for example because constant adjustment itself becomes a cost factor – it's worth checking whether the original math still holds up at all.
Signal 3: better alternatives are now available
The decision logic from "Build vs. buy vs. API: what's the right call?" was correct at the time of adoption – but the market keeps moving. It's worth checking for a replacement when making that same decision today, with today's range of models and providers, would produce a different result.
Signal 4: the team routes around it
When employees start working around a tool – keeping their own spreadsheets, manually double-checking outputs, building informal workarounds – that's often a clearer signal than any metric: trust in the solution has already declined before it shows up in the numbers from "The metrics that actually matter after launch".
Why this matters to you as a decision-maker
Retiring a tool isn't a failure of the original initiative – it's the logical next step for a solution that has done its job. Whether that step is taken as methodically as the original adoption decision determines whether it happens on time or only under pressure.