Anomaly models drift. The operating discipline around them is what keeps the alerts worth reading.
The pattern
Condition monitoring deployments follow a recognisable arc. The first weeks are lively — the model flags things, the team investigates, and a few genuine faults get caught early. Then comes a quiet month with no failures, a scattering of alerts that lead nowhere, and a slow drift toward ignoring the dashboard.
By month four the system is technically running and operationally dead. Nothing broke. Nobody decided to stop using it.
Why the quiet month is the hard part
During the quiet period the only visible output is false positives. Without a defined response, each one costs an engineer's afternoon and returns nothing, which teaches the team that the alerts are not worth opening.
The failure is not statistical. A model with a good false-positive rate still dies here, because the cost of investigating an alert is being paid by someone who has no protocol for what to do with it.
A model nobody acts on is not monitoring. It is telemetry with ambitions.
What holds it together
Three practices, none of them technical. Every alert class has a named owner and a written first action — check this, then this, then escalate or close. Every closed alert records why it closed, which is the only training data that improves the model. And thresholds are reviewed on a schedule, not when someone complains.
Retraining follows the same discipline: a planned cadence with a held-out comparison, so a change to the model is a decision with evidence rather than a reaction to a bad week.
What good looks like at six months
Fewer alerts than month one, each one opened by someone who expects it to mean something, and a closure log that explains the last quarter. That is an unspectacular outcome and it is the one worth engineering for.
Written by the software delivery team. Published articles carry a named author once attribution is confirmed.
