Walk onto any modern plant floor and you will find a monitoring system nobody believes. Operators mute it. Engineers explain away its alarms. Supervisors have a reflexive shrug that means "ignore that one, it always does it." The anomaly detection system became background noise itself, which is worse than useless: it actively trains people to dismiss genuine signals.
I have watched this pattern repeat across machining lines, paint shops, heat treatment cells and assembly plants, and the root cause is rarely the algorithm. It is the failure to distinguish process behaviour from sensor behaviour before a single threshold was ever set. Fix that sequencing and most of the alarm fatigue disappears as a consequence.
This article lays out the working order: characterise the signal first, use static thresholds only where they belong, apply unsupervised methods where they genuinely earn their keep, engineer the alert stream, and validate against real faults before asking anyone to trust the system.
What You Are Actually Detecting
Before choosing any method, be honest about what your data contains. A temperature trace from a furnace thermocouple carries at least four components: the true process value, the sensor's own noise floor, transient artefacts from sampling and transmission, and occasionally genuine process deviations. Most failed monitoring systems fail because someone treated the combined signal as if it were the process itself.
Start with characterisation, not detection. Park the process in a known-good state and record hours of data. Examine the noise distribution: is it Gaussian, or does it have heavy tails from electrical interference? Is the variance constant, or does it scale with the measured value? A pressure transducer whose noise doubles at the top of its range needs different treatment from one with a flat noise floor.
Check autocorrelation too. If consecutive samples are highly correlated, your effective sample size is far smaller than your raw count, and any threshold tuned on raw samples will be far too tight. This single mistake explains a large share of detectors that fire constantly in the first week of operation.
Sensor artefacts deserve their own catalogue: stuck values where the reading freezes on a plateau, dropouts that read as zero or a rail voltage, spike trains from solenoid switching nearby, quantisation steps from coarsely digitised channels. Each will fire a naive detector relentlessly, and each is a sensor problem, not a process problem. Classify them separately in the data pipeline and route them to maintenance, not to the alarm queue.
Static Thresholds: Necessary, Brutal, Insufficient
Every plant starts with fixed limits because they are simple and auditable: a high limit, a low limit, a trip point, done. Static thresholds have a legitimate place in hard safety interlocks, gross out-of-range detection and guarding against sensor failure itself. What they cannot do is respect process context.
A coolant temperature of 78 °C might be perfectly normal during a heavy cutting cycle and a genuine problem during idle. A fixed threshold either alarms during the normal cycle or sleeps through the abnormal one. Context is not a refinement of threshold logic; it is a requirement.
The refinement most plants eventually adopt is the band around a reference trajectory: upper and lower envelopes that follow the expected process state rather than a flat line. This works well for cyclical processes such as press strokes, oven ramps and wash cycles, where you template the expected curve and measure deviation from it. The catch is that the envelope must be built from data that is itself clean. Build the template from a week that included a failing sensor and you have institutionalised the fault.
Rate-of-change limits are the underrated third option. Many real process failures announce themselves through slope before they announce themselves through level: a degrading bearing shows a climbing vibration trend long before any absolute limit is crossed. The trade-off is that derivative signals amplify noise, so you must filter before differentiating, and the filter's lag becomes part of your detection delay. Engineer that trade-off deliberately rather than by accident.
Characterise before you detect
- 01CharacterisePark the process in a known-good state; record hours of data and study noise distribution and autocorrelation.
- 02Catalogue artefactsStuck values, dropouts, spike trains, quantisation steps — routed to maintenance, not alarms.
- 03Set contextual limitsStatic for safety interlocks; envelope bands for cyclical trajectories; slope limits for incipient failures.
- 04Add multivariate modelsOnly on curated, characterised data with known-good training windows.
- 05Validate against faultsTest against a labelled anomaly library before anyone is asked to trust the system.
Unsupervised Methods and Where They Earn Their Keep
Once you move past thresholds, the natural step is unsupervised learning: methods that build a model of normality from historical data and flag departures from it. Isolation forests, one-class SVMs, autoencoders, PCA with distance metrics, density-based clustering — the family is large and the marketing is louder than the engineering. In my experience these methods are excellent at one thing and frequently misused for another.
They are excellent at multivariate detection: the case where five individual channels are each within limits but their combination is impossible. Bearing temperature slightly high, motor current slightly high, vibration spectrum slightly shifted — no single channel trips, yet the joint pattern announces a developing mechanical fault. PCA with Hotelling's T-squared and Q-residuals remains my workhorse because its diagnostics tell you which variables are driving the excursion, not merely that one occurred.
Where unsupervised methods disappoint is when people feed them raw, uncharacterised data and expect magic. Train an autoencoder on a period that included seasonal drift, tooling wear-in and a miscalibrated sensor, and it will happily learn all of it as normal, then flag good product as anomalous when conditions return to nominal. Deep models in particular absorb whatever regime dominates the training window.
The discipline is in training-data curation: known-good states, labelled exclusions of sensor faults, and enough coverage of legitimate operating modes that the model does not treat a valid recipe change as an anomaly. That curation is more work than model selection, and it is where the outcome is actually decided.
Alert Fatigue: The Failure That Kills the System
An anomaly detection system is judged by exactly one metric in the eyes of its users: the fraction of alarms worth getting out of a chair for. Once operators conclude that most alarms are rubbish, you have lost them, and no amount of retraining posters will win them back. Alert fatigue is not a soft cultural problem; it is a quantifiable engineering failure with a measurable signature.
What is a reasonable error rate for one sensor becomes intolerable across a plant.
The arithmetic of fatigue is unforgiving. Suppose your process generates thousands of monitored points and you set a threshold with even a modest per-point false alarm rate. Multiply through and alarms arrive faster than anyone can triage them — each individually defensible, collectively destructive. The Bonferroni logic applies: either accept a much stricter threshold per point, aggregate decisions across related points, or batch and consolidate so a single underlying cause produces one alert rather than forty.
Suppression logic matters as much as detection logic. Deadbands on alarm clearing stop a signal hovering near a limit from chattering. Delay timers stop a transient that resolves within seconds from ever reaching a human. Alarm shelving with mandatory expiry lets an operator silence a known issue without it being forgotten permanently. Prioritisation tiers ensure a multivariate excursion that historically preceded equipment failure outranks a marginal single-channel blip. These are not conveniences; they are the difference between a monitoring system and an ignored system.
Validation Before Trust
The uncomfortable truth is that you cannot tune a detector on normal data alone. False positive rates you can estimate from clean history; false negative rates you cannot, because you need labelled anomalies to count the ones you missed. This is where most deployments quietly go wrong: the system is validated only against the noise it produces, never against the faults it should catch.
Build an anomaly library deliberately. When a real event occurs — a blocked nozzle, a drifting servo, a contaminated batch — go back to the historical data, find the onset, and test whether your detector would have fired, when, and how early. Seed faults during planned maintenance windows where it is safe: pull a sensor connector, partially block a filter, introduce a known offset. Each becomes a labelled case that lets you measure detection delay and sensitivity honestly.
Review performance as a routine, not a project. Weekly or monthly, sit with the alarm log and classify outcomes: true process event, sensor artefact, threshold too tight, model drift. Watch for regime change — tooling generations, recipe changes, seasonal ambient conditions — that shifts the normality baseline and quietly turns yesterday's well-tuned detector into today's noise generator. Retraining on a rolling window helps, but only if the window reflects what normal currently means, and only if you revalidate against the anomaly library after each retrain.
Designing for the Operator, Not the Data Scientist
The final failure mode is presentation. A detection system that outputs a score and a chart forces the responder to do the diagnosis before deciding whether to respond. Operators triage by answerability: can I act on this in the next five minutes? If the alarm cannot say what is affected, which asset, which probable cause category, the honest engineering answer is that it will be ignored — and rightly so.
Alarms that get answered
Alarm as data
- Score exceeds threshold: 0.87
- Chart attached, no interpretation
- Asset ID only
- No suggested action
- Onset time unknown
Alarm as instruction
- Channel combination that fired, with current value against envelope
- Asset, probable cause category, time since onset
- Correlated channels moving in sympathy
- Coarse response path: check sensor vs check process vs shut down
Make every alarm carry context automatically: which channel or combination fired, the current value against its expected envelope, the time since onset, and any correlated channels moving in sympathy. Rich alarms let a responder classify in seconds. Even a coarse suggested response path distinguishing "check sensor" from "check process" from "shut down and inspect" changes the economics of every alert received.
Deploy with the expectation of proving itself. Run the new system in shadow mode alongside the old, compare alarm volumes and outcomes openly, and show the floor the results — including the detections shadow mode caught that the old system missed, and the false alarms you eliminated. Trust in an alarm system is built exactly the way trust in anything else is built: by demonstrated behaviour, reviewed honestly, corrected when wrong.
A system nobody trusts was usually a system nobody was ever given evidence to trust. Close that loop — characterisation, contextual limits, curated models, engineered alerts, honest validation — and the shrug on the plant floor gives way to response, because the alarms have finally earned it.