Find the problem that no fixed threshold will ever fire on.
A fixed threshold is wrong twice a day: too high for a quiet link at 3 a.m. and too low for a busy one at 9. A detector that knows what normal looks like for each hour of the week finds the failures in between.
Recognize it
You’re here if
- Users report problems that no alert explains.
- You raise thresholds to quiet noise and then miss real issues.
- Peaks look normal on Mondays and abnormal on Saturdays.
Do this
5 steps, in order
See how much it is catching
Open Anomaly & ML and read the count of anomalies caught that never crossed a static threshold.
Open Anomaly & ML- Look for
- A number for the last 24 hours, with the mode: observe-only or enforce.
- What it means
- That is the gap your thresholds leave.
Inspect one detection
Open a detection and read its baseline for that hour of the week and how far the value sits from it.
Open Anomaly & ML- Look for
- A deviation measured against that entity’s own normal, with the evidence behind it.
- What it means
- Deviation from its own normal is the claim. Judge whether it is believable.
Cross-check against what users felt
Open the same entity in Explorer on a 7-day window.
Open Explorer- Look for
- The shape that explains the detection.
- What it means
- If the picture does not show it, do not promote the detection.
See it in context
Check the banner on Situations that counts anomalies the ML detector caught.
Open Situations- Look for
- A banner stating whether they are observe-only (not paged) or folded into situations.
- What it means
- While observe-only, nothing pages. In augment or enforce, they join the situations.
Promote deliberately
Move from observe-only to a stronger mode only after a week where the detections were consistently ones you would have wanted.
Open Anomaly & ML- Look for
- A mode change that you can reverse.
- What it means
- Start with one domain. Keep a record of what you saw.
Then decide
What you found decides what you do
IfDetections match real problems
ThenPromote the mode and add the entity to a board. Build your NOC wall →
IfDetections look like ordinary daily peaks
ThenLeave it observe-only and raise the seasonality sensitivity.
Watch out
Where this goes wrong
- Observe-only means nothing pages. A detector you never promote is a report nobody reads.
You know the size of the gap, you have judged a real detection against what users felt, and you have a promotion decision.
Keep going
Often next
Calls are choppy, links are green
Find the silent brownout that never trips an outage alarm, and bring the carrier evidence.
~10 minNOC engineer, Network lead, Service ownerGet ahead of itGet warned before a breach
Read a forecast, act on its ETA, and check the track record before you trust it.
~10 minNetwork lead, NOC engineerToo much noiseToo many alerts
Measure the noise, preview each suppression policy, then enforce deliberately.
~20 minNOC engineer, Network lead