Use casesGet ahead of it

Find the problem that no fixed threshold will ever fire on.

A fixed threshold is wrong twice a day: too high for a quiet link at 3 a.m. and too low for a busy one at 9. A detector that knows what normal looks like for each hour of the week finds the failures in between.

  • Timeabout 15 minutes
  • ForNOC engineer, Network lead
  • ScreensAnomaly & ML, Explorer, Situations

Recognize it

You’re here if

  • Users report problems that no alert explains.
  • You raise thresholds to quiet noise and then miss real issues.
  • Peaks look normal on Mondays and abnormal on Saturdays.

Do this

5 steps, in order

  1. See how much it is catching

    Open Anomaly & ML and read the count of anomalies caught that never crossed a static threshold.

    Open Anomaly & ML
    Look for
    A number for the last 24 hours, with the mode: observe-only or enforce.
    What it means
    That is the gap your thresholds leave.
  2. Inspect one detection

    Open a detection and read its baseline for that hour of the week and how far the value sits from it.

    Open Anomaly & ML
    Look for
    A deviation measured against that entity’s own normal, with the evidence behind it.
    What it means
    Deviation from its own normal is the claim. Judge whether it is believable.
  3. Cross-check against what users felt

    Open the same entity in Explorer on a 7-day window.

    Open Explorer
    Look for
    The shape that explains the detection.
    What it means
    If the picture does not show it, do not promote the detection.
  4. See it in context

    Check the banner on Situations that counts anomalies the ML detector caught.

    Open Situations
    Look for
    A banner stating whether they are observe-only (not paged) or folded into situations.
    What it means
    While observe-only, nothing pages. In augment or enforce, they join the situations.
  5. Promote deliberately

    Move from observe-only to a stronger mode only after a week where the detections were consistently ones you would have wanted.

    Open Anomaly & ML
    Look for
    A mode change that you can reverse.
    What it means
    Start with one domain. Keep a record of what you saw.

Then decide

What you found decides what you do

  • IfDetections match real problems

    ThenPromote the mode and add the entity to a board. Build your NOC wall →

  • IfDetections look like ordinary daily peaks

    ThenLeave it observe-only and raise the seasonality sensitivity.

Watch out

Where this goes wrong

  • Observe-only means nothing pages. A detector you never promote is a report nobody reads.
You’re done when

You know the size of the gap, you have judged a real detection against what users felt, and you have a promotion decision.