Use casesToo much noise

The queue is thousands deep. Cut it to what needs a human.

Suppression you cannot see is suppression you cannot trust. The safe route is to measure first, run the policies in observe-only mode, where they only report what they would hide, and enforce only after you have read that list.

  • Timeabout 20 minutes
  • ForNOC engineer, Network lead
  • ScreensPipeline, Noise Agent

Recognize it

You’re here if

  • The queue gets triaged by position, not by priority.
  • The same alerts fire, resolve and fire again within a day.
  • A real problem was missed last month because it was buried.

Do this

5 steps, in order

  1. Measure before you touch anything

    Open Pipeline and read the stages: raw fires, de-duplicated, firing, situations, paging.

    Open Pipeline
    Look for
    How many raw fires collapse into how many things a person is asked to look at.
    What it means
    That ratio is your baseline. Write it down.
  2. See what each policy would remove

    Open the Noise Agent. Make sure it is in observe-only mode (shown as shadow in the app), then read the compression waterfall and the list of alerts each policy would suppress.

    Open Noise Agent
    Look for
    Seasonality, cooldown, storm and chronic policies, each with a count and an example list.
    What it means
    Observe-only reports what would be hidden without hiding anything.
  3. Check the exemptions

    Confirm security alerts are exempt. This is on by default.

    Open Noise Agent
    Look for
    The exempt-security setting enabled.
    What it means
    Keep it on. Only security is exempt by default, so read the suppressed list for anything critical that is not a security alert.
  4. Tune one policy at a time

    Change a single threshold, such as the cooldown window or the chronic recurrence count, and compare the waterfall.

    Open Noise Agent
    Look for
    A before and after for that policy alone.
    What it means
    Two changes at once and you will not know which one hid the alert you wanted.
  5. Enforce on purpose

    After a full week observe-only with nothing in the suppressed list you would have wanted, switch to enforce.

    Open Noise Agent
    Look for
    Suppressed alerts marked as noise and kept out of the queue.
    What it means
    Put a weekly review of the suppressed list on someone’s calendar.

Then decide

What you found decides what you do

  • IfA policy suppresses something you would have wanted

    ThenRaise its threshold, or turn the policy off and keep the rest.

  • IfAlerts stay noisy after policies

    ThenThe source is a flapping circuit or a threshold set wrongly. Fix the cause. A circuit keeps flapping →

Watch out

Where this goes wrong

  • Cooldown and chronic rules look at recurrence, not severity. Read the suppressed list before you enforce, and again after the first week.
  • A change-freeze window mutes pages for the sites it covers. Keep windows short and scoped to what is actually being changed.
You’re done when

You know the baseline ratio, every policy has been previewed, and enforcement is a decision rather than an accident.