The queue is thousands deep. Cut it to what needs a human.
Suppression you cannot see is suppression you cannot trust. The safe route is to measure first, run the policies in observe-only mode, where they only report what they would hide, and enforce only after you have read that list.
Recognize it
You’re here if
- The queue gets triaged by position, not by priority.
- The same alerts fire, resolve and fire again within a day.
- A real problem was missed last month because it was buried.
Do this
5 steps, in order
Measure before you touch anything
Open Pipeline and read the stages: raw fires, de-duplicated, firing, situations, paging.
Open Pipeline- Look for
- How many raw fires collapse into how many things a person is asked to look at.
- What it means
- That ratio is your baseline. Write it down.
See what each policy would remove
Open the Noise Agent. Make sure it is in observe-only mode (shown as shadow in the app), then read the compression waterfall and the list of alerts each policy would suppress.
Open Noise Agent- Look for
- Seasonality, cooldown, storm and chronic policies, each with a count and an example list.
- What it means
- Observe-only reports what would be hidden without hiding anything.
Check the exemptions
Confirm security alerts are exempt. This is on by default.
Open Noise Agent- Look for
- The exempt-security setting enabled.
- What it means
- Keep it on. Only security is exempt by default, so read the suppressed list for anything critical that is not a security alert.
Tune one policy at a time
Change a single threshold, such as the cooldown window or the chronic recurrence count, and compare the waterfall.
Open Noise Agent- Look for
- A before and after for that policy alone.
- What it means
- Two changes at once and you will not know which one hid the alert you wanted.
Enforce on purpose
After a full week observe-only with nothing in the suppressed list you would have wanted, switch to enforce.
Open Noise Agent- Look for
- Suppressed alerts marked as noise and kept out of the queue.
- What it means
- Put a weekly review of the suppressed list on someone’s calendar.
Then decide
What you found decides what you do
IfA policy suppresses something you would have wanted
ThenRaise its threshold, or turn the policy off and keep the rest.
IfAlerts stay noisy after policies
ThenThe source is a flapping circuit or a threshold set wrongly. Fix the cause. A circuit keeps flapping →
Watch out
Where this goes wrong
- Cooldown and chronic rules look at recurrence, not severity. Read the suppressed list before you enforce, and again after the first week.
- A change-freeze window mutes pages for the sites it covers. Keep windows short and scoped to what is actually being changed.
You know the baseline ratio, every policy has been previewed, and enforcement is a decision rather than an accident.
Keep going
Often next
A circuit keeps flapping
Stop the path-change storm, then turn a month of micro-outages into a repair and a credit.
~15 minNOC engineer, Network lead, Finance and vendor managementSecurityA security alert fired
Read the shape of the event, check whether the network is also hurt, and pick the containment.
~15 minSecurity analyst, NOC engineerSomething is downA whole region went red
Collapse a wall of site alerts into one provider fault and escalate once, not forty times.
~15 minNOC engineer, Network lead, Service owner