Every site in a region went red at once. Find the one thing that’s actually broken.
When a carrier backbone falls over, every site behind it lights up and the queue fills with symptoms. The job is to stop triaging site by site, name the common factor, and make one call.
Recognize it
You’re here if
- Many primaries across several sites go unavailable within the same few minutes.
- The sites share nothing obvious except where they are.
- Pages are arriving per site and the on-call is already behind.
Do this
5 steps, in order
Stop triaging site by site
Open Situations. A burst of per-site breaches should already be folded into one situation with one suspected root.
Open Situations- Look for
- One situation listing many affected sites and links, correlated by shared provider and timing, with the children inhibited.
- What it means
- If you still see a page per site, grouping did not catch it. Look at step 5.
Confirm the common factor
Look at the provider and link counts for the affected sites.
Open Topology- Look for
- The same provider on every failing primary.
- What it means
- One provider across sites in several locations is a carrier fault, not N circuit faults. NetPulse only names a provider as root when at least three links degrade on transport metrics across at least two sites.
Check the backups can take the load
For two or three affected sites, open the Internet backup’s Utilization.
Open Explorer- Look for
- Backups carrying a whole region’s traffic at once.
- What it means
- Shared Internet breakout can saturate. If it does, expect a second wave of application degradation after the first outage settles.
Declare one incident and escalate once
Open the incident. Confirm it is a single parent with the child alerts suppressed, then use the correlated link list as the carrier’s evidence.
Open Governed Brain- Look for
- One ticket, one priority, the full affected-site list.
- What it means
- One call to the carrier NOC with the list, instead of N tickets with N different stories.
Tune grouping if it split
If one outage produced several situations, review the time window and provider-sharing rules for this domain.
Open Correlation- Look for
- The correlation methods and thresholds in force, editable per domain.
- What it means
- Widen the window a little and re-run before the next outage, not during it.
Then decide
What you found decides what you do
IfBackups are stable and traffic is flowing
ThenKeep the incident open against the carrier and watch for the primaries to recover.
IfBackups are saturating
ThenShed bulk traffic and prioritize real-time applications now. Voice dies at the same hour daily →
IfThe outage is over
ThenCollect the downtime per circuit for the SLA claim. Claim an SLA credit →
Watch out
Where this goes wrong
- Resist closing child alerts by hand. They are suppressed under the parent on purpose, and closing them early hides which sites recovered when.
- Applications are always symptoms. If NetPulse names an application as root, treat it as a bug in grouping, not a finding.
There is one incident, one carrier ticket with the full list, and the backups are confirmed stable.
Keep going
Often next
A circuit went down
Confirm traffic actually moved, size the damage on backup, and set the failback condition.
~10 minNOC engineer, Network leadToo much noiseToo many alerts
Measure the noise, preview each suppression policy, then enforce deliberately.
~20 minNOC engineer, Network leadMoney and contractsClaim an SLA credit
Find the breaches, assemble dated evidence, and build the scorecard for the renewal.
~20 minFinance and vendor management, Network lead, Service owner