Use casesSomething is down

Every site in a region went red at once. Find the one thing that’s actually broken.

When a carrier backbone falls over, every site behind it lights up and the queue fills with symptoms. The job is to stop triaging site by site, name the common factor, and make one call.

  • Timeabout 15 minutes
  • ForNOC engineer, Network lead, Service owner
  • ScreensSituations, Topology, Explorer, Governed Brain, Correlation

Recognize it

You’re here if

  • Many primaries across several sites go unavailable within the same few minutes.
  • The sites share nothing obvious except where they are.
  • Pages are arriving per site and the on-call is already behind.

Do this

5 steps, in order

  1. Stop triaging site by site

    Open Situations. A burst of per-site breaches should already be folded into one situation with one suspected root.

    Open Situations
    Look for
    One situation listing many affected sites and links, correlated by shared provider and timing, with the children inhibited.
    What it means
    If you still see a page per site, grouping did not catch it. Look at step 5.
  2. Confirm the common factor

    Look at the provider and link counts for the affected sites.

    Open Topology
    Look for
    The same provider on every failing primary.
    What it means
    One provider across sites in several locations is a carrier fault, not N circuit faults. NetPulse only names a provider as root when at least three links degrade on transport metrics across at least two sites.
  3. Check the backups can take the load

    For two or three affected sites, open the Internet backup’s Utilization.

    Open Explorer
    Look for
    Backups carrying a whole region’s traffic at once.
    What it means
    Shared Internet breakout can saturate. If it does, expect a second wave of application degradation after the first outage settles.
  4. Declare one incident and escalate once

    Open the incident. Confirm it is a single parent with the child alerts suppressed, then use the correlated link list as the carrier’s evidence.

    Open Governed Brain
    Look for
    One ticket, one priority, the full affected-site list.
    What it means
    One call to the carrier NOC with the list, instead of N tickets with N different stories.
  5. Tune grouping if it split

    If one outage produced several situations, review the time window and provider-sharing rules for this domain.

    Open Correlation
    Look for
    The correlation methods and thresholds in force, editable per domain.
    What it means
    Widen the window a little and re-run before the next outage, not during it.

Then decide

What you found decides what you do

  • IfBackups are stable and traffic is flowing

    ThenKeep the incident open against the carrier and watch for the primaries to recover.

  • IfBackups are saturating

    ThenShed bulk traffic and prioritize real-time applications now. Voice dies at the same hour daily →

  • IfThe outage is over

    ThenCollect the downtime per circuit for the SLA claim. Claim an SLA credit →

Watch out

Where this goes wrong

  • Resist closing child alerts by hand. They are suppressed under the parent on purpose, and closing them early hides which sites recovered when.
  • Applications are always symptoms. If NetPulse names an application as root, treat it as a bug in grouping, not a finding.
You’re done when

There is one incident, one carrier ticket with the full list, and the backups are confirmed stable.