Use casesSomething is down

A circuit just went down. Confirm the failover and own the next hour.

A hard-down is the easy part to see and the easy part to get wrong afterwards. Did the tunnel really move? What is the backup costing you? When does traffic go home? This gets you to a confident answer fast.

  • Timeabout 10 minutes
  • ForNOC engineer, Network lead
  • ScreensAlerts, Situations, Explorer, Governed Brain, Remediation

Recognize it

You’re here if

  • An availability alert fires on a primary link, usually P1.
  • Users at the site report a short drop, seconds rather than minutes, then carry on.
  • The site’s tunnel now reports a different active path.

Do this

5 steps, in order

  1. Confirm what is actually down

    Filter to Critical and find the link. Note the site and the time it first fired.

    Open Alerts
    Look for
    One link-availability finding on a single primary link, tied to its site.
    What it means
    One link is a circuit problem. Several links across sites at the same moment is a different playbook: go to “Every site in a region went red at once”.
  2. Check that traffic really moved

    Open the situation for that site and read its timeline: root fault begins, detected and correlated, governed as an incident.

    Open Situations
    Look for
    The tunnel’s active path switching to the backup after a brief tunnel-down interval at cutover.
    What it means
    If the tunnel never moved to the backup you have an outage, not a failover. Page the site owner now.
  3. Size the damage on backup

    Pick the site, then look at the backup link’s Utilization and the primary’s Availability over the widest window.

    Open Explorer
    Look for
    Backup utilization against its throughput ceiling, and any overage cost building on a metered 4G or 5G path.
    What it means
    A backup running near its ceiling will starve voice and ERP before the primary is fixed. Prioritize real-time applications or shed bulk traffic.
  4. Open the carrier ticket from the incident

    The incident already carries a ticket, a priority and the suggested runbook. Use its evidence block as the carrier ticket.

    Open Governed Brain
    Look for
    Priority, business impact, and the “what changed?” attribution.
    What it means
    Ask for a physical-repair dispatch on the local loop and quote the outage start time from the timeline.
  5. Set the failback condition

    Read the recommended failback. The standard rule is to return to the primary once its availability has held for 15 minutes.

    Open Remediation
    Look for
    A recommendation with a rollback plan, gated for your approval.
    What it means
    Write the condition on the ticket so nobody has to remember it at 3 a.m.

Then decide

What you found decides what you do

  • IfThe primary is back and the tunnel returned to it

    ThenClose out. Record the outage minutes for the monthly SLA review. Claim an SLA credit →

  • IfThe primary is back but the tunnel is still on the backup

    ThenFailback is stuck and a metered link is billing you for nothing. Fix the hold-down. Find the WAN money leaks →

  • IfThe primary drops again every few minutes

    ThenThis is flapping, not an outage. Contain it first. A circuit keeps flapping →

Watch out

Where this goes wrong

  • Failover that takes much longer than roughly 30 seconds is itself a finding. Ask why before you close the ticket.
  • Do not read “backup is up” as “users are fine”. A metered backup often has a fraction of the primary’s bandwidth.
You’re done when

You know the tunnel moved, what the backup is carrying, who has the carrier ticket, and when traffic goes home.