A circuit just went down. Confirm the failover and own the next hour.
A hard-down is the easy part to see and the easy part to get wrong afterwards. Did the tunnel really move? What is the backup costing you? When does traffic go home? This gets you to a confident answer fast.
Recognize it
You’re here if
- An availability alert fires on a primary link, usually P1.
- Users at the site report a short drop, seconds rather than minutes, then carry on.
- The site’s tunnel now reports a different active path.
Do this
5 steps, in order
Confirm what is actually down
Filter to Critical and find the link. Note the site and the time it first fired.
Open Alerts- Look for
- One link-availability finding on a single primary link, tied to its site.
- What it means
- One link is a circuit problem. Several links across sites at the same moment is a different playbook: go to “Every site in a region went red at once”.
Check that traffic really moved
Open the situation for that site and read its timeline: root fault begins, detected and correlated, governed as an incident.
Open Situations- Look for
- The tunnel’s active path switching to the backup after a brief tunnel-down interval at cutover.
- What it means
- If the tunnel never moved to the backup you have an outage, not a failover. Page the site owner now.
Size the damage on backup
Pick the site, then look at the backup link’s Utilization and the primary’s Availability over the widest window.
Open Explorer- Look for
- Backup utilization against its throughput ceiling, and any overage cost building on a metered 4G or 5G path.
- What it means
- A backup running near its ceiling will starve voice and ERP before the primary is fixed. Prioritize real-time applications or shed bulk traffic.
Open the carrier ticket from the incident
The incident already carries a ticket, a priority and the suggested runbook. Use its evidence block as the carrier ticket.
Open Governed Brain- Look for
- Priority, business impact, and the “what changed?” attribution.
- What it means
- Ask for a physical-repair dispatch on the local loop and quote the outage start time from the timeline.
Set the failback condition
Read the recommended failback. The standard rule is to return to the primary once its availability has held for 15 minutes.
Open Remediation- Look for
- A recommendation with a rollback plan, gated for your approval.
- What it means
- Write the condition on the ticket so nobody has to remember it at 3 a.m.
Then decide
What you found decides what you do
IfThe primary is back and the tunnel returned to it
ThenClose out. Record the outage minutes for the monthly SLA review. Claim an SLA credit →
IfThe primary is back but the tunnel is still on the backup
ThenFailback is stuck and a metered link is billing you for nothing. Fix the hold-down. Find the WAN money leaks →
IfThe primary drops again every few minutes
ThenThis is flapping, not an outage. Contain it first. A circuit keeps flapping →
Watch out
Where this goes wrong
- Failover that takes much longer than roughly 30 seconds is itself a finding. Ask why before you close the ticket.
- Do not read “backup is up” as “users are fine”. A metered backup often has a fraction of the primary’s bandwidth.
You know the tunnel moved, what the backup is carrying, who has the carrier ticket, and when traffic goes home.
Keep going
Often next
A circuit keeps flapping
Stop the path-change storm, then turn a month of micro-outages into a repair and a credit.
~15 minNOC engineer, Network lead, Finance and vendor managementMoney and contractsFind the WAN money leaks
Catch stuck-on-backup and idle circuits, and bank the savings with a detector that keeps watching.
~15 minFinance and vendor management, Network leadSomething is downA whole region went red
Collapse a wall of site alerts into one provider fault and escalate once, not forty times.
~15 minNOC engineer, Network lead, Service owner