Use casesIt’s up, but it’s bad
The tunnel says Up, but traffic isn’t getting through.
Escalating to the carrier when the fault is in your own overlay burns hours. The discriminator is simple: a tunnel that is losing traffic over an underlay that is not.
Recognize it
You’re here if
- Inter-HQ application flows are slow or failing.
- The tunnel reports Up, and both carriers report clean.
- Edge CPU is higher than usual.
Do this
4 steps, in order
Compare tunnel loss with underlay loss
Drill to the site, then open the tunnel and its underlay links. Put tunnel Packet loss next to the link’s Packet loss.
Open Dashboards- Look for
- Tunnel loss rising to a few percent while the underlay stays under 0.1%.
- What it means
- A clean underlay with a lossy tunnel puts the fault in your overlay, not the carrier.
Check the crypto cost
Look at the site’s Device CPU on the same window.
Open Explorer- Look for
- CPU climbing alongside the tunnel loss, a sign of rekey thrash.
- What it means
- Repeated rekeying burns CPU and drops packets in the gaps.
Fix the overlay, not the carrier
Review the recommended action: force a security-association renegotiation and clamp MSS or lower tunnel MTU.
Open Remediation- Look for
- A gated, reversible recommendation.
- What it means
- Approve it, then watch tunnel loss fall back toward the underlay’s. Up only means the control channel is alive; it says nothing about the data.
Add the missing alarm
In KPI Studio, define a KPI on tunnel packet loss, test it against history, then operationalize it.
Open KPI Studio- Look for
- A backtest showing when it would have fired.
- What it means
- A tunnel that reports Up and is losing traffic should not wait for a user to report it.
Then decide
What you found decides what you do
IfUnderlay is clean and tunnel loss is high
ThenOverlay fault. Renegotiate and fix MTU or MSS.
RB-TUNNEL-OVERLAYIfUnderlay loss is also high
ThenCarrier or last mile. Go back to the brownout playbook. Calls are choppy, links are green →
Watch out
Where this goes wrong
- Do not open a carrier ticket until you have shown the underlay is lossy. A ticket with clean underlay data will be returned to you.
You have shown whose fault it is, the overlay is renegotiated, and a divergence check exists.
Keep going
Often next
Calls are choppy, links are green
Find the silent brownout that never trips an outage alarm, and bring the carrier evidence.
~10 minNOC engineer, Network lead, Service ownerRun the NOCApprove or hold a fix
Read the recommendation, the rollback plan and the blast radius, then decide.
~10 minNetwork lead, NOC engineer, Security analystGet ahead of itFind what thresholds miss
Let a seasonality-aware detector flag what is wrong for this hour of this week, then decide whether to promote it.
~15 minNOC engineer, Network lead