Use casesIt’s up, but it’s bad

The tunnel says Up, but traffic isn’t getting through.

Escalating to the carrier when the fault is in your own overlay burns hours. The discriminator is simple: a tunnel that is losing traffic over an underlay that is not.

  • Timeabout 15 minutes
  • ForNetwork lead, NOC engineer
  • ScreensDashboards, Explorer, Remediation, KPI Studio

Recognize it

You’re here if

  • Inter-HQ application flows are slow or failing.
  • The tunnel reports Up, and both carriers report clean.
  • Edge CPU is higher than usual.

Do this

4 steps, in order

  1. Compare tunnel loss with underlay loss

    Drill to the site, then open the tunnel and its underlay links. Put tunnel Packet loss next to the link’s Packet loss.

    Open Dashboards
    Look for
    Tunnel loss rising to a few percent while the underlay stays under 0.1%.
    What it means
    A clean underlay with a lossy tunnel puts the fault in your overlay, not the carrier.
  2. Check the crypto cost

    Look at the site’s Device CPU on the same window.

    Open Explorer
    Look for
    CPU climbing alongside the tunnel loss, a sign of rekey thrash.
    What it means
    Repeated rekeying burns CPU and drops packets in the gaps.
  3. Fix the overlay, not the carrier

    Review the recommended action: force a security-association renegotiation and clamp MSS or lower tunnel MTU.

    Open Remediation
    Look for
    A gated, reversible recommendation.
    What it means
    Approve it, then watch tunnel loss fall back toward the underlay’s. Up only means the control channel is alive; it says nothing about the data.
  4. Add the missing alarm

    In KPI Studio, define a KPI on tunnel packet loss, test it against history, then operationalize it.

    Open KPI Studio
    Look for
    A backtest showing when it would have fired.
    What it means
    A tunnel that reports Up and is losing traffic should not wait for a user to report it.

Then decide

What you found decides what you do

  • IfUnderlay is clean and tunnel loss is high

    ThenOverlay fault. Renegotiate and fix MTU or MSS. RB-TUNNEL-OVERLAY

  • IfUnderlay loss is also high

    ThenCarrier or last mile. Go back to the brownout playbook. Calls are choppy, links are green →

Watch out

Where this goes wrong

  • Do not open a carrier ticket until you have shown the underlay is lossy. A ticket with clean underlay data will be returned to you.
You’re done when

You have shown whose fault it is, the overlay is renegotiated, and a divergence check exists.