Use casesIt’s up, but it’s bad

Users say calls are choppy. Every link shows green.

Availability is the wrong question here. The link is up, so nothing pages, and meanwhile latency and jitter have been climbing for hours. You need quality over time, not status.

  • Timeabout 10 minutes
  • ForNOC engineer, Network lead, Service owner
  • ScreensExplorer, Anomaly & ML, Situations, Remediation, KPI Studio

Recognize it

You’re here if

  • Teams or Zoom users complain, no outage ticket exists.
  • The site’s links all read available.
  • Voice MOS or application SLA has been sliding for hours.

Do this

6 steps, in order

  1. Look at quality, not up/down

    Pick the site. Step through Latency, Packet loss and Voice MOS on the widest window.

    Open Explorer
    Look for
    Latency and jitter ramping over hours while availability stays true, and MOS sliding with them.
    What it means
    A gradual ramp with no outage is a brownout. A step is a change. A repeating shape is congestion.
  2. Separate provider from your own edge

    Compare when loss and error rate move against when latency moved.

    Open Explorer
    Look for
    Latency and jitter first, loss trailing behind by tens of minutes, and error rate staying clean.
    What it means
    That order isolates the fault to the provider’s middle mile. Errors would point at your last mile or your CPE.
  3. Ask what static thresholds ignored

    Open Anomaly & ML and look for the link flagged against its own baseline for that hour.

    Open Anomaly & ML
    Look for
    A grey failure: degraded for its hour of the week, never crossing a fixed threshold.
    What it means
    This is the evidence that explains why no alarm fired, and the reason to raise the detection mode.
  4. Build the carrier’s evidence

    Open the situation timeline for the site and note when the ramp began and its p95 latency step.

    Open Situations
    Look for
    A dated onset and a measurable breach.
    What it means
    Carriers act on a start time and a number, not on “users are unhappy”.
  5. Protect the real-time traffic meanwhile

    Review the recommended steering: raise SLA weighting so voice and video prefer the Internet path.

    Open Remediation
    Look for
    A reversible recommendation with a rollback plan.
    What it means
    Fail back once MPLS latency returns to its baseline, often about 25 ms.
  6. Make the next one visible earlier

    In KPI Studio, define a KPI on applications for MOS sliding while the SLA still looks green: slope(mos_score) * (avg(sla_compliance_percent) > 95). Backtest it on history, then operationalize it.

    Open KPI Studio
    Look for
    A backtest showing when it would have fired on this event.
    What it means
    That KPI exists for exactly this case: the dashboards are green and the experience is not.

Then decide

What you found decides what you do

  • IfLatency tracks utilization, loss stays near zero, utilization is only 60 to 85%

    ThenBufferbloat: packets are queued, not dropped. Enable queue management and shape just below line rate. RB-CAPACITY-SATURATION

  • IfQuality fell right after a configuration change on the edge, and the incident’s “what changed?” names it

    ThenConfiguration drift. Diff against the template and re-push. RB-CONFIG-DRIFT

  • IfLatency, jitter and loss all rise on one provider’s links only

    ThenProvider middle-mile brownout. Escalate with the evidence. RB-LINK-BROWNOUT

Watch out

Where this goes wrong

  • A fixed threshold would have fired late or never. If this keeps happening, check the Anomaly agent’s mode: while it is observe-only it records and does not page.
You’re done when

You have a start time, a measurable breach, a diagnosis of whose fault it is, and real-time traffic is off the bad path.