Use casesIt’s up, but it’s bad
Users say calls are choppy. Every link shows green.
Availability is the wrong question here. The link is up, so nothing pages, and meanwhile latency and jitter have been climbing for hours. You need quality over time, not status.
Recognize it
You’re here if
- Teams or Zoom users complain, no outage ticket exists.
- The site’s links all read available.
- Voice MOS or application SLA has been sliding for hours.
Do this
6 steps, in order
Look at quality, not up/down
Pick the site. Step through Latency, Packet loss and Voice MOS on the widest window.
Open Explorer- Look for
- Latency and jitter ramping over hours while availability stays true, and MOS sliding with them.
- What it means
- A gradual ramp with no outage is a brownout. A step is a change. A repeating shape is congestion.
Separate provider from your own edge
Compare when loss and error rate move against when latency moved.
Open Explorer- Look for
- Latency and jitter first, loss trailing behind by tens of minutes, and error rate staying clean.
- What it means
- That order isolates the fault to the provider’s middle mile. Errors would point at your last mile or your CPE.
Ask what static thresholds ignored
Open Anomaly & ML and look for the link flagged against its own baseline for that hour.
Open Anomaly & ML- Look for
- A grey failure: degraded for its hour of the week, never crossing a fixed threshold.
- What it means
- This is the evidence that explains why no alarm fired, and the reason to raise the detection mode.
Build the carrier’s evidence
Open the situation timeline for the site and note when the ramp began and its p95 latency step.
Open Situations- Look for
- A dated onset and a measurable breach.
- What it means
- Carriers act on a start time and a number, not on “users are unhappy”.
Protect the real-time traffic meanwhile
Review the recommended steering: raise SLA weighting so voice and video prefer the Internet path.
Open Remediation- Look for
- A reversible recommendation with a rollback plan.
- What it means
- Fail back once MPLS latency returns to its baseline, often about 25 ms.
Make the next one visible earlier
In KPI Studio, define a KPI on applications for MOS sliding while the SLA still looks green:
Open KPI Studioslope(mos_score) * (avg(sla_compliance_percent) > 95). Backtest it on history, then operationalize it.- Look for
- A backtest showing when it would have fired on this event.
- What it means
- That KPI exists for exactly this case: the dashboards are green and the experience is not.
Then decide
What you found decides what you do
IfLatency tracks utilization, loss stays near zero, utilization is only 60 to 85%
ThenBufferbloat: packets are queued, not dropped. Enable queue management and shape just below line rate.
RB-CAPACITY-SATURATIONIfQuality fell right after a configuration change on the edge, and the incident’s “what changed?” names it
ThenConfiguration drift. Diff against the template and re-push.
RB-CONFIG-DRIFTIfLatency, jitter and loss all rise on one provider’s links only
ThenProvider middle-mile brownout. Escalate with the evidence.
RB-LINK-BROWNOUT
Watch out
Where this goes wrong
- A fixed threshold would have fired late or never. If this keeps happening, check the Anomaly agent’s mode: while it is observe-only it records and does not page.
You have a start time, a measurable breach, a diagnosis of whose fault it is, and real-time traffic is off the bad path.
Keep going
Often next
Find what thresholds miss
Let a seasonality-aware detector flag what is wrong for this hour of this week, then decide whether to promote it.
~15 minNOC engineer, Network leadIt’s up, but it’s badVoice dies at the same hour daily
Find the bulk transfer starving real-time traffic, and fix it without buying bandwidth.
~15 minNOC engineer, Network leadMoney and contractsClaim an SLA credit
Find the breaches, assemble dated evidence, and build the scorecard for the renewal.
~20 minFinance and vendor management, Network lead, Service owner