An edge device is unhealthy. Is it heat, a memory leak, or the controller?
A rebooting edge looks the same on a dashboard whatever the cause, and the fixes are completely different: a technician, a rollback, or a controller restart. Read temperature, memory and CPU on one window and the cause usually announces itself.
Recognize it
You’re here if
- An edge reboots repeatedly, or its tunnels drop and re-establish.
- Control connections fall to zero for a reading or two.
- A site is intermittently out, with no carrier issue behind it.
Do this
5 steps, in order
Read three signals on one window
Pick the site and step through Temperature, then Device CPU, on the same window. Then open the device in Dashboards for Memory.
Open Explorer- Look for
- Which of the three moves, and whether CPU moves with it.
- What it means
- Heat with flat CPU is hardware. Memory climbing steadily is software. Neither, on many devices at once, points at the controller.
Check what changed
Open the incident for the site and read its “what changed?” attribution.
Open Governed Brain- Look for
- A software upgrade or configuration change shortly before onset.
- What it means
- A leak that starts at an upgrade is a rollback decision, not a debugging session.
Check the blast radius
Drill from the fleet to the site, then to the device. Compare against other devices of the same model.
Open Dashboards- Look for
- One device, a whole site, or a whole cohort.
- What it means
- A cohort means the cause is in the image or the controller. Hold the image before the next site upgrades.
Act on the cause
Open the recommended playbook for the incident. Each action shows its rollback plan and whether it needs your approval.
Open Remediation- Look for
- A gated recommendation, not an automatic change.
- What it means
- Approve, or hold with a reason. See the approval playbook for how the gates work.
Protect the rest of the fleet
In KPI Studio, define a device KPI on the trend of memory utilization, backtest it on this device’s history, then operationalize it.
Open KPI Studio- Look for
- The KPI firing on this device before its first reboot, in the backtest.
- What it means
- Catch the next leak while the device is still at 70% memory, not at its first reboot.
Then decide
What you found decides what you do
IfTemperature ramps (45 to near 80 °C) while CPU stays flat, and the device restarts
ThenThermal. Dispatch hands to restore cooling and pre-stage a spare.
RB-DEVICE-THERMALIfMemory climbs about 2% an hour with a sawtooth reset at each reboot, starting at an upgrade
ThenMemory leak. Roll back to the last known-good image and hold the cohort.
RB-DEVICE-MEMLEAKIfMany edges lose control connections while their own CPU is normal and uptime keeps counting up
ThenController saturation. Look at the controller, not the edges.
RB-CONTROLLER-SAT
Watch out
Where this goes wrong
- Uptime tells you which story you are in. Reset to near zero means the device crashed. Still counting means the device is fine and cut off.
- A controller problem has the widest blast radius in the fleet. Do not spend an hour on one edge before you have looked at the controller.
You know which of the three it is, who owns the fix, and whether anything else in the fleet is on the same slope.
Keep going
Often next
Approve or hold a fix
Read the recommendation, the rollback plan and the blast radius, then decide.
~10 minNetwork lead, NOC engineer, Security analystGet ahead of itFind what thresholds miss
Let a seasonality-aware detector flag what is wrong for this hour of this week, then decide whether to promote it.
~15 minNOC engineer, Network leadSomething is downA circuit went down
Confirm traffic actually moved, size the damage on backup, and set the failback condition.
~10 minNOC engineer, Network lead