Use casesSomething is down

An edge device is unhealthy. Is it heat, a memory leak, or the controller?

A rebooting edge looks the same on a dashboard whatever the cause, and the fixes are completely different: a technician, a rollback, or a controller restart. Read temperature, memory and CPU on one window and the cause usually announces itself.

  • Timeabout 15 minutes
  • ForNOC engineer, Network lead
  • ScreensExplorer, Governed Brain, Dashboards, Remediation, KPI Studio

Recognize it

You’re here if

  • An edge reboots repeatedly, or its tunnels drop and re-establish.
  • Control connections fall to zero for a reading or two.
  • A site is intermittently out, with no carrier issue behind it.

Do this

5 steps, in order

  1. Read three signals on one window

    Pick the site and step through Temperature, then Device CPU, on the same window. Then open the device in Dashboards for Memory.

    Open Explorer
    Look for
    Which of the three moves, and whether CPU moves with it.
    What it means
    Heat with flat CPU is hardware. Memory climbing steadily is software. Neither, on many devices at once, points at the controller.
  2. Check what changed

    Open the incident for the site and read its “what changed?” attribution.

    Open Governed Brain
    Look for
    A software upgrade or configuration change shortly before onset.
    What it means
    A leak that starts at an upgrade is a rollback decision, not a debugging session.
  3. Check the blast radius

    Drill from the fleet to the site, then to the device. Compare against other devices of the same model.

    Open Dashboards
    Look for
    One device, a whole site, or a whole cohort.
    What it means
    A cohort means the cause is in the image or the controller. Hold the image before the next site upgrades.
  4. Act on the cause

    Open the recommended playbook for the incident. Each action shows its rollback plan and whether it needs your approval.

    Open Remediation
    Look for
    A gated recommendation, not an automatic change.
    What it means
    Approve, or hold with a reason. See the approval playbook for how the gates work.
  5. Protect the rest of the fleet

    In KPI Studio, define a device KPI on the trend of memory utilization, backtest it on this device’s history, then operationalize it.

    Open KPI Studio
    Look for
    The KPI firing on this device before its first reboot, in the backtest.
    What it means
    Catch the next leak while the device is still at 70% memory, not at its first reboot.

Then decide

What you found decides what you do

  • IfTemperature ramps (45 to near 80 °C) while CPU stays flat, and the device restarts

    ThenThermal. Dispatch hands to restore cooling and pre-stage a spare. RB-DEVICE-THERMAL

  • IfMemory climbs about 2% an hour with a sawtooth reset at each reboot, starting at an upgrade

    ThenMemory leak. Roll back to the last known-good image and hold the cohort. RB-DEVICE-MEMLEAK

  • IfMany edges lose control connections while their own CPU is normal and uptime keeps counting up

    ThenController saturation. Look at the controller, not the edges. RB-CONTROLLER-SAT

Watch out

Where this goes wrong

  • Uptime tells you which story you are in. Reset to near zero means the device crashed. Still counting means the device is fine and cut off.
  • A controller problem has the widest blast radius in the fleet. Do not spend an hour on one edge before you have looked at the controller.
You’re done when

You know which of the three it is, who owns the fix, and whether anything else in the fleet is on the same slope.