A security alert fired. Is it real, and what do you isolate?
Four very different things all arrive as a pile of security events. Their shapes are different: a burst, a flood of low-severity noise, a quiet nightly cadence, and an edge device behaving oddly. Classify by shape first and the containment follows.
Recognize it
You’re here if
- A burst of Critical or High security events on one site.
- An IPS or firewall storm after a rule update.
- A quiet site with a handful of Medium events every night.
Do this
5 steps, in order
Classify by shape, not by name
Open the alert for the site and note the count, the severity mix and whether the source is internal or external.
Open Alerts- Look for
- Hundreds in minutes, thousands of mostly low-severity, a few each night, or a small number to one unknown address.
- What it means
- The shape picks the row in the table below.
Check whether the network is also hurt
At the same site, step through Utilization, Packet loss and Security events per hour.
Open Explorer- Look for
- Utilization in the nineties and loss of several percent alongside the event burst.
- What it means
- A saturated Internet edge during a burst is a flood. Normal network metrics with a security signal is an intrusion or a rule problem.
Check the device
Open the site’s Device CPU on the same window as the event burst.
Open Explorer- Look for
- CPU in the nineties for as long as the events continue.
- What it means
- A device that is saturated while it talks to an unknown external address is a candidate for compromise, not for a capacity upgrade.
Find the few that matter
In a storm, filter the situation by severity and source. Set aside internal low-severity noise.
Open Situations- Look for
- A handful of High events from an external address among thousands of Low ones.
- What it means
- Those few are the investigation. The thousands are a tuning job.
Contain and record
Open the incident. Confirm it has a page and ticket and read the decision ledger.
Open Governed Brain- Look for
- Every decision NetPulse took, with its reason.
- What it means
- Anything that changes a device waits for your approval.
Then decide
What you found decides what you do
IfAbout 400 critical events, utilization 95 to 99%, loss 3 to 8%
ThenVolumetric DDoS. Engage upstream scrubbing, rate-limit the edge, shift spokes to the private underlay.
RB-SEC-DDOSIfThousands of IPS hits, nearly all low severity from internal addresses, a few High from outside, after a ruleset update
ThenSignature storm. Roll back or tune the signatures and investigate the external Highs.
RB-SEC-IPS-STORMIfA handful of events each night to one constant external address, tiny volume, no outage
ThenCommand-and-control beaconing. Quarantine the host, blocklist the address, rotate credentials.
RB-SEC-C2-EXFILIfDevice CPU 90 to 98% with a High-severity stream of events to an unknown external address
ThenCompromised edge. Isolate, capture forensics, reimage, rotate certificates.
RB-SEC-COMPROMISE
Watch out
Where this goes wrong
- With beaconing the risk is dwell time, not downtime. A site that is quiet in the NOC queue may be the one to look at.
- A change-freeze window mutes pages for the sites it covers, security included. Keep windows narrow.
You have classified the event, confirmed whether service is affected, and started the matching containment with a ticket.
Keep going
Often next
Too many alerts
Measure the noise, preview each suppression policy, then enforce deliberately.
~20 minNOC engineer, Network leadRun the NOCApprove or hold a fix
Read the recommendation, the rollback plan and the blast radius, then decide.
~10 minNetwork lead, NOC engineer, Security analystSomething is downA device is unhealthy
Three failure shapes look alike from the NOC. Read three signals together and pick the right fix.
~15 minNOC engineer, Network lead