The difference between a contained security incident and a full breach disclosure often comes down to the first 30 minutes of response. Teams without a rehearsed runbook waste that critical window figuring out what to do instead of doing it. Here's the sequence we use for AWS security incidents, and why the order matters as much as the actions themselves.
The Order of Operations Matters
A common mistake under pressure is jumping straight to eradication — killing the compromised resource, rotating credentials, deleting suspicious infrastructure — before preserving evidence or fully understanding the scope. This can destroy the forensic trail you need to understand how the attacker got in, what else they touched, and whether they still have access through a path you haven't found yet.
Phase 1: Detection and Initial Triage
Confirm the incident is real before escalating broadly. False positives from security tooling are common; a brief validation step (checking whether the flagged activity has a legitimate explanation) prevents unnecessary panic and resource disruption, but don't let this validation step drag on if genuine signs of compromise are present.
Establish an incident commander immediately. One person owns the decision-making during the incident, coordinating rather than everyone acting independently. This matters more than it sounds — uncoordinated parallel actions during a live incident (two people independently rotating the same credentials, for example) create confusion and can actually slow containment.
Open a dedicated communication channel (a specific Slack channel or bridge call) separate from normal team communication, so incident-related discussion doesn't get lost in unrelated chatter.
Phase 2: Containment (Without Destroying Evidence)
Isolate, don't immediately terminate. For a compromised EC2 instance, modify its security group to remove all inbound/outbound access except from your forensics tooling, rather than terminating it immediately — termination destroys volatile memory and can complicate forensic analysis. Take an EBS snapshot before any further action.
Revoke and rotate credentials that show signs of compromise, but do this deliberately — if you suspect an IAM user or role is compromised, disable it (attach a deny-all policy or delete active sessions) rather than deleting it outright, preserving the ability to review its recent activity history.
Contain the blast radius using AWS Organizations SCPs if the incident might span multiple accounts — a Service Control Policy can restrict actions across an entire OU immediately if you suspect lateral movement between accounts.
Preserve CloudTrail logs immediately. Confirm CloudTrail logging hasn't been disabled or tampered with (a common attacker move to cover tracks), and export relevant log ranges to a separate, access-restricted location before any remediation actions might overwrite or affect log retention.
Phase 3: Investigation and Scope Determination
Reconstruct the timeline using CloudTrail, VPC Flow Logs, and GuardDuty findings. Determine the initial access vector (leaked credential, misconfigured resource, compromised dependency), what actions the attacker took, and — critically — whether they accessed anything requiring separate breach disclosure obligations (customer data, PII, regulated data).
Check for persistence mechanisms. Attackers who gain IAM access often create new access keys, IAM users, or roles as backup access in case their initial foothold is discovered and removed. Audit for unexpected IAM changes across the entire account, not just the resource where the incident was initially detected.
Determine whether the incident is contained to a single account or has spread, particularly relevant in AWS Organizations setups where cross-account roles could allow lateral movement you haven't yet detected in the account where the incident was first noticed.
Phase 4: Eradication
Only after scope is reasonably well understood: remove attacker-created resources (rogue IAM users, unauthorized access keys, unexpected EC2 instances or Lambda functions), patch or remediate the actual vulnerability that enabled initial access, and rotate every credential that was potentially exposed — not just the ones you're certain were used.
Phase 5: Recovery and Post-Incident Review
Restore services from known-clean state, verifying integrity rather than assuming a quick restart is sufficient if the underlying vulnerability isn't yet fully remediated.
Conduct a blameless post-incident review documenting the timeline, root cause, what worked well in the response, and what should change — in the runbook, in monitoring coverage, or in the underlying infrastructure that allowed the initial access.
Update detection rules and monitoring based on what this specific incident revealed about gaps in your existing coverage — every real incident is also a test of your detection capability, and gaps found during a real incident should directly inform monitoring improvements.
Building This Runbook Before You Need It
The single highest-value activity most teams skip: running a tabletop exercise simulating a realistic AWS security incident before a real one happens. This surfaces gaps in your actual runbook, your team's familiarity with the tools involved (do people actually know how to isolate a security group or export CloudTrail logs under pressure?), and your communication process — all of which are far better discovered in a low-stakes drill than during a real incident.
Rutagon builds incident response runbooks and security automation for AWS environments, and helps teams run realistic tabletop exercises to validate them. Contact us to discuss a security incident response engagement.
Frequently Asked Questions
What's the first thing I should do when I suspect an AWS security incident?
Confirm the incident is real, establish a single incident commander, and open a dedicated communication channel — before taking any containment or remediation action. Rushing straight to eradication without this initial structure often creates confusion and can destroy forensic evidence.
Should I terminate a compromised EC2 instance immediately?
Generally no. Isolate it first (restrict security group access) and take an EBS snapshot before further action. Immediate termination destroys volatile memory and can complicate forensic investigation into how the compromise occurred.
How do I know if a security incident requires breach disclosure?
This depends on whether the attacker accessed customer data, PII, or other regulated data during the incident — determined during the investigation and scope phase. Work with legal counsel to confirm specific disclosure obligations based on what data was actually accessed, not just what was theoretically accessible.
What should I check for persistence mechanisms after containing an incident?
Audit for unexpected IAM changes across the entire account — new access keys, new IAM users or roles, and modified permission policies — since attackers often create backup access in case their initial foothold is discovered and removed.
How often should we run incident response tabletop exercises?
At least annually for most teams, with more frequent exercises for organizations with higher regulatory requirements or larger, more complex AWS environments. Tabletop exercises surface runbook gaps and team readiness issues far more safely than discovering them during a real incident.