mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
462 lines
16 KiB
Markdown
462 lines
16 KiB
Markdown
---
|
|
name: incident-management
|
|
description: Implement incident management processes and escalation procedures. Configure on-call schedules and post-incident reviews. Use when managing production incidents.
|
|
license: MIT
|
|
metadata:
|
|
author: devops-skills
|
|
version: "1.0"
|
|
---
|
|
|
|
# Incident Management
|
|
|
|
Implement effective incident management processes including severity definitions, escalation matrices, war room procedures, and blameless post-mortem templates.
|
|
|
|
## When to Use
|
|
|
|
- Establishing incident management processes for production systems
|
|
- Defining severity levels and escalation procedures
|
|
- Running war rooms and coordinating incident response
|
|
- Conducting blameless post-incident reviews
|
|
- Building on-call schedules and notification workflows
|
|
- Meeting compliance requirements for incident response (SOC 2, HIPAA, PCI DSS)
|
|
|
|
## Severity Levels
|
|
|
|
```yaml
|
|
severity_definitions:
|
|
SEV1_critical:
|
|
impact: "Complete service outage or data breach affecting all/most customers"
|
|
examples:
|
|
- Production site completely down
|
|
- Data breach confirmed or suspected
|
|
- Complete loss of a critical business function
|
|
- Security incident with active exploitation
|
|
response_time: "Immediate (within 5 minutes)"
|
|
update_frequency: "Every 15-30 minutes"
|
|
who_is_paged: "On-call engineer, engineering manager, incident commander, executive on-call"
|
|
communication: "Status page update, customer email, executive notification"
|
|
resolution_target: "< 1 hour to mitigate"
|
|
|
|
SEV2_major:
|
|
impact: "Major feature broken or severe degradation affecting many customers"
|
|
examples:
|
|
- Key feature completely non-functional
|
|
- Significant performance degradation (>5x latency)
|
|
- Data processing pipeline completely stalled
|
|
- Partial outage affecting a region or segment
|
|
response_time: "Within 15 minutes"
|
|
update_frequency: "Every 30-60 minutes"
|
|
who_is_paged: "On-call engineer, engineering manager"
|
|
communication: "Status page update if customer-facing"
|
|
resolution_target: "< 4 hours to mitigate"
|
|
|
|
SEV3_moderate:
|
|
impact: "Minor feature impaired or degradation affecting some customers"
|
|
examples:
|
|
- Non-critical feature broken
|
|
- Moderate performance degradation
|
|
- Elevated error rate (below threshold for SEV2)
|
|
- Single-customer impact on non-critical function
|
|
response_time: "Within 1 hour during business hours"
|
|
update_frequency: "Every 2-4 hours"
|
|
who_is_paged: "On-call engineer"
|
|
communication: "Internal only unless customer inquires"
|
|
resolution_target: "< 1 business day"
|
|
|
|
SEV4_low:
|
|
impact: "Cosmetic issue, minor inconvenience, or non-customer-facing problem"
|
|
examples:
|
|
- UI cosmetic bug
|
|
- Non-critical monitoring gap
|
|
- Internal tool degradation
|
|
- Documentation inaccuracy in production
|
|
response_time: "Next business day"
|
|
update_frequency: "As needed"
|
|
who_is_paged: "None (ticket created)"
|
|
communication: "None"
|
|
resolution_target: "Within sprint planning cycle"
|
|
```
|
|
|
|
## Escalation Matrix
|
|
|
|
```yaml
|
|
escalation_matrix:
|
|
tier_1_on_call_engineer:
|
|
reached_via: "PagerDuty / OpsGenie alert"
|
|
responsibilities:
|
|
- Acknowledge alert within 5 minutes
|
|
- Assess severity and impact
|
|
- Begin troubleshooting
|
|
- Escalate to Tier 2 if unable to resolve within 30 minutes (SEV1/2)
|
|
escalation_trigger: "Cannot resolve, needs additional expertise, or severity upgrade"
|
|
|
|
tier_2_team_lead_or_sme:
|
|
reached_via: "PagerDuty escalation or direct page"
|
|
responsibilities:
|
|
- Provide subject matter expertise
|
|
- Assist with diagnosis and resolution
|
|
- Coordinate with other teams if cross-service issue
|
|
- Escalate to Tier 3 if broader coordination needed
|
|
escalation_trigger: "Multi-service issue, needs executive decision, or customer-facing SEV1"
|
|
|
|
tier_3_engineering_management:
|
|
reached_via: "PagerDuty escalation or direct call"
|
|
responsibilities:
|
|
- Assign incident commander (if not already)
|
|
- Allocate additional resources
|
|
- Make business decisions (feature disable, rollback, etc.)
|
|
- Coordinate external communication
|
|
escalation_trigger: "Business impact decision, extended outage, or PR/legal concern"
|
|
|
|
tier_4_executive:
|
|
reached_via: "Direct phone call"
|
|
responsibilities:
|
|
- Authorize extraordinary measures
|
|
- Manage board/investor communication
|
|
- Approve public statements
|
|
- Engage external resources (vendors, consultants)
|
|
escalation_trigger: "Major breach, extended SEV1, regulatory or legal implication"
|
|
|
|
time_based_escalation:
|
|
sev1:
|
|
"15 min no ack": "Re-page on-call + backup on-call"
|
|
"30 min unresolved": "Page team lead"
|
|
"1 hour unresolved": "Page engineering manager + executive on-call"
|
|
"2 hours unresolved": "All-hands engineering involvement"
|
|
sev2:
|
|
"30 min no ack": "Re-page on-call + backup on-call"
|
|
"1 hour unresolved": "Page team lead"
|
|
"4 hours unresolved": "Page engineering manager"
|
|
```
|
|
|
|
## War Room Procedures
|
|
|
|
```yaml
|
|
war_room:
|
|
activation: "Automatically for SEV1, on-demand for SEV2"
|
|
|
|
setup:
|
|
communication_channel:
|
|
primary: "Dedicated Slack channel (#incident-YYYY-MM-DD-brief-name)"
|
|
voice: "Zoom/Google Meet bridge (persistent link)"
|
|
backup: "Phone conference bridge"
|
|
channel_rules:
|
|
- "Only incident-related communication in the channel"
|
|
- "Use threads for side discussions"
|
|
- "Prefix messages with role (IC:, COMMS:, ENG:)"
|
|
|
|
roles:
|
|
incident_commander:
|
|
responsibilities:
|
|
- Own the incident from declaration to resolution
|
|
- Coordinate all response activities
|
|
- Make decisions on response actions
|
|
- Assign tasks to responders
|
|
- Determine when incident is resolved
|
|
- Schedule post-mortem
|
|
selection: "On-call IC roster, or senior engineer who declares the incident"
|
|
|
|
communications_lead:
|
|
responsibilities:
|
|
- Draft and publish status page updates
|
|
- Coordinate customer notifications
|
|
- Handle internal stakeholder updates
|
|
- Manage executive communication
|
|
- Document timeline in real-time
|
|
selection: "Designated from on-call comms roster or engineering manager"
|
|
|
|
technical_lead:
|
|
responsibilities:
|
|
- Lead technical diagnosis and troubleshooting
|
|
- Coordinate technical responders
|
|
- Recommend mitigation and resolution actions
|
|
- Verify fix effectiveness
|
|
selection: "Senior engineer with relevant system expertise"
|
|
|
|
scribe:
|
|
responsibilities:
|
|
- Document all actions, decisions, and findings
|
|
- Maintain real-time timeline
|
|
- Record who did what and when
|
|
- Capture screenshots and log excerpts
|
|
selection: "Any available team member (can be rotated)"
|
|
|
|
workflow:
|
|
1_declare:
|
|
- "IC declares incident with severity level"
|
|
- "War room channel and bridge created"
|
|
- "Roles assigned"
|
|
- "First status update posted"
|
|
|
|
2_assess:
|
|
- "Determine scope and customer impact"
|
|
- "Identify affected systems and services"
|
|
- "Establish working hypothesis"
|
|
|
|
3_mitigate:
|
|
- "Focus on restoring service first, root cause second"
|
|
- "IC approves all changes to production"
|
|
- "Changes documented in real-time"
|
|
- "Rollback if mitigation makes things worse"
|
|
|
|
4_resolve:
|
|
- "Confirm service restored to normal"
|
|
- "Verify monitoring shows healthy metrics"
|
|
- "IC declares incident resolved"
|
|
- "Final status page update"
|
|
|
|
5_follow_up:
|
|
- "Schedule post-mortem within 48 hours"
|
|
- "Assign action items from immediate findings"
|
|
- "Send internal summary"
|
|
```
|
|
|
|
## On-Call Configuration
|
|
|
|
```yaml
|
|
on_call_schedule:
|
|
rotation_structure:
|
|
primary:
|
|
rotation: "Weekly"
|
|
handoff: "Monday 10:00 AM local time"
|
|
team_size: "Minimum 5 engineers in rotation"
|
|
secondary:
|
|
rotation: "Weekly (offset from primary)"
|
|
activation: "If primary does not acknowledge within 10 minutes"
|
|
|
|
expectations:
|
|
response_time: "Acknowledge alert within 5 minutes"
|
|
availability: "Reachable by phone and laptop within 15 minutes"
|
|
handoff: "Document any ongoing issues during handoff"
|
|
compensation: "Per company on-call compensation policy"
|
|
|
|
health:
|
|
max_consecutive_weeks: 2
|
|
minimum_gap_between_rotations: "2 weeks"
|
|
post_incident_rest: "If engaged for 4+ hours overnight, late start next day"
|
|
burnout_monitoring: "Track pages per person per week, rebalance if needed"
|
|
|
|
pagerduty_configuration:
|
|
escalation_policy:
|
|
- level_1:
|
|
target: "Primary on-call"
|
|
timeout: "5 minutes"
|
|
- level_2:
|
|
target: "Secondary on-call"
|
|
timeout: "10 minutes"
|
|
- level_3:
|
|
target: "Engineering manager"
|
|
timeout: "15 minutes"
|
|
|
|
notification_rules:
|
|
high_urgency:
|
|
- "Push notification immediately"
|
|
- "Phone call after 1 minute"
|
|
- "SMS after 2 minutes"
|
|
low_urgency:
|
|
- "Push notification"
|
|
- "Email after 5 minutes"
|
|
```
|
|
|
|
## Post-Mortem Template
|
|
|
|
```markdown
|
|
# Post-Incident Review: [Incident Title]
|
|
|
|
**Date:** YYYY-MM-DD
|
|
**Severity:** SEV[1-4]
|
|
**Duration:** [Start time] to [End time] ([X hours Y minutes])
|
|
**Incident Commander:** [Name]
|
|
**Author:** [Name]
|
|
**Status:** Draft / In Review / Final
|
|
|
|
## Executive Summary
|
|
[2-3 sentence summary of what happened, the impact, and the resolution]
|
|
|
|
## Impact
|
|
- **Customer impact:** [Number/percentage of customers affected, what they experienced]
|
|
- **Duration of impact:** [How long customers were affected]
|
|
- **Revenue impact:** [Estimated financial impact, if applicable]
|
|
- **Data impact:** [Any data loss or corruption]
|
|
- **SLA impact:** [Any SLA breaches]
|
|
|
|
## Timeline (all times UTC)
|
|
| Time | Event |
|
|
|------|-------|
|
|
| HH:MM | [First anomaly detected by monitoring] |
|
|
| HH:MM | [Alert fired / customer report received] |
|
|
| HH:MM | [On-call engineer acknowledged] |
|
|
| HH:MM | [Incident declared at SEV level] |
|
|
| HH:MM | [War room established] |
|
|
| HH:MM | [Root cause identified] |
|
|
| HH:MM | [Mitigation applied] |
|
|
| HH:MM | [Service restored] |
|
|
| HH:MM | [Incident resolved] |
|
|
|
|
## Root Cause
|
|
[Detailed technical explanation of what caused the incident]
|
|
|
|
## Detection
|
|
- **How was the incident detected?** [Monitoring alert / customer report / manual observation]
|
|
- **Time to detect:** [Time from first anomaly to detection]
|
|
- **Could we have detected sooner?** [Yes/No, with explanation]
|
|
|
|
## Response
|
|
- **What went well:**
|
|
- [List things that worked effectively during response]
|
|
- [E.g., "Runbook for database failover was accurate and followed successfully"]
|
|
- [E.g., "Communication to customers was timely and clear"]
|
|
|
|
- **What could be improved:**
|
|
- [List things that slowed or hindered response]
|
|
- [E.g., "Took 20 minutes to identify the correct service owner"]
|
|
- [E.g., "Monitoring did not alert on the specific failure mode"]
|
|
|
|
## Contributing Factors
|
|
[List all factors that contributed to the incident occurring or being worse than it could have been. This is not about blame - it is about understanding the system.]
|
|
|
|
1. [Factor 1: e.g., "Configuration change was not tested in staging"]
|
|
2. [Factor 2: e.g., "Alert threshold was too high to catch gradual degradation"]
|
|
3. [Factor 3: e.g., "No circuit breaker between Service A and Service B"]
|
|
|
|
## Action Items
|
|
| ID | Action | Owner | Priority | Due Date | Status |
|
|
|----|--------|-------|----------|----------|--------|
|
|
| 1 | [Preventive action] | [Name] | P1 | YYYY-MM-DD | Open |
|
|
| 2 | [Detection improvement] | [Name] | P2 | YYYY-MM-DD | Open |
|
|
| 3 | [Process improvement] | [Name] | P2 | YYYY-MM-DD | Open |
|
|
| 4 | [Runbook update] | [Name] | P3 | YYYY-MM-DD | Open |
|
|
|
|
## Lessons Learned
|
|
[Key takeaways that should be shared broadly]
|
|
|
|
## Appendix
|
|
- [Link to monitoring dashboards during incident]
|
|
- [Link to relevant log queries]
|
|
- [Link to war room channel archive]
|
|
```
|
|
|
|
## Post-Mortem Process
|
|
|
|
```yaml
|
|
post_mortem_process:
|
|
scheduling:
|
|
sev1: "Within 48 hours of resolution"
|
|
sev2: "Within 1 week of resolution"
|
|
sev3: "Within 2 weeks (optional, based on learning potential)"
|
|
sev4: "Not required"
|
|
|
|
meeting_format:
|
|
duration: "60-90 minutes"
|
|
attendees:
|
|
required: "IC, technical lead, scribe, involved engineers"
|
|
optional: "Engineering manager, product manager, affected team leads"
|
|
agenda:
|
|
- "5 min: Review timeline and facts"
|
|
- "15 min: Walk through root cause and contributing factors"
|
|
- "15 min: Discuss what went well"
|
|
- "15 min: Discuss what could be improved"
|
|
- "15 min: Define and assign action items"
|
|
- "5 min: Identify lessons learned and sharing plan"
|
|
|
|
principles:
|
|
- "Blameless: Focus on systems and processes, not individuals"
|
|
- "Factual: Base discussion on data, logs, and observations"
|
|
- "Forward-looking: Prioritize preventive actions over assigning fault"
|
|
- "Complete: Address detection, response, and prevention"
|
|
- "Actionable: Every finding should produce a tracked action item"
|
|
|
|
action_item_tracking:
|
|
- "All action items entered into issue tracker (Jira, GitHub Issues)"
|
|
- "Priority assigned based on risk reduction potential"
|
|
- "Owner assigned with due date"
|
|
- "Reviewed in team standups and sprint planning"
|
|
- "Tracked to completion"
|
|
- "Monthly review of open post-mortem action items"
|
|
```
|
|
|
|
## Incident Metrics
|
|
|
|
```yaml
|
|
incident_metrics:
|
|
mttr:
|
|
name: "Mean Time to Resolve"
|
|
definition: "Average time from incident detection to resolution"
|
|
target: "SEV1: <1h, SEV2: <4h"
|
|
trending: "Track monthly, aim for improvement"
|
|
|
|
mttd:
|
|
name: "Mean Time to Detect"
|
|
definition: "Average time from incident start to detection"
|
|
target: "< 5 minutes for SEV1/2"
|
|
trending: "Monitors effectiveness of alerting"
|
|
|
|
mtta:
|
|
name: "Mean Time to Acknowledge"
|
|
definition: "Average time from alert to engineer acknowledgment"
|
|
target: "< 5 minutes"
|
|
trending: "Monitors on-call responsiveness"
|
|
|
|
incident_frequency:
|
|
name: "Incidents per week/month by severity"
|
|
target: "Trending downward"
|
|
trending: "Monitors system reliability improvement"
|
|
|
|
action_item_completion:
|
|
name: "Post-mortem action item completion rate"
|
|
target: "> 90% completed on time"
|
|
trending: "Monitors follow-through on improvements"
|
|
|
|
recurring_incidents:
|
|
name: "Percentage of incidents with same root cause as previous incident"
|
|
target: "< 10%"
|
|
trending: "Monitors effectiveness of preventive actions"
|
|
```
|
|
|
|
## Incident Management Checklist
|
|
|
|
```yaml
|
|
incident_management_checklist:
|
|
process_setup:
|
|
- [ ] Severity levels defined with clear criteria
|
|
- [ ] Escalation matrix documented
|
|
- [ ] On-call schedule established and staffed
|
|
- [ ] War room procedures documented
|
|
- [ ] Post-mortem template created
|
|
- [ ] Communication templates prepared (status page, email)
|
|
- [ ] Incident management tool configured (PagerDuty, OpsGenie)
|
|
|
|
per_incident:
|
|
- [ ] Incident declared with severity level
|
|
- [ ] War room established (SEV1/2)
|
|
- [ ] Roles assigned (IC, comms, technical lead, scribe)
|
|
- [ ] Timeline maintained in real-time
|
|
- [ ] Status page updated (customer-facing impact)
|
|
- [ ] Stakeholders notified per communication plan
|
|
- [ ] Resolution verified with monitoring
|
|
- [ ] Post-mortem scheduled
|
|
- [ ] Post-mortem conducted and published
|
|
- [ ] Action items tracked to completion
|
|
|
|
compliance:
|
|
- [ ] All SEV1/2 incidents have post-mortems
|
|
- [ ] Incident log maintained for audit evidence
|
|
- [ ] Metrics reported monthly
|
|
- [ ] On-call health monitored (pages per person)
|
|
- [ ] Annual incident response training conducted
|
|
- [ ] Annual incident response plan test completed
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
- Define severity levels with concrete examples so there is no ambiguity during an active incident
|
|
- Implement time-based escalation: if the on-call does not acknowledge, automatically escalate
|
|
- Focus on mitigation first, root cause second: restore service before investigating why it failed
|
|
- Run blameless post-mortems: the goal is to improve systems, not to assign fault to individuals
|
|
- Track post-mortem action items to completion: an unfinished action item means the same incident can recur
|
|
- Monitor incident metrics (MTTR, MTTD, frequency) as leading indicators of system reliability
|
|
- Protect on-call health: track page volume per person and redistribute if someone is overburdened
|
|
- Separate the incident commander role from the technical lead role in SEV1/2 incidents
|
|
- Practice incident response regularly with game days or chaos engineering exercises
|
|
- Archive incident records and post-mortems for compliance evidence and organizational learning
|