External/DevOps-Security-Agent-Skills
1
0
Fork 0
mirror of https://github.com/BagelHole/DevOps-Security-Agent-Skills.git synced 2026-08-22 12:49:53 +02:00
Code Issues Packages Projects Releases Wiki Activity

V2

Browse Source
This commit is contained in:
Toby
2026-03-24 18:02:50 -04:00
parent 2d209b9258
commit ba9e489584
111 changed files with 48382 additions and 3099 deletions
Download Patch File Download Diff File Expand all files Collapse all files
compliance/continuity/business-continuity/SKILL.md
+411 -57
View File
@@ -9,78 +9,432 @@ metadata:
# Business Continuity Planning
Develop and maintain business continuity capabilities.
Develop and maintain business continuity capabilities including Business Impact Analysis, communication plans, recovery procedures, and testing schedules for organizational resilience.
## When to Use
- Developing a formal Business Continuity Plan (BCP) for the organization
- Conducting a Business Impact Analysis (BIA) to prioritize recovery efforts
- Establishing communication plans for crisis scenarios
- Defining recovery procedures for critical business processes
- Scheduling and conducting BCP exercises and tests
- Meeting compliance requirements for continuity planning (SOC 2, ISO 27001, HIPAA, FedRAMP)
## BCP Framework
```yaml
bcp_phases:
1_analysis:
- Business Impact Analysis (BIA)
- Risk assessment
- Critical process identification
2_planning:
- Recovery strategies
- Resource requirements
- Communication plans
3_implementation:
- Procedure documentation
- Training
- Technology setup
4_testing:
- Plan exercises
- Gap identification
- Continuous improvement
1_governance:
actions:
- Obtain executive sponsorship and funding
- Assign BCP coordinator and team
- Define BCP scope and policy
- Establish BCP committee with cross-functional representation
deliverables:
- BCP policy statement
- BCP team charter and roster
- Scope document
2_analysis:
actions:
- Conduct Business Impact Analysis (BIA)
- Perform risk assessment for continuity threats
- Identify critical business processes and dependencies
- Determine recovery priorities and resource requirements
deliverables:
- BIA report
- Risk assessment report
- Critical process inventory
3_strategy:
actions:
- Select recovery strategies for each critical process
- Identify alternate work arrangements (remote, alternate site)
- Define technology recovery strategies (DR plan)
- Establish vendor and supply chain contingencies
deliverables:
- Recovery strategy document
- Technology recovery plan
- Alternate site arrangements
4_plan_development:
actions:
- Write detailed recovery procedures
- Develop communication plans (internal and external)
- Create emergency response procedures
- Document roles, responsibilities, and contact information
deliverables:
- Business Continuity Plan document
- Communication plan
- Emergency response procedures
- Contact lists and call trees
5_testing:
actions:
- Develop test plan and schedule
- Conduct exercises (tabletop, functional, full-scale)
- Evaluate results and identify gaps
- Update plans based on lessons learned
deliverables:
- Test plan
- Exercise reports
- Updated BCP based on findings
6_maintenance:
actions:
- Review and update BCP annually (minimum)
- Update after significant organizational changes
- Refresh BIA when business processes change
- Maintain training and awareness program
deliverables:
- Annual BCP review record
- Updated BIA (if changes occurred)
- Training completion records
```
## Business Impact Analysis
## Business Impact Analysis Template
```yaml
process_classification:
critical:
max_downtime: 4 hours
examples: Payment processing, authentication
essential:
max_downtime: 24 hours
examples: Customer support, reporting
necessary:
max_downtime: 72 hours
examples: Internal tools, analytics
desirable:
max_downtime: 7 days
examples: Development environments
bia_template:
process_assessment:
process_name: ""
process_owner: ""
department: ""
description: ""
criticality_classification:
mission_critical:
max_tolerable_downtime: "0-4 hours"
description: "Failure causes immediate, severe impact to customers or revenue"
examples:
- Payment processing
- Authentication and authorization
- Core API serving customer requests
- Order fulfillment
essential:
max_tolerable_downtime: "4-24 hours"
description: "Failure causes significant degradation but not complete loss"
examples:
- Customer support systems
- Reporting and dashboards
- Email and notifications
- Billing and invoicing
important:
max_tolerable_downtime: "1-3 days"
description: "Failure causes inconvenience and workarounds are available"
examples:
- Internal collaboration tools
- Analytics and BI platforms
- HR self-service systems
- Knowledge base
non_essential:
max_tolerable_downtime: "3-7 days"
description: "Failure has minimal operational impact"
examples:
- Development and test environments
- Training platforms
- Archive systems
impact_categories:
financial:
revenue_loss_per_hour: ""
penalty_or_fine_risk: ""
recovery_cost_estimate: ""
operational:
affected_employees: ""
affected_customers: ""
workaround_available: "yes/no"
workaround_description: ""
reputational:
customer_visibility: "high/medium/low"
media_attention_risk: "high/medium/low"
regulatory_reporting_required: "yes/no"
legal_regulatory:
compliance_impact: ""
contractual_sla_breach: "yes/no"
sla_penalty_details: ""
dependencies:
technology:
- system: ""
rto: ""
rpo: ""
dr_strategy: ""
people:
- role: ""
minimum_staff: ""
remote_capable: "yes/no"
vendors:
- vendor: ""
service: ""
sla: ""
alternative: ""
facilities:
- location: ""
alternative: ""
recovery_requirements:
rto: ""
rpo: ""
minimum_recovery_level: "Description of minimum acceptable service"
full_recovery_target: "Time to full normal operations"
```
## Communication Plan
```yaml
communication:
internal:
- Executive notification
- Team communication
- Status updates
external:
- Customer notification
- Regulatory reporting
- Media relations
channels:
- Primary: Slack/Teams
- Secondary: Email
- Emergency: Phone tree
communication_plan:
activation_criteria:
- Event affecting multiple critical systems
- Physical facility unavailable
- Pandemic or workforce availability crisis
- Major vendor/partner outage
- Cybersecurity incident with operational impact
internal_communication:
executive_notification:
who: "CEO, CTO, CFO, VP Engineering, VP Operations"
when: "Within 15 minutes of BCP activation"
method: "Phone call (primary), SMS (secondary)"
message_template: |
BUSINESS CONTINUITY EVENT ACTIVATED
Incident: [Brief description]
Impact: [Systems/processes affected]
Status: [Current state]
Next update: [Time]
Bridge call: [Number/link]
team_notification:
who: "All affected department leads and their teams"
when: "Within 30 minutes of BCP activation"
method: "Slack/Teams (primary), Email (secondary), SMS (tertiary)"
message_template: |
BCP ACTIVATED - [Event Type]
What happened: [Description]
What is affected: [Systems/services]
What to do: [Immediate actions for your team]
Status updates: [Channel/frequency]
Questions: Contact [BCP coordinator]
all_staff_notification:
who: "All employees"
when: "Within 1 hour of BCP activation"
method: "Email, Slack/Teams announcement, intranet"
content: "Situation summary, impact on work, expectations"
status_updates:
frequency: "Every 2 hours during active event, daily after stabilization"
channel: "Dedicated Slack channel, email distribution list"
content: "Current status, actions taken, next steps, timeline"
external_communication:
customers:
who: "Affected customers"
when: "Within 2 hours of BCP activation (if customer-facing impact)"
method: "Status page update, email, in-app notification"
message_template: |
We are currently experiencing [issue description].
Impact: [What customers may notice]
Status: We are actively working to resolve this.
Updates: Follow our status page at status.example.com
ETA: [Estimated resolution time or "investigating"]
regulatory:
who: "Applicable regulatory bodies"
when: "Per regulatory requirements (e.g., 72 hours for GDPR breach)"
method: "Formal notification per regulatory procedure"
media:
who: "Press inquiries"
when: "Only if media attention occurs"
method: "Prepared statement through communications team"
rule: "All media inquiries routed to designated spokesperson"
vendors_partners:
who: "Critical vendors and business partners"
when: "Within 4 hours if partner services affected"
method: "Direct contact via relationship manager"
contact_lists:
maintenance: "Updated quarterly"
storage: "Accessible offline (printed, mobile app, cloud-independent)"
includes:
- BCP team members (name, role, phone, email, alternate phone)
- Executive team
- Department leads
- Key vendor contacts
- Regulatory contacts
- Legal counsel
- Insurance broker
- PR/communications firm
```
## Recovery Procedures
```yaml
recovery_procedures:
immediate_response:
step_1: "Incident commander assesses situation and declares BCP activation"
step_2: "Notify BCP team and establish command structure"
step_3: "Activate communication plan"
step_4: "Assess damage and determine scope of disruption"
step_5: "Initiate appropriate recovery procedures based on scenario"
scenario_specific:
data_center_or_region_outage:
- Activate DR failover procedures
- Redirect traffic to DR region
- Verify service restoration
- Communicate status to stakeholders
- Plan return to primary when available
cybersecurity_incident:
- Engage incident response team
- Contain the threat (isolate affected systems)
- Assess data impact and potential breach
- Activate forensic investigation
- Restore from known-good backups if needed
- Notify legal and regulatory as required
pandemic_workforce_disruption:
- Activate remote work procedures
- Verify VPN and remote access capacity
- Redistribute critical functions if staff unavailable
- Implement shift rotations to maintain coverage
- Assess vendor ability to maintain service levels
key_vendor_failure:
- Assess impact on dependent business processes
- Activate vendor contingency plan
- Engage alternate vendor if available
- Implement manual workarounds as needed
- Communicate impact to affected stakeholders
facility_unavailable:
- Account for all personnel safety
- Activate alternate work site arrangements
- Redirect mail and deliveries
- Set up temporary communication channels
- Assess timeline for facility restoration
stabilization:
- Monitor recovered services continuously
- Address any residual issues
- Begin planning return to normal operations
- Continue stakeholder communication
- Document all actions and decisions
return_to_normal:
- Develop return-to-normal plan
- Execute failback procedures (if DR was activated)
- Verify data consistency and integrity
- Restore standard operating procedures
- Conduct post-event review
- Update BCP based on lessons learned
```
## Testing Schedule and Types
```yaml
testing_schedule:
tabletop_exercise:
frequency: "Quarterly"
duration: "2-3 hours"
participants: "BCP team, department leads, executive sponsor"
format: "Facilitated discussion of a scenario"
scenarios_to_rotate:
- Major cloud provider region outage
- Ransomware attack on production systems
- Key employee unavailability (bus factor scenario)
- Critical vendor goes out of business
- Office building inaccessible
output: "Exercise report with findings and action items"
functional_exercise:
frequency: "Semi-annually"
duration: "4-8 hours"
participants: "BCP team, IT operations, affected departments"
format: "Execute specific recovery procedures without full disruption"
examples:
- "Activate remote work for one department for a day"
- "Failover a non-production database and verify application connectivity"
- "Execute communication plan and verify contact list accuracy"
- "Restore a critical system from backup in an isolated environment"
output: "Functional test report with measured recovery times"
full_scale_exercise:
frequency: "Annually"
duration: "1-2 days"
participants: "All BCP team members, IT, communications, management"
format: "Simulate a major disruption and execute full recovery"
includes:
- "Activate BCP command structure"
- "Execute DR failover for production systems"
- "Activate communication plan"
- "Operate from alternate arrangements for set period"
- "Execute failback and return to normal"
output: "Full exercise report with comprehensive metrics and lessons learned"
testing_metrics:
- "Time to activate BCP command structure"
- "Time to complete communication notifications"
- "Contact list accuracy (% reachable)"
- "Actual RTO vs. target RTO per system"
- "Actual RPO vs. target RPO per system"
- "Number of issues identified"
- "Number of runbook corrections needed"
```
## BCP Maintenance Checklist
```yaml
bcp_maintenance_checklist:
quarterly:
- [ ] Contact lists verified and updated
- [ ] Tabletop exercise conducted
- [ ] BCP team roster reviewed
- [ ] Vendor contact information verified
- [ ] Communication channels tested
semi_annually:
- [ ] Functional exercise conducted
- [ ] Recovery procedures reviewed for accuracy
- [ ] Technology dependencies verified
- [ ] Vendor continuity capabilities confirmed
annually:
- [ ] Full-scale exercise conducted
- [ ] Business Impact Analysis refreshed
- [ ] Risk assessment updated
- [ ] BCP document fully reviewed and updated
- [ ] Executive review and sign-off obtained
- [ ] Training completed for all BCP team members
- [ ] Lessons learned from all exercises incorporated
triggered_by_change:
- [ ] New critical business process added
- [ ] Major organizational restructuring
- [ ] Technology platform migration
- [ ] New regulatory requirement
- [ ] Significant vendor change
- [ ] Actual disruption event (post-event update)
```
## Best Practices
- Annual BIA updates
- Regular plan testing
- Clear roles and responsibilities
- Multiple communication channels
- Executive sponsorship
- Secure executive sponsorship: BCP without leadership commitment will not be taken seriously
- Base recovery priorities on Business Impact Analysis, not assumptions or technical preferences
- Test the communication plan independently: it fails more often than the technology recovery
- Maintain contact lists as if your primary systems are unavailable (offline copies, mobile access)
- Conduct tabletop exercises quarterly at minimum: they are low-cost and high-value for identifying gaps
- Include non-IT scenarios in planning (pandemic, facility loss, key personnel unavailability)
- Define clear activation criteria so there is no ambiguity about when to invoke the BCP
- Keep the BCP document practical and actionable, not a shelf document written for auditors
- Update the BCP after every significant organizational or technology change
- Review and incorporate lessons from every exercise and every real event into the plan
compliance/continuity/disaster-recovery/SKILL.md
+532 -50
View File
@@ -9,65 +9,547 @@ metadata:
# Disaster Recovery
Implement disaster recovery strategies and procedures.
Implement disaster recovery strategies including RTO/RPO planning, AWS cross-region failover patterns, DR testing procedures, and automated failover scripts.
## DR Metrics
## When to Use
- Defining RTO and RPO targets for critical systems
- Designing multi-region or multi-cloud disaster recovery architectures
- Implementing automated failover and failback procedures
- Conducting DR tests (tabletop, component, full failover)
- Meeting compliance requirements for contingency planning (SOC 2, HIPAA, FedRAMP, ISO 27001)
## RTO/RPO Planning
```yaml
recovery_metrics:
RTO: Recovery Time Objective
- Maximum acceptable downtime
- How long to restore service
RPO: Recovery Point Objective
- Maximum acceptable data loss
- How much data can be lost
RTO:
definition: "Recovery Time Objective - maximum acceptable downtime"
measurement: "From incident declaration to service restoration"
factors:
- Failover automation maturity
- Data replication lag
- DNS propagation time
- Application warm-up time
- Verification procedures
RPO:
definition: "Recovery Point Objective - maximum acceptable data loss"
measurement: "Time gap between last good backup and the incident"
factors:
- Backup frequency
- Replication method (sync vs. async)
- Transaction log shipping interval
- Cross-region replication lag
service_tier_targets:
tier_1_critical:
examples: "Authentication, payment processing, core API"
rto: "< 15 minutes"
rpo: "< 1 minute (near-zero)"
strategy: "Multi-site active-active or warm standby"
replication: "Synchronous or near-synchronous"
testing: "Quarterly failover test"
tier_2_essential:
examples: "Customer dashboards, reporting, notifications"
rto: "< 1 hour"
rpo: "< 15 minutes"
strategy: "Warm standby or pilot light"
replication: "Asynchronous with short interval"
testing: "Semi-annual failover test"
tier_3_standard:
examples: "Internal tools, analytics, batch processing"
rto: "< 4 hours"
rpo: "< 1 hour"
strategy: "Pilot light or backup and restore"
replication: "Periodic snapshots"
testing: "Annual failover test"
tier_4_non_essential:
examples: "Development environments, documentation sites"
rto: "< 24 hours"
rpo: "< 24 hours"
strategy: "Backup and restore"
replication: "Daily backups"
testing: "Annual backup restore verification"
```
## DR Strategies
| Strategy | RTO | RPO | Cost |
|----------|-----|-----|------|
| Backup & Restore | Hours | Hours | $ |
| Pilot Light | Minutes-Hours | Minutes | $$ |
| Warm Standby | Minutes | Seconds | $$$ |
| Multi-Site Active | Near-zero | Near-zero | $$$$ |
## AWS Multi-Region
```bash
# Cross-region RDS replica
aws rds create-db-instance-read-replica \
--db-instance-identifier dr-replica \
--source-db-instance-identifier prod-db \
--source-region us-east-1 \
--region us-west-2
# S3 cross-region replication
aws s3api put-bucket-replication \
--bucket source-bucket \
--replication-configuration file://replication.json
```
## DR Testing
## DR Strategies Comparison
```yaml
dr_test_schedule:
tabletop: Quarterly
component_failover: Monthly
full_failover: Annually
test_checklist:
- [ ] Verify backup integrity
- [ ] Test failover procedures
- [ ] Validate data consistency
- [ ] Measure actual RTO/RPO
- [ ] Document lessons learned
strategies:
backup_and_restore:
rto: "Hours"
rpo: "Hours (depends on backup frequency)"
cost: "$"
description: "Regular backups stored in DR region. Restore from backup when needed."
aws_services:
- "S3 cross-region replication for backups"
- "RDS automated snapshots copied to DR region"
- "AMI copies in DR region"
- "Terraform/CloudFormation for infrastructure rebuild"
pros: "Lowest cost, simplest to maintain"
cons: "Longest recovery time, highest data loss potential"
pilot_light:
rto: "Minutes to hours"
rpo: "Minutes"
cost: "$$"
description: "Core infrastructure running in DR region (databases replicated). Scale up compute on failover."
aws_services:
- "RDS cross-region read replica (always running)"
- "S3 cross-region replication"
- "AMIs pre-built in DR region"
- "Auto Scaling groups at zero/minimal capacity"
pros: "Fast database recovery, moderate cost"
cons: "Compute scale-up adds to recovery time"
warm_standby:
rto: "Minutes"
rpo: "Seconds to minutes"
cost: "$$$"
description: "Scaled-down but functional environment in DR region. Scale up on failover."
aws_services:
- "RDS cross-region read replica"
- "ECS/EKS running at reduced capacity"
- "Route53 health checks for automated DNS failover"
- "Global Accelerator for traffic management"
pros: "Fast failover, reduced risk"
cons: "Higher baseline cost for idle resources"
multi_site_active:
rto: "Near-zero"
rpo: "Near-zero"
cost: "$$$$"
description: "Active-active across regions. Traffic served from both regions simultaneously."
aws_services:
- "DynamoDB Global Tables or Aurora Global Database"
- "Route53 latency/weighted routing"
- "CloudFront with multi-origin"
- "Global Accelerator"
- "ECS/EKS in both regions"
pros: "Minimal downtime and data loss"
cons: "Highest cost, most complex to operate"
```
## AWS Cross-Region DR Implementation
```bash
# === Database Replication ===
# Create cross-region RDS read replica
aws rds create-db-instance-read-replica \
--db-instance-identifier prod-db-dr-replica \
--source-db-instance-identifier arn:aws:rds:us-east-1:123456789012:db:prod-db \
--db-instance-class db.r6g.large \
--region us-west-2 \
--kms-key-id arn:aws:kms:us-west-2:123456789012:alias/rds-dr-key \
--multi-az \
--tags Key=Purpose,Value=DR Key=Environment,Value=production
# Create Aurora Global Database for near-zero RPO
aws rds create-global-cluster \
--global-cluster-identifier prod-global-db \
--source-db-cluster-identifier arn:aws:rds:us-east-1:123456789012:cluster:prod-aurora-cluster \
--region us-east-1
# Add secondary region to Aurora Global Database
aws rds create-db-cluster \
--db-cluster-identifier prod-aurora-dr \
--global-cluster-identifier prod-global-db \
--engine aurora-postgresql \
--region us-west-2 \
--kms-key-id arn:aws:kms:us-west-2:123456789012:alias/aurora-dr-key
# === Storage Replication ===
# S3 cross-region replication
cat > /tmp/replication-config.json << 'EOF'
{
"Role": "arn:aws:iam::123456789012:role/s3-replication-role",
"Rules": [
{
"ID": "ReplicateAll",
"Status": "Enabled",
"Filter": {"Prefix": ""},
"Destination": {
"Bucket": "arn:aws:s3:::prod-data-dr-usw2",
"StorageClass": "STANDARD",
"EncryptionConfiguration": {
"ReplicaKmsKeyID": "arn:aws:kms:us-west-2:123456789012:alias/s3-dr-key"
}
},
"DeleteMarkerReplication": {"Status": "Enabled"}
}
]
}
EOF
aws s3api put-bucket-replication \
--bucket prod-data-use1 \
--replication-configuration file:///tmp/replication-config.json
# === DNS Failover ===
# Route53 health check for primary region
aws route53 create-health-check --caller-reference "prod-health-$(date +%s)" \
--health-check-config '{
"Type": "HTTPS",
"FullyQualifiedDomainName": "api.example.com",
"Port": 443,
"ResourcePath": "/health",
"RequestInterval": 10,
"FailureThreshold": 3,
"EnableSNI": true
}'
# Configure failover routing
aws route53 change-resource-record-sets --hosted-zone-id Z123456 \
--change-batch '{
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "api.example.com",
"Type": "A",
"SetIdentifier": "primary",
"Failover": "PRIMARY",
"AliasTarget": {
"HostedZoneId": "Z1234PRIMARY",
"DNSName": "primary-alb.us-east-1.elb.amazonaws.com",
"EvaluateTargetHealth": true
},
"HealthCheckId": "health-check-id-primary"
}
},
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "api.example.com",
"Type": "A",
"SetIdentifier": "secondary",
"Failover": "SECONDARY",
"AliasTarget": {
"HostedZoneId": "Z5678SECONDARY",
"DNSName": "dr-alb.us-west-2.elb.amazonaws.com",
"EvaluateTargetHealth": true
}
}
}
]
}'
```
## Failover Script
```bash
#!/usr/bin/env bash
# dr-failover.sh - Execute disaster recovery failover to DR region
set -euo pipefail
DR_REGION="us-west-2"
PRIMARY_REGION="us-east-1"
SLACK_WEBHOOK="${DR_SLACK_WEBHOOK}"
LOG_FILE="/var/log/dr-failover-$(date +%Y%m%d-%H%M%S).log"
log() {
echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $1" | tee -a "$LOG_FILE"
}
notify() {
curl -s -X POST "$SLACK_WEBHOOK" \
-H "Content-Type: application/json" \
-d "{\"text\":\"DR FAILOVER: $1\"}" > /dev/null
}
log "=== DR Failover Initiated ==="
notify "DR failover initiated to $DR_REGION"
# Step 1: Promote RDS read replica
log "Step 1: Promoting RDS read replica in $DR_REGION"
aws rds promote-read-replica \
--db-instance-identifier prod-db-dr-replica \
--region "$DR_REGION"
log "Waiting for RDS promotion to complete..."
aws rds wait db-instance-available \
--db-instance-identifier prod-db-dr-replica \
--region "$DR_REGION"
log "RDS promotion complete"
notify "RDS read replica promoted to primary in $DR_REGION"
# Step 2: Scale up application in DR region
log "Step 2: Scaling up application in $DR_REGION"
aws ecs update-service \
--cluster prod-cluster-dr \
--service api-service \
--desired-count 4 \
--region "$DR_REGION"
log "Waiting for ECS service to stabilize..."
aws ecs wait services-stable \
--cluster prod-cluster-dr \
--services api-service \
--region "$DR_REGION"
log "ECS service scaled up and stable"
notify "Application scaled up in $DR_REGION"
# Step 3: Verify health
log "Step 3: Verifying health in $DR_REGION"
for i in $(seq 1 10); do
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://dr-alb.us-west-2.elb.amazonaws.com/health")
if [ "$STATUS" = "200" ]; then
log "Health check passed (attempt $i)"
break
fi
log "Health check failed (attempt $i, status $STATUS), retrying..."
sleep 10
done
if [ "$STATUS" != "200" ]; then
log "ERROR: Health check failed after 10 attempts"
notify "ALERT: DR health check failing - manual intervention required"
exit 1
fi
# Step 4: Update DNS (if not using automatic Route53 failover)
log "Step 4: DNS failover (Route53 automatic failover should handle this)"
log "Verifying DNS resolution..."
DR_IP=$(dig +short api.example.com)
log "api.example.com resolves to: $DR_IP"
# Step 5: Verify end-to-end
log "Step 5: End-to-end verification"
RESPONSE=$(curl -s "https://api.example.com/health")
log "Health response: $RESPONSE"
log "=== DR Failover Complete ==="
notify "DR failover to $DR_REGION complete. Service restored."
# Generate failover report
cat > "/var/log/dr-failover-report-$(date +%Y%m%d).md" << EOF
# DR Failover Report
- **Date:** $(date -u +%Y-%m-%dT%H:%M:%SZ)
- **Primary Region:** $PRIMARY_REGION
- **DR Region:** $DR_REGION
- **RTO Actual:** Calculate from incident declaration
- **RPO Actual:** Check replication lag at time of incident
- **Status:** Operational in DR region
- **Actions Required:**
- [ ] Monitor error rates and latency
- [ ] Plan failback when primary region is restored
- [ ] Conduct post-incident review
EOF
```
## DR Testing Procedures
```yaml
dr_test_types:
tabletop_exercise:
frequency: Quarterly
duration: "1-2 hours"
participants: "Engineering, SRE, management, communications"
process:
- Present a disaster scenario (region outage, data corruption, etc.)
- Walk through the response step by step
- Identify gaps in runbooks and communication plans
- Document action items
output: "Tabletop exercise report with findings and action items"
component_failover:
frequency: Monthly
duration: "1-4 hours"
scope: "Individual component failover (database, single service)"
process:
- Select component for testing
- Execute failover procedure from runbook
- Measure actual RTO and RPO
- Execute failback procedure
- Document results
output: "Component test report with measured RTO/RPO"
full_failover:
frequency: Annually
duration: "4-8 hours (scheduled maintenance window)"
scope: "Complete regional failover of all tier 1 and tier 2 services"
process:
1_preparation:
- Schedule maintenance window and notify stakeholders
- Verify DR environment is healthy
- Brief all participating teams
- Set up war room communication channel
2_execute:
- Simulate primary region failure
- Execute failover runbooks for all services
- Record timestamps at each milestone
3_verify:
- Run end-to-end test suite against DR environment
- Verify data consistency
- Check monitoring and alerting in DR region
- Confirm external integrations work
4_failback:
- Restore primary region
- Re-establish replication
- Execute failback to primary
- Verify data consistency post-failback
5_report:
- Document actual RTO and RPO for each service
- Compare against targets
- List all issues encountered
- Create action items for improvements
output: "Full DR test report with measured vs. target metrics"
dr_test_checklist:
before_test:
- [ ] Test plan documented and approved
- [ ] Maintenance window scheduled and communicated
- [ ] All DR runbooks reviewed and updated
- [ ] DR environment health verified
- [ ] Monitoring configured in DR region
- [ ] Communication channel established
- [ ] Rollback plan confirmed
during_test:
- [ ] Timestamps recorded for each step
- [ ] Screenshots captured for evidence
- [ ] Issues logged in real-time
- [ ] Data consistency verified
- [ ] External integrations tested
- [ ] Health checks passing in DR
after_test:
- [ ] Failback completed successfully
- [ ] Primary region replication re-established
- [ ] Data consistency verified post-failback
- [ ] Test report written with metrics
- [ ] Action items created and assigned
- [ ] Runbooks updated based on findings
- [ ] Results presented to management
```
## Terraform DR Infrastructure
```hcl
# DR region infrastructure
provider "aws" {
alias = "dr"
region = "us-west-2"
}
resource "aws_db_instance" "dr_replica" {
provider = aws.dr
identifier = "prod-db-dr-replica"
replicate_source_db = aws_db_instance.primary.arn
instance_class = "db.r6g.large"
storage_encrypted = true
kms_key_id = aws_kms_key.dr_rds.arn
multi_az = true
deletion_protection = true
skip_final_snapshot = false
tags = {
Purpose = "DR"
Environment = "production"
}
}
resource "aws_route53_health_check" "primary" {
fqdn = "primary-alb.us-east-1.elb.amazonaws.com"
port = 443
type = "HTTPS"
resource_path = "/health"
failure_threshold = 3
request_interval = 10
enable_sni = true
tags = {
Name = "primary-health-check"
}
}
resource "aws_route53_record" "failover_primary" {
zone_id = aws_route53_zone.main.zone_id
name = "api.example.com"
type = "A"
set_identifier = "primary"
failover_routing_policy {
type = "PRIMARY"
}
alias {
name = aws_lb.primary.dns_name
zone_id = aws_lb.primary.zone_id
evaluate_target_health = true
}
health_check_id = aws_route53_health_check.primary.id
}
resource "aws_route53_record" "failover_secondary" {
zone_id = aws_route53_zone.main.zone_id
name = "api.example.com"
type = "A"
set_identifier = "secondary"
failover_routing_policy {
type = "SECONDARY"
}
alias {
name = aws_lb.dr.dns_name
zone_id = aws_lb.dr.zone_id
evaluate_target_health = true
}
}
```
## DR Compliance Checklist
```yaml
dr_compliance_checklist:
planning:
- [ ] RTO and RPO targets defined per service tier
- [ ] DR strategy selected based on targets and budget
- [ ] DR architecture documented with diagrams
- [ ] Failover and failback runbooks written
- [ ] Communication plan for DR events documented
- [ ] DR roles and responsibilities assigned
implementation:
- [ ] Cross-region database replication configured
- [ ] Storage replication configured (S3, EBS snapshots)
- [ ] DNS failover routing configured
- [ ] DR region infrastructure provisioned (IaC)
- [ ] Monitoring and alerting configured in DR region
- [ ] Secrets and credentials available in DR region
testing:
- [ ] Tabletop exercises conducted quarterly
- [ ] Component failover tests conducted monthly
- [ ] Full failover test conducted annually
- [ ] Actual RTO/RPO measured and compared to targets
- [ ] Test results documented and reviewed
- [ ] Runbooks updated based on test findings
operational:
- [ ] Replication lag monitored with alerting
- [ ] DR environment health checked regularly
- [ ] Backup integrity verified monthly
- [ ] DR runbooks reviewed and updated quarterly
- [ ] DR test evidence archived for compliance audits
```
## Best Practices
- Regular DR testing
- Automate failover where possible
- Document all procedures
- Update runbooks after tests
- Define RTO and RPO targets based on business impact analysis, not technical convenience
- Choose the DR strategy that matches your targets and budget: do not over-engineer or under-invest
- Automate failover as much as possible to reduce human error and recovery time
- Test DR procedures regularly at increasing levels of complexity (tabletop, component, full)
- Measure actual RTO and RPO during tests and compare against targets every time
- Include failback procedures in your DR plan: getting back to normal is as important as failing over
- Monitor replication lag continuously and alert when it exceeds RPO thresholds
- Keep DR infrastructure managed by the same IaC as production to prevent configuration drift
- Practice DR in non-emergency conditions so the team is prepared when a real disaster occurs
- Archive DR test results as compliance evidence for SOC 2, HIPAA, and other frameworks
compliance/continuity/incident-management/SKILL.md
+426 -55
View File
@@ -9,82 +9,453 @@ metadata:
# Incident Management
Implement effective incident management processes.
Implement effective incident management processes including severity definitions, escalation matrices, war room procedures, and blameless post-mortem templates.
## Incident Severity
## When to Use
| Severity | Impact | Response | Example |
|----------|--------|----------|---------|
| SEV1 | Total outage | Immediate, all-hands | Site down |
| SEV2 | Major degradation | Urgent, on-call | Feature broken |
| SEV3 | Minor impact | Standard | Slow performance |
| SEV4 | Minimal | Next business day | Cosmetic issue |
- Establishing incident management processes for production systems
- Defining severity levels and escalation procedures
- Running war rooms and coordinating incident response
- Conducting blameless post-incident reviews
- Building on-call schedules and notification workflows
- Meeting compliance requirements for incident response (SOC 2, HIPAA, PCI DSS)
## Incident Process
## Severity Levels
```yaml
incident_workflow:
1_detect:
- Alerting triggers
- Customer reports
- Monitoring anomalies
2_triage:
- Severity assessment
- Impact determination
- Team notification
3_respond:
- Incident commander assigned
- Communication established
- Mitigation started
4_resolve:
- Root cause addressed
- Service restored
- Customer notified
5_review:
- Timeline documented
- Root cause analysis
- Action items created
severity_definitions:
SEV1_critical:
impact: "Complete service outage or data breach affecting all/most customers"
examples:
- Production site completely down
- Data breach confirmed or suspected
- Complete loss of a critical business function
- Security incident with active exploitation
response_time: "Immediate (within 5 minutes)"
update_frequency: "Every 15-30 minutes"
who_is_paged: "On-call engineer, engineering manager, incident commander, executive on-call"
communication: "Status page update, customer email, executive notification"
resolution_target: "< 1 hour to mitigate"
SEV2_major:
impact: "Major feature broken or severe degradation affecting many customers"
examples:
- Key feature completely non-functional
- Significant performance degradation (>5x latency)
- Data processing pipeline completely stalled
- Partial outage affecting a region or segment
response_time: "Within 15 minutes"
update_frequency: "Every 30-60 minutes"
who_is_paged: "On-call engineer, engineering manager"
communication: "Status page update if customer-facing"
resolution_target: "< 4 hours to mitigate"
SEV3_moderate:
impact: "Minor feature impaired or degradation affecting some customers"
examples:
- Non-critical feature broken
- Moderate performance degradation
- Elevated error rate (below threshold for SEV2)
- Single-customer impact on non-critical function
response_time: "Within 1 hour during business hours"
update_frequency: "Every 2-4 hours"
who_is_paged: "On-call engineer"
communication: "Internal only unless customer inquires"
resolution_target: "< 1 business day"
SEV4_low:
impact: "Cosmetic issue, minor inconvenience, or non-customer-facing problem"
examples:
- UI cosmetic bug
- Non-critical monitoring gap
- Internal tool degradation
- Documentation inaccuracy in production
response_time: "Next business day"
update_frequency: "As needed"
who_is_paged: "None (ticket created)"
communication: "None"
resolution_target: "Within sprint planning cycle"
```
## Incident Commander
## Escalation Matrix
```yaml
ic_responsibilities:
- Own incident resolution
- Coordinate response teams
- Manage communication
- Make escalation decisions
- Schedule post-mortem
escalation_matrix:
tier_1_on_call_engineer:
reached_via: "PagerDuty / OpsGenie alert"
responsibilities:
- Acknowledge alert within 5 minutes
- Assess severity and impact
- Begin troubleshooting
- Escalate to Tier 2 if unable to resolve within 30 minutes (SEV1/2)
escalation_trigger: "Cannot resolve, needs additional expertise, or severity upgrade"
tier_2_team_lead_or_sme:
reached_via: "PagerDuty escalation or direct page"
responsibilities:
- Provide subject matter expertise
- Assist with diagnosis and resolution
- Coordinate with other teams if cross-service issue
- Escalate to Tier 3 if broader coordination needed
escalation_trigger: "Multi-service issue, needs executive decision, or customer-facing SEV1"
tier_3_engineering_management:
reached_via: "PagerDuty escalation or direct call"
responsibilities:
- Assign incident commander (if not already)
- Allocate additional resources
- Make business decisions (feature disable, rollback, etc.)
- Coordinate external communication
escalation_trigger: "Business impact decision, extended outage, or PR/legal concern"
tier_4_executive:
reached_via: "Direct phone call"
responsibilities:
- Authorize extraordinary measures
- Manage board/investor communication
- Approve public statements
- Engage external resources (vendors, consultants)
escalation_trigger: "Major breach, extended SEV1, regulatory or legal implication"
time_based_escalation:
sev1:
"15 min no ack": "Re-page on-call + backup on-call"
"30 min unresolved": "Page team lead"
"1 hour unresolved": "Page engineering manager + executive on-call"
"2 hours unresolved": "All-hands engineering involvement"
sev2:
"30 min no ack": "Re-page on-call + backup on-call"
"1 hour unresolved": "Page team lead"
"4 hours unresolved": "Page engineering manager"
```
## Post-Incident Review
## War Room Procedures
```yaml
war_room:
activation: "Automatically for SEV1, on-demand for SEV2"
setup:
communication_channel:
primary: "Dedicated Slack channel (#incident-YYYY-MM-DD-brief-name)"
voice: "Zoom/Google Meet bridge (persistent link)"
backup: "Phone conference bridge"
channel_rules:
- "Only incident-related communication in the channel"
- "Use threads for side discussions"
- "Prefix messages with role (IC:, COMMS:, ENG:)"
roles:
incident_commander:
responsibilities:
- Own the incident from declaration to resolution
- Coordinate all response activities
- Make decisions on response actions
- Assign tasks to responders
- Determine when incident is resolved
- Schedule post-mortem
selection: "On-call IC roster, or senior engineer who declares the incident"
communications_lead:
responsibilities:
- Draft and publish status page updates
- Coordinate customer notifications
- Handle internal stakeholder updates
- Manage executive communication
- Document timeline in real-time
selection: "Designated from on-call comms roster or engineering manager"
technical_lead:
responsibilities:
- Lead technical diagnosis and troubleshooting
- Coordinate technical responders
- Recommend mitigation and resolution actions
- Verify fix effectiveness
selection: "Senior engineer with relevant system expertise"
scribe:
responsibilities:
- Document all actions, decisions, and findings
- Maintain real-time timeline
- Record who did what and when
- Capture screenshots and log excerpts
selection: "Any available team member (can be rotated)"
workflow:
1_declare:
- "IC declares incident with severity level"
- "War room channel and bridge created"
- "Roles assigned"
- "First status update posted"
2_assess:
- "Determine scope and customer impact"
- "Identify affected systems and services"
- "Establish working hypothesis"
3_mitigate:
- "Focus on restoring service first, root cause second"
- "IC approves all changes to production"
- "Changes documented in real-time"
- "Rollback if mitigation makes things worse"
4_resolve:
- "Confirm service restored to normal"
- "Verify monitoring shows healthy metrics"
- "IC declares incident resolved"
- "Final status page update"
5_follow_up:
- "Schedule post-mortem within 48 hours"
- "Assign action items from immediate findings"
- "Send internal summary"
```
## On-Call Configuration
```yaml
on_call_schedule:
rotation_structure:
primary:
rotation: "Weekly"
handoff: "Monday 10:00 AM local time"
team_size: "Minimum 5 engineers in rotation"
secondary:
rotation: "Weekly (offset from primary)"
activation: "If primary does not acknowledge within 10 minutes"
expectations:
response_time: "Acknowledge alert within 5 minutes"
availability: "Reachable by phone and laptop within 15 minutes"
handoff: "Document any ongoing issues during handoff"
compensation: "Per company on-call compensation policy"
health:
max_consecutive_weeks: 2
minimum_gap_between_rotations: "2 weeks"
post_incident_rest: "If engaged for 4+ hours overnight, late start next day"
burnout_monitoring: "Track pages per person per week, rebalance if needed"
pagerduty_configuration:
escalation_policy:
- level_1:
target: "Primary on-call"
timeout: "5 minutes"
- level_2:
target: "Secondary on-call"
timeout: "10 minutes"
- level_3:
target: "Engineering manager"
timeout: "15 minutes"
notification_rules:
high_urgency:
- "Push notification immediately"
- "Phone call after 1 minute"
- "SMS after 2 minutes"
low_urgency:
- "Push notification"
- "Email after 5 minutes"
```
## Post-Mortem Template
```markdown
## Incident Summary
- Duration:
- Impact:
- Severity:
# Post-Incident Review: [Incident Title]
## Timeline
**Date:** YYYY-MM-DD
**Severity:** SEV[1-4]
**Duration:** [Start time] to [End time] ([X hours Y minutes])
**Incident Commander:** [Name]
**Author:** [Name]
**Status:** Draft / In Review / Final
## Executive Summary
[2-3 sentence summary of what happened, the impact, and the resolution]
## Impact
- **Customer impact:** [Number/percentage of customers affected, what they experienced]
- **Duration of impact:** [How long customers were affected]
- **Revenue impact:** [Estimated financial impact, if applicable]
- **Data impact:** [Any data loss or corruption]
- **SLA impact:** [Any SLA breaches]
## Timeline (all times UTC)
| Time | Event |
|------|-------|
| HH:MM | [First anomaly detected by monitoring] |
| HH:MM | [Alert fired / customer report received] |
| HH:MM | [On-call engineer acknowledged] |
| HH:MM | [Incident declared at SEV level] |
| HH:MM | [War room established] |
| HH:MM | [Root cause identified] |
| HH:MM | [Mitigation applied] |
| HH:MM | [Service restored] |
| HH:MM | [Incident resolved] |
## Root Cause
[Detailed technical explanation of what caused the incident]
## What Went Well
## Detection
- **How was the incident detected?** [Monitoring alert / customer report / manual observation]
- **Time to detect:** [Time from first anomaly to detection]
- **Could we have detected sooner?** [Yes/No, with explanation]
## What Could Be Improved
## Response
- **What went well:**
- [List things that worked effectively during response]
- [E.g., "Runbook for database failover was accurate and followed successfully"]
- [E.g., "Communication to customers was timely and clear"]
- **What could be improved:**
- [List things that slowed or hindered response]
- [E.g., "Took 20 minutes to identify the correct service owner"]
- [E.g., "Monitoring did not alert on the specific failure mode"]
## Contributing Factors
[List all factors that contributed to the incident occurring or being worse than it could have been. This is not about blame - it is about understanding the system.]
1. [Factor 1: e.g., "Configuration change was not tested in staging"]
2. [Factor 2: e.g., "Alert threshold was too high to catch gradual degradation"]
3. [Factor 3: e.g., "No circuit breaker between Service A and Service B"]
## Action Items
| Item | Owner | Due Date |
| ID | Action | Owner | Priority | Due Date | Status |
|----|--------|-------|----------|----------|--------|
| 1 | [Preventive action] | [Name] | P1 | YYYY-MM-DD | Open |
| 2 | [Detection improvement] | [Name] | P2 | YYYY-MM-DD | Open |
| 3 | [Process improvement] | [Name] | P2 | YYYY-MM-DD | Open |
| 4 | [Runbook update] | [Name] | P3 | YYYY-MM-DD | Open |
## Lessons Learned
[Key takeaways that should be shared broadly]
## Appendix
- [Link to monitoring dashboards during incident]
- [Link to relevant log queries]
- [Link to war room channel archive]
```
## Post-Mortem Process
```yaml
post_mortem_process:
scheduling:
sev1: "Within 48 hours of resolution"
sev2: "Within 1 week of resolution"
sev3: "Within 2 weeks (optional, based on learning potential)"
sev4: "Not required"
meeting_format:
duration: "60-90 minutes"
attendees:
required: "IC, technical lead, scribe, involved engineers"
optional: "Engineering manager, product manager, affected team leads"
agenda:
- "5 min: Review timeline and facts"
- "15 min: Walk through root cause and contributing factors"
- "15 min: Discuss what went well"
- "15 min: Discuss what could be improved"
- "15 min: Define and assign action items"
- "5 min: Identify lessons learned and sharing plan"
principles:
- "Blameless: Focus on systems and processes, not individuals"
- "Factual: Base discussion on data, logs, and observations"
- "Forward-looking: Prioritize preventive actions over assigning fault"
- "Complete: Address detection, response, and prevention"
- "Actionable: Every finding should produce a tracked action item"
action_item_tracking:
- "All action items entered into issue tracker (Jira, GitHub Issues)"
- "Priority assigned based on risk reduction potential"
- "Owner assigned with due date"
- "Reviewed in team standups and sprint planning"
- "Tracked to completion"
- "Monthly review of open post-mortem action items"
```
## Incident Metrics
```yaml
incident_metrics:
mttr:
name: "Mean Time to Resolve"
definition: "Average time from incident detection to resolution"
target: "SEV1: <1h, SEV2: <4h"
trending: "Track monthly, aim for improvement"
mttd:
name: "Mean Time to Detect"
definition: "Average time from incident start to detection"
target: "< 5 minutes for SEV1/2"
trending: "Monitors effectiveness of alerting"
mtta:
name: "Mean Time to Acknowledge"
definition: "Average time from alert to engineer acknowledgment"
target: "< 5 minutes"
trending: "Monitors on-call responsiveness"
incident_frequency:
name: "Incidents per week/month by severity"
target: "Trending downward"
trending: "Monitors system reliability improvement"
action_item_completion:
name: "Post-mortem action item completion rate"
target: "> 90% completed on time"
trending: "Monitors follow-through on improvements"
recurring_incidents:
name: "Percentage of incidents with same root cause as previous incident"
target: "< 10%"
trending: "Monitors effectiveness of preventive actions"
```
## Incident Management Checklist
```yaml
incident_management_checklist:
process_setup:
- [ ] Severity levels defined with clear criteria
- [ ] Escalation matrix documented
- [ ] On-call schedule established and staffed
- [ ] War room procedures documented
- [ ] Post-mortem template created
- [ ] Communication templates prepared (status page, email)
- [ ] Incident management tool configured (PagerDuty, OpsGenie)
per_incident:
- [ ] Incident declared with severity level
- [ ] War room established (SEV1/2)
- [ ] Roles assigned (IC, comms, technical lead, scribe)
- [ ] Timeline maintained in real-time
- [ ] Status page updated (customer-facing impact)
- [ ] Stakeholders notified per communication plan
- [ ] Resolution verified with monitoring
- [ ] Post-mortem scheduled
- [ ] Post-mortem conducted and published
- [ ] Action items tracked to completion
compliance:
- [ ] All SEV1/2 incidents have post-mortems
- [ ] Incident log maintained for audit evidence
- [ ] Metrics reported monthly
- [ ] On-call health monitored (pages per person)
- [ ] Annual incident response training conducted
- [ ] Annual incident response plan test completed
```
## Best Practices
- Clear severity definitions
- Defined escalation paths
- Blameless post-mortems
- Action item tracking
- Regular training
- Define severity levels with concrete examples so there is no ambiguity during an active incident
- Implement time-based escalation: if the on-call does not acknowledge, automatically escalate
- Focus on mitigation first, root cause second: restore service before investigating why it failed
- Run blameless post-mortems: the goal is to improve systems, not to assign fault to individuals
- Track post-mortem action items to completion: an unfinished action item means the same incident can recur
- Monitor incident metrics (MTTR, MTTD, frequency) as leading indicators of system reliability
- Protect on-call health: track page volume per person and redistribute if someone is overburdened
- Separate the incident commander role from the technical lead role in SEV1/2 incidents
- Practice incident response regularly with game days or chaos engineering exercises
- Archive incident records and post-mortems for compliance evidence and organizational learning
compliance/continuity/runbook-creation/SKILL.md
+441 -52
View File
@@ -9,88 +9,477 @@ metadata:
# Runbook Creation
Create effective operational runbooks and procedures.
Create effective operational runbooks, standard operating procedures, and
troubleshooting guides that any on-call engineer can follow under pressure.
## Runbook Structure
## Runbook Template — Full Structure
```markdown
# Runbook: [Service/Process Name]
````markdown
# Runbook: [Service / Process Name]
**Owner:** [Team or individual]
**Last Reviewed:** YYYY-MM-DD
**Version:** X.Y
**Severity if unavailable:** SEV[1-4]
---
## Overview
Brief description of the service and runbook purpose.
Brief description of the service, why this runbook exists, and when to
use it.
## Prerequisites
- Required access
- Tools needed
- Knowledge required
- [ ] Required access / IAM role: [details]
- [ ] Tools installed: [kubectl, aws-cli, psql, etc.]
- [ ] VPN connected to [environment]
- [ ] Communication channel open: [Slack #channel]
## Procedure
Step-by-step instructions with commands.
## Verification
How to confirm success.
### Step 1 — [Action Name]
## Rollback
Steps to undo if needed.
[Explanation of what this step does and why.]
## Escalation
When and how to escalate.
## Related Runbooks
Links to related procedures.
```bash
# command here
```
## Example Runbook
**Expected output:** [describe what success looks like]
```markdown
# Runbook: Database Failover
### Step 2 — [Action Name]
```bash
# command here
```
**Expected output:** [description]
*(Continue with numbered steps...)*
## Verification
How to confirm the procedure succeeded:
- [ ] [Check 1 — e.g., health endpoint returns 200]
- [ ] [Check 2 — e.g., no errors in logs for 5 minutes]
- [ ] [Check 3 — e.g., metrics return to baseline]
## Rollback
If the procedure fails or causes unexpected issues:
### Rollback Step 1
```bash
# rollback command
```
### Rollback Step 2
```bash
# rollback command
```
## Troubleshooting
| Symptom | Likely Cause | Resolution |
|---------|-------------|------------|
| [symptom 1] | [cause] | [fix] |
| [symptom 2] | [cause] | [fix] |
## Escalation
If unresolved after [X] minutes:
- **Primary:** @[team-lead] — [phone/Slack]
- **Secondary:** @[manager] — [phone/Slack]
## Related Runbooks
- [Link to related runbook 1]
- [Link to related runbook 2]
## Change Log
| Date | Author | Change |
|------|--------|--------|
| YYYY-MM-DD | [Name] | Initial version |
````
## Example Runbook — Database Failover
````markdown
# Runbook: PostgreSQL Database Failover
**Owner:** Platform / DBA team
**Last Reviewed:** 2025-06-15
**Version:** 2.1
**Severity if unavailable:** SEV1
---
## Overview
Procedure to failover PostgreSQL to replica.
Failover the primary PostgreSQL instance to the synchronous replica when
the primary is unreachable or degraded. This runbook covers both planned
(maintenance) and unplanned (emergency) failover.
## Prerequisites
- [ ] DBA access to primary and replica
- [ ] VPN connected
- [ ] DBA or SRE-level access to primary and replica hosts
- [ ] `psql` client installed (v14+)
- [ ] VPN connected to production network
- [ ] Slack channel #db-ops open
- [ ] Confirm replica is in sync: replication lag < 1 MB
## Procedure
### 1. Verify Replica Status
\`\`\`bash
psql -h replica -c "SELECT pg_is_in_recovery();"
# Should return 't'
\`\`\`
### Step 1 — Verify Replica Health
### 2. Stop Application Writes
\`\`\`bash
kubectl scale deployment app --replicas=0
\`\`\`
```bash
psql -h replica.db.internal -U dba -d postgres -c \
"SELECT pg_is_in_recovery(), pg_last_wal_replay_lsn();"
```
### 3. Promote Replica
\`\`\`bash
psql -h replica -c "SELECT pg_promote();"
\`\`\`
**Expected output:** `pg_is_in_recovery = t`, LSN advancing.
### 4. Update DNS
\`\`\`bash
aws route53 change-resource-record-sets ...
\`\`\`
### Step 2 — Stop Application Writes
```bash
kubectl scale deployment api-server --replicas=0 -n production
kubectl scale deployment worker --replicas=0 -n production
```
**Expected output:** Deployments scaled to 0 pods.
### Step 3 — Confirm Write Quiesce
```bash
psql -h primary.db.internal -U dba -d postgres -c \
"SELECT count(*) FROM pg_stat_activity WHERE state = 'active' AND query !~ 'pg_stat';"
```
**Expected output:** Count = 0 (no active queries).
### Step 4 — Promote Replica
```bash
psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_promote();"
```
Wait up to 30 seconds, then confirm:
```bash
psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_is_in_recovery();"
```
**Expected output:** `pg_is_in_recovery = f` (no longer a replica).
### Step 5 — Update DNS
```bash
aws route53 change-resource-record-sets \
--hosted-zone-id Z1234567890 \
--change-batch '{
"Changes": [{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "db.internal.example.com",
"Type": "CNAME",
"TTL": 60,
"ResourceRecords": [{"Value": "replica.db.internal"}]
}
}]
}'
```
### Step 6 — Restart Application
```bash
kubectl scale deployment api-server --replicas=6 -n production
kubectl scale deployment worker --replicas=4 -n production
```
## Verification
- [ ] Application connects to new primary
- [ ] No replication lag errors
- [ ] Transactions completing
- [ ] `psql -h db.internal.example.com -c "SELECT 1;"` returns successfully
- [ ] Application logs show successful DB connections (no errors for 5 min)
- [ ] Transaction throughput returns to baseline on Grafana dashboard
- [ ] No replication-lag alerts firing
## Rollback
If the promoted replica has issues, restore from the most recent backup:
```bash
# Restore latest automated snapshot (RDS example)
aws rds restore-db-instance-from-db-snapshot \
--db-instance-identifier prod-db-restored \
--db-snapshot-identifier prod-db-latest-snapshot
```
## Escalation
If issues persist after 15 minutes, escalate to:
- Primary: @dba-lead
- Secondary: @platform-oncall
If unresolved after 15 minutes:
- **Primary:** @dba-lead — +1-555-0101
- **Secondary:** @platform-oncall — +1-555-0102
````
## Automation Scripts for Common Operations
### Service Health Check
```bash
#!/usr/bin/env bash
# health-check.sh — Check health of critical services
set -euo pipefail
SERVICES=(
"https://api.example.com/healthz"
"https://app.example.com/healthz"
"https://admin.example.com/healthz"
)
EXIT_CODE=0
for url in "${SERVICES[@]}"; do
HTTP_CODE=$(curl -so /dev/null -w '%{http_code}' --max-time 5 "$url" 2>/dev/null || echo "000")
if [ "$HTTP_CODE" -eq 200 ]; then
printf " OK %s\n" "$url"
else
printf " FAIL %s (HTTP %s)\n" "$url" "$HTTP_CODE"
EXIT_CODE=1
fi
done
exit $EXIT_CODE
```
### Log Collection for Incident Investigation
```bash
#!/usr/bin/env bash
# collect-logs.sh — Gather logs from multiple sources for incident review
set -euo pipefail
INCIDENT_ID="${1:?Usage: collect-logs.sh <incident-id>}"
OUTDIR="/tmp/incident-${INCIDENT_ID}"
mkdir -p "$OUTDIR"
echo "Collecting logs for incident $INCIDENT_ID..."
# Kubernetes pod logs (last 30 min)
kubectl logs -l app=api-server -n production --since=30m \
> "${OUTDIR}/api-server-pods.log" 2>&1
# CloudWatch Logs (last 30 min)
aws logs filter-log-events \
--log-group-name /ecs/production/api \
--start-time "$(date -d '30 minutes ago' +%s)000" \
--output text > "${OUTDIR}/cloudwatch-api.log" 2>&1
# Database slow query log
psql -h db.internal -U dba -d postgres -c \
"SELECT * FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start;" \
> "${OUTDIR}/db-active-queries.log" 2>&1
# System resource snapshot
kubectl top pods -n production > "${OUTDIR}/pod-resources.log" 2>&1
echo "Logs saved to $OUTDIR"
tar czf "${OUTDIR}.tar.gz" -C /tmp "incident-${INCIDENT_ID}"
echo "Archive: ${OUTDIR}.tar.gz"
```
### Certificate Expiry Check
```bash
#!/usr/bin/env bash
# cert-check.sh — Warn if TLS certificates expire within 30 days
set -euo pipefail
DOMAINS=(
"api.example.com"
"app.example.com"
"admin.example.com"
)
WARN_DAYS=30
TODAY=$(date +%s)
EXIT_CODE=0
for domain in "${DOMAINS[@]}"; do
EXPIRY=$(echo | openssl s_client -servername "$domain" -connect "${domain}:443" 2>/dev/null \
| openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s 2>/dev/null || echo 0)
DAYS_LEFT=$(( (EXPIRY_EPOCH - TODAY) / 86400 ))
if [ "$DAYS_LEFT" -lt "$WARN_DAYS" ]; then
printf " WARN %s expires in %d days (%s)\n" "$domain" "$DAYS_LEFT" "$EXPIRY"
EXIT_CODE=1
else
printf " OK %s — %d days remaining\n" "$domain" "$DAYS_LEFT"
fi
done
exit $EXIT_CODE
```
### Disk Space Cleanup
```bash
#!/usr/bin/env bash
# disk-cleanup.sh — Free disk space on a host
set -euo pipefail
echo "=== Disk Usage Before ==="
df -h /
# Remove old journal logs (> 7 days)
journalctl --vacuum-time=7d 2>/dev/null || true
# Clean Docker artifacts
docker system prune -f --volumes 2>/dev/null || true
# Remove old log files
find /var/log -name "*.gz" -mtime +7 -delete 2>/dev/null || true
find /tmp -type f -mtime +3 -delete 2>/dev/null || true
echo "=== Disk Usage After ==="
df -h /
```
## Runbook Review Checklist
Use this checklist every time a runbook is created or updated.
```yaml
content_review:
- [ ] Title clearly identifies the service and operation
- [ ] Overview explains WHEN and WHY to use this runbook
- [ ] Prerequisites list all required access, tools, and setup
- [ ] Every step has a concrete command (no vague instructions)
- [ ] Expected output is documented for each step
- [ ] Verification section confirms success with specific checks
- [ ] Rollback section exists and has been tested
- [ ] Escalation contacts are current (names, phones, Slack handles)
- [ ] Troubleshooting table covers the top 3-5 known failure modes
usability_review:
- [ ] A new team member can follow the runbook without tribal knowledge
- [ ] Steps are numbered and sequential (no branching without clear labels)
- [ ] Commands can be copy-pasted (no placeholder values without explanation)
- [ ] Time estimates included for long-running steps
- [ ] No jargon or acronyms used without definition
maintenance_review:
- [ ] Owner and last-reviewed date are set
- [ ] Version number incremented
- [ ] Change log entry added
- [ ] Related runbooks section is up to date
- [ ] Links to dashboards and docs are valid (not broken)
```
## Runbook Testing Procedures
```yaml
testing_strategy:
dry_run:
frequency: "Every time a runbook is created or substantially edited"
method: "Walk through each step in a staging environment"
goal: "Verify commands work and output matches documentation"
peer_review:
frequency: "Every edit"
method: "Another engineer follows the runbook in staging without help"
goal: "Confirm the runbook is self-contained and unambiguous"
scheduled_validation:
frequency: "Quarterly"
method: "SRE team picks 5 runbooks at random, executes in staging"
goal: "Catch runbooks that have drifted from production reality"
incident_triggered:
trigger: "Any time a runbook is used in a real incident"
method: "Post-mortem includes runbook accuracy assessment"
goal: "Capture improvements while the experience is fresh"
automation_testing:
method: "CI pipeline validates bash scripts with shellcheck and dry-run"
example: |
# .github/workflows/runbook-lint.yml
name: Lint Runbook Scripts
on: [pull_request]
jobs:
shellcheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: ShellCheck
run: |
find runbooks/ -name "*.sh" -exec shellcheck {} +
```
## Versioning Strategy
```yaml
versioning:
storage: "Git repository — one directory per service, one file per runbook"
naming: "runbooks/<service>/<operation>.md"
branching: "PRs required for all changes; reviewed by service owner"
version_scheme:
format: "MAJOR.MINOR"
major_bump: "Procedure changes that alter the steps or their order"
minor_bump: "Clarifications, typo fixes, updated contact info"
directory_layout: |
runbooks/
api-server/
deploy.md
rollback.md
scale-up.md
database/
failover.md
backup-restore.md
vacuum-maintenance.md
infrastructure/
dns-update.md
certificate-renewal.md
disk-cleanup.md
review_requirements:
- PR must be approved by the service owner
- CI must pass (shellcheck for scripts, markdown lint)
- Reviewer confirms they can follow the steps independently
retention: "Git history serves as full audit trail — never delete old versions"
```
## Runbook Index Template
Keep a top-level index so engineers can find the right runbook quickly.
```markdown
# Runbook Index
| Service | Runbook | Severity | Owner | Last Tested |
|---------|---------|----------|-------|-------------|
| API Server | [Deploy](api-server/deploy.md) | — | @platform | 2025-05-01 |
| API Server | [Rollback](api-server/rollback.md) | SEV1 | @platform | 2025-05-01 |
| Database | [Failover](database/failover.md) | SEV1 | @dba | 2025-04-15 |
| Database | [Backup Restore](database/backup-restore.md) | SEV2 | @dba | 2025-04-15 |
| Infra | [DNS Update](infrastructure/dns-update.md) | SEV2 | @sre | 2025-06-01 |
| Infra | [Cert Renewal](infrastructure/certificate-renewal.md) | SEV3 | @sre | 2025-06-01 |
```
## Best Practices
- Keep procedures simple and clear
- Include verification steps
- Test runbooks regularly
- Version control runbooks
- Include troubleshooting tips
- Write runbooks for the engineer at 3 AM — clear, sequential, copy-pasteable
- Include expected output so the operator knows if a step succeeded
- Always provide a rollback path; every action should be reversible
- Test runbooks in staging before they are needed in production
- Keep runbooks in version control alongside the code they support
- Assign an owner to every runbook; ownerless runbooks rot fast
- After every incident, update the relevant runbook with lessons learned
- Automate repetitive runbook steps into scripts, but keep the runbook as
the orchestration guide so operators understand the "why"