mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
67 lines
2.1 KiB
Markdown
67 lines
2.1 KiB
Markdown
---
|
|
name: ai-sre-incident-response
|
|
description: Build AI-focused SRE incident response practices for LLM outages, degraded quality, runaway cost events, and safety regressions.
|
|
license: MIT
|
|
metadata:
|
|
author: devops-skills
|
|
version: "1.0"
|
|
---
|
|
|
|
# AI SRE Incident Response
|
|
|
|
Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.
|
|
|
|
## AI Incident Classes
|
|
|
|
- **Availability incident**: model/provider unavailable, timeout storm.
|
|
- **Quality incident**: answer accuracy or tool success drops below SLO.
|
|
- **Safety incident**: harmful or policy-violating outputs increase.
|
|
- **Cost incident**: unexpected token or provider spend spike.
|
|
|
|
## Severity Framework (Example)
|
|
|
|
- **SEV1**: user-facing outage, critical compliance risk, or active data leak.
|
|
- **SEV2**: major degradation affecting key flows.
|
|
- **SEV3**: limited impact or internal-only issue.
|
|
|
|
## Golden Signals for AI Services
|
|
|
|
- Request success rate
|
|
- Latency (queue + generation + tool execution)
|
|
- Hallucination/groundedness proxy metrics
|
|
- Cost per minute and per tenant
|
|
- Guardrail violation rate
|
|
|
|
## Response Playbooks
|
|
|
|
### Model Outage
|
|
1. Freeze deployments.
|
|
2. Shift traffic to fallback model/provider.
|
|
3. Enforce stricter rate limits.
|
|
4. Communicate ETA and mitigation.
|
|
|
|
### Quality Regression
|
|
1. Roll back prompt/model version.
|
|
2. Disable risky optimization flags.
|
|
3. Increase sampling for trace review.
|
|
4. Re-run latest eval baseline.
|
|
|
|
### Cost Spike
|
|
1. Identify top tenants/routes/models.
|
|
2. Enable cache + cheaper fallback path.
|
|
3. Apply temporary token caps.
|
|
4. Open postmortem with prevention actions.
|
|
|
|
## Postmortem Requirements
|
|
|
|
- Timeline with detector and responder timestamps
|
|
- Blast radius by tenant and feature
|
|
- Missed signals and alert tuning actions
|
|
- Concrete hardening tasks with owners and due dates
|
|
|
|
## Related Skills
|
|
|
|
- [incident-response](../../../security/operations/incident-response/) - Standard incident process and evidence
|
|
- [alerting-oncall](../../observability/alerting-oncall/) - Paging and escalation policy
|
|
- [llm-cost-optimization](../llm-cost-optimization/) - Spend controls and efficiency patterns
|