Add 10 in-depth SEO-focused DevOps, security, and AI infra skills

2026-08-22 12:49:53 +02:00 · 2026-03-03 12:11:58 -05:00
parent 4401692295
commit 87bd457b23
11 changed files with 619 additions and 0 deletions
@@ -0,0 +1,66 @@
+---
+name: ai-sre-incident-response
+description: Build AI-focused SRE incident response practices for LLM outages, degraded quality, runaway cost events, and safety regressions.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# AI SRE Incident Response
+
+Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.
+
+## AI Incident Classes
+
+- **Availability incident**: model/provider unavailable, timeout storm.
+- **Quality incident**: answer accuracy or tool success drops below SLO.
+- **Safety incident**: harmful or policy-violating outputs increase.
+- **Cost incident**: unexpected token or provider spend spike.
+
+## Severity Framework (Example)
+
+- **SEV1**: user-facing outage, critical compliance risk, or active data leak.
+- **SEV2**: major degradation affecting key flows.
+- **SEV3**: limited impact or internal-only issue.
+
+## Golden Signals for AI Services
+
+- Request success rate
+- Latency (queue + generation + tool execution)
+- Hallucination/groundedness proxy metrics
+- Cost per minute and per tenant
+- Guardrail violation rate
+
+## Response Playbooks
+
+### Model Outage
+1. Freeze deployments.
+2. Shift traffic to fallback model/provider.
+3. Enforce stricter rate limits.
+4. Communicate ETA and mitigation.
+
+### Quality Regression
+1. Roll back prompt/model version.
+2. Disable risky optimization flags.
+3. Increase sampling for trace review.
+4. Re-run latest eval baseline.
+
+### Cost Spike
+1. Identify top tenants/routes/models.
+2. Enable cache + cheaper fallback path.
+3. Apply temporary token caps.
+4. Open postmortem with prevention actions.
+
+## Postmortem Requirements
+
+- Timeline with detector and responder timestamps
+- Blast radius by tenant and feature
+- Missed signals and alert tuning actions
+- Concrete hardening tasks with owners and due dates
+
+## Related Skills
+
+- [incident-response](../../../security/operations/incident-response/) - Standard incident process and evidence
+- [alerting-oncall](../../observability/alerting-oncall/) - Paging and escalation policy
+- [llm-cost-optimization](../llm-cost-optimization/) - Spend controls and efficiency patterns