mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
Add 10 in-depth SEO-focused DevOps, security, and AI infra skills
This commit is contained in:
Toby
@@ -0,0 +1,66 @@
|
||||
---
|
||||
name: ai-sre-incident-response
|
||||
description: Build AI-focused SRE incident response practices for LLM outages, degraded quality, runaway cost events, and safety regressions.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# AI SRE Incident Response
|
||||
|
||||
Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.
|
||||
|
||||
## AI Incident Classes
|
||||
|
||||
- **Availability incident**: model/provider unavailable, timeout storm.
|
||||
- **Quality incident**: answer accuracy or tool success drops below SLO.
|
||||
- **Safety incident**: harmful or policy-violating outputs increase.
|
||||
- **Cost incident**: unexpected token or provider spend spike.
|
||||
|
||||
## Severity Framework (Example)
|
||||
|
||||
- **SEV1**: user-facing outage, critical compliance risk, or active data leak.
|
||||
- **SEV2**: major degradation affecting key flows.
|
||||
- **SEV3**: limited impact or internal-only issue.
|
||||
|
||||
## Golden Signals for AI Services
|
||||
|
||||
- Request success rate
|
||||
- Latency (queue + generation + tool execution)
|
||||
- Hallucination/groundedness proxy metrics
|
||||
- Cost per minute and per tenant
|
||||
- Guardrail violation rate
|
||||
|
||||
## Response Playbooks
|
||||
|
||||
### Model Outage
|
||||
1. Freeze deployments.
|
||||
2. Shift traffic to fallback model/provider.
|
||||
3. Enforce stricter rate limits.
|
||||
4. Communicate ETA and mitigation.
|
||||
|
||||
### Quality Regression
|
||||
1. Roll back prompt/model version.
|
||||
2. Disable risky optimization flags.
|
||||
3. Increase sampling for trace review.
|
||||
4. Re-run latest eval baseline.
|
||||
|
||||
### Cost Spike
|
||||
1. Identify top tenants/routes/models.
|
||||
2. Enable cache + cheaper fallback path.
|
||||
3. Apply temporary token caps.
|
||||
4. Open postmortem with prevention actions.
|
||||
|
||||
## Postmortem Requirements
|
||||
|
||||
- Timeline with detector and responder timestamps
|
||||
- Blast radius by tenant and feature
|
||||
- Missed signals and alert tuning actions
|
||||
- Concrete hardening tasks with owners and due dates
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [incident-response](../../../security/operations/incident-response/) - Standard incident process and evidence
|
||||
- [alerting-oncall](../../observability/alerting-oncall/) - Paging and escalation policy
|
||||
- [llm-cost-optimization](../llm-cost-optimization/) - Spend controls and efficiency patterns
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
name: llmops-platform-engineering
|
||||
description: Build production LLMOps platforms with CI/CD, model promotion workflows, evaluation gates, rollback, and governance across cloud and self-hosted inference.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# LLMOps Platform Engineering
|
||||
|
||||
Design and operate an internal LLM platform that supports rapid experimentation without compromising reliability, cost, or compliance.
|
||||
|
||||
## Outcomes
|
||||
|
||||
- Standardized path from experiment to production
|
||||
- Safe model rollout with quality and safety gates
|
||||
- Repeatable infra modules for inference, vector DB, and observability
|
||||
- Clear ownership model across platform, app, and security teams
|
||||
|
||||
## Reference Architecture
|
||||
|
||||
1. **Control Plane**: model registry, prompt/version catalog, policy checks, eval pipeline.
|
||||
2. **Data Plane**: inference gateway, vector database, cache, feature store.
|
||||
3. **Ops Plane**: telemetry, alerting, SLO dashboards, cost analytics.
|
||||
4. **Security Plane**: IAM boundaries, secret rotation, content filters, audit logs.
|
||||
|
||||
## Golden Delivery Workflow
|
||||
|
||||
1. Train/fine-tune or onboard provider model.
|
||||
2. Register artifact and metadata (license, intended use, constraints).
|
||||
3. Run automated eval suite (quality + safety + latency + cost).
|
||||
4. Deploy canary behind gateway with strict traffic policy.
|
||||
5. Promote after SLO and business KPI thresholds pass.
|
||||
6. Keep rollback target hot for fast reversion.
|
||||
|
||||
## CI/CD Design for AI Services
|
||||
|
||||
- Build immutable containers with pinned dependencies and model hashes.
|
||||
- Use environment promotion: `dev -> stage -> prod`.
|
||||
- Fail deployment if:
|
||||
- regression evals drop below baseline,
|
||||
- safety tests exceed risk threshold,
|
||||
- p95 latency exceeds SLO budget.
|
||||
- Store deployment evidence for audits (commit SHA, eval report, approver).
|
||||
|
||||
## Operational SLOs
|
||||
|
||||
- Availability: `99.9%` for synchronous inference endpoints.
|
||||
- Latency: p95 under product-specific target (for example, `<1200ms`).
|
||||
- Cost: per-request and per-tenant budget ceilings.
|
||||
- Quality: task success rate and groundedness thresholds.
|
||||
|
||||
## Platform Guardrails
|
||||
|
||||
- Enforce tenant quotas and model allow-lists.
|
||||
- Require structured output contracts for automation paths.
|
||||
- Default to low-risk model settings for critical workflows.
|
||||
- Disable unconstrained tool execution in production.
|
||||
|
||||
## Tooling Stack (Example)
|
||||
|
||||
- **Orchestration**: Argo Workflows / GitHub Actions / Airflow.
|
||||
- **Model Registry**: MLflow / custom metadata DB.
|
||||
- **Gateway**: LiteLLM / Envoy-based API gateway.
|
||||
- **Observability**: OpenTelemetry + Prometheus + Grafana + Langfuse.
|
||||
- **Policy**: OPA/Rego for deployment and runtime checks.
|
||||
|
||||
## Incident Readiness
|
||||
|
||||
- Runbooks for model outage, provider timeout spikes, and cost surges.
|
||||
- Chaos drills for provider failover and vector DB degradation.
|
||||
- Pre-approved rollback path with one-command execution.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [ai-pipeline-orchestration](../ai-pipeline-orchestration/) - Orchestrate ingestion and inference workflows
|
||||
- [agent-evals](../agent-evals/) - Build evaluation gates for releases
|
||||
- [llm-gateway](../../../infrastructure/networking/llm-gateway/) - Route and control LLM traffic
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
name: model-registry-governance
|
||||
description: Establish model registry standards, governance controls, metadata schemas, approvals, and lifecycle policies for enterprise AI deployments.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# Model Registry Governance
|
||||
|
||||
Create a trustworthy system of record for model artifacts, prompts, adapters, and evaluation evidence.
|
||||
|
||||
## Core Principles
|
||||
|
||||
- **Traceability**: every production model maps to source code, data snapshot, and evaluation results.
|
||||
- **Reproducibility**: builds are deterministic with pinned dependencies.
|
||||
- **Policy-driven promotion**: no manual bypass for critical safety checks.
|
||||
- **Lifecycle hygiene**: stale, vulnerable, or unowned models are retired automatically.
|
||||
|
||||
## Required Metadata Schema
|
||||
|
||||
Track at minimum:
|
||||
|
||||
- Model name, semantic version, checksum, and storage URI
|
||||
- Base model lineage and fine-tune method
|
||||
- Training/eval datasets and time windows
|
||||
- License, allowed use cases, prohibited use cases
|
||||
- Security risk rating and mitigation controls
|
||||
- Owner, backup owner, and escalation contact
|
||||
|
||||
## Approval Workflow
|
||||
|
||||
1. Registration request created from CI.
|
||||
2. Security checks (artifact scan, dependency scan, provenance).
|
||||
3. Evaluation package uploaded (quality, toxicity, jailbreak, bias, latency, cost).
|
||||
4. Required approvals: platform + product + security (as policy dictates).
|
||||
5. Promotion to stage/prod based on signed decision record.
|
||||
|
||||
## Lifecycle States
|
||||
|
||||
- `draft`: internal experimentation.
|
||||
- `candidate`: passed baseline tests.
|
||||
- `approved`: authorized for production rollout.
|
||||
- `deprecated`: replacement announced, new usage blocked.
|
||||
- `retired`: no serving allowed, archived for audit.
|
||||
|
||||
## Governance Policies
|
||||
|
||||
- Reject artifacts without SBOM/provenance.
|
||||
- Block promotion if known critical CVEs remain unresolved.
|
||||
- Require refreshed evals after prompt/template changes.
|
||||
- Expire approvals after a configurable period (for example 90 days).
|
||||
|
||||
## Audit Readiness
|
||||
|
||||
Maintain immutable records of:
|
||||
|
||||
- Who approved and when
|
||||
- Which policy checks executed
|
||||
- Which exceptions were granted
|
||||
- What model/version served each customer request window
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [sbom-supply-chain](../../../security/scanning/sbom-supply-chain/) - Provenance and signing
|
||||
- [policy-as-code](../../../compliance/governance/policy-as-code/) - Enforce governance with policy engines
|
||||
- [llm-fine-tuning](../../../infrastructure/local-ai/llm-fine-tuning/) - Version adapters and training outputs
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: rag-observability-evals
|
||||
description: Monitor and evaluate RAG systems with retrieval quality metrics, groundedness checks, hallucination detection, and continuous regression testing.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# RAG Observability and Evaluations
|
||||
|
||||
Run retrieval-augmented generation like a measurable production system, not a black box.
|
||||
|
||||
## What to Measure
|
||||
|
||||
### Retrieval Quality
|
||||
- Recall@k and MRR for top-k chunks
|
||||
- Citation coverage and source freshness
|
||||
- Embedding drift and index staleness
|
||||
|
||||
### Generation Quality
|
||||
- Groundedness score (answer supported by retrieved context)
|
||||
- Hallucination rate by route/use case
|
||||
- Instruction adherence and format validity
|
||||
|
||||
### Reliability and Cost
|
||||
- p50/p95 latency split by retrieval vs generation
|
||||
- Token usage per stage
|
||||
- Cache hit rate and cost per successful answer
|
||||
|
||||
## Evaluation Pipeline
|
||||
|
||||
1. Curate a benchmark set with gold answers and source docs.
|
||||
2. Run nightly offline evals for every retriever/model configuration.
|
||||
3. Execute online shadow evals on sampled production traffic.
|
||||
4. Gate releases on minimum quality + safety + latency thresholds.
|
||||
|
||||
## Alerting Strategy
|
||||
|
||||
Page on:
|
||||
- sharp decline in groundedness,
|
||||
- spike in unanswered or fallback responses,
|
||||
- index freshness SLA breach,
|
||||
- cost-per-answer anomaly.
|
||||
|
||||
## Practical Guardrails
|
||||
|
||||
- Force citations for high-risk domains.
|
||||
- Return abstain/fallback when confidence is below threshold.
|
||||
- Re-rank retrieved chunks before final generation.
|
||||
- Use query rewriting only with strict regression tests.
|
||||
|
||||
## Incident Triage Checklist
|
||||
|
||||
- Did embedding model change?
|
||||
- Did chunking/indexing logic change?
|
||||
- Did source corpus ingestion fail?
|
||||
- Did gateway route to unintended model tier?
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [rag-infrastructure](../../../infrastructure/local-ai/rag-infrastructure/) - Deploy robust RAG backends
|
||||
- [agent-observability](../agent-observability/) - Instrument requests, traces, and costs
|
||||
- [agent-evals](../agent-evals/) - Build repeatable eval suites
|
||||
Reference in New Issue
Block a user