Add 10 in-depth SEO-focused DevOps, security, and AI infra skills

devops/ai/rag-observability-evals/SKILL.md

@@ -0,0 +1,64 @@
+---
+name: rag-observability-evals
+description: Monitor and evaluate RAG systems with retrieval quality metrics, groundedness checks, hallucination detection, and continuous regression testing.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# RAG Observability and Evaluations
+
+Run retrieval-augmented generation like a measurable production system, not a black box.
+
+## What to Measure
+
+### Retrieval Quality
+- Recall@k and MRR for top-k chunks
+- Citation coverage and source freshness
+- Embedding drift and index staleness
+
+### Generation Quality
+- Groundedness score (answer supported by retrieved context)
+- Hallucination rate by route/use case
+- Instruction adherence and format validity
+
+### Reliability and Cost
+- p50/p95 latency split by retrieval vs generation
+- Token usage per stage
+- Cache hit rate and cost per successful answer
+
+## Evaluation Pipeline
+
+1. Curate a benchmark set with gold answers and source docs.
+2. Run nightly offline evals for every retriever/model configuration.
+3. Execute online shadow evals on sampled production traffic.
+4. Gate releases on minimum quality + safety + latency thresholds.
+
+## Alerting Strategy
+
+Page on:
+- sharp decline in groundedness,
+- spike in unanswered or fallback responses,
+- index freshness SLA breach,
+- cost-per-answer anomaly.
+
+## Practical Guardrails
+
+- Force citations for high-risk domains.
+- Return abstain/fallback when confidence is below threshold.
+- Re-rank retrieved chunks before final generation.
+- Use query rewriting only with strict regression tests.
+
+## Incident Triage Checklist
+
+- Did embedding model change?
+- Did chunking/indexing logic change?
+- Did source corpus ingestion fail?
+- Did gateway route to unintended model tier?
+
+## Related Skills
+
+- [rag-infrastructure](../../../infrastructure/local-ai/rag-infrastructure/) - Deploy robust RAG backends
+- [agent-observability](../agent-observability/) - Instrument requests, traces, and costs
+- [agent-evals](../agent-evals/) - Build repeatable eval suites