mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
65 lines
2.0 KiB
Markdown
65 lines
2.0 KiB
Markdown
---
|
|
name: rag-observability-evals
|
|
description: Monitor and evaluate RAG systems with retrieval quality metrics, groundedness checks, hallucination detection, and continuous regression testing.
|
|
license: MIT
|
|
metadata:
|
|
author: devops-skills
|
|
version: "1.0"
|
|
---
|
|
|
|
# RAG Observability and Evaluations
|
|
|
|
Run retrieval-augmented generation like a measurable production system, not a black box.
|
|
|
|
## What to Measure
|
|
|
|
### Retrieval Quality
|
|
- Recall@k and MRR for top-k chunks
|
|
- Citation coverage and source freshness
|
|
- Embedding drift and index staleness
|
|
|
|
### Generation Quality
|
|
- Groundedness score (answer supported by retrieved context)
|
|
- Hallucination rate by route/use case
|
|
- Instruction adherence and format validity
|
|
|
|
### Reliability and Cost
|
|
- p50/p95 latency split by retrieval vs generation
|
|
- Token usage per stage
|
|
- Cache hit rate and cost per successful answer
|
|
|
|
## Evaluation Pipeline
|
|
|
|
1. Curate a benchmark set with gold answers and source docs.
|
|
2. Run nightly offline evals for every retriever/model configuration.
|
|
3. Execute online shadow evals on sampled production traffic.
|
|
4. Gate releases on minimum quality + safety + latency thresholds.
|
|
|
|
## Alerting Strategy
|
|
|
|
Page on:
|
|
- sharp decline in groundedness,
|
|
- spike in unanswered or fallback responses,
|
|
- index freshness SLA breach,
|
|
- cost-per-answer anomaly.
|
|
|
|
## Practical Guardrails
|
|
|
|
- Force citations for high-risk domains.
|
|
- Return abstain/fallback when confidence is below threshold.
|
|
- Re-rank retrieved chunks before final generation.
|
|
- Use query rewriting only with strict regression tests.
|
|
|
|
## Incident Triage Checklist
|
|
|
|
- Did embedding model change?
|
|
- Did chunking/indexing logic change?
|
|
- Did source corpus ingestion fail?
|
|
- Did gateway route to unintended model tier?
|
|
|
|
## Related Skills
|
|
|
|
- [rag-infrastructure](../../../infrastructure/local-ai/rag-infrastructure/) - Deploy robust RAG backends
|
|
- [agent-observability](../agent-observability/) - Instrument requests, traces, and costs
|
|
- [agent-evals](../agent-evals/) - Build repeatable eval suites
|