Fix skill cross-links and add missing sre-dashboards skill

This commit is contained in:
Toby
2026-05-22 09:02:40 -04:00
parent ba9e489584
commit f7d2562069
7 changed files with 131 additions and 13 deletions
+3 -3
View File
@@ -256,7 +256,7 @@ nightly_refresh = ScheduleDefinition(
## Related Skills
- [rag-infrastructure](../../infrastructure/local-ai/rag-infrastructure/) - RAG system setup
- [llm-fine-tuning](../../infrastructure/local-ai/llm-fine-tuning/) - Training jobs
- [rag-infrastructure](../../../infrastructure/local-ai/rag-infrastructure/) - RAG system setup
- [llm-fine-tuning](../../../infrastructure/local-ai/llm-fine-tuning/) - Training jobs
- [agent-observability](../agent-observability/) - Pipeline monitoring
- [kubernetes-ops](../orchestration/kubernetes-ops/) - Running pipeline pods on K8s
- [kubernetes-ops](../../orchestration/kubernetes-ops/) - Running pipeline pods on K8s
+2 -2
View File
@@ -304,6 +304,6 @@ tcp-keepalive 60
## Related Skills
- [llm-cost-optimization](../llm-cost-optimization/) - Full cost strategy
- [llm-gateway](../../infrastructure/networking/llm-gateway/) - Gateway-level caching
- [vector-database-ops](../../infrastructure/databases/vector-database-ops/) - Qdrant setup
- [llm-gateway](../../../infrastructure/networking/llm-gateway/) - Gateway-level caching
- [vector-database-ops](../../../infrastructure/databases/vector-database-ops/) - Qdrant setup
- [agent-observability](../agent-observability/) - Cache metrics dashboards
+2 -2
View File
@@ -280,7 +280,7 @@ def track_call(model, team, task_type, response):
## Related Skills
- [llm-gateway](../../infrastructure/networking/llm-gateway/) - Centralized cost control
- [llm-gateway](../../../infrastructure/networking/llm-gateway/) - Centralized cost control
- [llm-caching](../llm-caching/) - Semantic caching patterns
- [vllm-server](../../infrastructure/local-ai/vllm-server/) - Self-hosted inference
- [vllm-server](../../../infrastructure/local-ai/vllm-server/) - Self-hosted inference
- [agent-observability](../agent-observability/) - Token and cost telemetry
@@ -0,0 +1,118 @@
---
name: sre-dashboards
description: Design and operationalize SRE dashboards that surface reliability, latency, error, saturation, and capacity signals across services. Use when building observability views for SLOs, incident response, and executive reliability reporting.
license: MIT
metadata:
author: devops-skills
version: "1.0"
---
# SRE Dashboards
Build dashboards that help teams detect, triage, and prevent reliability incidents.
## When to Use This Skill
Use this skill when:
- Defining service-level dashboards for production systems
- Tracking SLO health and error-budget burn
- Creating incident command-center views
- Standardizing dashboard patterns across teams
## Prerequisites
- Metrics pipeline (Prometheus, OpenTelemetry, or vendor equivalent)
- Logs/traces linked to services and environments
- Agreed service taxonomy (team, service, tier, environment)
## Dashboard Architecture
Structure dashboards in layers:
1. **Executive Reliability View**: SLO attainment, incident counts, MTTR trends.
2. **Service Health View**: RED/USE metrics, dependency health, release markers.
3. **Deep-Dive View**: Per-endpoint latency, resource saturation, error categories.
Keep each view answer-oriented:
- *Are customers impacted?*
- *What changed?*
- *Where is the bottleneck?*
## Core SRE Panels
### Golden Signals
- **Latency**: p50/p95/p99 request duration by endpoint
- **Traffic**: request throughput and queue depth
- **Errors**: 5xx rate, failed jobs, timeout ratio
- **Saturation**: CPU, memory, disk I/O, thread/connection pool exhaustion
### SLO Panels
- Current SLI value (rolling windows: 5m, 1h, 24h, 30d)
- Error-budget remaining (%)
- Burn-rate panels (fast and slow windows)
- Multi-window burn alert status
### Change Correlation
- Deployment markers and config-change annotations
- Feature flag state overlays
- Upstream/downstream dependency error rates
## Example PromQL Snippets
```promql
# API error rate (%)
100 * sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
```
```promql
# p95 latency by route
histogram_quantile(0.95,
sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)
```
```promql
# Fast burn rate (5m / 1h)
(
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
)
/
(
sum(rate(http_requests_total{status=~"5.."}[1h]))
/ sum(rate(http_requests_total[1h]))
)
```
## Operational Guidelines
- Use consistent color semantics (green=healthy, yellow=degrading, red=breach)
- Label units explicitly (ms, req/s, %, cores)
- Default time windows to incident-friendly ranges (15m, 1h, 6h, 24h)
- Minimize panel count per dashboard to reduce cognitive load
- Add runbook links directly in panel descriptions
## Troubleshooting
### Panel appears flat or empty
- Verify label cardinality and filters (`service`, `env`, `region`)
- Confirm scrape/ingest latency is within expected range
- Check metric rename regressions after instrumentation updates
### High cardinality slows dashboards
- Aggregate by stable dimensions (`service`, `route_group`) instead of raw IDs
- Use recording rules for expensive percentile and ratio queries
- Split deep-dive dashboards from NOC summary dashboards
## Related Skills
- [prometheus-grafana](../prometheus-grafana/) - Dashboard implementation and PromQL
- [opentelemetry](../opentelemetry/) - Standardized telemetry instrumentation
- [alerting-oncall](../alerting-oncall/) - Reliability alert routing and escalation
- [agent-observability](../../ai/agent-observability/) - AI workload reliability telemetry
@@ -308,7 +308,7 @@ kubectl get inferenceservice llama-3-8b -n models -w
## Related Skills
- [vllm-server](../../infrastructure/local-ai/vllm-server/) - vLLM for LLM serving
- [llm-inference-scaling](../../infrastructure/local-ai/llm-inference-scaling/) - KEDA autoscaling
- [kubernetes-ops](./kubernetes-ops/) - Core Kubernetes operations
- [gpu-server-management](../../infrastructure/servers/gpu-server-management/) - GPU nodes
- [vllm-server](../../../infrastructure/local-ai/vllm-server/) - vLLM for LLM serving
- [llm-inference-scaling](../../../infrastructure/local-ai/llm-inference-scaling/) - KEDA autoscaling
- [kubernetes-ops](../kubernetes-ops/) - Core Kubernetes operations
- [gpu-server-management](../../../infrastructure/servers/gpu-server-management/) - GPU nodes