diff --git a/devops/ai/ai-pipeline-orchestration/SKILL.md b/devops/ai/ai-pipeline-orchestration/SKILL.md index 4d97ad8..4edb133 100644 --- a/devops/ai/ai-pipeline-orchestration/SKILL.md +++ b/devops/ai/ai-pipeline-orchestration/SKILL.md @@ -256,7 +256,7 @@ nightly_refresh = ScheduleDefinition( ## Related Skills -- [rag-infrastructure](../../infrastructure/local-ai/rag-infrastructure/) - RAG system setup -- [llm-fine-tuning](../../infrastructure/local-ai/llm-fine-tuning/) - Training jobs +- [rag-infrastructure](../../../infrastructure/local-ai/rag-infrastructure/) - RAG system setup +- [llm-fine-tuning](../../../infrastructure/local-ai/llm-fine-tuning/) - Training jobs - [agent-observability](../agent-observability/) - Pipeline monitoring -- [kubernetes-ops](../orchestration/kubernetes-ops/) - Running pipeline pods on K8s +- [kubernetes-ops](../../orchestration/kubernetes-ops/) - Running pipeline pods on K8s diff --git a/devops/ai/llm-caching/SKILL.md b/devops/ai/llm-caching/SKILL.md index ea0e8ba..72dff54 100644 --- a/devops/ai/llm-caching/SKILL.md +++ b/devops/ai/llm-caching/SKILL.md @@ -304,6 +304,6 @@ tcp-keepalive 60 ## Related Skills - [llm-cost-optimization](../llm-cost-optimization/) - Full cost strategy -- [llm-gateway](../../infrastructure/networking/llm-gateway/) - Gateway-level caching -- [vector-database-ops](../../infrastructure/databases/vector-database-ops/) - Qdrant setup +- [llm-gateway](../../../infrastructure/networking/llm-gateway/) - Gateway-level caching +- [vector-database-ops](../../../infrastructure/databases/vector-database-ops/) - Qdrant setup - [agent-observability](../agent-observability/) - Cache metrics dashboards diff --git a/devops/ai/llm-cost-optimization/SKILL.md b/devops/ai/llm-cost-optimization/SKILL.md index ae17da2..e8fc990 100644 --- a/devops/ai/llm-cost-optimization/SKILL.md +++ b/devops/ai/llm-cost-optimization/SKILL.md @@ -280,7 +280,7 @@ def track_call(model, team, task_type, response): ## Related Skills -- [llm-gateway](../../infrastructure/networking/llm-gateway/) - Centralized cost control +- [llm-gateway](../../../infrastructure/networking/llm-gateway/) - Centralized cost control - [llm-caching](../llm-caching/) - Semantic caching patterns -- [vllm-server](../../infrastructure/local-ai/vllm-server/) - Self-hosted inference +- [vllm-server](../../../infrastructure/local-ai/vllm-server/) - Self-hosted inference - [agent-observability](../agent-observability/) - Token and cost telemetry diff --git a/devops/observability/sre-dashboards/SKILL.md b/devops/observability/sre-dashboards/SKILL.md new file mode 100644 index 0000000..6512750 --- /dev/null +++ b/devops/observability/sre-dashboards/SKILL.md @@ -0,0 +1,118 @@ +--- +name: sre-dashboards +description: Design and operationalize SRE dashboards that surface reliability, latency, error, saturation, and capacity signals across services. Use when building observability views for SLOs, incident response, and executive reliability reporting. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# SRE Dashboards + +Build dashboards that help teams detect, triage, and prevent reliability incidents. + +## When to Use This Skill + +Use this skill when: +- Defining service-level dashboards for production systems +- Tracking SLO health and error-budget burn +- Creating incident command-center views +- Standardizing dashboard patterns across teams + +## Prerequisites + +- Metrics pipeline (Prometheus, OpenTelemetry, or vendor equivalent) +- Logs/traces linked to services and environments +- Agreed service taxonomy (team, service, tier, environment) + +## Dashboard Architecture + +Structure dashboards in layers: + +1. **Executive Reliability View**: SLO attainment, incident counts, MTTR trends. +2. **Service Health View**: RED/USE metrics, dependency health, release markers. +3. **Deep-Dive View**: Per-endpoint latency, resource saturation, error categories. + +Keep each view answer-oriented: +- *Are customers impacted?* +- *What changed?* +- *Where is the bottleneck?* + +## Core SRE Panels + +### Golden Signals + +- **Latency**: p50/p95/p99 request duration by endpoint +- **Traffic**: request throughput and queue depth +- **Errors**: 5xx rate, failed jobs, timeout ratio +- **Saturation**: CPU, memory, disk I/O, thread/connection pool exhaustion + +### SLO Panels + +- Current SLI value (rolling windows: 5m, 1h, 24h, 30d) +- Error-budget remaining (%) +- Burn-rate panels (fast and slow windows) +- Multi-window burn alert status + +### Change Correlation + +- Deployment markers and config-change annotations +- Feature flag state overlays +- Upstream/downstream dependency error rates + +## Example PromQL Snippets + +```promql +# API error rate (%) +100 * sum(rate(http_requests_total{status=~"5.."}[5m])) + / sum(rate(http_requests_total[5m])) +``` + +```promql +# p95 latency by route +histogram_quantile(0.95, + sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])) +) +``` + +```promql +# Fast burn rate (5m / 1h) +( + sum(rate(http_requests_total{status=~"5.."}[5m])) + / sum(rate(http_requests_total[5m])) +) +/ +( + sum(rate(http_requests_total{status=~"5.."}[1h])) + / sum(rate(http_requests_total[1h])) +) +``` + +## Operational Guidelines + +- Use consistent color semantics (green=healthy, yellow=degrading, red=breach) +- Label units explicitly (ms, req/s, %, cores) +- Default time windows to incident-friendly ranges (15m, 1h, 6h, 24h) +- Minimize panel count per dashboard to reduce cognitive load +- Add runbook links directly in panel descriptions + +## Troubleshooting + +### Panel appears flat or empty + +- Verify label cardinality and filters (`service`, `env`, `region`) +- Confirm scrape/ingest latency is within expected range +- Check metric rename regressions after instrumentation updates + +### High cardinality slows dashboards + +- Aggregate by stable dimensions (`service`, `route_group`) instead of raw IDs +- Use recording rules for expensive percentile and ratio queries +- Split deep-dive dashboards from NOC summary dashboards + +## Related Skills + +- [prometheus-grafana](../prometheus-grafana/) - Dashboard implementation and PromQL +- [opentelemetry](../opentelemetry/) - Standardized telemetry instrumentation +- [alerting-oncall](../alerting-oncall/) - Reliability alert routing and escalation +- [agent-observability](../../ai/agent-observability/) - AI workload reliability telemetry diff --git a/devops/orchestration/model-serving-kubernetes/SKILL.md b/devops/orchestration/model-serving-kubernetes/SKILL.md index affc644..b052df0 100644 --- a/devops/orchestration/model-serving-kubernetes/SKILL.md +++ b/devops/orchestration/model-serving-kubernetes/SKILL.md @@ -308,7 +308,7 @@ kubectl get inferenceservice llama-3-8b -n models -w ## Related Skills -- [vllm-server](../../infrastructure/local-ai/vllm-server/) - vLLM for LLM serving -- [llm-inference-scaling](../../infrastructure/local-ai/llm-inference-scaling/) - KEDA autoscaling -- [kubernetes-ops](./kubernetes-ops/) - Core Kubernetes operations -- [gpu-server-management](../../infrastructure/servers/gpu-server-management/) - GPU nodes +- [vllm-server](../../../infrastructure/local-ai/vllm-server/) - vLLM for LLM serving +- [llm-inference-scaling](../../../infrastructure/local-ai/llm-inference-scaling/) - KEDA autoscaling +- [kubernetes-ops](../kubernetes-ops/) - Core Kubernetes operations +- [gpu-server-management](../../../infrastructure/servers/gpu-server-management/) - GPU nodes diff --git a/infrastructure/databases/vector-database-ops/SKILL.md b/infrastructure/databases/vector-database-ops/SKILL.md index f879b2a..f38d81f 100644 --- a/infrastructure/databases/vector-database-ops/SKILL.md +++ b/infrastructure/databases/vector-database-ops/SKILL.md @@ -281,5 +281,5 @@ while True: ## Related Skills - [rag-infrastructure](../../local-ai/rag-infrastructure/) - Full RAG pipeline -- [databases](../databases/) - General database management +- [databases](../) - General database management - [postgresql](../postgresql/) - pgvector host database ops diff --git a/infrastructure/local-ai/openclaw-security-hardening/SKILL.md b/infrastructure/local-ai/openclaw-security-hardening/SKILL.md index 2c6dd7c..03b27fc 100644 --- a/infrastructure/local-ai/openclaw-security-hardening/SKILL.md +++ b/infrastructure/local-ai/openclaw-security-hardening/SKILL.md @@ -106,4 +106,4 @@ Minimum operational runbook: - [openclaw-local-mac-mini](../openclaw-local-mac-mini/) - Local OpenClaw hosting setup - [multi-tenant-llm-hosting](../multi-tenant-llm-hosting/) - Multi-tenant AI isolation patterns -- [zero-trust](../../networking/zero-trust/) - Private access and identity-aware network controls +- [zero-trust](../../../security/network/zero-trust/) - Private access and identity-aware network controls