mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
Add OpenClaw local and Mac mini setup skill
This commit is contained in:
@@ -0,0 +1,39 @@
|
||||
---
|
||||
name: agent-evals
|
||||
description: Build automated evaluation suites for AI agents using golden datasets, rubrics, and regression gates.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# Agent Evals
|
||||
|
||||
Create repeatable checks so agent behavior improves safely over time.
|
||||
|
||||
## Evaluation Layers
|
||||
|
||||
- Unit evals: prompt-level correctness
|
||||
- Tool evals: API/tool call decision quality
|
||||
- End-to-end evals: realistic multi-step tasks
|
||||
- Safety evals: prompt injection and data leak resistance
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
```bash
|
||||
# Example eval pipeline steps
|
||||
make evals-smoke
|
||||
make evals-regression
|
||||
make evals-safety
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Version datasets with expected outputs.
|
||||
- Track pass rates and score drift over time.
|
||||
- Block deploys on critical safety regressions.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [github-actions](../../ci-cd/github-actions/) - Eval automation in CI
|
||||
- [ai-agent-security](../../../security/ai/ai-agent-security/) - Security-focused eval cases
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: agent-observability
|
||||
description: Instrument AI agents with tracing, token metrics, latency, and cost visibility. Use for reliability and debugging.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# Agent Observability
|
||||
|
||||
Monitor AI agent behavior with logs, traces, metrics, and cost telemetry.
|
||||
|
||||
## Track Core Signals
|
||||
|
||||
- Request latency (p50/p95/p99)
|
||||
- Token usage (prompt/completion/cached)
|
||||
- Tool call success and failure rates
|
||||
- Cost per task and per customer
|
||||
- Hallucination and retry frequency
|
||||
|
||||
## Implementation Pattern
|
||||
|
||||
1. Add trace IDs to every user request.
|
||||
2. Capture each LLM call and tool call as child spans.
|
||||
3. Emit structured logs with model, temperature, and response status.
|
||||
4. Create SLOs for success rate and median response time.
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Redact PII before exporting traces.
|
||||
- Keep a replayable request envelope for incident review.
|
||||
- Alert on abnormal token spikes and tool error bursts.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [alerting-oncall](../../observability/alerting-oncall/) - Alert workflows
|
||||
- [agent-evals](../agent-evals/) - Quality verification
|
||||
Reference in New Issue
Block a user