mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
Add 12 AI infrastructure and LLM operations skills
New skills covering hot-topic AI engineering subjects: Local AI Infrastructure: - vllm-server: High-throughput LLM serving with PagedAttention, tensor parallelism, quantization - llm-inference-scaling: KEDA-based GPU autoscaling for LLM inference on Kubernetes - rag-infrastructure: Production RAG with hybrid search, reranking, and embedding pipelines - llm-fine-tuning: QLoRA/LoRA fine-tuning with Axolotl, DeepSpeed ZeRO-3, and DPO alignment Infrastructure: - gpu-server-management: NVIDIA driver setup, MIG partitioning, DCGM monitoring - vector-database-ops: Qdrant, Weaviate, pgvector for production AI search - llm-gateway: LiteLLM-based API gateway with rate limiting, virtual keys, fallback routing DevOps/AI: - llm-cost-optimization: Model right-sizing, prompt/semantic caching, batch API, break-even analysis - llm-caching: Multi-layer exact + semantic + provider caching to cut costs 30-70% - ai-pipeline-orchestration: Prefect/Airflow/Dagster for RAG ingestion and training workflows Orchestration: - model-serving-kubernetes: KServe + Triton with canary deployments and GPU autoscaling Security: - ai-security-hardening: Prompt injection defense, PII scrubbing, model weight verification https://claude.ai/code/session_011MN1C4PrkCeg2Qmi7q1ZUe
This commit is contained in:
Claude
@@ -0,0 +1,270 @@
|
||||
---
|
||||
name: llm-inference-scaling
|
||||
description: Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# LLM Inference Scaling
|
||||
|
||||
Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- LLM API traffic is unpredictable and you need to scale up/down automatically
|
||||
- Managing a fleet of vLLM or TGI inference pods on Kubernetes
|
||||
- Reducing inference costs with spot/preemptible GPU instances
|
||||
- Implementing queue-based autoscaling for batch inference jobs
|
||||
- Building a multi-model serving platform that shares GPU resources
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes cluster with GPU nodes (NVIDIA operator installed)
|
||||
- KEDA (Kubernetes Event-Driven Autoscaler) installed
|
||||
- Prometheus with GPU metrics (`dcgm-exporter` or `gpu-operator`)
|
||||
- Helm 3+ for chart deployments
|
||||
|
||||
## GPU Node Setup
|
||||
|
||||
```bash
|
||||
# Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)
|
||||
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
|
||||
helm repo update
|
||||
|
||||
helm install gpu-operator nvidia/gpu-operator \
|
||||
--namespace gpu-operator \
|
||||
--create-namespace \
|
||||
--set driver.enabled=true \
|
||||
--set dcgm.enabled=true \
|
||||
--set devicePlugin.enabled=true
|
||||
|
||||
# Verify GPU nodes are recognized
|
||||
kubectl get nodes -l nvidia.com/gpu.present=true
|
||||
kubectl describe node <gpu-node> | grep nvidia
|
||||
```
|
||||
|
||||
## vLLM Deployment with GPU Resources
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: vllm-llama-8b
|
||||
labels:
|
||||
app: vllm
|
||||
model: llama-3.1-8b
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: vllm
|
||||
model: llama-3.1-8b
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: vllm
|
||||
model: llama-3.1-8b
|
||||
spec:
|
||||
nodeSelector:
|
||||
nvidia.com/gpu.present: "true"
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
containers:
|
||||
- name: vllm
|
||||
image: vllm/vllm-openai:latest
|
||||
args:
|
||||
- "--model"
|
||||
- "meta-llama/Llama-3.1-8B-Instruct"
|
||||
- "--tensor-parallel-size"
|
||||
- "1"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.90"
|
||||
- "--max-num-seqs"
|
||||
- "128"
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
memory: "20Gi"
|
||||
cpu: "4"
|
||||
limits:
|
||||
nvidia.com/gpu: "1"
|
||||
memory: "24Gi"
|
||||
cpu: "8"
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8000
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
env:
|
||||
- name: HUGGING_FACE_HUB_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token
|
||||
key: token
|
||||
```
|
||||
|
||||
## KEDA Autoscaling on Prometheus Metrics
|
||||
|
||||
```yaml
|
||||
apiVersion: keda.sh/v1alpha1
|
||||
kind: ScaledObject
|
||||
metadata:
|
||||
name: vllm-scaledobject
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
name: vllm-llama-8b
|
||||
minReplicaCount: 1
|
||||
maxReplicaCount: 8
|
||||
cooldownPeriod: 300 # 5 min before scale-down
|
||||
pollingInterval: 15
|
||||
triggers:
|
||||
- type: prometheus
|
||||
metadata:
|
||||
serverAddress: http://prometheus-server.monitoring:9090
|
||||
metricName: vllm_num_requests_waiting
|
||||
threshold: "10" # scale up if >10 requests waiting
|
||||
query: |
|
||||
sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
|
||||
- type: prometheus
|
||||
metadata:
|
||||
serverAddress: http://prometheus-server.monitoring:9090
|
||||
metricName: vllm_gpu_cache_usage
|
||||
threshold: "0.8" # scale up if KV cache >80% full
|
||||
query: |
|
||||
avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})
|
||||
```
|
||||
|
||||
## Queue-Based Scaling (Redis + KEDA)
|
||||
|
||||
```yaml
|
||||
# ScaledJob for async batch inference
|
||||
apiVersion: keda.sh/v1alpha1
|
||||
kind: ScaledJob
|
||||
metadata:
|
||||
name: llm-batch-inference
|
||||
spec:
|
||||
jobTargetRef:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: inference-worker
|
||||
image: myapp/inference-worker:latest
|
||||
env:
|
||||
- name: REDIS_URL
|
||||
value: redis://redis:6379
|
||||
- name: QUEUE_NAME
|
||||
value: inference-jobs
|
||||
restartPolicy: OnFailure
|
||||
minReplicaCount: 0
|
||||
maxReplicaCount: 20
|
||||
pollingInterval: 5
|
||||
successfulJobsHistoryLimit: 3
|
||||
triggers:
|
||||
- type: redis
|
||||
metadata:
|
||||
address: redis:6379
|
||||
listName: inference-jobs
|
||||
listLength: "5" # 1 worker per 5 queued jobs
|
||||
```
|
||||
|
||||
## Spot Instance Strategy
|
||||
|
||||
```yaml
|
||||
# Mixed node pool: on-demand + spot GPUs
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: cluster-autoscaler-priority-config
|
||||
data:
|
||||
priorities: |
|
||||
10: # low priority = prefer
|
||||
- .*spot.*
|
||||
50:
|
||||
- .*on-demand.*
|
||||
---
|
||||
# Node affinity for spot with on-demand fallback
|
||||
spec:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 80
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: node.kubernetes.io/lifecycle
|
||||
operator: In
|
||||
values: [spot]
|
||||
- weight: 20
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: node.kubernetes.io/lifecycle
|
||||
operator: In
|
||||
values: [on-demand]
|
||||
```
|
||||
|
||||
## Cluster Autoscaler for GPU Nodes
|
||||
|
||||
```bash
|
||||
# AWS EKS — enable cluster autoscaler for GPU node group
|
||||
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
|
||||
--namespace kube-system \
|
||||
--set autoDiscovery.clusterName=my-cluster \
|
||||
--set awsRegion=us-east-1 \
|
||||
--set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole \
|
||||
--set extraArgs.skip-nodes-with-local-storage=false \
|
||||
--set extraArgs.expander=least-waste
|
||||
|
||||
# Annotate GPU node group for autoscaler
|
||||
kubectl annotate node <node> \
|
||||
cluster-autoscaler.kubernetes.io/safe-to-evict="false"
|
||||
```
|
||||
|
||||
## Scaling Metrics to Monitor
|
||||
|
||||
```bash
|
||||
# Prometheus queries for scaling decisions
|
||||
# Requests waiting in vLLM queue
|
||||
sum(vllm:num_requests_waiting) by (model)
|
||||
|
||||
# GPU KV cache utilization (>80% = bottleneck)
|
||||
avg(vllm:gpu_cache_usage_perc) by (pod)
|
||||
|
||||
# Tokens per second throughput
|
||||
sum(rate(vllm:generation_tokens_total[5m])) by (model)
|
||||
|
||||
# P99 time-to-first-token
|
||||
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| Pods stuck in `Pending` | No GPU nodes available | Check cluster autoscaler logs; verify node group limits |
|
||||
| Scale-up too slow | Cluster autoscaler delay + model load time | Pre-warm replicas; increase `minReplicaCount` |
|
||||
| GPU fragmentation | Multiple small models on large GPUs | Use MIG partitioning or consolidate model sizes |
|
||||
| Spot eviction causes errors | Spot instance reclamation | Add `PodDisruptionBudget`; use graceful shutdown |
|
||||
| KEDA not scaling | Prometheus query returns no data | Test query in Prometheus UI first |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Set `minReplicaCount: 1` to avoid cold starts; scale to 0 only for batch jobs.
|
||||
- Use `PodDisruptionBudget` with `minAvailable: 1` to survive spot evictions.
|
||||
- Pre-pull model weights into a shared PVC to speed up pod startup by 5–10×.
|
||||
- Separate model families across node pools (A10G for 7B, A100 for 70B).
|
||||
- Use Kubernetes VPA for CPU/memory right-sizing alongside KEDA for replica count.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [vllm-server](../vllm-server/) - vLLM configuration and tuning
|
||||
- [gpu-server-management](../../servers/gpu-server-management/) - GPU node setup
|
||||
- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - KServe
|
||||
- [kubernetes-ops](../../../devops/orchestration/kubernetes-ops/) - Core Kubernetes
|
||||
- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost strategies
|
||||
Reference in New Issue
Block a user