mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
Add 12 AI infrastructure and LLM operations skills
New skills covering hot-topic AI engineering subjects: Local AI Infrastructure: - vllm-server: High-throughput LLM serving with PagedAttention, tensor parallelism, quantization - llm-inference-scaling: KEDA-based GPU autoscaling for LLM inference on Kubernetes - rag-infrastructure: Production RAG with hybrid search, reranking, and embedding pipelines - llm-fine-tuning: QLoRA/LoRA fine-tuning with Axolotl, DeepSpeed ZeRO-3, and DPO alignment Infrastructure: - gpu-server-management: NVIDIA driver setup, MIG partitioning, DCGM monitoring - vector-database-ops: Qdrant, Weaviate, pgvector for production AI search - llm-gateway: LiteLLM-based API gateway with rate limiting, virtual keys, fallback routing DevOps/AI: - llm-cost-optimization: Model right-sizing, prompt/semantic caching, batch API, break-even analysis - llm-caching: Multi-layer exact + semantic + provider caching to cut costs 30-70% - ai-pipeline-orchestration: Prefect/Airflow/Dagster for RAG ingestion and training workflows Orchestration: - model-serving-kubernetes: KServe + Triton with canary deployments and GPU autoscaling Security: - ai-security-hardening: Prompt injection defense, PII scrubbing, model weight verification https://claude.ai/code/session_011MN1C4PrkCeg2Qmi7q1ZUe
This commit is contained in:
Claude
@@ -0,0 +1,312 @@
|
||||
---
|
||||
name: llm-fine-tuning
|
||||
description: Set up infrastructure for fine-tuning LLMs with QLoRA, LoRA, and full fine-tuning using Hugging Face TRL, Axolotl, and distributed training with DeepSpeed or FSDP. Covers dataset prep, training runs, and model export.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# LLM Fine-Tuning Infrastructure
|
||||
|
||||
Train and fine-tune open-source LLMs efficiently — from LoRA on a single GPU to distributed full fine-tuning across multi-node clusters.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Fine-tuning an LLM on domain-specific data (legal, medical, code, support)
|
||||
- Running QLoRA to fine-tune 70B models on consumer GPUs
|
||||
- Setting up distributed training with DeepSpeed or FSDP
|
||||
- Exporting fine-tuned adapters for production serving
|
||||
- Implementing RLHF, DPO, or instruction tuning pipelines
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- NVIDIA GPU(s) with 24GB+ VRAM (RTX 4090 / A100 / H100)
|
||||
- CUDA 12.1+ and `nvidia-smi` working
|
||||
- Python 3.10+ with `pip`
|
||||
- Hugging Face account and `HF_TOKEN` for gated models
|
||||
- 500GB+ disk for model weights and training data
|
||||
|
||||
## Quick Start: QLoRA Fine-Tuning
|
||||
|
||||
```bash
|
||||
pip install transformers datasets trl peft bitsandbytes accelerate
|
||||
|
||||
python - <<'EOF'
|
||||
from datasets import load_dataset
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
|
||||
from peft import LoraConfig, get_peft_model
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
import torch
|
||||
|
||||
model_id = "meta-llama/Llama-3.1-8B-Instruct"
|
||||
|
||||
# 4-bit quantization (QLoRA)
|
||||
bnb_config = BitsAndBytesConfig(
|
||||
load_in_4bit=True,
|
||||
bnb_4bit_quant_type="nf4",
|
||||
bnb_4bit_compute_dtype=torch.bfloat16,
|
||||
bnb_4bit_use_double_quant=True,
|
||||
)
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id, quantization_config=bnb_config, device_map="auto"
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
|
||||
# LoRA configuration
|
||||
peft_config = LoraConfig(
|
||||
r=16, # rank
|
||||
lora_alpha=32,
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
|
||||
"gate_proj", "up_proj", "down_proj"],
|
||||
lora_dropout=0.05,
|
||||
bias="none",
|
||||
task_type="CAUSAL_LM",
|
||||
)
|
||||
|
||||
dataset = load_dataset("your-org/your-dataset", split="train")
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model=model,
|
||||
args=SFTConfig(
|
||||
output_dir="./output",
|
||||
num_train_epochs=3,
|
||||
per_device_train_batch_size=2,
|
||||
gradient_accumulation_steps=8,
|
||||
learning_rate=2e-4,
|
||||
bf16=True,
|
||||
logging_steps=10,
|
||||
save_strategy="epoch",
|
||||
report_to="wandb",
|
||||
),
|
||||
train_dataset=dataset,
|
||||
peft_config=peft_config,
|
||||
processing_class=tokenizer,
|
||||
)
|
||||
trainer.train()
|
||||
trainer.save_model("./fine-tuned-model")
|
||||
EOF
|
||||
```
|
||||
|
||||
## Axolotl (Production Fine-Tuning Framework)
|
||||
|
||||
```yaml
|
||||
# config.yaml — Axolotl QLoRA config for Llama 3.1
|
||||
base_model: meta-llama/Llama-3.1-8B-Instruct
|
||||
model_type: LlamaForCausalLM
|
||||
tokenizer_type: PreTrainedTokenizerFast
|
||||
|
||||
load_in_4bit: true
|
||||
adapter: qlora
|
||||
lora_r: 32
|
||||
lora_alpha: 64
|
||||
lora_dropout: 0.05
|
||||
lora_target_modules:
|
||||
- q_proj
|
||||
- k_proj
|
||||
- v_proj
|
||||
- o_proj
|
||||
- gate_proj
|
||||
- up_proj
|
||||
- down_proj
|
||||
|
||||
datasets:
|
||||
- path: your-org/your-dataset
|
||||
type: alpaca # or sharegpt, chat_template, etc.
|
||||
|
||||
dataset_prepared_path: ./prepared-data
|
||||
val_set_size: 0.05
|
||||
output_dir: ./output
|
||||
|
||||
sequence_len: 4096
|
||||
sample_packing: true # pack multiple short samples for efficiency
|
||||
|
||||
micro_batch_size: 2
|
||||
gradient_accumulation_steps: 8
|
||||
num_epochs: 3
|
||||
learning_rate: 2e-4
|
||||
optimizer: adamw_bnb_8bit
|
||||
lr_scheduler: cosine
|
||||
warmup_ratio: 0.05
|
||||
|
||||
bf16: true
|
||||
flash_attention: true
|
||||
|
||||
logging_steps: 10
|
||||
eval_steps: 100
|
||||
save_steps: 200
|
||||
wandb_project: my-fine-tune
|
||||
```
|
||||
|
||||
```bash
|
||||
# Run with Axolotl
|
||||
pip install axolotl[flash-attn,deepspeed]
|
||||
accelerate launch -m axolotl.cli.train config.yaml
|
||||
```
|
||||
|
||||
## Distributed Training with DeepSpeed
|
||||
|
||||
```json
|
||||
// deepspeed_zero3.json — ZeRO Stage 3 (split optimizer + gradients + params)
|
||||
{
|
||||
"zero_optimization": {
|
||||
"stage": 3,
|
||||
"offload_optimizer": {"device": "cpu", "pin_memory": true},
|
||||
"offload_param": {"device": "cpu", "pin_memory": true},
|
||||
"overlap_comm": true,
|
||||
"contiguous_gradients": true,
|
||||
"sub_group_size": 1e9,
|
||||
"reduce_bucket_size": "auto",
|
||||
"stage3_prefetch_bucket_size": "auto",
|
||||
"stage3_param_persistence_threshold": "auto",
|
||||
"stage3_max_live_parameters": 1e9,
|
||||
"stage3_max_reuse_distance": 1e9,
|
||||
"gather_16bit_weights_on_model_save": true
|
||||
},
|
||||
"bf16": {"enabled": true},
|
||||
"gradient_clipping": 1.0,
|
||||
"train_batch_size": "auto",
|
||||
"train_micro_batch_size_per_gpu": "auto"
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
# Launch 4-GPU DeepSpeed training
|
||||
deepspeed --num_gpus=4 train.py \
|
||||
--deepspeed deepspeed_zero3.json \
|
||||
--model_name meta-llama/Llama-3.1-70B-Instruct \
|
||||
--output_dir ./output
|
||||
```
|
||||
|
||||
## DPO / RLHF Alignment
|
||||
|
||||
```python
|
||||
from trl import DPOTrainer, DPOConfig
|
||||
from datasets import load_dataset
|
||||
|
||||
# Dataset format: {"prompt": ..., "chosen": ..., "rejected": ...}
|
||||
dataset = load_dataset("your-org/preference-data")
|
||||
|
||||
trainer = DPOTrainer(
|
||||
model=model,
|
||||
ref_model=None, # None = implicit reference with peft
|
||||
args=DPOConfig(
|
||||
output_dir="./dpo-output",
|
||||
beta=0.1, # KL divergence weight
|
||||
num_train_epochs=1,
|
||||
per_device_train_batch_size=1,
|
||||
gradient_accumulation_steps=16,
|
||||
learning_rate=5e-7,
|
||||
bf16=True,
|
||||
),
|
||||
train_dataset=dataset["train"],
|
||||
peft_config=peft_config,
|
||||
processing_class=tokenizer,
|
||||
)
|
||||
trainer.train()
|
||||
```
|
||||
|
||||
## Merging LoRA Adapters for Deployment
|
||||
|
||||
```python
|
||||
from peft import PeftModel
|
||||
from transformers import AutoModelForCausalLM
|
||||
|
||||
# Load base model in full precision
|
||||
base_model = AutoModelForCausalLM.from_pretrained(
|
||||
"meta-llama/Llama-3.1-8B-Instruct",
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="cpu",
|
||||
)
|
||||
|
||||
# Load and merge LoRA adapter
|
||||
model = PeftModel.from_pretrained(base_model, "./fine-tuned-model")
|
||||
merged_model = model.merge_and_unload()
|
||||
|
||||
# Save merged model (ready for vLLM serving)
|
||||
merged_model.save_pretrained("./merged-model", safe_serialization=True)
|
||||
tokenizer.save_pretrained("./merged-model")
|
||||
|
||||
# Push to Hugging Face Hub
|
||||
merged_model.push_to_hub("your-org/your-fine-tuned-model")
|
||||
```
|
||||
|
||||
## Kubernetes Training Job
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: llm-fine-tune
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
nodeSelector:
|
||||
nvidia.com/gpu.product: A100-SXM4-80GB
|
||||
containers:
|
||||
- name: trainer
|
||||
image: nvcr.io/nvidia/pytorch:24.05-py3
|
||||
command: ["accelerate", "launch", "-m", "axolotl.cli.train", "/config/config.yaml"]
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: "4"
|
||||
memory: "320Gi"
|
||||
requests:
|
||||
nvidia.com/gpu: "4"
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: model-cache
|
||||
mountPath: /root/.cache/huggingface
|
||||
- name: output
|
||||
mountPath: /output
|
||||
env:
|
||||
- name: HUGGING_FACE_HUB_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token
|
||||
key: token
|
||||
- name: WANDB_API_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: wandb-token
|
||||
key: key
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: axolotl-config
|
||||
- name: model-cache
|
||||
persistentVolumeClaim:
|
||||
claimName: model-cache-pvc
|
||||
- name: output
|
||||
persistentVolumeClaim:
|
||||
claimName: training-output-pvc
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `CUDA out of memory` | Batch too large | Reduce `micro_batch_size`; increase `gradient_accumulation_steps` |
|
||||
| Training loss NaN | Learning rate too high | Lower LR to `1e-4` or `5e-5`; add warmup |
|
||||
| Slow training | No Flash Attention | Install `flash-attn`; enable `flash_attention: true` |
|
||||
| Poor fine-tune quality | Bad data formatting | Validate dataset format; check `sample_packing` compatibility |
|
||||
| Adapter merge errors | Mixed quantization | Merge in bf16 on CPU, not in 4-bit |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use Flash Attention 2 — it's 2–4× faster and uses less memory.
|
||||
- Monitor training loss/eval loss via W&B or MLflow; overfit = more dropout or less data.
|
||||
- Validate with a held-out eval set (5–10%); MMLU or custom evals for quality gates.
|
||||
- Start with LoRA r=16 before increasing — higher rank = more parameters, diminishing returns.
|
||||
- Use `sample_packing` in Axolotl to maximize GPU utilization on short sequences.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [vllm-server](../vllm-server/) - Serve fine-tuned models
|
||||
- [gpu-server-management](../../servers/gpu-server-management/) - GPU setup
|
||||
- [llm-inference-scaling](../llm-inference-scaling/) - Deploy at scale
|
||||
- [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Training pipelines
|
||||
@@ -0,0 +1,270 @@
|
||||
---
|
||||
name: llm-inference-scaling
|
||||
description: Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# LLM Inference Scaling
|
||||
|
||||
Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- LLM API traffic is unpredictable and you need to scale up/down automatically
|
||||
- Managing a fleet of vLLM or TGI inference pods on Kubernetes
|
||||
- Reducing inference costs with spot/preemptible GPU instances
|
||||
- Implementing queue-based autoscaling for batch inference jobs
|
||||
- Building a multi-model serving platform that shares GPU resources
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes cluster with GPU nodes (NVIDIA operator installed)
|
||||
- KEDA (Kubernetes Event-Driven Autoscaler) installed
|
||||
- Prometheus with GPU metrics (`dcgm-exporter` or `gpu-operator`)
|
||||
- Helm 3+ for chart deployments
|
||||
|
||||
## GPU Node Setup
|
||||
|
||||
```bash
|
||||
# Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)
|
||||
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
|
||||
helm repo update
|
||||
|
||||
helm install gpu-operator nvidia/gpu-operator \
|
||||
--namespace gpu-operator \
|
||||
--create-namespace \
|
||||
--set driver.enabled=true \
|
||||
--set dcgm.enabled=true \
|
||||
--set devicePlugin.enabled=true
|
||||
|
||||
# Verify GPU nodes are recognized
|
||||
kubectl get nodes -l nvidia.com/gpu.present=true
|
||||
kubectl describe node <gpu-node> | grep nvidia
|
||||
```
|
||||
|
||||
## vLLM Deployment with GPU Resources
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: vllm-llama-8b
|
||||
labels:
|
||||
app: vllm
|
||||
model: llama-3.1-8b
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: vllm
|
||||
model: llama-3.1-8b
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: vllm
|
||||
model: llama-3.1-8b
|
||||
spec:
|
||||
nodeSelector:
|
||||
nvidia.com/gpu.present: "true"
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
containers:
|
||||
- name: vllm
|
||||
image: vllm/vllm-openai:latest
|
||||
args:
|
||||
- "--model"
|
||||
- "meta-llama/Llama-3.1-8B-Instruct"
|
||||
- "--tensor-parallel-size"
|
||||
- "1"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.90"
|
||||
- "--max-num-seqs"
|
||||
- "128"
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
memory: "20Gi"
|
||||
cpu: "4"
|
||||
limits:
|
||||
nvidia.com/gpu: "1"
|
||||
memory: "24Gi"
|
||||
cpu: "8"
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8000
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
env:
|
||||
- name: HUGGING_FACE_HUB_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token
|
||||
key: token
|
||||
```
|
||||
|
||||
## KEDA Autoscaling on Prometheus Metrics
|
||||
|
||||
```yaml
|
||||
apiVersion: keda.sh/v1alpha1
|
||||
kind: ScaledObject
|
||||
metadata:
|
||||
name: vllm-scaledobject
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
name: vllm-llama-8b
|
||||
minReplicaCount: 1
|
||||
maxReplicaCount: 8
|
||||
cooldownPeriod: 300 # 5 min before scale-down
|
||||
pollingInterval: 15
|
||||
triggers:
|
||||
- type: prometheus
|
||||
metadata:
|
||||
serverAddress: http://prometheus-server.monitoring:9090
|
||||
metricName: vllm_num_requests_waiting
|
||||
threshold: "10" # scale up if >10 requests waiting
|
||||
query: |
|
||||
sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
|
||||
- type: prometheus
|
||||
metadata:
|
||||
serverAddress: http://prometheus-server.monitoring:9090
|
||||
metricName: vllm_gpu_cache_usage
|
||||
threshold: "0.8" # scale up if KV cache >80% full
|
||||
query: |
|
||||
avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})
|
||||
```
|
||||
|
||||
## Queue-Based Scaling (Redis + KEDA)
|
||||
|
||||
```yaml
|
||||
# ScaledJob for async batch inference
|
||||
apiVersion: keda.sh/v1alpha1
|
||||
kind: ScaledJob
|
||||
metadata:
|
||||
name: llm-batch-inference
|
||||
spec:
|
||||
jobTargetRef:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: inference-worker
|
||||
image: myapp/inference-worker:latest
|
||||
env:
|
||||
- name: REDIS_URL
|
||||
value: redis://redis:6379
|
||||
- name: QUEUE_NAME
|
||||
value: inference-jobs
|
||||
restartPolicy: OnFailure
|
||||
minReplicaCount: 0
|
||||
maxReplicaCount: 20
|
||||
pollingInterval: 5
|
||||
successfulJobsHistoryLimit: 3
|
||||
triggers:
|
||||
- type: redis
|
||||
metadata:
|
||||
address: redis:6379
|
||||
listName: inference-jobs
|
||||
listLength: "5" # 1 worker per 5 queued jobs
|
||||
```
|
||||
|
||||
## Spot Instance Strategy
|
||||
|
||||
```yaml
|
||||
# Mixed node pool: on-demand + spot GPUs
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: cluster-autoscaler-priority-config
|
||||
data:
|
||||
priorities: |
|
||||
10: # low priority = prefer
|
||||
- .*spot.*
|
||||
50:
|
||||
- .*on-demand.*
|
||||
---
|
||||
# Node affinity for spot with on-demand fallback
|
||||
spec:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 80
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: node.kubernetes.io/lifecycle
|
||||
operator: In
|
||||
values: [spot]
|
||||
- weight: 20
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: node.kubernetes.io/lifecycle
|
||||
operator: In
|
||||
values: [on-demand]
|
||||
```
|
||||
|
||||
## Cluster Autoscaler for GPU Nodes
|
||||
|
||||
```bash
|
||||
# AWS EKS — enable cluster autoscaler for GPU node group
|
||||
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
|
||||
--namespace kube-system \
|
||||
--set autoDiscovery.clusterName=my-cluster \
|
||||
--set awsRegion=us-east-1 \
|
||||
--set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole \
|
||||
--set extraArgs.skip-nodes-with-local-storage=false \
|
||||
--set extraArgs.expander=least-waste
|
||||
|
||||
# Annotate GPU node group for autoscaler
|
||||
kubectl annotate node <node> \
|
||||
cluster-autoscaler.kubernetes.io/safe-to-evict="false"
|
||||
```
|
||||
|
||||
## Scaling Metrics to Monitor
|
||||
|
||||
```bash
|
||||
# Prometheus queries for scaling decisions
|
||||
# Requests waiting in vLLM queue
|
||||
sum(vllm:num_requests_waiting) by (model)
|
||||
|
||||
# GPU KV cache utilization (>80% = bottleneck)
|
||||
avg(vllm:gpu_cache_usage_perc) by (pod)
|
||||
|
||||
# Tokens per second throughput
|
||||
sum(rate(vllm:generation_tokens_total[5m])) by (model)
|
||||
|
||||
# P99 time-to-first-token
|
||||
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| Pods stuck in `Pending` | No GPU nodes available | Check cluster autoscaler logs; verify node group limits |
|
||||
| Scale-up too slow | Cluster autoscaler delay + model load time | Pre-warm replicas; increase `minReplicaCount` |
|
||||
| GPU fragmentation | Multiple small models on large GPUs | Use MIG partitioning or consolidate model sizes |
|
||||
| Spot eviction causes errors | Spot instance reclamation | Add `PodDisruptionBudget`; use graceful shutdown |
|
||||
| KEDA not scaling | Prometheus query returns no data | Test query in Prometheus UI first |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Set `minReplicaCount: 1` to avoid cold starts; scale to 0 only for batch jobs.
|
||||
- Use `PodDisruptionBudget` with `minAvailable: 1` to survive spot evictions.
|
||||
- Pre-pull model weights into a shared PVC to speed up pod startup by 5–10×.
|
||||
- Separate model families across node pools (A10G for 7B, A100 for 70B).
|
||||
- Use Kubernetes VPA for CPU/memory right-sizing alongside KEDA for replica count.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [vllm-server](../vllm-server/) - vLLM configuration and tuning
|
||||
- [gpu-server-management](../../servers/gpu-server-management/) - GPU node setup
|
||||
- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - KServe
|
||||
- [kubernetes-ops](../../../devops/orchestration/kubernetes-ops/) - Core Kubernetes
|
||||
- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost strategies
|
||||
@@ -0,0 +1,253 @@
|
||||
---
|
||||
name: rag-infrastructure
|
||||
description: Build and operate Retrieval-Augmented Generation (RAG) infrastructure with vector stores, embedding pipelines, and hybrid search. Covers ingestion, chunking strategies, reranking, and production deployment patterns.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# RAG Infrastructure
|
||||
|
||||
Production infrastructure for Retrieval-Augmented Generation: ingest documents, generate embeddings, store in vector databases, and serve grounded LLM responses.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Building a knowledge base Q&A system over internal documents
|
||||
- Implementing semantic search over large document collections
|
||||
- Reducing LLM hallucinations with retrieved context
|
||||
- Setting up embedding pipelines and vector store infrastructure
|
||||
- Deploying hybrid search (dense + sparse/BM25)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.10+ with `pip`
|
||||
- A vector database (Qdrant, Weaviate, Pinecone, or pgvector)
|
||||
- An embedding model (OpenAI, Cohere, or local via `sentence-transformers`)
|
||||
- An LLM endpoint (OpenAI API or self-hosted vLLM)
|
||||
- Docker for local vector DB deployment
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
Documents → Chunker → Embedder → Vector Store
|
||||
↓
|
||||
User Query → Embedder → Vector Store (search) → Reranker → LLM → Answer
|
||||
```
|
||||
|
||||
## Embedding Pipeline
|
||||
|
||||
```python
|
||||
from sentence_transformers import SentenceTransformer
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.models import Distance, VectorParams, PointStruct
|
||||
import uuid
|
||||
|
||||
# Local embedding model (no API cost)
|
||||
model = SentenceTransformer("BAAI/bge-large-en-v1.5")
|
||||
|
||||
# Connect to Qdrant
|
||||
client = QdrantClient("http://localhost:6333")
|
||||
|
||||
# Create collection
|
||||
client.create_collection(
|
||||
collection_name="knowledge-base",
|
||||
vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
|
||||
)
|
||||
|
||||
def ingest_documents(docs: list[dict]):
|
||||
"""Chunk, embed, and upsert documents."""
|
||||
points = []
|
||||
for doc in docs:
|
||||
chunks = chunk_text(doc["text"], chunk_size=512, overlap=50)
|
||||
embeddings = model.encode(chunks, batch_size=32, show_progress_bar=True)
|
||||
for chunk, embedding in zip(chunks, embeddings):
|
||||
points.append(PointStruct(
|
||||
id=str(uuid.uuid4()),
|
||||
vector=embedding.tolist(),
|
||||
payload={"text": chunk, "source": doc["source"], "title": doc["title"]},
|
||||
))
|
||||
client.upsert(collection_name="knowledge-base", points=points)
|
||||
print(f"Ingested {len(points)} chunks")
|
||||
```
|
||||
|
||||
## Chunking Strategies
|
||||
|
||||
```python
|
||||
from langchain.text_splitter import RecursiveCharacterTextSplitter
|
||||
|
||||
def chunk_text(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]:
|
||||
"""Recursive character splitter — best general-purpose strategy."""
|
||||
splitter = RecursiveCharacterTextSplitter(
|
||||
chunk_size=chunk_size,
|
||||
chunk_overlap=overlap,
|
||||
separators=["\n\n", "\n", ". ", " ", ""],
|
||||
)
|
||||
return splitter.split_text(text)
|
||||
|
||||
# For code/markdown — use language-aware splitter
|
||||
from langchain.text_splitter import MarkdownHeaderTextSplitter
|
||||
|
||||
headers = [("#", "H1"), ("##", "H2"), ("###", "H3")]
|
||||
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
|
||||
```
|
||||
|
||||
## Hybrid Search (Dense + Sparse)
|
||||
|
||||
```python
|
||||
from qdrant_client.models import SparseVector, SparseVectorParams, NamedSparseVector
|
||||
from fastembed import SparseTextEmbedding
|
||||
|
||||
# Qdrant hybrid collection (dense + BM25 sparse)
|
||||
client.create_collection(
|
||||
collection_name="hybrid-kb",
|
||||
vectors_config={"dense": VectorParams(size=1024, distance=Distance.COSINE)},
|
||||
sparse_vectors_config={"sparse": SparseVectorParams()},
|
||||
)
|
||||
|
||||
sparse_model = SparseTextEmbedding("prithivida/Splade_PP_en_v1")
|
||||
|
||||
def hybrid_search(query: str, top_k: int = 10) -> list[dict]:
|
||||
dense_vec = model.encode(query).tolist()
|
||||
sparse_vec = list(sparse_model.embed(query))[0]
|
||||
|
||||
results = client.query_points(
|
||||
collection_name="hybrid-kb",
|
||||
prefetch=[
|
||||
{"query": dense_vec, "using": "dense", "limit": 20},
|
||||
{"query": SparseVector(indices=sparse_vec.indices.tolist(),
|
||||
values=sparse_vec.values.tolist()),
|
||||
"using": "sparse", "limit": 20},
|
||||
],
|
||||
query={"fusion": "rrf"}, # Reciprocal Rank Fusion
|
||||
limit=top_k,
|
||||
)
|
||||
return [{"text": p.payload["text"], "score": p.score} for p in results.points]
|
||||
```
|
||||
|
||||
## Reranking
|
||||
|
||||
```python
|
||||
import cohere
|
||||
|
||||
co = cohere.Client("your-api-key")
|
||||
|
||||
def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
|
||||
"""Rerank retrieved chunks for relevance (improves RAG quality ~20-30%)."""
|
||||
response = co.rerank(
|
||||
model="rerank-english-v3.0",
|
||||
query=query,
|
||||
documents=candidates,
|
||||
top_n=top_n,
|
||||
)
|
||||
return [candidates[r.index] for r in response.results]
|
||||
|
||||
# Alternative: local reranker (no API cost)
|
||||
from sentence_transformers import CrossEncoder
|
||||
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
|
||||
|
||||
def local_rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
|
||||
pairs = [[query, c] for c in candidates]
|
||||
scores = reranker.predict(pairs)
|
||||
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
|
||||
return [text for text, _ in ranked[:top_n]]
|
||||
```
|
||||
|
||||
## RAG Query Pipeline
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
llm = OpenAI(base_url="http://localhost:8000/v1", api_key="your-key")
|
||||
|
||||
def rag_query(user_question: str) -> str:
|
||||
# 1. Retrieve
|
||||
candidates = hybrid_search(user_question, top_k=20)
|
||||
texts = [c["text"] for c in candidates]
|
||||
|
||||
# 2. Rerank
|
||||
top_chunks = local_rerank(user_question, texts, top_n=5)
|
||||
|
||||
# 3. Generate
|
||||
context = "\n\n---\n\n".join(top_chunks)
|
||||
response = llm.chat.completions.create(
|
||||
model="meta-llama/Llama-3.1-8B-Instruct",
|
||||
messages=[
|
||||
{"role": "system", "content": (
|
||||
"Answer the question using only the provided context. "
|
||||
"If the answer isn't in the context, say so.\n\nContext:\n" + context
|
||||
)},
|
||||
{"role": "user", "content": user_question},
|
||||
],
|
||||
temperature=0.1,
|
||||
max_tokens=1024,
|
||||
)
|
||||
return response.choices[0].message.content
|
||||
```
|
||||
|
||||
## Docker Compose: Full RAG Stack
|
||||
|
||||
```yaml
|
||||
services:
|
||||
qdrant:
|
||||
image: qdrant/qdrant:latest
|
||||
volumes:
|
||||
- qdrant-data:/qdrant/storage
|
||||
ports:
|
||||
- "6333:6333"
|
||||
restart: unless-stopped
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
restart: unless-stopped
|
||||
|
||||
ingestion-worker:
|
||||
build: ./ingestion
|
||||
environment:
|
||||
- QDRANT_URL=http://qdrant:6333
|
||||
- REDIS_URL=redis://redis:6379
|
||||
depends_on: [qdrant, redis]
|
||||
restart: unless-stopped
|
||||
|
||||
rag-api:
|
||||
build: ./api
|
||||
ports:
|
||||
- "8080:8080"
|
||||
environment:
|
||||
- QDRANT_URL=http://qdrant:6333
|
||||
- LLM_BASE_URL=http://vllm:8000/v1
|
||||
depends_on: [qdrant]
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
qdrant-data:
|
||||
redis-data:
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| Poor retrieval quality | Chunk size too large | Try 256–512 tokens; overlap 10–15% |
|
||||
| LLM ignores retrieved context | Context too long | Rerank and keep top 3–5 chunks |
|
||||
| Slow ingestion | Sequential embedding | Use `batch_size=64` and async upserts |
|
||||
| Stale documents | No re-ingestion pipeline | Track `doc_hash`; re-embed on change |
|
||||
| High embedding costs | All chunks re-embedded | Cache embeddings with hash-based dedup |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use `BAAI/bge-large-en-v1.5` or `nomic-embed-text` for strong free embeddings.
|
||||
- Always rerank before passing to LLM — 5 precise chunks beat 20 noisy ones.
|
||||
- Store source metadata (URL, page, section) in vector payloads for citations.
|
||||
- Use namespace/tenant isolation in the vector store for multi-tenant RAG.
|
||||
- Evaluate with RAGAS metrics: faithfulness, answer relevancy, context precision.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [vector-database-ops](../../databases/vector-database-ops/) - Qdrant/Weaviate management
|
||||
- [vllm-server](../vllm-server/) - Self-hosted LLM endpoint
|
||||
- [ollama-stack](../ollama-stack/) - Local LLM for development
|
||||
- [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Ingestion pipelines
|
||||
@@ -0,0 +1,219 @@
|
||||
---
|
||||
name: vllm-server
|
||||
description: Deploy and manage vLLM for high-throughput LLM inference. Configure continuous batching, tensor parallelism, quantization, and OpenAI-compatible API endpoints for production LLM serving.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# vLLM Server Management
|
||||
|
||||
Deploy production-grade LLM inference servers with vLLM — the fastest open-source LLM serving engine with PagedAttention and continuous batching.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Serving open-source LLMs (Llama, Mistral, Qwen, Gemma) at scale
|
||||
- Building an OpenAI-compatible API endpoint for self-hosted models
|
||||
- Optimizing LLM throughput and latency for production traffic
|
||||
- Running multi-GPU inference with tensor or pipeline parallelism
|
||||
- Deploying quantized models to reduce GPU memory requirements
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- NVIDIA GPU(s) with CUDA 12.1+ (A100/H100 recommended for production)
|
||||
- Docker or Python 3.9+ with pip
|
||||
- 40GB+ VRAM for 70B models; 8GB+ for 7B models
|
||||
- `nvidia-container-toolkit` for Docker GPU passthrough
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Install vLLM
|
||||
pip install vllm
|
||||
|
||||
# Serve a model (OpenAI-compatible API)
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--api-key your-secret-key
|
||||
|
||||
# Test the endpoint
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer your-secret-key" \
|
||||
-d '{
|
||||
"model": "meta-llama/Llama-3.1-8B-Instruct",
|
||||
"messages": [{"role": "user", "content": "Hello!"}]
|
||||
}'
|
||||
```
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
```bash
|
||||
docker run --runtime nvidia --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-p 8000:8000 \
|
||||
--ipc=host \
|
||||
vllm/vllm-openai:latest \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--api-key your-secret-key
|
||||
```
|
||||
|
||||
## Docker Compose (Production)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
vllm:
|
||||
image: vllm/vllm-openai:latest
|
||||
runtime: nvidia
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=all
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
|
||||
volumes:
|
||||
- model-cache:/root/.cache/huggingface
|
||||
ports:
|
||||
- "8000:8000"
|
||||
ipc: host
|
||||
command: >
|
||||
--model meta-llama/Llama-3.1-70B-Instruct
|
||||
--tensor-parallel-size 2
|
||||
--max-model-len 32768
|
||||
--gpu-memory-utilization 0.90
|
||||
--api-key ${VLLM_API_KEY}
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
model-cache:
|
||||
```
|
||||
|
||||
## Key Configuration Options
|
||||
|
||||
### Multi-GPU Tensor Parallelism
|
||||
|
||||
```bash
|
||||
# Split one model across 4 GPUs
|
||||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.90
|
||||
```
|
||||
|
||||
### Quantization (Lower VRAM)
|
||||
|
||||
```bash
|
||||
# AWQ quantization (70B on 2x A100 40GB)
|
||||
vllm serve casperhansen/llama-3-70b-instruct-awq \
|
||||
--quantization awq \
|
||||
--tensor-parallel-size 2
|
||||
|
||||
# GPTQ quantization
|
||||
vllm serve TheBloke/Llama-2-70B-Chat-GPTQ \
|
||||
--quantization gptq
|
||||
|
||||
# FP8 (H100 NVL native)
|
||||
vllm serve meta-llama/Llama-3.1-405B-Instruct \
|
||||
--quantization fp8 \
|
||||
--tensor-parallel-size 8
|
||||
```
|
||||
|
||||
### Structured Output & Tools
|
||||
|
||||
```bash
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser llama3_json \
|
||||
--guided-decoding-backend outlines
|
||||
```
|
||||
|
||||
### LoRA Adapters
|
||||
|
||||
```bash
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--enable-lora \
|
||||
--lora-modules sql-lora=/path/to/sql-lora \
|
||||
code-lora=/path/to/code-lora \
|
||||
--max-lora-rank 64
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
```bash
|
||||
# Maximize throughput for batch workloads
|
||||
vllm serve <model> \
|
||||
--max-num-seqs 256 \ # max concurrent sequences
|
||||
--max-num-batched-tokens 8192 \ # tokens per batch
|
||||
--gpu-memory-utilization 0.95 \ # use 95% VRAM
|
||||
--swap-space 4 # CPU swap (GiB)
|
||||
|
||||
# Minimize latency for interactive use
|
||||
vllm serve <model> \
|
||||
--max-num-seqs 32 \
|
||||
--enforce-eager # disable CUDA graph capture
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
# Install benchmark tool
|
||||
pip install vllm
|
||||
|
||||
# Run throughput benchmark
|
||||
python -m vllm.entrypoints.openai.run_batch \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--input-file prompts.jsonl \
|
||||
--output-file results.jsonl
|
||||
|
||||
# Benchmark with vllm bench
|
||||
vllm bench throughput \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--num-prompts 1000 \
|
||||
--input-len 512 \
|
||||
--output-len 128
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Check running server stats
|
||||
curl http://localhost:8000/metrics # Prometheus metrics
|
||||
|
||||
# Key metrics to watch:
|
||||
# vllm:num_requests_running - active requests
|
||||
# vllm:gpu_cache_usage_perc - KV cache utilization
|
||||
# vllm:generation_tokens_per_s - throughput
|
||||
# vllm:time_to_first_token_ms - TTFT latency
|
||||
# vllm:e2e_request_latency_seconds - end-to-end latency
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `CUDA out of memory` | Model too large for VRAM | Add `--quantization awq` or reduce `--gpu-memory-utilization` |
|
||||
| Slow cold start | Model not cached | Pre-pull with `huggingface-cli download <model>` |
|
||||
| Low throughput | Too few concurrent requests | Increase `--max-num-seqs` |
|
||||
| KV cache full errors | Context length too long | Set `--max-model-len` lower |
|
||||
| `tokenizer error` | Tokenizer mismatch | Use `--tokenizer` to specify correct tokenizer |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use `--gpu-memory-utilization 0.90` to leave headroom for CUDA kernels.
|
||||
- Pin model versions with `--revision` for reproducible deployments.
|
||||
- Set `HF_HUB_OFFLINE=1` in production to prevent unexpected downloads.
|
||||
- Use AWQ or GPTQ quantization before tensor parallelism — lower VRAM first.
|
||||
- Enable `--enable-chunked-prefill` for long-context workloads.
|
||||
- Monitor `gpu_cache_usage_perc` — above 95% causes queuing.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [llm-inference-scaling](../llm-inference-scaling/) - Auto-scaling vLLM deployments
|
||||
- [gpu-server-management](../../servers/gpu-server-management/) - GPU driver setup
|
||||
- [llm-gateway](../../networking/llm-gateway/) - Load balancing across vLLM instances
|
||||
- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost management
|
||||
- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - K8s deployment
|
||||
Reference in New Issue
Block a user