mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
Add 12 AI infrastructure and LLM operations skills
New skills covering hot-topic AI engineering subjects: Local AI Infrastructure: - vllm-server: High-throughput LLM serving with PagedAttention, tensor parallelism, quantization - llm-inference-scaling: KEDA-based GPU autoscaling for LLM inference on Kubernetes - rag-infrastructure: Production RAG with hybrid search, reranking, and embedding pipelines - llm-fine-tuning: QLoRA/LoRA fine-tuning with Axolotl, DeepSpeed ZeRO-3, and DPO alignment Infrastructure: - gpu-server-management: NVIDIA driver setup, MIG partitioning, DCGM monitoring - vector-database-ops: Qdrant, Weaviate, pgvector for production AI search - llm-gateway: LiteLLM-based API gateway with rate limiting, virtual keys, fallback routing DevOps/AI: - llm-cost-optimization: Model right-sizing, prompt/semantic caching, batch API, break-even analysis - llm-caching: Multi-layer exact + semantic + provider caching to cut costs 30-70% - ai-pipeline-orchestration: Prefect/Airflow/Dagster for RAG ingestion and training workflows Orchestration: - model-serving-kubernetes: KServe + Triton with canary deployments and GPU autoscaling Security: - ai-security-hardening: Prompt injection defense, PII scrubbing, model weight verification https://claude.ai/code/session_011MN1C4PrkCeg2Qmi7q1ZUe
This commit is contained in:
Claude
@@ -0,0 +1,285 @@
|
||||
---
|
||||
name: vector-database-ops
|
||||
description: Deploy, manage, and optimize vector databases for AI applications. Covers Qdrant, Weaviate, pgvector, and Pinecone — collection management, indexing strategies, backup, and performance tuning for production RAG and semantic search workloads.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# Vector Database Operations
|
||||
|
||||
Run production vector databases for AI-powered search, RAG, and recommendation systems.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Setting up a vector database for a RAG or semantic search application
|
||||
- Choosing between Qdrant, Weaviate, pgvector, or Pinecone
|
||||
- Managing collections, indexes, and data migrations
|
||||
- Optimizing query performance and indexing for production loads
|
||||
- Implementing multi-tenant vector search with namespace isolation
|
||||
|
||||
## Vector Database Comparison
|
||||
|
||||
| Database | Best For | Hosting | Filtering | Scale |
|
||||
|----------|----------|---------|-----------|-------|
|
||||
| **Qdrant** | High-performance, rich filtering, self-hosted | Self / Cloud | Excellent | Very High |
|
||||
| **Weaviate** | Schema-first, hybrid search, multi-modal | Self / Cloud | Good | High |
|
||||
| **pgvector** | Already on Postgres, simple use cases | Self | Good | Medium |
|
||||
| **Pinecone** | Zero-ops managed, serverless | Managed only | Good | Very High |
|
||||
| **Chroma** | Local dev, prototyping | Self only | Basic | Low-Medium |
|
||||
|
||||
## Qdrant — Production Deployment
|
||||
|
||||
```bash
|
||||
# Docker (single node)
|
||||
docker run -d \
|
||||
--name qdrant \
|
||||
-p 6333:6333 \
|
||||
-p 6334:6334 \
|
||||
-v $(pwd)/qdrant-data:/qdrant/storage \
|
||||
qdrant/qdrant:latest
|
||||
|
||||
# With custom config
|
||||
docker run -d \
|
||||
--name qdrant \
|
||||
-p 6333:6333 \
|
||||
-v $(pwd)/qdrant-data:/qdrant/storage \
|
||||
-v $(pwd)/qdrant-config.yaml:/qdrant/config/production.yaml \
|
||||
qdrant/qdrant:latest
|
||||
```
|
||||
|
||||
```yaml
|
||||
# qdrant-config.yaml
|
||||
storage:
|
||||
storage_path: /qdrant/storage
|
||||
on_disk_payload: true # store payload on disk (saves RAM)
|
||||
|
||||
service:
|
||||
max_request_size_mb: 32
|
||||
|
||||
hnsw_index:
|
||||
m: 16 # graph connections per node
|
||||
ef_construct: 100 # accuracy vs build time trade-off
|
||||
full_scan_threshold: 10000 # switch to brute force below this
|
||||
|
||||
quantization:
|
||||
scalar:
|
||||
type: int8
|
||||
quantile: 0.99
|
||||
always_ram: true # keep quantized index in RAM
|
||||
|
||||
telemetry_disabled: true
|
||||
```
|
||||
|
||||
## Qdrant Collection Management
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.models import (
|
||||
Distance, VectorParams, HnswConfigDiff,
|
||||
ScalarQuantizationConfig, ScalarType, QuantizationConfig
|
||||
)
|
||||
|
||||
client = QdrantClient("http://localhost:6333")
|
||||
|
||||
# Create optimized collection
|
||||
client.create_collection(
|
||||
collection_name="documents",
|
||||
vectors_config=VectorParams(
|
||||
size=1536, # OpenAI ada-002 / text-embedding-3-small
|
||||
distance=Distance.COSINE,
|
||||
on_disk=True, # save RAM — vectors stored on disk
|
||||
),
|
||||
hnsw_config=HnswConfigDiff(
|
||||
m=32, # higher = better recall, more RAM
|
||||
ef_construct=200,
|
||||
on_disk=False, # keep HNSW graph in RAM for speed
|
||||
),
|
||||
quantization_config=QuantizationConfig(
|
||||
scalar=ScalarQuantizationConfig(
|
||||
type=ScalarType.INT8,
|
||||
quantile=0.99,
|
||||
always_ram=True,
|
||||
)
|
||||
),
|
||||
)
|
||||
|
||||
# Create payload index for fast filtering
|
||||
client.create_payload_index(
|
||||
collection_name="documents",
|
||||
field_name="tenant_id",
|
||||
field_schema="keyword",
|
||||
)
|
||||
client.create_payload_index(
|
||||
collection_name="documents",
|
||||
field_name="created_at",
|
||||
field_schema="datetime",
|
||||
)
|
||||
|
||||
# Collection info
|
||||
info = client.get_collection("documents")
|
||||
print(f"Vectors: {info.vectors_count}, Status: {info.status}")
|
||||
```
|
||||
|
||||
## Qdrant Filtered Search
|
||||
|
||||
```python
|
||||
from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
|
||||
|
||||
# Tenant-isolated search (multi-tenant RAG)
|
||||
results = client.query_points(
|
||||
collection_name="documents",
|
||||
query=query_embedding,
|
||||
query_filter=Filter(
|
||||
must=[
|
||||
FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")),
|
||||
FieldCondition(key="doc_type", match=MatchValue(value="contract")),
|
||||
],
|
||||
should=[
|
||||
FieldCondition(key="created_at", range=Range(gte="2024-01-01")),
|
||||
],
|
||||
),
|
||||
limit=10,
|
||||
with_payload=True,
|
||||
)
|
||||
```
|
||||
|
||||
## pgvector — PostgreSQL Extension
|
||||
|
||||
```sql
|
||||
-- Enable extension
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
|
||||
-- Create table with vector column
|
||||
CREATE TABLE documents (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
content TEXT NOT NULL,
|
||||
embedding VECTOR(1536),
|
||||
metadata JSONB DEFAULT '{}',
|
||||
tenant_id TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Create HNSW index (faster queries, more memory)
|
||||
CREATE INDEX ON documents
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
WITH (m = 16, ef_construction = 64);
|
||||
|
||||
-- Create IVFFlat index (less memory, slower build)
|
||||
-- CREATE INDEX ON documents
|
||||
-- USING ivfflat (embedding vector_cosine_ops)
|
||||
-- WITH (lists = 100);
|
||||
|
||||
-- Semantic search with metadata filtering
|
||||
SELECT id, content, metadata,
|
||||
1 - (embedding <=> $1::vector) AS similarity
|
||||
FROM documents
|
||||
WHERE tenant_id = 'acme-corp'
|
||||
AND metadata->>'doc_type' = 'contract'
|
||||
ORDER BY embedding <=> $1::vector
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
```bash
|
||||
# Deploy pgvector via Docker
|
||||
docker run -d \
|
||||
--name pgvector \
|
||||
-e POSTGRES_PASSWORD=secret \
|
||||
-e POSTGRES_DB=vectordb \
|
||||
-p 5432:5432 \
|
||||
-v pgvector-data:/var/lib/postgresql/data \
|
||||
pgvector/pgvector:pg16
|
||||
```
|
||||
|
||||
## Weaviate Deployment
|
||||
|
||||
```yaml
|
||||
# docker-compose for Weaviate
|
||||
services:
|
||||
weaviate:
|
||||
image: semitechnologies/weaviate:latest
|
||||
ports:
|
||||
- "8080:8080"
|
||||
- "50051:50051"
|
||||
environment:
|
||||
QUERY_DEFAULTS_LIMIT: 25
|
||||
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "false"
|
||||
AUTHENTICATION_APIKEY_ENABLED: "true"
|
||||
AUTHENTICATION_APIKEY_ALLOWED_KEYS: "${WEAVIATE_API_KEY}"
|
||||
AUTHENTICATION_APIKEY_USERS: "admin"
|
||||
PERSISTENCE_DATA_PATH: /var/lib/weaviate
|
||||
ENABLE_MODULES: text2vec-openai,generative-openai
|
||||
OPENAI_APIKEY: "${OPENAI_API_KEY}"
|
||||
CLUSTER_HOSTNAME: node1
|
||||
volumes:
|
||||
- weaviate-data:/var/lib/weaviate
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
weaviate-data:
|
||||
```
|
||||
|
||||
## Backup and Restore
|
||||
|
||||
```bash
|
||||
# Qdrant — snapshot backup
|
||||
curl -X POST "http://localhost:6333/collections/documents/snapshots"
|
||||
# Download snapshot
|
||||
curl -O "http://localhost:6333/collections/documents/snapshots/documents-snapshot.snapshot"
|
||||
# Restore
|
||||
curl -X POST "http://localhost:6333/collections/documents/snapshots/recover" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"location": "/qdrant/snapshots/documents-snapshot.snapshot"}'
|
||||
|
||||
# pgvector — standard pg_dump
|
||||
pg_dump -h localhost -U postgres -d vectordb \
|
||||
--table=documents --format=custom > documents-backup.dump
|
||||
|
||||
# Restore
|
||||
pg_restore -h localhost -U postgres -d vectordb documents-backup.dump
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
```python
|
||||
# Qdrant — optimize collection after bulk load
|
||||
client.update_collection(
|
||||
collection_name="documents",
|
||||
optimizer_config={"indexing_threshold": 0}, # force indexing now
|
||||
)
|
||||
|
||||
# Wait for optimization to complete
|
||||
import time
|
||||
while True:
|
||||
info = client.get_collection("documents")
|
||||
if info.status.value == "green":
|
||||
break
|
||||
time.sleep(5)
|
||||
print(f"Optimizing... segments: {info.segments_count}")
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| Slow queries | No HNSW index built yet | Wait for indexing; check `status == green` |
|
||||
| High RAM usage | Vectors in memory | Enable `on_disk=True` for vectors |
|
||||
| Poor recall | Low `ef` search param | Increase `ef` in search request (at query time) |
|
||||
| pgvector slow | Using IVFFlat without vacuum | Run `VACUUM ANALYZE documents` |
|
||||
| Weaviate OOM | Too many objects | Enable async indexing; increase heap |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use cosine distance for normalized embeddings; dot product for unnormalized.
|
||||
- Always create payload indexes on filter fields (`tenant_id`, `doc_type`).
|
||||
- For datasets >10M vectors, use `on_disk` vectors + `always_ram` quantization.
|
||||
- Benchmark with your actual query patterns before choosing IVFFlat vs HNSW.
|
||||
- Snapshot before any bulk delete or migration operation.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [rag-infrastructure](../../local-ai/rag-infrastructure/) - Full RAG pipeline
|
||||
- [databases](../databases/) - General database management
|
||||
- [postgresql](../postgresql/) - pgvector host database ops
|
||||
@@ -0,0 +1,312 @@
|
||||
---
|
||||
name: llm-fine-tuning
|
||||
description: Set up infrastructure for fine-tuning LLMs with QLoRA, LoRA, and full fine-tuning using Hugging Face TRL, Axolotl, and distributed training with DeepSpeed or FSDP. Covers dataset prep, training runs, and model export.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# LLM Fine-Tuning Infrastructure
|
||||
|
||||
Train and fine-tune open-source LLMs efficiently — from LoRA on a single GPU to distributed full fine-tuning across multi-node clusters.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Fine-tuning an LLM on domain-specific data (legal, medical, code, support)
|
||||
- Running QLoRA to fine-tune 70B models on consumer GPUs
|
||||
- Setting up distributed training with DeepSpeed or FSDP
|
||||
- Exporting fine-tuned adapters for production serving
|
||||
- Implementing RLHF, DPO, or instruction tuning pipelines
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- NVIDIA GPU(s) with 24GB+ VRAM (RTX 4090 / A100 / H100)
|
||||
- CUDA 12.1+ and `nvidia-smi` working
|
||||
- Python 3.10+ with `pip`
|
||||
- Hugging Face account and `HF_TOKEN` for gated models
|
||||
- 500GB+ disk for model weights and training data
|
||||
|
||||
## Quick Start: QLoRA Fine-Tuning
|
||||
|
||||
```bash
|
||||
pip install transformers datasets trl peft bitsandbytes accelerate
|
||||
|
||||
python - <<'EOF'
|
||||
from datasets import load_dataset
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
|
||||
from peft import LoraConfig, get_peft_model
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
import torch
|
||||
|
||||
model_id = "meta-llama/Llama-3.1-8B-Instruct"
|
||||
|
||||
# 4-bit quantization (QLoRA)
|
||||
bnb_config = BitsAndBytesConfig(
|
||||
load_in_4bit=True,
|
||||
bnb_4bit_quant_type="nf4",
|
||||
bnb_4bit_compute_dtype=torch.bfloat16,
|
||||
bnb_4bit_use_double_quant=True,
|
||||
)
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id, quantization_config=bnb_config, device_map="auto"
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
|
||||
# LoRA configuration
|
||||
peft_config = LoraConfig(
|
||||
r=16, # rank
|
||||
lora_alpha=32,
|
||||
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
|
||||
"gate_proj", "up_proj", "down_proj"],
|
||||
lora_dropout=0.05,
|
||||
bias="none",
|
||||
task_type="CAUSAL_LM",
|
||||
)
|
||||
|
||||
dataset = load_dataset("your-org/your-dataset", split="train")
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model=model,
|
||||
args=SFTConfig(
|
||||
output_dir="./output",
|
||||
num_train_epochs=3,
|
||||
per_device_train_batch_size=2,
|
||||
gradient_accumulation_steps=8,
|
||||
learning_rate=2e-4,
|
||||
bf16=True,
|
||||
logging_steps=10,
|
||||
save_strategy="epoch",
|
||||
report_to="wandb",
|
||||
),
|
||||
train_dataset=dataset,
|
||||
peft_config=peft_config,
|
||||
processing_class=tokenizer,
|
||||
)
|
||||
trainer.train()
|
||||
trainer.save_model("./fine-tuned-model")
|
||||
EOF
|
||||
```
|
||||
|
||||
## Axolotl (Production Fine-Tuning Framework)
|
||||
|
||||
```yaml
|
||||
# config.yaml — Axolotl QLoRA config for Llama 3.1
|
||||
base_model: meta-llama/Llama-3.1-8B-Instruct
|
||||
model_type: LlamaForCausalLM
|
||||
tokenizer_type: PreTrainedTokenizerFast
|
||||
|
||||
load_in_4bit: true
|
||||
adapter: qlora
|
||||
lora_r: 32
|
||||
lora_alpha: 64
|
||||
lora_dropout: 0.05
|
||||
lora_target_modules:
|
||||
- q_proj
|
||||
- k_proj
|
||||
- v_proj
|
||||
- o_proj
|
||||
- gate_proj
|
||||
- up_proj
|
||||
- down_proj
|
||||
|
||||
datasets:
|
||||
- path: your-org/your-dataset
|
||||
type: alpaca # or sharegpt, chat_template, etc.
|
||||
|
||||
dataset_prepared_path: ./prepared-data
|
||||
val_set_size: 0.05
|
||||
output_dir: ./output
|
||||
|
||||
sequence_len: 4096
|
||||
sample_packing: true # pack multiple short samples for efficiency
|
||||
|
||||
micro_batch_size: 2
|
||||
gradient_accumulation_steps: 8
|
||||
num_epochs: 3
|
||||
learning_rate: 2e-4
|
||||
optimizer: adamw_bnb_8bit
|
||||
lr_scheduler: cosine
|
||||
warmup_ratio: 0.05
|
||||
|
||||
bf16: true
|
||||
flash_attention: true
|
||||
|
||||
logging_steps: 10
|
||||
eval_steps: 100
|
||||
save_steps: 200
|
||||
wandb_project: my-fine-tune
|
||||
```
|
||||
|
||||
```bash
|
||||
# Run with Axolotl
|
||||
pip install axolotl[flash-attn,deepspeed]
|
||||
accelerate launch -m axolotl.cli.train config.yaml
|
||||
```
|
||||
|
||||
## Distributed Training with DeepSpeed
|
||||
|
||||
```json
|
||||
// deepspeed_zero3.json — ZeRO Stage 3 (split optimizer + gradients + params)
|
||||
{
|
||||
"zero_optimization": {
|
||||
"stage": 3,
|
||||
"offload_optimizer": {"device": "cpu", "pin_memory": true},
|
||||
"offload_param": {"device": "cpu", "pin_memory": true},
|
||||
"overlap_comm": true,
|
||||
"contiguous_gradients": true,
|
||||
"sub_group_size": 1e9,
|
||||
"reduce_bucket_size": "auto",
|
||||
"stage3_prefetch_bucket_size": "auto",
|
||||
"stage3_param_persistence_threshold": "auto",
|
||||
"stage3_max_live_parameters": 1e9,
|
||||
"stage3_max_reuse_distance": 1e9,
|
||||
"gather_16bit_weights_on_model_save": true
|
||||
},
|
||||
"bf16": {"enabled": true},
|
||||
"gradient_clipping": 1.0,
|
||||
"train_batch_size": "auto",
|
||||
"train_micro_batch_size_per_gpu": "auto"
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
# Launch 4-GPU DeepSpeed training
|
||||
deepspeed --num_gpus=4 train.py \
|
||||
--deepspeed deepspeed_zero3.json \
|
||||
--model_name meta-llama/Llama-3.1-70B-Instruct \
|
||||
--output_dir ./output
|
||||
```
|
||||
|
||||
## DPO / RLHF Alignment
|
||||
|
||||
```python
|
||||
from trl import DPOTrainer, DPOConfig
|
||||
from datasets import load_dataset
|
||||
|
||||
# Dataset format: {"prompt": ..., "chosen": ..., "rejected": ...}
|
||||
dataset = load_dataset("your-org/preference-data")
|
||||
|
||||
trainer = DPOTrainer(
|
||||
model=model,
|
||||
ref_model=None, # None = implicit reference with peft
|
||||
args=DPOConfig(
|
||||
output_dir="./dpo-output",
|
||||
beta=0.1, # KL divergence weight
|
||||
num_train_epochs=1,
|
||||
per_device_train_batch_size=1,
|
||||
gradient_accumulation_steps=16,
|
||||
learning_rate=5e-7,
|
||||
bf16=True,
|
||||
),
|
||||
train_dataset=dataset["train"],
|
||||
peft_config=peft_config,
|
||||
processing_class=tokenizer,
|
||||
)
|
||||
trainer.train()
|
||||
```
|
||||
|
||||
## Merging LoRA Adapters for Deployment
|
||||
|
||||
```python
|
||||
from peft import PeftModel
|
||||
from transformers import AutoModelForCausalLM
|
||||
|
||||
# Load base model in full precision
|
||||
base_model = AutoModelForCausalLM.from_pretrained(
|
||||
"meta-llama/Llama-3.1-8B-Instruct",
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="cpu",
|
||||
)
|
||||
|
||||
# Load and merge LoRA adapter
|
||||
model = PeftModel.from_pretrained(base_model, "./fine-tuned-model")
|
||||
merged_model = model.merge_and_unload()
|
||||
|
||||
# Save merged model (ready for vLLM serving)
|
||||
merged_model.save_pretrained("./merged-model", safe_serialization=True)
|
||||
tokenizer.save_pretrained("./merged-model")
|
||||
|
||||
# Push to Hugging Face Hub
|
||||
merged_model.push_to_hub("your-org/your-fine-tuned-model")
|
||||
```
|
||||
|
||||
## Kubernetes Training Job
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: llm-fine-tune
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
nodeSelector:
|
||||
nvidia.com/gpu.product: A100-SXM4-80GB
|
||||
containers:
|
||||
- name: trainer
|
||||
image: nvcr.io/nvidia/pytorch:24.05-py3
|
||||
command: ["accelerate", "launch", "-m", "axolotl.cli.train", "/config/config.yaml"]
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: "4"
|
||||
memory: "320Gi"
|
||||
requests:
|
||||
nvidia.com/gpu: "4"
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: model-cache
|
||||
mountPath: /root/.cache/huggingface
|
||||
- name: output
|
||||
mountPath: /output
|
||||
env:
|
||||
- name: HUGGING_FACE_HUB_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token
|
||||
key: token
|
||||
- name: WANDB_API_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: wandb-token
|
||||
key: key
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: axolotl-config
|
||||
- name: model-cache
|
||||
persistentVolumeClaim:
|
||||
claimName: model-cache-pvc
|
||||
- name: output
|
||||
persistentVolumeClaim:
|
||||
claimName: training-output-pvc
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `CUDA out of memory` | Batch too large | Reduce `micro_batch_size`; increase `gradient_accumulation_steps` |
|
||||
| Training loss NaN | Learning rate too high | Lower LR to `1e-4` or `5e-5`; add warmup |
|
||||
| Slow training | No Flash Attention | Install `flash-attn`; enable `flash_attention: true` |
|
||||
| Poor fine-tune quality | Bad data formatting | Validate dataset format; check `sample_packing` compatibility |
|
||||
| Adapter merge errors | Mixed quantization | Merge in bf16 on CPU, not in 4-bit |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use Flash Attention 2 — it's 2–4× faster and uses less memory.
|
||||
- Monitor training loss/eval loss via W&B or MLflow; overfit = more dropout or less data.
|
||||
- Validate with a held-out eval set (5–10%); MMLU or custom evals for quality gates.
|
||||
- Start with LoRA r=16 before increasing — higher rank = more parameters, diminishing returns.
|
||||
- Use `sample_packing` in Axolotl to maximize GPU utilization on short sequences.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [vllm-server](../vllm-server/) - Serve fine-tuned models
|
||||
- [gpu-server-management](../../servers/gpu-server-management/) - GPU setup
|
||||
- [llm-inference-scaling](../llm-inference-scaling/) - Deploy at scale
|
||||
- [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Training pipelines
|
||||
@@ -0,0 +1,270 @@
|
||||
---
|
||||
name: llm-inference-scaling
|
||||
description: Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# LLM Inference Scaling
|
||||
|
||||
Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- LLM API traffic is unpredictable and you need to scale up/down automatically
|
||||
- Managing a fleet of vLLM or TGI inference pods on Kubernetes
|
||||
- Reducing inference costs with spot/preemptible GPU instances
|
||||
- Implementing queue-based autoscaling for batch inference jobs
|
||||
- Building a multi-model serving platform that shares GPU resources
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes cluster with GPU nodes (NVIDIA operator installed)
|
||||
- KEDA (Kubernetes Event-Driven Autoscaler) installed
|
||||
- Prometheus with GPU metrics (`dcgm-exporter` or `gpu-operator`)
|
||||
- Helm 3+ for chart deployments
|
||||
|
||||
## GPU Node Setup
|
||||
|
||||
```bash
|
||||
# Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)
|
||||
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
|
||||
helm repo update
|
||||
|
||||
helm install gpu-operator nvidia/gpu-operator \
|
||||
--namespace gpu-operator \
|
||||
--create-namespace \
|
||||
--set driver.enabled=true \
|
||||
--set dcgm.enabled=true \
|
||||
--set devicePlugin.enabled=true
|
||||
|
||||
# Verify GPU nodes are recognized
|
||||
kubectl get nodes -l nvidia.com/gpu.present=true
|
||||
kubectl describe node <gpu-node> | grep nvidia
|
||||
```
|
||||
|
||||
## vLLM Deployment with GPU Resources
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: vllm-llama-8b
|
||||
labels:
|
||||
app: vllm
|
||||
model: llama-3.1-8b
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: vllm
|
||||
model: llama-3.1-8b
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: vllm
|
||||
model: llama-3.1-8b
|
||||
spec:
|
||||
nodeSelector:
|
||||
nvidia.com/gpu.present: "true"
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
containers:
|
||||
- name: vllm
|
||||
image: vllm/vllm-openai:latest
|
||||
args:
|
||||
- "--model"
|
||||
- "meta-llama/Llama-3.1-8B-Instruct"
|
||||
- "--tensor-parallel-size"
|
||||
- "1"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.90"
|
||||
- "--max-num-seqs"
|
||||
- "128"
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
memory: "20Gi"
|
||||
cpu: "4"
|
||||
limits:
|
||||
nvidia.com/gpu: "1"
|
||||
memory: "24Gi"
|
||||
cpu: "8"
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8000
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
env:
|
||||
- name: HUGGING_FACE_HUB_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token
|
||||
key: token
|
||||
```
|
||||
|
||||
## KEDA Autoscaling on Prometheus Metrics
|
||||
|
||||
```yaml
|
||||
apiVersion: keda.sh/v1alpha1
|
||||
kind: ScaledObject
|
||||
metadata:
|
||||
name: vllm-scaledobject
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
name: vllm-llama-8b
|
||||
minReplicaCount: 1
|
||||
maxReplicaCount: 8
|
||||
cooldownPeriod: 300 # 5 min before scale-down
|
||||
pollingInterval: 15
|
||||
triggers:
|
||||
- type: prometheus
|
||||
metadata:
|
||||
serverAddress: http://prometheus-server.monitoring:9090
|
||||
metricName: vllm_num_requests_waiting
|
||||
threshold: "10" # scale up if >10 requests waiting
|
||||
query: |
|
||||
sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
|
||||
- type: prometheus
|
||||
metadata:
|
||||
serverAddress: http://prometheus-server.monitoring:9090
|
||||
metricName: vllm_gpu_cache_usage
|
||||
threshold: "0.8" # scale up if KV cache >80% full
|
||||
query: |
|
||||
avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})
|
||||
```
|
||||
|
||||
## Queue-Based Scaling (Redis + KEDA)
|
||||
|
||||
```yaml
|
||||
# ScaledJob for async batch inference
|
||||
apiVersion: keda.sh/v1alpha1
|
||||
kind: ScaledJob
|
||||
metadata:
|
||||
name: llm-batch-inference
|
||||
spec:
|
||||
jobTargetRef:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: inference-worker
|
||||
image: myapp/inference-worker:latest
|
||||
env:
|
||||
- name: REDIS_URL
|
||||
value: redis://redis:6379
|
||||
- name: QUEUE_NAME
|
||||
value: inference-jobs
|
||||
restartPolicy: OnFailure
|
||||
minReplicaCount: 0
|
||||
maxReplicaCount: 20
|
||||
pollingInterval: 5
|
||||
successfulJobsHistoryLimit: 3
|
||||
triggers:
|
||||
- type: redis
|
||||
metadata:
|
||||
address: redis:6379
|
||||
listName: inference-jobs
|
||||
listLength: "5" # 1 worker per 5 queued jobs
|
||||
```
|
||||
|
||||
## Spot Instance Strategy
|
||||
|
||||
```yaml
|
||||
# Mixed node pool: on-demand + spot GPUs
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: cluster-autoscaler-priority-config
|
||||
data:
|
||||
priorities: |
|
||||
10: # low priority = prefer
|
||||
- .*spot.*
|
||||
50:
|
||||
- .*on-demand.*
|
||||
---
|
||||
# Node affinity for spot with on-demand fallback
|
||||
spec:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 80
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: node.kubernetes.io/lifecycle
|
||||
operator: In
|
||||
values: [spot]
|
||||
- weight: 20
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: node.kubernetes.io/lifecycle
|
||||
operator: In
|
||||
values: [on-demand]
|
||||
```
|
||||
|
||||
## Cluster Autoscaler for GPU Nodes
|
||||
|
||||
```bash
|
||||
# AWS EKS — enable cluster autoscaler for GPU node group
|
||||
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
|
||||
--namespace kube-system \
|
||||
--set autoDiscovery.clusterName=my-cluster \
|
||||
--set awsRegion=us-east-1 \
|
||||
--set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole \
|
||||
--set extraArgs.skip-nodes-with-local-storage=false \
|
||||
--set extraArgs.expander=least-waste
|
||||
|
||||
# Annotate GPU node group for autoscaler
|
||||
kubectl annotate node <node> \
|
||||
cluster-autoscaler.kubernetes.io/safe-to-evict="false"
|
||||
```
|
||||
|
||||
## Scaling Metrics to Monitor
|
||||
|
||||
```bash
|
||||
# Prometheus queries for scaling decisions
|
||||
# Requests waiting in vLLM queue
|
||||
sum(vllm:num_requests_waiting) by (model)
|
||||
|
||||
# GPU KV cache utilization (>80% = bottleneck)
|
||||
avg(vllm:gpu_cache_usage_perc) by (pod)
|
||||
|
||||
# Tokens per second throughput
|
||||
sum(rate(vllm:generation_tokens_total[5m])) by (model)
|
||||
|
||||
# P99 time-to-first-token
|
||||
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| Pods stuck in `Pending` | No GPU nodes available | Check cluster autoscaler logs; verify node group limits |
|
||||
| Scale-up too slow | Cluster autoscaler delay + model load time | Pre-warm replicas; increase `minReplicaCount` |
|
||||
| GPU fragmentation | Multiple small models on large GPUs | Use MIG partitioning or consolidate model sizes |
|
||||
| Spot eviction causes errors | Spot instance reclamation | Add `PodDisruptionBudget`; use graceful shutdown |
|
||||
| KEDA not scaling | Prometheus query returns no data | Test query in Prometheus UI first |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Set `minReplicaCount: 1` to avoid cold starts; scale to 0 only for batch jobs.
|
||||
- Use `PodDisruptionBudget` with `minAvailable: 1` to survive spot evictions.
|
||||
- Pre-pull model weights into a shared PVC to speed up pod startup by 5–10×.
|
||||
- Separate model families across node pools (A10G for 7B, A100 for 70B).
|
||||
- Use Kubernetes VPA for CPU/memory right-sizing alongside KEDA for replica count.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [vllm-server](../vllm-server/) - vLLM configuration and tuning
|
||||
- [gpu-server-management](../../servers/gpu-server-management/) - GPU node setup
|
||||
- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - KServe
|
||||
- [kubernetes-ops](../../../devops/orchestration/kubernetes-ops/) - Core Kubernetes
|
||||
- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost strategies
|
||||
@@ -0,0 +1,253 @@
|
||||
---
|
||||
name: rag-infrastructure
|
||||
description: Build and operate Retrieval-Augmented Generation (RAG) infrastructure with vector stores, embedding pipelines, and hybrid search. Covers ingestion, chunking strategies, reranking, and production deployment patterns.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# RAG Infrastructure
|
||||
|
||||
Production infrastructure for Retrieval-Augmented Generation: ingest documents, generate embeddings, store in vector databases, and serve grounded LLM responses.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Building a knowledge base Q&A system over internal documents
|
||||
- Implementing semantic search over large document collections
|
||||
- Reducing LLM hallucinations with retrieved context
|
||||
- Setting up embedding pipelines and vector store infrastructure
|
||||
- Deploying hybrid search (dense + sparse/BM25)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.10+ with `pip`
|
||||
- A vector database (Qdrant, Weaviate, Pinecone, or pgvector)
|
||||
- An embedding model (OpenAI, Cohere, or local via `sentence-transformers`)
|
||||
- An LLM endpoint (OpenAI API or self-hosted vLLM)
|
||||
- Docker for local vector DB deployment
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
Documents → Chunker → Embedder → Vector Store
|
||||
↓
|
||||
User Query → Embedder → Vector Store (search) → Reranker → LLM → Answer
|
||||
```
|
||||
|
||||
## Embedding Pipeline
|
||||
|
||||
```python
|
||||
from sentence_transformers import SentenceTransformer
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.models import Distance, VectorParams, PointStruct
|
||||
import uuid
|
||||
|
||||
# Local embedding model (no API cost)
|
||||
model = SentenceTransformer("BAAI/bge-large-en-v1.5")
|
||||
|
||||
# Connect to Qdrant
|
||||
client = QdrantClient("http://localhost:6333")
|
||||
|
||||
# Create collection
|
||||
client.create_collection(
|
||||
collection_name="knowledge-base",
|
||||
vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
|
||||
)
|
||||
|
||||
def ingest_documents(docs: list[dict]):
|
||||
"""Chunk, embed, and upsert documents."""
|
||||
points = []
|
||||
for doc in docs:
|
||||
chunks = chunk_text(doc["text"], chunk_size=512, overlap=50)
|
||||
embeddings = model.encode(chunks, batch_size=32, show_progress_bar=True)
|
||||
for chunk, embedding in zip(chunks, embeddings):
|
||||
points.append(PointStruct(
|
||||
id=str(uuid.uuid4()),
|
||||
vector=embedding.tolist(),
|
||||
payload={"text": chunk, "source": doc["source"], "title": doc["title"]},
|
||||
))
|
||||
client.upsert(collection_name="knowledge-base", points=points)
|
||||
print(f"Ingested {len(points)} chunks")
|
||||
```
|
||||
|
||||
## Chunking Strategies
|
||||
|
||||
```python
|
||||
from langchain.text_splitter import RecursiveCharacterTextSplitter
|
||||
|
||||
def chunk_text(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]:
|
||||
"""Recursive character splitter — best general-purpose strategy."""
|
||||
splitter = RecursiveCharacterTextSplitter(
|
||||
chunk_size=chunk_size,
|
||||
chunk_overlap=overlap,
|
||||
separators=["\n\n", "\n", ". ", " ", ""],
|
||||
)
|
||||
return splitter.split_text(text)
|
||||
|
||||
# For code/markdown — use language-aware splitter
|
||||
from langchain.text_splitter import MarkdownHeaderTextSplitter
|
||||
|
||||
headers = [("#", "H1"), ("##", "H2"), ("###", "H3")]
|
||||
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
|
||||
```
|
||||
|
||||
## Hybrid Search (Dense + Sparse)
|
||||
|
||||
```python
|
||||
from qdrant_client.models import SparseVector, SparseVectorParams, NamedSparseVector
|
||||
from fastembed import SparseTextEmbedding
|
||||
|
||||
# Qdrant hybrid collection (dense + BM25 sparse)
|
||||
client.create_collection(
|
||||
collection_name="hybrid-kb",
|
||||
vectors_config={"dense": VectorParams(size=1024, distance=Distance.COSINE)},
|
||||
sparse_vectors_config={"sparse": SparseVectorParams()},
|
||||
)
|
||||
|
||||
sparse_model = SparseTextEmbedding("prithivida/Splade_PP_en_v1")
|
||||
|
||||
def hybrid_search(query: str, top_k: int = 10) -> list[dict]:
|
||||
dense_vec = model.encode(query).tolist()
|
||||
sparse_vec = list(sparse_model.embed(query))[0]
|
||||
|
||||
results = client.query_points(
|
||||
collection_name="hybrid-kb",
|
||||
prefetch=[
|
||||
{"query": dense_vec, "using": "dense", "limit": 20},
|
||||
{"query": SparseVector(indices=sparse_vec.indices.tolist(),
|
||||
values=sparse_vec.values.tolist()),
|
||||
"using": "sparse", "limit": 20},
|
||||
],
|
||||
query={"fusion": "rrf"}, # Reciprocal Rank Fusion
|
||||
limit=top_k,
|
||||
)
|
||||
return [{"text": p.payload["text"], "score": p.score} for p in results.points]
|
||||
```
|
||||
|
||||
## Reranking
|
||||
|
||||
```python
|
||||
import cohere
|
||||
|
||||
co = cohere.Client("your-api-key")
|
||||
|
||||
def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
|
||||
"""Rerank retrieved chunks for relevance (improves RAG quality ~20-30%)."""
|
||||
response = co.rerank(
|
||||
model="rerank-english-v3.0",
|
||||
query=query,
|
||||
documents=candidates,
|
||||
top_n=top_n,
|
||||
)
|
||||
return [candidates[r.index] for r in response.results]
|
||||
|
||||
# Alternative: local reranker (no API cost)
|
||||
from sentence_transformers import CrossEncoder
|
||||
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
|
||||
|
||||
def local_rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
|
||||
pairs = [[query, c] for c in candidates]
|
||||
scores = reranker.predict(pairs)
|
||||
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
|
||||
return [text for text, _ in ranked[:top_n]]
|
||||
```
|
||||
|
||||
## RAG Query Pipeline
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
llm = OpenAI(base_url="http://localhost:8000/v1", api_key="your-key")
|
||||
|
||||
def rag_query(user_question: str) -> str:
|
||||
# 1. Retrieve
|
||||
candidates = hybrid_search(user_question, top_k=20)
|
||||
texts = [c["text"] for c in candidates]
|
||||
|
||||
# 2. Rerank
|
||||
top_chunks = local_rerank(user_question, texts, top_n=5)
|
||||
|
||||
# 3. Generate
|
||||
context = "\n\n---\n\n".join(top_chunks)
|
||||
response = llm.chat.completions.create(
|
||||
model="meta-llama/Llama-3.1-8B-Instruct",
|
||||
messages=[
|
||||
{"role": "system", "content": (
|
||||
"Answer the question using only the provided context. "
|
||||
"If the answer isn't in the context, say so.\n\nContext:\n" + context
|
||||
)},
|
||||
{"role": "user", "content": user_question},
|
||||
],
|
||||
temperature=0.1,
|
||||
max_tokens=1024,
|
||||
)
|
||||
return response.choices[0].message.content
|
||||
```
|
||||
|
||||
## Docker Compose: Full RAG Stack
|
||||
|
||||
```yaml
|
||||
services:
|
||||
qdrant:
|
||||
image: qdrant/qdrant:latest
|
||||
volumes:
|
||||
- qdrant-data:/qdrant/storage
|
||||
ports:
|
||||
- "6333:6333"
|
||||
restart: unless-stopped
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
restart: unless-stopped
|
||||
|
||||
ingestion-worker:
|
||||
build: ./ingestion
|
||||
environment:
|
||||
- QDRANT_URL=http://qdrant:6333
|
||||
- REDIS_URL=redis://redis:6379
|
||||
depends_on: [qdrant, redis]
|
||||
restart: unless-stopped
|
||||
|
||||
rag-api:
|
||||
build: ./api
|
||||
ports:
|
||||
- "8080:8080"
|
||||
environment:
|
||||
- QDRANT_URL=http://qdrant:6333
|
||||
- LLM_BASE_URL=http://vllm:8000/v1
|
||||
depends_on: [qdrant]
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
qdrant-data:
|
||||
redis-data:
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| Poor retrieval quality | Chunk size too large | Try 256–512 tokens; overlap 10–15% |
|
||||
| LLM ignores retrieved context | Context too long | Rerank and keep top 3–5 chunks |
|
||||
| Slow ingestion | Sequential embedding | Use `batch_size=64` and async upserts |
|
||||
| Stale documents | No re-ingestion pipeline | Track `doc_hash`; re-embed on change |
|
||||
| High embedding costs | All chunks re-embedded | Cache embeddings with hash-based dedup |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use `BAAI/bge-large-en-v1.5` or `nomic-embed-text` for strong free embeddings.
|
||||
- Always rerank before passing to LLM — 5 precise chunks beat 20 noisy ones.
|
||||
- Store source metadata (URL, page, section) in vector payloads for citations.
|
||||
- Use namespace/tenant isolation in the vector store for multi-tenant RAG.
|
||||
- Evaluate with RAGAS metrics: faithfulness, answer relevancy, context precision.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [vector-database-ops](../../databases/vector-database-ops/) - Qdrant/Weaviate management
|
||||
- [vllm-server](../vllm-server/) - Self-hosted LLM endpoint
|
||||
- [ollama-stack](../ollama-stack/) - Local LLM for development
|
||||
- [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Ingestion pipelines
|
||||
@@ -0,0 +1,219 @@
|
||||
---
|
||||
name: vllm-server
|
||||
description: Deploy and manage vLLM for high-throughput LLM inference. Configure continuous batching, tensor parallelism, quantization, and OpenAI-compatible API endpoints for production LLM serving.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# vLLM Server Management
|
||||
|
||||
Deploy production-grade LLM inference servers with vLLM — the fastest open-source LLM serving engine with PagedAttention and continuous batching.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Serving open-source LLMs (Llama, Mistral, Qwen, Gemma) at scale
|
||||
- Building an OpenAI-compatible API endpoint for self-hosted models
|
||||
- Optimizing LLM throughput and latency for production traffic
|
||||
- Running multi-GPU inference with tensor or pipeline parallelism
|
||||
- Deploying quantized models to reduce GPU memory requirements
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- NVIDIA GPU(s) with CUDA 12.1+ (A100/H100 recommended for production)
|
||||
- Docker or Python 3.9+ with pip
|
||||
- 40GB+ VRAM for 70B models; 8GB+ for 7B models
|
||||
- `nvidia-container-toolkit` for Docker GPU passthrough
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Install vLLM
|
||||
pip install vllm
|
||||
|
||||
# Serve a model (OpenAI-compatible API)
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--api-key your-secret-key
|
||||
|
||||
# Test the endpoint
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer your-secret-key" \
|
||||
-d '{
|
||||
"model": "meta-llama/Llama-3.1-8B-Instruct",
|
||||
"messages": [{"role": "user", "content": "Hello!"}]
|
||||
}'
|
||||
```
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
```bash
|
||||
docker run --runtime nvidia --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-p 8000:8000 \
|
||||
--ipc=host \
|
||||
vllm/vllm-openai:latest \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--api-key your-secret-key
|
||||
```
|
||||
|
||||
## Docker Compose (Production)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
vllm:
|
||||
image: vllm/vllm-openai:latest
|
||||
runtime: nvidia
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=all
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
|
||||
volumes:
|
||||
- model-cache:/root/.cache/huggingface
|
||||
ports:
|
||||
- "8000:8000"
|
||||
ipc: host
|
||||
command: >
|
||||
--model meta-llama/Llama-3.1-70B-Instruct
|
||||
--tensor-parallel-size 2
|
||||
--max-model-len 32768
|
||||
--gpu-memory-utilization 0.90
|
||||
--api-key ${VLLM_API_KEY}
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
model-cache:
|
||||
```
|
||||
|
||||
## Key Configuration Options
|
||||
|
||||
### Multi-GPU Tensor Parallelism
|
||||
|
||||
```bash
|
||||
# Split one model across 4 GPUs
|
||||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.90
|
||||
```
|
||||
|
||||
### Quantization (Lower VRAM)
|
||||
|
||||
```bash
|
||||
# AWQ quantization (70B on 2x A100 40GB)
|
||||
vllm serve casperhansen/llama-3-70b-instruct-awq \
|
||||
--quantization awq \
|
||||
--tensor-parallel-size 2
|
||||
|
||||
# GPTQ quantization
|
||||
vllm serve TheBloke/Llama-2-70B-Chat-GPTQ \
|
||||
--quantization gptq
|
||||
|
||||
# FP8 (H100 NVL native)
|
||||
vllm serve meta-llama/Llama-3.1-405B-Instruct \
|
||||
--quantization fp8 \
|
||||
--tensor-parallel-size 8
|
||||
```
|
||||
|
||||
### Structured Output & Tools
|
||||
|
||||
```bash
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser llama3_json \
|
||||
--guided-decoding-backend outlines
|
||||
```
|
||||
|
||||
### LoRA Adapters
|
||||
|
||||
```bash
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--enable-lora \
|
||||
--lora-modules sql-lora=/path/to/sql-lora \
|
||||
code-lora=/path/to/code-lora \
|
||||
--max-lora-rank 64
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
```bash
|
||||
# Maximize throughput for batch workloads
|
||||
vllm serve <model> \
|
||||
--max-num-seqs 256 \ # max concurrent sequences
|
||||
--max-num-batched-tokens 8192 \ # tokens per batch
|
||||
--gpu-memory-utilization 0.95 \ # use 95% VRAM
|
||||
--swap-space 4 # CPU swap (GiB)
|
||||
|
||||
# Minimize latency for interactive use
|
||||
vllm serve <model> \
|
||||
--max-num-seqs 32 \
|
||||
--enforce-eager # disable CUDA graph capture
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
# Install benchmark tool
|
||||
pip install vllm
|
||||
|
||||
# Run throughput benchmark
|
||||
python -m vllm.entrypoints.openai.run_batch \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--input-file prompts.jsonl \
|
||||
--output-file results.jsonl
|
||||
|
||||
# Benchmark with vllm bench
|
||||
vllm bench throughput \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--num-prompts 1000 \
|
||||
--input-len 512 \
|
||||
--output-len 128
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Check running server stats
|
||||
curl http://localhost:8000/metrics # Prometheus metrics
|
||||
|
||||
# Key metrics to watch:
|
||||
# vllm:num_requests_running - active requests
|
||||
# vllm:gpu_cache_usage_perc - KV cache utilization
|
||||
# vllm:generation_tokens_per_s - throughput
|
||||
# vllm:time_to_first_token_ms - TTFT latency
|
||||
# vllm:e2e_request_latency_seconds - end-to-end latency
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `CUDA out of memory` | Model too large for VRAM | Add `--quantization awq` or reduce `--gpu-memory-utilization` |
|
||||
| Slow cold start | Model not cached | Pre-pull with `huggingface-cli download <model>` |
|
||||
| Low throughput | Too few concurrent requests | Increase `--max-num-seqs` |
|
||||
| KV cache full errors | Context length too long | Set `--max-model-len` lower |
|
||||
| `tokenizer error` | Tokenizer mismatch | Use `--tokenizer` to specify correct tokenizer |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use `--gpu-memory-utilization 0.90` to leave headroom for CUDA kernels.
|
||||
- Pin model versions with `--revision` for reproducible deployments.
|
||||
- Set `HF_HUB_OFFLINE=1` in production to prevent unexpected downloads.
|
||||
- Use AWQ or GPTQ quantization before tensor parallelism — lower VRAM first.
|
||||
- Enable `--enable-chunked-prefill` for long-context workloads.
|
||||
- Monitor `gpu_cache_usage_perc` — above 95% causes queuing.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [llm-inference-scaling](../llm-inference-scaling/) - Auto-scaling vLLM deployments
|
||||
- [gpu-server-management](../../servers/gpu-server-management/) - GPU driver setup
|
||||
- [llm-gateway](../../networking/llm-gateway/) - Load balancing across vLLM instances
|
||||
- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost management
|
||||
- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - K8s deployment
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
name: llm-gateway
|
||||
description: Deploy an API gateway for LLM traffic with load balancing, rate limiting, key management, semantic caching, fallback routing, and cost tracking. Covers LiteLLM Proxy, OpenRouter-compatible setup, and custom Nginx/Traefik patterns.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# LLM Gateway
|
||||
|
||||
A unified API gateway that routes LLM requests across providers and self-hosted models — with rate limiting, cost tracking, caching, and failover.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Running multiple LLM backends (OpenAI, Anthropic, vLLM, Ollama) behind a single endpoint
|
||||
- Enforcing per-team or per-user rate limits and spend budgets
|
||||
- Implementing automatic fallback when a provider is down
|
||||
- Adding semantic caching to reduce API costs by 20–50%
|
||||
- Centralizing API key management instead of distributing keys to every app
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker and Docker Compose
|
||||
- A PostgreSQL or SQLite database (for LiteLLM state)
|
||||
- LLM API keys (OpenAI, Anthropic, etc.) or self-hosted vLLM endpoints
|
||||
- Optional: Redis for caching and rate limiting
|
||||
|
||||
## LiteLLM Proxy — Quick Start
|
||||
|
||||
LiteLLM is the de facto open-source LLM gateway with OpenAI-compatible API.
|
||||
|
||||
```bash
|
||||
# Run with Docker
|
||||
docker run -d \
|
||||
--name litellm-proxy \
|
||||
-p 4000:4000 \
|
||||
-e OPENAI_API_KEY=$OPENAI_API_KEY \
|
||||
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
|
||||
-v $(pwd)/litellm-config.yaml:/app/config.yaml \
|
||||
ghcr.io/berriai/litellm:main-latest \
|
||||
--config /app/config.yaml \
|
||||
--detailed_debug
|
||||
```
|
||||
|
||||
## LiteLLM Configuration
|
||||
|
||||
```yaml
|
||||
# litellm-config.yaml
|
||||
model_list:
|
||||
# OpenAI models
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
rpm: 10000
|
||||
tpm: 2000000
|
||||
|
||||
- model_name: gpt-4o-mini
|
||||
litellm_params:
|
||||
model: openai/gpt-4o-mini
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
# Anthropic
|
||||
- model_name: claude-sonnet-4-6
|
||||
litellm_params:
|
||||
model: anthropic/claude-sonnet-4-6
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
|
||||
# Self-hosted vLLM instances (load balanced)
|
||||
- model_name: llama-3.1-8b
|
||||
litellm_params:
|
||||
model: openai/meta-llama/Llama-3.1-8B-Instruct
|
||||
api_base: http://vllm-1:8000/v1
|
||||
api_key: fake # vLLM key
|
||||
- model_name: llama-3.1-8b
|
||||
litellm_params:
|
||||
model: openai/meta-llama/Llama-3.1-8B-Instruct
|
||||
api_base: http://vllm-2:8000/v1 # second replica — auto load balanced
|
||||
api_key: fake
|
||||
|
||||
# Fallback: cheap model if primary fails
|
||||
- model_name: gpt-4o
|
||||
litellm_params:
|
||||
model: openai/gpt-4o-mini # fallback to cheaper model
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
router_settings:
|
||||
routing_strategy: least-busy # or: latency-based, simple-shuffle
|
||||
num_retries: 3
|
||||
retry_after: 5
|
||||
allowed_fails: 2
|
||||
cooldown_time: 60
|
||||
|
||||
# Fallback configuration
|
||||
fallbacks:
|
||||
- gpt-4o: [claude-sonnet-4-6]
|
||||
- claude-sonnet-4-6: [gpt-4o]
|
||||
|
||||
litellm_settings:
|
||||
# Semantic caching
|
||||
cache: true
|
||||
cache_params:
|
||||
type: redis
|
||||
host: redis
|
||||
port: 6379
|
||||
similarity_threshold: 0.90 # cache if >90% semantic similarity
|
||||
|
||||
# Logging
|
||||
success_callback: ["langfuse"]
|
||||
failure_callback: ["langfuse"]
|
||||
langfuse_public_key: os.environ/LANGFUSE_PUBLIC_KEY
|
||||
langfuse_secret_key: os.environ/LANGFUSE_SECRET_KEY
|
||||
|
||||
general_settings:
|
||||
master_key: os.environ/LITELLM_MASTER_KEY
|
||||
database_url: postgresql://litellm:password@postgres:5432/litellm
|
||||
store_model_in_db: true
|
||||
```
|
||||
|
||||
## Docker Compose: Full Gateway Stack
|
||||
|
||||
```yaml
|
||||
services:
|
||||
litellm:
|
||||
image: ghcr.io/berriai/litellm:main-latest
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
volumes:
|
||||
- ./litellm-config.yaml:/app/config.yaml
|
||||
ports:
|
||||
- "4000:4000"
|
||||
environment:
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
|
||||
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
|
||||
- DATABASE_URL=postgresql://litellm:password@postgres:5432/litellm
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_started
|
||||
restart: unless-stopped
|
||||
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_DB: litellm
|
||||
POSTGRES_USER: litellm
|
||||
POSTGRES_PASSWORD: password
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U litellm"]
|
||||
interval: 5s
|
||||
retries: 5
|
||||
restart: unless-stopped
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
redis-data:
|
||||
```
|
||||
|
||||
## Virtual Keys & Rate Limiting
|
||||
|
||||
```bash
|
||||
# Create a virtual API key for a team (via LiteLLM API)
|
||||
curl -X POST http://localhost:4000/key/generate \
|
||||
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"team_id": "team-backend",
|
||||
"key_alias": "backend-team-key",
|
||||
"models": ["gpt-4o-mini", "llama-3.1-8b"],
|
||||
"max_budget": 100, # USD limit
|
||||
"budget_duration": "monthly",
|
||||
"rpm_limit": 100, # requests per minute
|
||||
"tpm_limit": 500000 # tokens per minute
|
||||
}'
|
||||
|
||||
# View spend
|
||||
curl http://localhost:4000/spend/keys \
|
||||
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
|
||||
```
|
||||
|
||||
## Nginx Load Balancer (Alternative/Complement)
|
||||
|
||||
```nginx
|
||||
# nginx.conf — round-robin across vLLM replicas
|
||||
upstream vllm_backends {
|
||||
least_conn;
|
||||
server vllm-1:8000 max_fails=3 fail_timeout=30s;
|
||||
server vllm-2:8000 max_fails=3 fail_timeout=30s;
|
||||
server vllm-3:8000 max_fails=3 fail_timeout=30s;
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name llm-api.internal;
|
||||
|
||||
# Rate limiting
|
||||
limit_req_zone $http_authorization zone=per_key:10m rate=100r/m;
|
||||
limit_req zone=per_key burst=20 nodelay;
|
||||
|
||||
location /v1/ {
|
||||
proxy_pass http://vllm_backends;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_read_timeout 300s; # long timeout for streaming
|
||||
proxy_buffering off; # required for SSE streaming
|
||||
proxy_cache_bypass 1;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Monitoring Gateway Health
|
||||
|
||||
```bash
|
||||
# Check LiteLLM health
|
||||
curl http://localhost:4000/health
|
||||
|
||||
# Model-level health
|
||||
curl http://localhost:4000/health/liveliness
|
||||
|
||||
# Spend by model
|
||||
curl http://localhost:4000/spend/models \
|
||||
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
|
||||
|
||||
# Active virtual keys
|
||||
curl http://localhost:4000/key/list \
|
||||
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `ConnectionRefusedError` to backend | Backend not reachable | Check `api_base` URL; verify backend is healthy |
|
||||
| Rate limit errors (429) | Budget/RPM exceeded | Increase limits or rotate to fallback model |
|
||||
| Slow streaming responses | `proxy_buffering` enabled | Set `proxy_buffering off` in Nginx |
|
||||
| Cache miss rate high | Threshold too strict | Lower `similarity_threshold` to `0.85` |
|
||||
| Postgres connection errors | DB not ready | Add `depends_on` with `condition: service_healthy` |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use virtual keys per team/app — never expose raw provider API keys.
|
||||
- Enable `cache: true` with Redis for repeated or similar queries; can cut costs 30–50%.
|
||||
- Set `num_retries: 3` with fallbacks to handle provider outages gracefully.
|
||||
- Log all requests to Langfuse or OpenTelemetry for cost attribution and debugging.
|
||||
- Use `least-busy` routing strategy for self-hosted models to avoid GPU saturation.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [vllm-server](../../local-ai/vllm-server/) - Backend inference server
|
||||
- [llm-inference-scaling](../../local-ai/llm-inference-scaling/) - Auto-scaling backends
|
||||
- [llm-caching](../../../devops/ai/llm-caching/) - Semantic cache patterns
|
||||
- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost management
|
||||
@@ -0,0 +1,221 @@
|
||||
---
|
||||
name: gpu-server-management
|
||||
description: Set up and manage NVIDIA GPU servers for AI workloads — driver installation, CUDA toolkit, container toolkit, MIG partitioning, GPU health monitoring, and multi-GPU configuration for LLM inference and training.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# GPU Server Management
|
||||
|
||||
Provision, configure, and monitor NVIDIA GPU servers for AI inference and training workloads.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Setting up a new GPU server for LLM inference or model training
|
||||
- Installing or upgrading NVIDIA drivers and CUDA toolkit
|
||||
- Configuring Docker with NVIDIA Container Toolkit for GPU workloads
|
||||
- Partitioning A100/H100 GPUs with MIG for multi-tenant workloads
|
||||
- Troubleshooting GPU errors, driver issues, or thermal throttling
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Ubuntu 22.04 LTS (recommended) or RHEL 8/9
|
||||
- NVIDIA GPU (A10G, A100, H100, RTX 4090, or L40S recommended)
|
||||
- Root or sudo access
|
||||
- Internet access for package downloads
|
||||
|
||||
## Driver Installation (Ubuntu)
|
||||
|
||||
```bash
|
||||
# Remove old drivers
|
||||
sudo apt purge -y 'nvidia*' 'cuda*' 'libcuda*'
|
||||
sudo apt autoremove -y
|
||||
|
||||
# Add NVIDIA package repository
|
||||
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
|
||||
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
||||
|
||||
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
|
||||
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
|
||||
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
||||
|
||||
sudo apt update
|
||||
|
||||
# Install latest driver (560.x as of 2025)
|
||||
sudo apt install -y nvidia-driver-560 cuda-toolkit-12-6
|
||||
|
||||
# Install NVIDIA Container Toolkit (Docker GPU support)
|
||||
sudo apt install -y nvidia-container-toolkit
|
||||
sudo nvidia-ctk runtime configure --runtime=docker
|
||||
sudo systemctl restart docker
|
||||
|
||||
# Verify
|
||||
nvidia-smi
|
||||
nvcc --version
|
||||
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi
|
||||
```
|
||||
|
||||
## Post-Install Configuration
|
||||
|
||||
```bash
|
||||
# Enable persistence mode (reduces driver initialization latency)
|
||||
sudo nvidia-smi -pm 1
|
||||
|
||||
# Set power limit (reduce heat/noise on inference servers)
|
||||
sudo nvidia-smi -pl 350 # watts; check TDP for your GPU model
|
||||
|
||||
# Disable ECC on inference servers (frees ~6% VRAM, less safe)
|
||||
sudo nvidia-smi --ecc-config=0 # requires reboot
|
||||
|
||||
# Enable P2P for multi-GPU NVLink training
|
||||
sudo nvidia-smi topo -m # check NVLink topology
|
||||
```
|
||||
|
||||
## GPU Health Monitoring
|
||||
|
||||
```bash
|
||||
# Real-time monitoring (like htop for GPUs)
|
||||
watch -n 1 nvidia-smi
|
||||
|
||||
# Detailed stats
|
||||
nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,\
|
||||
utilization.memory,memory.used,memory.free,power.draw,clocks.current.graphics \
|
||||
--format=csv --loop=1
|
||||
|
||||
# DCGM — production monitoring daemon (for clusters)
|
||||
sudo apt install -y datacenter-gpu-manager
|
||||
sudo systemctl start dcgm
|
||||
dcgmi discovery -l # list GPUs
|
||||
dcgmi diag -r 1 # quick health check
|
||||
dcgmi diag -r 3 # full diagnostic (takes ~20 min)
|
||||
|
||||
# Check GPU errors (XID errors — important for stability)
|
||||
sudo dmesg | grep -i "NVRM\|nvidia\|XID"
|
||||
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total \
|
||||
--format=csv,noheader
|
||||
```
|
||||
|
||||
## Prometheus GPU Metrics (DCGM Exporter)
|
||||
|
||||
```bash
|
||||
# Deploy DCGM Exporter for Prometheus scraping
|
||||
docker run -d \
|
||||
--name dcgm-exporter \
|
||||
--gpus all \
|
||||
--cap-add SYS_ADMIN \
|
||||
-p 9400:9400 \
|
||||
--restart unless-stopped \
|
||||
nvcr.io/nvidia/k8s/dcgm-exporter:latest
|
||||
|
||||
# Key metrics exposed:
|
||||
# DCGM_FI_DEV_GPU_UTIL - GPU utilization %
|
||||
# DCGM_FI_DEV_MEM_COPY_UTIL - Memory bandwidth utilization
|
||||
# DCGM_FI_DEV_FB_USED - Framebuffer memory used (MB)
|
||||
# DCGM_FI_DEV_SM_CLOCK - SM clock speed (MHz)
|
||||
# DCGM_FI_DEV_GPU_TEMP - Temperature (°C)
|
||||
# DCGM_FI_DEV_POWER_USAGE - Power draw (W)
|
||||
# DCGM_FI_DEV_XID_ERRORS - XID error count (0 = healthy)
|
||||
```
|
||||
|
||||
## MIG Partitioning (A100/H100)
|
||||
|
||||
MIG (Multi-Instance GPU) allows slicing one GPU into isolated smaller GPUs.
|
||||
|
||||
```bash
|
||||
# Enable MIG mode (requires reboot or restart of all processes)
|
||||
sudo nvidia-smi -mig 1
|
||||
sudo systemctl restart nvidia-persistenced
|
||||
|
||||
# List available MIG profiles (A100 80GB example)
|
||||
nvidia-smi mig -lgip
|
||||
# 1g.10gb — 1 slice, 10GB (max 7 instances)
|
||||
# 2g.20gb — 2 slices, 20GB (max 3 instances)
|
||||
# 3g.40gb — 3 slices, 40GB (max 2 instances)
|
||||
# 7g.80gb — full GPU, 80GB (max 1 instance)
|
||||
|
||||
# Create MIG instances (e.g., 3× 2g.20gb + 1× 2g.20gb = multi-tenant)
|
||||
sudo nvidia-smi mig -cgi 2g.20gb,2g.20gb,2g.20gb,2g.20gb -C
|
||||
|
||||
# List created instances
|
||||
nvidia-smi mig -lgi
|
||||
nvidia-smi mig -lcgi
|
||||
|
||||
# Use in Docker
|
||||
docker run --gpus '"device=MIG-GPU-xxx/0/0"' ...
|
||||
|
||||
# Disable MIG
|
||||
sudo nvidia-smi mig -i 0 -dci
|
||||
sudo nvidia-smi mig -i 0 -dgi
|
||||
sudo nvidia-smi -mig 0
|
||||
```
|
||||
|
||||
## Kernel & OS Tuning for GPU Servers
|
||||
|
||||
```bash
|
||||
# Increase file descriptor limits
|
||||
echo '* soft nofile 1048576' | sudo tee -a /etc/security/limits.conf
|
||||
echo '* hard nofile 1048576' | sudo tee -a /etc/security/limits.conf
|
||||
|
||||
# Disable transparent huge pages (reduces latency jitter)
|
||||
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
|
||||
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
|
||||
|
||||
# Persist via rc.local or systemd unit:
|
||||
cat <<'EOF' | sudo tee /etc/rc.local
|
||||
#!/bin/bash
|
||||
echo never > /sys/kernel/mm/transparent_hugepage/enabled
|
||||
echo never > /sys/kernel/mm/transparent_hugepage/defrag
|
||||
nvidia-smi -pm 1
|
||||
exit 0
|
||||
EOF
|
||||
sudo chmod +x /etc/rc.local
|
||||
|
||||
# PCIe performance mode
|
||||
sudo nvidia-smi --auto-boost-default=0
|
||||
sudo nvidia-smi --auto-boost-permission=0
|
||||
```
|
||||
|
||||
## Multi-GPU Topology Check
|
||||
|
||||
```bash
|
||||
# Check NVLink and PCIe topology
|
||||
nvidia-smi topo -m
|
||||
# Output shows interconnect type:
|
||||
# NV4 = NVLink 4.0 (H100 SXM)
|
||||
# NV2 = NVLink 2.0 (A100 SXM)
|
||||
# PHB = PCIe bus (slower; avoid for tensor parallel training)
|
||||
# PIX = same PCIe switch (fast)
|
||||
|
||||
# Bandwidth test between GPUs
|
||||
/usr/local/cuda/samples/bin/x86_64/linux/release/p2pBandwidthLatencyTest
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `nvidia-smi: command not found` | Driver not installed | Follow driver installation steps above |
|
||||
| Driver version mismatch | CUDA/driver incompatibility | Check compatibility matrix at developer.nvidia.com |
|
||||
| GPU temperature >85°C | Poor airflow or fan failure | Check `nvidia-smi -q -d TEMPERATURE`; reseat cooler |
|
||||
| XID 79 errors | GPU hardware error | Run `dcgmi diag -r 3`; may need GPU replacement |
|
||||
| `failed to open device` in container | Container toolkit not configured | Run `nvidia-ctk runtime configure --runtime=docker` |
|
||||
| Low PCIe bandwidth | Wrong slot or power limit | Check `nvidia-smi -q | grep PCIe`; use x16 slot |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Always enable persistence mode (`nvidia-smi -pm 1`) — reduces first-request latency.
|
||||
- Monitor XID errors; persistent XID 79/94 indicates hardware failure.
|
||||
- For training: use NVLink-connected GPUs; for inference: PCIe is usually fine.
|
||||
- Set up DCGM alerts on temperature >80°C and power draw near TDP.
|
||||
- Use MIG for multi-tenant inference to provide GPU isolation between models.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [vllm-server](../../local-ai/vllm-server/) - LLM inference on GPUs
|
||||
- [llm-fine-tuning](../../local-ai/llm-fine-tuning/) - GPU training setup
|
||||
- [linux-hardening](../../../security/hardening/linux-hardening/) - Secure the host OS
|
||||
- [prometheus-grafana](../../../devops/observability/prometheus-grafana/) - Metrics dashboards
|
||||
Reference in New Issue
Block a user