From dd77232b16c4f6445c51347eb516c3e98ac9c8d4 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 2 Mar 2026 01:18:29 +0000 Subject: [PATCH] Add 12 AI infrastructure and LLM operations skills New skills covering hot-topic AI engineering subjects: Local AI Infrastructure: - vllm-server: High-throughput LLM serving with PagedAttention, tensor parallelism, quantization - llm-inference-scaling: KEDA-based GPU autoscaling for LLM inference on Kubernetes - rag-infrastructure: Production RAG with hybrid search, reranking, and embedding pipelines - llm-fine-tuning: QLoRA/LoRA fine-tuning with Axolotl, DeepSpeed ZeRO-3, and DPO alignment Infrastructure: - gpu-server-management: NVIDIA driver setup, MIG partitioning, DCGM monitoring - vector-database-ops: Qdrant, Weaviate, pgvector for production AI search - llm-gateway: LiteLLM-based API gateway with rate limiting, virtual keys, fallback routing DevOps/AI: - llm-cost-optimization: Model right-sizing, prompt/semantic caching, batch API, break-even analysis - llm-caching: Multi-layer exact + semantic + provider caching to cut costs 30-70% - ai-pipeline-orchestration: Prefect/Airflow/Dagster for RAG ingestion and training workflows Orchestration: - model-serving-kubernetes: KServe + Triton with canary deployments and GPU autoscaling Security: - ai-security-hardening: Prompt injection defense, PII scrubbing, model weight verification https://claude.ai/code/session_011MN1C4PrkCeg2Qmi7q1ZUe --- README.md | 12 + devops/ai/ai-pipeline-orchestration/SKILL.md | 262 +++++++++++++++ devops/ai/llm-caching/SKILL.md | 309 +++++++++++++++++ devops/ai/llm-cost-optimization/SKILL.md | 286 ++++++++++++++++ .../model-serving-kubernetes/SKILL.md | 314 +++++++++++++++++ .../databases/vector-database-ops/SKILL.md | 285 ++++++++++++++++ .../local-ai/llm-fine-tuning/SKILL.md | 312 +++++++++++++++++ .../local-ai/llm-inference-scaling/SKILL.md | 270 +++++++++++++++ .../local-ai/rag-infrastructure/SKILL.md | 253 ++++++++++++++ infrastructure/local-ai/vllm-server/SKILL.md | 219 ++++++++++++ .../networking/llm-gateway/SKILL.md | 266 +++++++++++++++ .../servers/gpu-server-management/SKILL.md | 221 ++++++++++++ security/ai/ai-security-hardening/SKILL.md | 318 ++++++++++++++++++ 13 files changed, 3327 insertions(+) create mode 100644 devops/ai/ai-pipeline-orchestration/SKILL.md create mode 100644 devops/ai/llm-caching/SKILL.md create mode 100644 devops/ai/llm-cost-optimization/SKILL.md create mode 100644 devops/orchestration/model-serving-kubernetes/SKILL.md create mode 100644 infrastructure/databases/vector-database-ops/SKILL.md create mode 100644 infrastructure/local-ai/llm-fine-tuning/SKILL.md create mode 100644 infrastructure/local-ai/llm-inference-scaling/SKILL.md create mode 100644 infrastructure/local-ai/rag-infrastructure/SKILL.md create mode 100644 infrastructure/local-ai/vllm-server/SKILL.md create mode 100644 infrastructure/networking/llm-gateway/SKILL.md create mode 100644 infrastructure/servers/gpu-server-management/SKILL.md create mode 100644 security/ai/ai-security-hardening/SKILL.md diff --git a/README.md b/README.md index ac2c42e..2b590e6 100644 --- a/README.md +++ b/README.md @@ -195,6 +195,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's | [argocd-gitops](devops/orchestration/argocd-gitops/) | GitOps with ArgoCD | | [kustomize](devops/orchestration/kustomize/) | Kubernetes manifest customization | | [openshift](devops/orchestration/openshift/) | OpenShift cluster management | +| [model-serving-kubernetes](devops/orchestration/model-serving-kubernetes/) | KServe and Triton model serving with canary deployments and GPU autoscaling | ### Observability | Skill | Description | @@ -212,6 +213,9 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's |-------|-------------| | [agent-observability](devops/ai/agent-observability/) | Tracing, latency, token, and cost telemetry for agents | | [agent-evals](devops/ai/agent-evals/) | Automated regression and safety eval suites for agents | +| [llm-cost-optimization](devops/ai/llm-cost-optimization/) | Cut LLM API costs with caching, batching, model routing, and self-hosting | +| [llm-caching](devops/ai/llm-caching/) | Exact and semantic caching layers to reduce API calls by 30–70% | +| [ai-pipeline-orchestration](devops/ai/ai-pipeline-orchestration/) | Orchestrate RAG ingestion, training, and batch inference with Prefect/Airflow | ### Release Management | Skill | Description | @@ -276,6 +280,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's |-------|-------------| | [ai-agent-security](security/ai/ai-agent-security/) | Defend agents against injection, tool abuse, and exfiltration | | [llm-app-security](security/ai/llm-app-security/) | Harden LLM app inputs, outputs, and tenant isolation | +| [ai-security-hardening](security/ai/ai-security-hardening/) | Harden LLM deployments against prompt injection, model theft, and data exfiltration | @@ -334,6 +339,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's | [user-management](infrastructure/servers/user-management/) | Users, groups, sudo | | [systemd-services](infrastructure/servers/systemd-services/) | Services and timers | | [performance-tuning](infrastructure/servers/performance-tuning/) | System optimization | +| [gpu-server-management](infrastructure/servers/gpu-server-management/) | NVIDIA GPU driver setup, MIG partitioning, DCGM monitoring for AI workloads | ### Networking | Skill | Description | @@ -343,6 +349,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's | [cdn-setup](infrastructure/networking/cdn-setup/) | CloudFront, Cloudflare | | [reverse-proxy](infrastructure/networking/reverse-proxy/) | nginx, Traefik | | [service-mesh](infrastructure/networking/service-mesh/) | Istio, Linkerd | +| [llm-gateway](infrastructure/networking/llm-gateway/) | Unified LLM API gateway with routing, rate limiting, virtual keys, and semantic caching | ### Databases | Skill | Description | @@ -353,6 +360,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's | [mongodb](infrastructure/databases/mongodb/) | MongoDB clusters | | [redis](infrastructure/databases/redis/) | Redis caching | | [database-backups](infrastructure/databases/database-backups/) | Backup strategies | +| [vector-database-ops](infrastructure/databases/vector-database-ops/) | Qdrant, Weaviate, and pgvector for production AI search and RAG workloads | ### Storage | Skill | Description | @@ -375,6 +383,10 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's | [ollama-stack](infrastructure/local-ai/ollama-stack/) | Private local inference stack with Ollama | | [mac-mini-llm-lab](infrastructure/local-ai/mac-mini-llm-lab/) | Mac mini setup for always-on local LLM serving | | [openclaw-local-mac-mini](infrastructure/local-ai/openclaw-local-mac-mini/) | OpenClaw setup for local development and Mac mini hosting | +| [vllm-server](infrastructure/local-ai/vllm-server/) | High-throughput LLM serving with vLLM — PagedAttention, tensor parallelism, OpenAI API | +| [llm-inference-scaling](infrastructure/local-ai/llm-inference-scaling/) | Auto-scale LLM inference clusters on Kubernetes with KEDA and GPU-aware scheduling | +| [rag-infrastructure](infrastructure/local-ai/rag-infrastructure/) | Production RAG with vector stores, hybrid search, embedding pipelines, and reranking | +| [llm-fine-tuning](infrastructure/local-ai/llm-fine-tuning/) | QLoRA and full fine-tuning with Axolotl, DeepSpeed, and DPO alignment on GPU clusters | ### IT Operations | Skill | Description | diff --git a/devops/ai/ai-pipeline-orchestration/SKILL.md b/devops/ai/ai-pipeline-orchestration/SKILL.md new file mode 100644 index 0000000..4d97ad8 --- /dev/null +++ b/devops/ai/ai-pipeline-orchestration/SKILL.md @@ -0,0 +1,262 @@ +--- +name: ai-pipeline-orchestration +description: Orchestrate AI/ML pipelines for data ingestion, model training, batch inference, and RAG indexing using Prefect, Airflow, or Dagster. Build reliable, observable, and retriable workflows for production AI systems. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# AI Pipeline Orchestration + +Build reliable, observable AI workflows — from document ingestion to batch inference to model training pipelines. + +## When to Use This Skill + +Use this skill when: +- Scheduling recurring RAG document ingestion and re-indexing +- Orchestrating multi-step batch LLM processing workflows +- Running nightly model evaluation and fine-tuning jobs +- Building ETL pipelines that feed into AI models +- Managing dependencies between data preparation and model serving + +## Tool Selection + +| Tool | Best For | Complexity | GPU Jobs | +|------|----------|------------|----------| +| **Prefect** | Modern Python-first; easy to adopt | Low | Good | +| **Airflow** | Complex DAGs; large teams; existing usage | High | Good | +| **Dagster** | Asset-centric; strong data lineage | Medium | Excellent | +| **Temporal** | Long-running workflows; reliability-first | Medium | Good | + +## Prefect — Quick Start + +```bash +pip install prefect prefect-kubernetes + +# Start Prefect server (or use Prefect Cloud) +prefect server start + +# In another terminal +prefect worker start --pool default-agent-pool +``` + +## Prefect: RAG Ingestion Pipeline + +```python +from prefect import flow, task, get_run_logger +from prefect.tasks import task_input_hash +from datetime import timedelta +import hashlib + +@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=24)) +def fetch_documents(source_url: str) -> list[dict]: + """Fetch documents from source; cached to avoid re-fetching.""" + logger = get_run_logger() + logger.info(f"Fetching from {source_url}") + # ... fetch logic + return documents + +@task(retries=3, retry_delay_seconds=30) +def chunk_and_embed(documents: list[dict]) -> list[dict]: + """Chunk documents and generate embeddings with retry on failure.""" + from sentence_transformers import SentenceTransformer + model = SentenceTransformer("BAAI/bge-large-en-v1.5") + chunks = [] + for doc in documents: + doc_chunks = chunk_text(doc["content"]) + embeddings = model.encode(doc_chunks, batch_size=64) + for chunk, emb in zip(doc_chunks, embeddings): + chunks.append({"text": chunk, "embedding": emb.tolist(), + "source": doc["url"], "doc_hash": doc["hash"]}) + return chunks + +@task(retries=2) +def upsert_to_vector_store(chunks: list[dict]) -> int: + """Upsert embeddings to Qdrant, skip unchanged documents.""" + from qdrant_client import QdrantClient + client = QdrantClient("http://qdrant:6333") + client.upsert(collection_name="knowledge-base", points=[...]) + return len(chunks) + +@flow(name="rag-ingestion", log_prints=True) +def rag_ingestion_pipeline(sources: list[str]): + """Full RAG ingestion flow — runs daily.""" + logger = get_run_logger() + total = 0 + for source in sources: + docs = fetch_documents(source) + chunks = chunk_and_embed(docs) + count = upsert_to_vector_store(chunks) + total += count + logger.info(f"Ingested {count} chunks from {source}") + logger.info(f"Pipeline complete: {total} total chunks indexed") + +if __name__ == "__main__": + rag_ingestion_pipeline.serve( + name="daily-rag-ingestion", + cron="0 2 * * *", # 2 AM daily + parameters={"sources": ["https://docs.myapp.com", "https://api.myapp.com/kb"]}, + ) +``` + +## Prefect: Batch LLM Inference Pipeline + +```python +from prefect import flow, task +from prefect.concurrency.sync import concurrency +import asyncio +from openai import AsyncOpenAI + +@task(retries=3, retry_delay_seconds=60) +async def process_batch(items: list[dict], model: str = "gpt-4o-mini") -> list[dict]: + """Process a batch of items through LLM with rate limit protection.""" + client = AsyncOpenAI() + async with concurrency("openai-api", occupy=len(items)): # rate limit + tasks = [ + client.chat.completions.create( + model=model, + messages=[{"role": "user", "content": item["prompt"]}], + max_tokens=256, + ) + for item in items + ] + responses = await asyncio.gather(*tasks, return_exceptions=True) + + results = [] + for item, response in zip(items, responses): + if isinstance(response, Exception): + results.append({**item, "error": str(response), "output": None}) + else: + results.append({**item, "output": response.choices[0].message.content}) + return results + +@flow(name="batch-llm-inference") +async def batch_inference_flow(input_file: str, output_file: str, batch_size: int = 50): + import json + items = [json.loads(line) for line in open(input_file)] + batches = [items[i:i+batch_size] for i in range(0, len(items), batch_size)] + + all_results = [] + for batch in batches: + results = await process_batch(batch) + all_results.extend(results) + + with open(output_file, "w") as f: + for result in all_results: + f.write(json.dumps(result) + "\n") + return len(all_results) +``` + +## Airflow: Model Training DAG + +```python +from airflow.decorators import dag, task +from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator +from datetime import datetime +from kubernetes.client import models as k8s + +@dag( + dag_id="llm_fine_tuning", + schedule="@weekly", + start_date=datetime(2025, 1, 1), + catchup=False, + tags=["ai", "training"], +) +def llm_fine_tuning_dag(): + + @task + def prepare_dataset() -> str: + """Download and preprocess training data.""" + # ... data prep logic + return "s3://my-bucket/training-data/2025-03-01/" + + train = KubernetesPodOperator( + task_id="train_model", + name="llm-training-job", + namespace="ml", + image="nvcr.io/nvidia/pytorch:24.05-py3", + cmds=["accelerate", "launch", "-m", "axolotl.cli.train", "/config/config.yaml"], + resources=k8s.V1ResourceRequirements( + limits={"nvidia.com/gpu": "4", "memory": "320Gi"}, + requests={"nvidia.com/gpu": "4"}, + ), + node_selector={"nvidia.com/gpu.product": "A100-SXM4-80GB"}, + volumes=[...], + volume_mounts=[...], + get_logs=True, + is_delete_operator_pod=True, + ) + + @task + def evaluate_model(dataset_path: str) -> dict: + """Run evals; fail pipeline if quality drops.""" + metrics = run_evals() + if metrics["accuracy"] < 0.85: + raise ValueError(f"Model quality too low: {metrics}") + return metrics + + @task + def deploy_model(metrics: dict): + """Push merged model to HF Hub and update vLLM config.""" + update_serving_config(new_model="org/fine-tuned-v2") + + dataset = prepare_dataset() + train.set_upstream(dataset) + eval_result = evaluate_model(dataset) + eval_result.set_upstream(train) + deploy_model(eval_result) + +llm_fine_tuning_dag() +``` + +## Dagster: Asset-Based AI Pipeline + +```python +from dagster import asset, AssetExecutionContext, define_asset_job, ScheduleDefinition + +@asset(description="Raw documents fetched from knowledge sources") +def raw_documents(context: AssetExecutionContext) -> list[dict]: + context.log.info("Fetching documents...") + return fetch_all_documents() + +@asset( + deps=[raw_documents], + description="Chunked and embedded document vectors", +) +def document_embeddings(context: AssetExecutionContext, raw_documents) -> int: + chunks = process_and_embed(raw_documents) + context.log.info(f"Generated {len(chunks)} embeddings") + upsert_to_qdrant(chunks) + return len(chunks) + +@asset( + deps=[document_embeddings], + description="RAG system quality metrics", +) +def rag_quality_metrics(context: AssetExecutionContext) -> dict: + metrics = evaluate_rag_system() + context.add_output_metadata({"ragas_score": metrics["ragas_score"]}) + return metrics + +# Schedule: refresh embeddings nightly +nightly_refresh = ScheduleDefinition( + job=define_asset_job("rag_refresh_job", [raw_documents, document_embeddings]), + cron_schedule="0 1 * * *", +) +``` + +## Best Practices + +- Use task-level retries for API calls; use flow-level retries for transient infra failures. +- Cache expensive steps (embedding generation, data fetching) to speed up reruns. +- Emit custom metrics from pipelines (chunk count, error rate, cost) to your observability stack. +- Use `concurrency` limits in Prefect or `pool` slots in Airflow to respect external rate limits. +- Separate ingestion, training, and deployment pipelines — don't couple them in one giant DAG. + +## Related Skills + +- [rag-infrastructure](../../infrastructure/local-ai/rag-infrastructure/) - RAG system setup +- [llm-fine-tuning](../../infrastructure/local-ai/llm-fine-tuning/) - Training jobs +- [agent-observability](../agent-observability/) - Pipeline monitoring +- [kubernetes-ops](../orchestration/kubernetes-ops/) - Running pipeline pods on K8s diff --git a/devops/ai/llm-caching/SKILL.md b/devops/ai/llm-caching/SKILL.md new file mode 100644 index 0000000..ea0e8ba --- /dev/null +++ b/devops/ai/llm-caching/SKILL.md @@ -0,0 +1,309 @@ +--- +name: llm-caching +description: Implement multi-layer LLM caching with exact match, semantic similarity, and provider-side prompt caching. Reduce API costs by 30–70%, cut latency, and improve throughput using Redis, GPTCache, and provider caching APIs. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# LLM Caching + +Cut LLM costs and latency with exact match, semantic, and provider-side caching layers. + +## When to Use This Skill + +Use this skill when: +- The same or similar queries are asked repeatedly (FAQ bots, support tools) +- LLM API costs are growing and you need immediate savings +- Serving high request volumes where repeated queries cause bottlenecks +- Implementing prompt caching for long system prompts (Anthropic/OpenAI) +- Building offline-capable AI features that need response persistence + +## Caching Layers + +``` +Request → Exact Cache → Semantic Cache → Provider Cache → LLM API + ↓ hit ↓ hit ↓ hit + instant ~5ms 50-80% cheaper +``` + +## Layer 1: Exact Match Cache (Redis) + +```python +import hashlib +import json +import redis +from openai import OpenAI + +r = redis.Redis(host="localhost", port=6379, decode_responses=True) +client = OpenAI() + +def build_cache_key(model: str, messages: list, temperature: float) -> str: + """Deterministic key from request parameters.""" + payload = json.dumps({ + "model": model, + "messages": messages, + "temperature": temperature, + }, sort_keys=True) + return f"llm:exact:{hashlib.sha256(payload.encode()).hexdigest()}" + +def cached_completion(model: str, messages: list, temperature: float = 0.0, + ttl: int = 3600) -> dict: + key = build_cache_key(model, messages, temperature) + + # Check cache + if cached := r.get(key): + return json.loads(cached) + + # Call API + response = client.chat.completions.create( + model=model, messages=messages, temperature=temperature + ) + result = response.model_dump() + + # Cache result (only cache deterministic responses) + if temperature == 0.0: + r.setex(key, ttl, json.dumps(result)) + + return result +``` + +## Layer 2: Semantic Cache (GPTCache) + +```python +from gptcache import cache, Config +from gptcache.adapter import openai +from gptcache.embedding import Onnx +from gptcache.manager import CacheBase, VectorBase, get_data_manager +from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation + +# Configure GPTCache with Qdrant backend +def init_gptcache(cache_obj, llm: str): + onnx = Onnx() # local embedding model + data_manager = get_data_manager( + CacheBase("redis"), # metadata store + VectorBase("qdrant", + host="localhost", + port=6333, + collection_name=f"llm-cache-{llm}", + dimension=onnx.dimension), + ) + cache_obj.init( + embedding_func=onnx.to_embeddings, + data_manager=data_manager, + similarity_evaluation=SearchDistanceEvaluation(), + config=Config(similarity_threshold=0.80), # 80% similarity = cache hit + ) + +cache.set_openai_key() +init_gptcache(cache, "gpt-4o-mini") + +# Now openai calls are automatically cached +response = openai.ChatCompletion.create( + model="gpt-4o-mini", + messages=[{"role": "user", "content": "What is machine learning?"}], +) +# Second call with similar question ("Explain machine learning") → cache hit +``` + +## Custom Semantic Cache (Production-Grade) + +```python +from sentence_transformers import SentenceTransformer +from qdrant_client import QdrantClient +from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, Range +import numpy as np +import uuid +import time + +embed_model = SentenceTransformer("BAAI/bge-small-en-v1.5") # fast, 33M params +qdrant = QdrantClient("http://localhost:6333") + +CACHE_COLLECTION = "semantic-cache" +SIMILARITY_THRESHOLD = 0.88 +CACHE_TTL_SECONDS = 86400 # 24h + +# Create collection once +qdrant.create_collection( + collection_name=CACHE_COLLECTION, + vectors_config=VectorParams(size=384, distance=Distance.COSINE), + on_disk_payload=True, +) + +def semantic_cache_lookup(query: str, model: str) -> str | None: + embedding = embed_model.encode(query).tolist() + results = qdrant.query_points( + collection_name=CACHE_COLLECTION, + query=embedding, + query_filter=Filter(must=[ + FieldCondition(key="model", match={"value": model}), + FieldCondition(key="expires_at", range=Range(gte=time.time())), + ]), + limit=1, + score_threshold=SIMILARITY_THRESHOLD, + ) + if results.points: + return results.points[0].payload["response"] + return None + +def semantic_cache_store(query: str, response: str, model: str): + embedding = embed_model.encode(query).tolist() + qdrant.upsert( + collection_name=CACHE_COLLECTION, + points=[PointStruct( + id=str(uuid.uuid4()), + vector=embedding, + payload={ + "query": query, + "response": response, + "model": model, + "created_at": time.time(), + "expires_at": time.time() + CACHE_TTL_SECONDS, + }, + )], + ) + +def smart_llm_call(query: str, model: str = "gpt-4o-mini") -> dict: + # 1. Semantic lookup + if cached_response := semantic_cache_lookup(query, model): + return {"response": cached_response, "source": "semantic_cache", "cost": 0} + + # 2. LLM call + response = client.chat.completions.create( + model=model, + messages=[{"role": "user", "content": query}], + ) + text = response.choices[0].message.content + cost = litellm.completion_cost(response) + + # 3. Store in cache + semantic_cache_store(query, text, model) + + return {"response": text, "source": "llm_api", "cost": cost} +``` + +## Layer 3: Provider-Side Prompt Caching + +```python +# Anthropic — cache long system prompts (saves 90% on cached input tokens) +import anthropic + +client = anthropic.Anthropic() + +# Long system prompt — mark for caching +SYSTEM_PROMPT = open("knowledge-base.txt").read() # e.g., 50k tokens + +def call_with_prompt_cache(user_question: str) -> str: + response = client.messages.create( + model="claude-sonnet-4-6", + max_tokens=1024, + system=[ + {"type": "text", "text": "You are a helpful assistant."}, + { + "type": "text", + "text": SYSTEM_PROMPT, + "cache_control": {"type": "ephemeral"}, # cache this block + } + ], + messages=[{"role": "user", "content": user_question}], + ) + # Log cache efficiency + usage = response.usage + cache_savings = usage.cache_read_input_tokens * 0.9 # 90% discount on cached + print(f"Cache hits: {usage.cache_read_input_tokens} tokens " + f"(saved ~${cache_savings * 3.0 / 1_000_000:.4f})") + return response.content[0].text + +# OpenAI — automatic for repeated prefixes (≥1,024 tokens) +# No code change needed; cached tokens appear in usage.prompt_tokens_details +response = client.chat.completions.create( + model="gpt-4o-mini", + messages=[ + {"role": "system", "content": LONG_SYSTEM_PROMPT}, # auto-cached + {"role": "user", "content": user_question}, + ] +) +cached = response.usage.prompt_tokens_details.cached_tokens +print(f"OpenAI cached {cached} tokens") +``` + +## Cache Warming + +```python +async def warm_cache(common_queries: list[str], model: str): + """Pre-populate cache with known frequent queries.""" + import asyncio + from openai import AsyncOpenAI + + aclient = AsyncOpenAI() + + async def warm_single(query: str): + if not semantic_cache_lookup(query, model): + response = await aclient.chat.completions.create( + model=model, + messages=[{"role": "user", "content": query}], + ) + text = response.choices[0].message.content + semantic_cache_store(query, text, model) + print(f"Warmed: {query[:50]}...") + + await asyncio.gather(*[warm_single(q) for q in common_queries]) + +# Warm on startup +import asyncio +asyncio.run(warm_cache(FREQUENT_QUERIES, "gpt-4o-mini")) +``` + +## Cache Metrics + +```python +from prometheus_client import Counter, Histogram + +cache_hits = Counter("llm_cache_hits_total", "Cache hits", ["cache_layer", "model"]) +cache_misses = Counter("llm_cache_misses_total", "Cache misses", ["model"]) +cache_savings_usd = Counter("llm_cache_savings_usd_total", "USD saved by cache", ["model"]) + +# Use in your smart_llm_call function +if source == "semantic_cache": + cache_hits.labels(cache_layer="semantic", model=model).inc() + cache_savings_usd.labels(model=model).inc(estimated_cost) +else: + cache_misses.labels(model=model).inc() +``` + +## Redis Configuration for LLM Caching + +```bash +# redis.conf tuning for LLM cache workload +maxmemory 8gb +maxmemory-policy allkeys-lru # evict least-recently-used when full +save "" # disable persistence (cache is ephemeral) +appendonly no +tcp-keepalive 60 +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| Low cache hit rate | Threshold too strict | Lower `SIMILARITY_THRESHOLD` to 0.82–0.85 | +| Stale cached responses | Long TTL | Use topic-specific TTLs; invalidate on data updates | +| Cache serving wrong answers | Threshold too loose | Raise threshold or add model-name filtering | +| Redis OOM | No eviction policy | Set `maxmemory` + `allkeys-lru` | +| Slow semantic lookup | Large cache collection | Add payload index on `model` + `expires_at` | + +## Best Practices + +- Start with exact cache — zero cost, instant wins for identical queries. +- Semantic threshold of 0.88–0.92 balances hit rate vs. accuracy; tune with your data. +- Set per-model TTLs: longer for stable knowledge (1 week), shorter for news/events (1 hour). +- Always filter by model name in semantic cache — different models give different answers. +- Log cache hit rate as a KPI; target 30%+ for FAQ-style applications. + +## Related Skills + +- [llm-cost-optimization](../llm-cost-optimization/) - Full cost strategy +- [llm-gateway](../../infrastructure/networking/llm-gateway/) - Gateway-level caching +- [vector-database-ops](../../infrastructure/databases/vector-database-ops/) - Qdrant setup +- [agent-observability](../agent-observability/) - Cache metrics dashboards diff --git a/devops/ai/llm-cost-optimization/SKILL.md b/devops/ai/llm-cost-optimization/SKILL.md new file mode 100644 index 0000000..ae17da2 --- /dev/null +++ b/devops/ai/llm-cost-optimization/SKILL.md @@ -0,0 +1,286 @@ +--- +name: llm-cost-optimization +description: Reduce LLM API and infrastructure costs through model selection, prompt caching, batching, caching, quantization, and self-hosting strategies. Track spend by team and model, set budgets, and implement cost-aware routing. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# LLM Cost Optimization + +Cut LLM costs by 50–90% with the right combination of caching, model selection, prompt optimization, and self-hosting. + +## When to Use This Skill + +Use this skill when: +- LLM API spend is growing faster than revenue +- You need to attribute AI costs to teams, products, or customers +- Implementing caching to avoid redundant LLM calls +- Deciding when to switch from API providers to self-hosted models +- Optimizing prompt length without sacrificing quality + +## Cost Levers by Impact + +| Strategy | Typical Savings | Effort | +|----------|-----------------|--------| +| Semantic caching | 20–50% | Low | +| Model right-sizing | 30–70% | Low | +| Prompt compression | 10–30% | Medium | +| Provider caching (prompt cache) | 10–25% | Low | +| Batching offline workloads | 50% (Batch API) | Medium | +| Self-hosting 7–8B models | 80–95% at scale | High | +| Quantization | 30–50% VRAM cost | Medium | + +## Track Costs First + +```python +# Use LiteLLM's cost tracking (automatic per-model pricing) +import litellm + +response = litellm.completion( + model="gpt-4o-mini", + messages=[{"role": "user", "content": "Hello"}], +) +cost = litellm.completion_cost(response) +print(f"Cost: ${cost:.6f}") + +# Add custom cost callbacks +def log_cost(kwargs, completion_response, start_time, end_time): + cost = kwargs.get("response_cost", 0) + model = kwargs.get("model") + user = kwargs.get("user") + # Send to your analytics DB + db.record_cost(user=user, model=model, cost=cost) + +litellm.success_callback = [log_cost] +``` + +## Model Right-Sizing + +```python +# Route by task complexity — don't use GPT-4o for everything +def get_model_for_task(task_type: str) -> str: + routing = { + "classification": "gpt-4o-mini", # ~30× cheaper than gpt-4o + "summarization": "gpt-4o-mini", + "extraction": "gpt-4o-mini", + "simple_qa": "gpt-4o-mini", + "complex_reasoning": "gpt-4o", + "code_generation": "claude-sonnet-4-6", + "creative_writing": "claude-opus-4-6", + } + return routing.get(task_type, "gpt-4o-mini") + +# Cost comparison (per 1M tokens, 2025 approx.) +# gpt-4o-mini: input $0.15 / output $0.60 +# gpt-4o: input $2.50 / output $10.00 +# claude-sonnet-4-6: input $3.00 / output $15.00 +# llama-3.1-8b (self): ~$0.05–0.10 all-in (GPU amortized) +``` + +## Prompt Caching (Provider-Side) + +```python +# Anthropic — cache long system prompts (saves 90% on cached tokens) +import anthropic + +client = anthropic.Anthropic() + +response = client.messages.create( + model="claude-sonnet-4-6", + max_tokens=1024, + system=[ + { + "type": "text", + "text": "You are a helpful assistant.", + }, + { + "type": "text", + "text": open("large-context.txt").read(), # large doc + "cache_control": {"type": "ephemeral"}, # cache this! + } + ], + messages=[{"role": "user", "content": "Summarize the key points."}], +) +# First call: full price. Subsequent calls: 90% discount on cached part. +print(f"Cache read tokens: {response.usage.cache_read_input_tokens}") + +# OpenAI — prompt caching is automatic for repeated prefixes >1024 tokens +# No code change needed; check usage.prompt_tokens_details.cached_tokens +``` + +## Batching with OpenAI Batch API (50% Discount) + +```python +import json +from openai import OpenAI + +client = OpenAI() + +# Prepare batch requests +requests = [ + { + "custom_id": f"task-{i}", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "model": "gpt-4o-mini", + "messages": [{"role": "user", "content": f"Classify: {text}"}], + "max_tokens": 50, + } + } + for i, text in enumerate(texts) +] + +# Write JSONL file +with open("batch.jsonl", "w") as f: + for req in requests: + f.write(json.dumps(req) + "\n") + +# Upload and create batch +batch_file = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch") +batch = client.batches.create( + input_file_id=batch_file.id, + endpoint="/v1/chat/completions", + completion_window="24h", +) +print(f"Batch ID: {batch.id}") # poll status with client.batches.retrieve(batch.id) +``` + +## Semantic Caching + +```python +import hashlib +import json +import redis +import numpy as np +from sentence_transformers import SentenceTransformer + +r = redis.Redis(host="localhost", port=6379) +embed_model = SentenceTransformer("BAAI/bge-small-en-v1.5") + +SIMILARITY_THRESHOLD = 0.92 +CACHE_TTL = 3600 * 24 # 24 hours + +def cached_llm_call(prompt: str, llm_fn) -> str: + # 1. Exact match (free) + exact_key = f"exact:{hashlib.sha256(prompt.encode()).hexdigest()}" + if cached := r.get(exact_key): + return cached.decode() + + # 2. Semantic match + query_vec = embed_model.encode(prompt) + cached_keys = r.keys("sem:*") + for key in cached_keys: + data = json.loads(r.get(key)) + similarity = np.dot(query_vec, data["embedding"]) / ( + np.linalg.norm(query_vec) * np.linalg.norm(data["embedding"]) + ) + if similarity >= SIMILARITY_THRESHOLD: + return data["response"] + + # 3. Cache miss — call LLM + response = llm_fn(prompt) + + # Store exact match + r.setex(exact_key, CACHE_TTL, response) + + # Store semantic embedding + sem_key = f"sem:{hashlib.sha256(prompt.encode()).hexdigest()}" + r.setex(sem_key, CACHE_TTL, json.dumps({ + "embedding": query_vec.tolist(), + "response": response, + "prompt": prompt, + })) + return response +``` + +## Prompt Compression + +```python +# LLMLingua — compress long prompts by 3–20× with minimal quality loss +from llmlingua import PromptCompressor + +compressor = PromptCompressor( + model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank", + device_map="cpu", +) + +compressed = compressor.compress_prompt( + long_context, + ratio=0.5, # keep 50% of tokens + rank_method="longllmlingua", +) +print(f"Original: {len(long_context.split())} words") +print(f"Compressed: {len(compressed['compressed_prompt'].split())} words") +print(f"Savings: {compressed['saving']}") +``` + +## Self-Hosting Break-Even Calculator + +```python +def break_even_analysis( + monthly_api_spend_usd: float, + gpu_cost_per_hour_usd: float = 2.50, # e.g., A10G on AWS + utilization: float = 0.70, # 70% GPU utilization +) -> dict: + monthly_gpu_cost = gpu_cost_per_hour_usd * 24 * 30 * utilization + break_even = monthly_gpu_cost / monthly_api_spend_usd + recommendation = ( + "Self-host now — strong ROI" if break_even < 0.5 else + "Self-host if traffic grows 2×" if break_even < 0.8 else + "Stick with API — not enough scale yet" + ) + return { + "monthly_gpu_cost": f"${monthly_gpu_cost:.0f}", + "monthly_api_spend": f"${monthly_api_spend_usd:.0f}", + "gpu_as_pct_of_api": f"{break_even*100:.0f}%", + "recommendation": recommendation, + } + +# Example: $5k/month on OpenAI, $2.50/hr A10G +print(break_even_analysis(5000)) +# → gpu_cost ~$1,260/mo = 25% of API spend → self-host now +``` + +## Cost Dashboard (Grafana) + +```python +# Emit cost metrics to Prometheus +from prometheus_client import Counter, Histogram + +llm_cost_total = Counter( + "llm_cost_usd_total", + "Total LLM spend in USD", + ["model", "team", "task_type"], +) +llm_tokens_total = Counter( + "llm_tokens_total", + "Total tokens used", + ["model", "token_type"], # token_type: prompt, completion, cached +) + +def track_call(model, team, task_type, response): + cost = calculate_cost(model, response.usage) + llm_cost_total.labels(model=model, team=team, task_type=task_type).inc(cost) + llm_tokens_total.labels(model=model, token_type="prompt").inc( + response.usage.prompt_tokens) + llm_tokens_total.labels(model=model, token_type="completion").inc( + response.usage.completion_tokens) +``` + +## Best Practices + +- Use `gpt-4o-mini` or `claude-haiku` for 80% of tasks — they're 10–30× cheaper. +- Enable prompt caching for system prompts >1,024 tokens (Anthropic) or >1,024 tokens (OpenAI). +- Audit your top 5 prompts by token count — compress or cache them. +- Set hard budget limits with LiteLLM virtual keys before costs spiral. +- Self-host 7B–8B models when monthly API spend exceeds $2k/month. + +## Related Skills + +- [llm-gateway](../../infrastructure/networking/llm-gateway/) - Centralized cost control +- [llm-caching](../llm-caching/) - Semantic caching patterns +- [vllm-server](../../infrastructure/local-ai/vllm-server/) - Self-hosted inference +- [agent-observability](../agent-observability/) - Token and cost telemetry diff --git a/devops/orchestration/model-serving-kubernetes/SKILL.md b/devops/orchestration/model-serving-kubernetes/SKILL.md new file mode 100644 index 0000000..affc644 --- /dev/null +++ b/devops/orchestration/model-serving-kubernetes/SKILL.md @@ -0,0 +1,314 @@ +--- +name: model-serving-kubernetes +description: Deploy ML models on Kubernetes with KServe (formerly KFServing) and NVIDIA Triton Inference Server. Includes canary deployments, autoscaling, model versioning, A/B testing, and GPU resource management for production model serving. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# Model Serving on Kubernetes + +Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling. + +## When to Use This Skill + +Use this skill when: +- Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale +- Implementing canary deployments and A/B testing for ML models +- Autoscaling inference pods based on request rate or GPU metrics +- Deploying LLMs with Triton or KServe on Kubernetes +- Managing multiple model versions with traffic splitting + +## Prerequisites + +- Kubernetes 1.28+ with GPU nodes +- KServe installed (or Triton standalone) +- `kubectl` and `helm` configured +- NVIDIA GPU Operator installed on cluster + +## KServe Installation + +```bash +# Install KServe with Helm +helm repo add kserve https://kserve.github.io/helm-charts +helm repo update + +helm install kserve kserve/kserve \ + --namespace kserve \ + --create-namespace \ + --set kserve.controller.gateway.ingressGateway.className=nginx + +# Verify +kubectl get pods -n kserve +kubectl get crd | grep kserve +``` + +## Basic InferenceService (KServe) + +```yaml +apiVersion: serving.kserve.io/v1beta1 +kind: InferenceService +metadata: + name: sklearn-iris + namespace: models +spec: + predictor: + sklearn: + storageUri: gs://kfserving-examples/models/sklearn/1.0/model + resources: + requests: + cpu: "1" + memory: 2Gi + limits: + cpu: "2" + memory: 4Gi +``` + +```bash +kubectl apply -f inference-service.yaml + +# Get inference service URL +kubectl get inferenceservice sklearn-iris -n models +# NAME URL READY ... +# sklearn-iris http://sklearn-iris.models.example.com True + +# Test prediction +curl -X POST http://sklearn-iris.models.example.com/v1/models/sklearn-iris:predict \ + -H "Content-Type: application/json" \ + -d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}' +``` + +## GPU-Enabled LLM InferenceService + +```yaml +apiVersion: serving.kserve.io/v1beta1 +kind: InferenceService +metadata: + name: llama-3-8b + namespace: models + annotations: + serving.kserve.io/enable-prometheus-scraping: "true" +spec: + predictor: + containers: + - name: vllm-container + image: vllm/vllm-openai:latest + args: + - "--model" + - "meta-llama/Llama-3.1-8B-Instruct" + - "--tensor-parallel-size" + - "1" + - "--gpu-memory-utilization" + - "0.90" + ports: + - containerPort: 8080 + protocol: TCP + resources: + requests: + nvidia.com/gpu: "1" + memory: "20Gi" + cpu: "4" + limits: + nvidia.com/gpu: "1" + memory: "24Gi" + cpu: "8" + readinessProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 60 + periodSeconds: 10 + env: + - name: HUGGING_FACE_HUB_TOKEN + valueFrom: + secretKeyRef: + name: hf-token + key: token + nodeSelector: + nvidia.com/gpu.present: "true" + transformer: + containers: + - name: kserve-container + image: kserve/kserve-transformer:latest +``` + +## Canary Deployment (Traffic Splitting) + +```yaml +apiVersion: serving.kserve.io/v1beta1 +kind: InferenceService +metadata: + name: llama-3-8b + namespace: models +spec: + predictor: + canaryTrafficPercent: 20 # 20% to new version, 80% to stable + containers: + - name: vllm-container + image: vllm/vllm-openai:latest + args: + - "--model" + - "meta-llama/Llama-3.1-8B-Instruct-v2" # new model version + resources: + limits: + nvidia.com/gpu: "1" +``` + +```bash +# Gradually increase canary traffic +kubectl patch inferenceservice llama-3-8b -n models \ + --type='json' \ + -p='[{"op":"replace","path":"/spec/predictor/canaryTrafficPercent","value":50}]' + +# Promote canary to stable +kubectl patch inferenceservice llama-3-8b -n models \ + --type='json' \ + -p='[{"op":"remove","path":"/spec/predictor/canaryTrafficPercent"}]' +``` + +## Autoscaling with KEDA + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: llama-scaler + namespace: models +spec: + scaleTargetRef: + apiVersion: serving.kserve.io/v1beta1 + kind: InferenceService + name: llama-3-8b + minReplicaCount: 1 + maxReplicaCount: 5 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus-server.monitoring:9090 + metricName: kserve_request_count + threshold: "10" + query: | + sum(rate(kserve_request_count_total{namespace="models", + service="llama-3-8b"}[1m])) +``` + +## NVIDIA Triton Inference Server + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: triton-server + namespace: models +spec: + replicas: 2 + selector: + matchLabels: + app: triton + template: + metadata: + labels: + app: triton + spec: + containers: + - name: triton + image: nvcr.io/nvidia/tritonserver:24.05-py3 + args: + - "tritonserver" + - "--model-store=s3://my-model-store/models" + - "--model-control-mode=poll" # auto-load new model versions + - "--repository-poll-secs=30" + - "--metrics-port=8002" + ports: + - containerPort: 8000 # HTTP + - containerPort: 8001 # gRPC + - containerPort: 8002 # Metrics + resources: + limits: + nvidia.com/gpu: "1" + readinessProbe: + httpGet: + path: /v2/health/ready + port: 8000 + initialDelaySeconds: 30 +``` + +## Triton Model Repository Structure + +``` +s3://my-model-store/models/ +├── text-classifier/ +│ ├── config.pbtxt +│ ├── 1/ +│ │ └── model.onnx +│ └── 2/ +│ └── model.onnx # new version; auto-loaded +├── embedding-model/ +│ ├── config.pbtxt +│ └── 1/ +│ └── model.onnx +``` + +```protobuf +# config.pbtxt for ONNX model +name: "text-classifier" +backend: "onnxruntime" +max_batch_size: 64 +dynamic_batching { + preferred_batch_size: [16, 32] + max_queue_delay_microseconds: 1000 +} +input [ + { name: "input_ids" data_type: TYPE_INT64 dims: [-1] } + { name: "attention_mask" data_type: TYPE_INT64 dims: [-1] } +] +output [ + { name: "logits" data_type: TYPE_FP32 dims: [-1] } +] +instance_group [ + { kind: KIND_GPU count: 2 } # 2 model instances on GPU +] +``` + +## Model Management Commands + +```bash +# List loaded models (Triton) +curl http://triton:8000/v2/models + +# Load a new model version +curl -X POST http://triton:8000/v2/repository/models/text-classifier/load + +# Unload a model +curl -X POST http://triton:8000/v2/repository/models/text-classifier/unload + +# KServe — watch rollout status +kubectl rollout status deployment/llama-3-8b-predictor -n models +kubectl get inferenceservice llama-3-8b -n models -w +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| `InferenceService not ready` | Model loading or OOM | Check predictor pod logs; increase memory limits | +| Canary stuck at 0% | KNative routing issue | Check `kubectl get ksvc -n models` | +| Triton missing model | S3 permissions or path | Verify IAM role; check `--model-store` path | +| Low GPU utilization | Dynamic batching off | Enable `dynamic_batching` in Triton config | +| Autoscaler not triggering | Prometheus query wrong | Test query in Prometheus UI | + +## Best Practices + +- Use canary deployments for all model updates — roll back in seconds if metrics degrade. +- Enable Triton dynamic batching — it can increase GPU throughput 5–10× for small models. +- Store models in S3/GCS with versioned paths (`s3://bucket/model/v1/`, `v2/`). +- Pin GPU node selectors to prevent model pods landing on CPU-only nodes. +- Monitor p99 latency and error rates per model version during canary rollouts. + +## Related Skills + +- [vllm-server](../../infrastructure/local-ai/vllm-server/) - vLLM for LLM serving +- [llm-inference-scaling](../../infrastructure/local-ai/llm-inference-scaling/) - KEDA autoscaling +- [kubernetes-ops](./kubernetes-ops/) - Core Kubernetes operations +- [gpu-server-management](../../infrastructure/servers/gpu-server-management/) - GPU nodes diff --git a/infrastructure/databases/vector-database-ops/SKILL.md b/infrastructure/databases/vector-database-ops/SKILL.md new file mode 100644 index 0000000..f879b2a --- /dev/null +++ b/infrastructure/databases/vector-database-ops/SKILL.md @@ -0,0 +1,285 @@ +--- +name: vector-database-ops +description: Deploy, manage, and optimize vector databases for AI applications. Covers Qdrant, Weaviate, pgvector, and Pinecone — collection management, indexing strategies, backup, and performance tuning for production RAG and semantic search workloads. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# Vector Database Operations + +Run production vector databases for AI-powered search, RAG, and recommendation systems. + +## When to Use This Skill + +Use this skill when: +- Setting up a vector database for a RAG or semantic search application +- Choosing between Qdrant, Weaviate, pgvector, or Pinecone +- Managing collections, indexes, and data migrations +- Optimizing query performance and indexing for production loads +- Implementing multi-tenant vector search with namespace isolation + +## Vector Database Comparison + +| Database | Best For | Hosting | Filtering | Scale | +|----------|----------|---------|-----------|-------| +| **Qdrant** | High-performance, rich filtering, self-hosted | Self / Cloud | Excellent | Very High | +| **Weaviate** | Schema-first, hybrid search, multi-modal | Self / Cloud | Good | High | +| **pgvector** | Already on Postgres, simple use cases | Self | Good | Medium | +| **Pinecone** | Zero-ops managed, serverless | Managed only | Good | Very High | +| **Chroma** | Local dev, prototyping | Self only | Basic | Low-Medium | + +## Qdrant — Production Deployment + +```bash +# Docker (single node) +docker run -d \ + --name qdrant \ + -p 6333:6333 \ + -p 6334:6334 \ + -v $(pwd)/qdrant-data:/qdrant/storage \ + qdrant/qdrant:latest + +# With custom config +docker run -d \ + --name qdrant \ + -p 6333:6333 \ + -v $(pwd)/qdrant-data:/qdrant/storage \ + -v $(pwd)/qdrant-config.yaml:/qdrant/config/production.yaml \ + qdrant/qdrant:latest +``` + +```yaml +# qdrant-config.yaml +storage: + storage_path: /qdrant/storage + on_disk_payload: true # store payload on disk (saves RAM) + +service: + max_request_size_mb: 32 + +hnsw_index: + m: 16 # graph connections per node + ef_construct: 100 # accuracy vs build time trade-off + full_scan_threshold: 10000 # switch to brute force below this + +quantization: + scalar: + type: int8 + quantile: 0.99 + always_ram: true # keep quantized index in RAM + +telemetry_disabled: true +``` + +## Qdrant Collection Management + +```python +from qdrant_client import QdrantClient +from qdrant_client.models import ( + Distance, VectorParams, HnswConfigDiff, + ScalarQuantizationConfig, ScalarType, QuantizationConfig +) + +client = QdrantClient("http://localhost:6333") + +# Create optimized collection +client.create_collection( + collection_name="documents", + vectors_config=VectorParams( + size=1536, # OpenAI ada-002 / text-embedding-3-small + distance=Distance.COSINE, + on_disk=True, # save RAM — vectors stored on disk + ), + hnsw_config=HnswConfigDiff( + m=32, # higher = better recall, more RAM + ef_construct=200, + on_disk=False, # keep HNSW graph in RAM for speed + ), + quantization_config=QuantizationConfig( + scalar=ScalarQuantizationConfig( + type=ScalarType.INT8, + quantile=0.99, + always_ram=True, + ) + ), +) + +# Create payload index for fast filtering +client.create_payload_index( + collection_name="documents", + field_name="tenant_id", + field_schema="keyword", +) +client.create_payload_index( + collection_name="documents", + field_name="created_at", + field_schema="datetime", +) + +# Collection info +info = client.get_collection("documents") +print(f"Vectors: {info.vectors_count}, Status: {info.status}") +``` + +## Qdrant Filtered Search + +```python +from qdrant_client.models import Filter, FieldCondition, MatchValue, Range + +# Tenant-isolated search (multi-tenant RAG) +results = client.query_points( + collection_name="documents", + query=query_embedding, + query_filter=Filter( + must=[ + FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")), + FieldCondition(key="doc_type", match=MatchValue(value="contract")), + ], + should=[ + FieldCondition(key="created_at", range=Range(gte="2024-01-01")), + ], + ), + limit=10, + with_payload=True, +) +``` + +## pgvector — PostgreSQL Extension + +```sql +-- Enable extension +CREATE EXTENSION IF NOT EXISTS vector; + +-- Create table with vector column +CREATE TABLE documents ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + content TEXT NOT NULL, + embedding VECTOR(1536), + metadata JSONB DEFAULT '{}', + tenant_id TEXT NOT NULL, + created_at TIMESTAMPTZ DEFAULT NOW() +); + +-- Create HNSW index (faster queries, more memory) +CREATE INDEX ON documents +USING hnsw (embedding vector_cosine_ops) +WITH (m = 16, ef_construction = 64); + +-- Create IVFFlat index (less memory, slower build) +-- CREATE INDEX ON documents +-- USING ivfflat (embedding vector_cosine_ops) +-- WITH (lists = 100); + +-- Semantic search with metadata filtering +SELECT id, content, metadata, + 1 - (embedding <=> $1::vector) AS similarity +FROM documents +WHERE tenant_id = 'acme-corp' + AND metadata->>'doc_type' = 'contract' +ORDER BY embedding <=> $1::vector +LIMIT 10; +``` + +```bash +# Deploy pgvector via Docker +docker run -d \ + --name pgvector \ + -e POSTGRES_PASSWORD=secret \ + -e POSTGRES_DB=vectordb \ + -p 5432:5432 \ + -v pgvector-data:/var/lib/postgresql/data \ + pgvector/pgvector:pg16 +``` + +## Weaviate Deployment + +```yaml +# docker-compose for Weaviate +services: + weaviate: + image: semitechnologies/weaviate:latest + ports: + - "8080:8080" + - "50051:50051" + environment: + QUERY_DEFAULTS_LIMIT: 25 + AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "false" + AUTHENTICATION_APIKEY_ENABLED: "true" + AUTHENTICATION_APIKEY_ALLOWED_KEYS: "${WEAVIATE_API_KEY}" + AUTHENTICATION_APIKEY_USERS: "admin" + PERSISTENCE_DATA_PATH: /var/lib/weaviate + ENABLE_MODULES: text2vec-openai,generative-openai + OPENAI_APIKEY: "${OPENAI_API_KEY}" + CLUSTER_HOSTNAME: node1 + volumes: + - weaviate-data:/var/lib/weaviate + restart: unless-stopped + +volumes: + weaviate-data: +``` + +## Backup and Restore + +```bash +# Qdrant — snapshot backup +curl -X POST "http://localhost:6333/collections/documents/snapshots" +# Download snapshot +curl -O "http://localhost:6333/collections/documents/snapshots/documents-snapshot.snapshot" +# Restore +curl -X POST "http://localhost:6333/collections/documents/snapshots/recover" \ + -H "Content-Type: application/json" \ + -d '{"location": "/qdrant/snapshots/documents-snapshot.snapshot"}' + +# pgvector — standard pg_dump +pg_dump -h localhost -U postgres -d vectordb \ + --table=documents --format=custom > documents-backup.dump + +# Restore +pg_restore -h localhost -U postgres -d vectordb documents-backup.dump +``` + +## Performance Tuning + +```python +# Qdrant — optimize collection after bulk load +client.update_collection( + collection_name="documents", + optimizer_config={"indexing_threshold": 0}, # force indexing now +) + +# Wait for optimization to complete +import time +while True: + info = client.get_collection("documents") + if info.status.value == "green": + break + time.sleep(5) + print(f"Optimizing... segments: {info.segments_count}") +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| Slow queries | No HNSW index built yet | Wait for indexing; check `status == green` | +| High RAM usage | Vectors in memory | Enable `on_disk=True` for vectors | +| Poor recall | Low `ef` search param | Increase `ef` in search request (at query time) | +| pgvector slow | Using IVFFlat without vacuum | Run `VACUUM ANALYZE documents` | +| Weaviate OOM | Too many objects | Enable async indexing; increase heap | + +## Best Practices + +- Use cosine distance for normalized embeddings; dot product for unnormalized. +- Always create payload indexes on filter fields (`tenant_id`, `doc_type`). +- For datasets >10M vectors, use `on_disk` vectors + `always_ram` quantization. +- Benchmark with your actual query patterns before choosing IVFFlat vs HNSW. +- Snapshot before any bulk delete or migration operation. + +## Related Skills + +- [rag-infrastructure](../../local-ai/rag-infrastructure/) - Full RAG pipeline +- [databases](../databases/) - General database management +- [postgresql](../postgresql/) - pgvector host database ops diff --git a/infrastructure/local-ai/llm-fine-tuning/SKILL.md b/infrastructure/local-ai/llm-fine-tuning/SKILL.md new file mode 100644 index 0000000..9f53ae5 --- /dev/null +++ b/infrastructure/local-ai/llm-fine-tuning/SKILL.md @@ -0,0 +1,312 @@ +--- +name: llm-fine-tuning +description: Set up infrastructure for fine-tuning LLMs with QLoRA, LoRA, and full fine-tuning using Hugging Face TRL, Axolotl, and distributed training with DeepSpeed or FSDP. Covers dataset prep, training runs, and model export. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# LLM Fine-Tuning Infrastructure + +Train and fine-tune open-source LLMs efficiently — from LoRA on a single GPU to distributed full fine-tuning across multi-node clusters. + +## When to Use This Skill + +Use this skill when: +- Fine-tuning an LLM on domain-specific data (legal, medical, code, support) +- Running QLoRA to fine-tune 70B models on consumer GPUs +- Setting up distributed training with DeepSpeed or FSDP +- Exporting fine-tuned adapters for production serving +- Implementing RLHF, DPO, or instruction tuning pipelines + +## Prerequisites + +- NVIDIA GPU(s) with 24GB+ VRAM (RTX 4090 / A100 / H100) +- CUDA 12.1+ and `nvidia-smi` working +- Python 3.10+ with `pip` +- Hugging Face account and `HF_TOKEN` for gated models +- 500GB+ disk for model weights and training data + +## Quick Start: QLoRA Fine-Tuning + +```bash +pip install transformers datasets trl peft bitsandbytes accelerate + +python - <<'EOF' +from datasets import load_dataset +from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig +from peft import LoraConfig, get_peft_model +from trl import SFTTrainer, SFTConfig +import torch + +model_id = "meta-llama/Llama-3.1-8B-Instruct" + +# 4-bit quantization (QLoRA) +bnb_config = BitsAndBytesConfig( + load_in_4bit=True, + bnb_4bit_quant_type="nf4", + bnb_4bit_compute_dtype=torch.bfloat16, + bnb_4bit_use_double_quant=True, +) + +model = AutoModelForCausalLM.from_pretrained( + model_id, quantization_config=bnb_config, device_map="auto" +) +tokenizer = AutoTokenizer.from_pretrained(model_id) + +# LoRA configuration +peft_config = LoraConfig( + r=16, # rank + lora_alpha=32, + target_modules=["q_proj", "k_proj", "v_proj", "o_proj", + "gate_proj", "up_proj", "down_proj"], + lora_dropout=0.05, + bias="none", + task_type="CAUSAL_LM", +) + +dataset = load_dataset("your-org/your-dataset", split="train") + +trainer = SFTTrainer( + model=model, + args=SFTConfig( + output_dir="./output", + num_train_epochs=3, + per_device_train_batch_size=2, + gradient_accumulation_steps=8, + learning_rate=2e-4, + bf16=True, + logging_steps=10, + save_strategy="epoch", + report_to="wandb", + ), + train_dataset=dataset, + peft_config=peft_config, + processing_class=tokenizer, +) +trainer.train() +trainer.save_model("./fine-tuned-model") +EOF +``` + +## Axolotl (Production Fine-Tuning Framework) + +```yaml +# config.yaml — Axolotl QLoRA config for Llama 3.1 +base_model: meta-llama/Llama-3.1-8B-Instruct +model_type: LlamaForCausalLM +tokenizer_type: PreTrainedTokenizerFast + +load_in_4bit: true +adapter: qlora +lora_r: 32 +lora_alpha: 64 +lora_dropout: 0.05 +lora_target_modules: + - q_proj + - k_proj + - v_proj + - o_proj + - gate_proj + - up_proj + - down_proj + +datasets: + - path: your-org/your-dataset + type: alpaca # or sharegpt, chat_template, etc. + +dataset_prepared_path: ./prepared-data +val_set_size: 0.05 +output_dir: ./output + +sequence_len: 4096 +sample_packing: true # pack multiple short samples for efficiency + +micro_batch_size: 2 +gradient_accumulation_steps: 8 +num_epochs: 3 +learning_rate: 2e-4 +optimizer: adamw_bnb_8bit +lr_scheduler: cosine +warmup_ratio: 0.05 + +bf16: true +flash_attention: true + +logging_steps: 10 +eval_steps: 100 +save_steps: 200 +wandb_project: my-fine-tune +``` + +```bash +# Run with Axolotl +pip install axolotl[flash-attn,deepspeed] +accelerate launch -m axolotl.cli.train config.yaml +``` + +## Distributed Training with DeepSpeed + +```json +// deepspeed_zero3.json — ZeRO Stage 3 (split optimizer + gradients + params) +{ + "zero_optimization": { + "stage": 3, + "offload_optimizer": {"device": "cpu", "pin_memory": true}, + "offload_param": {"device": "cpu", "pin_memory": true}, + "overlap_comm": true, + "contiguous_gradients": true, + "sub_group_size": 1e9, + "reduce_bucket_size": "auto", + "stage3_prefetch_bucket_size": "auto", + "stage3_param_persistence_threshold": "auto", + "stage3_max_live_parameters": 1e9, + "stage3_max_reuse_distance": 1e9, + "gather_16bit_weights_on_model_save": true + }, + "bf16": {"enabled": true}, + "gradient_clipping": 1.0, + "train_batch_size": "auto", + "train_micro_batch_size_per_gpu": "auto" +} +``` + +```bash +# Launch 4-GPU DeepSpeed training +deepspeed --num_gpus=4 train.py \ + --deepspeed deepspeed_zero3.json \ + --model_name meta-llama/Llama-3.1-70B-Instruct \ + --output_dir ./output +``` + +## DPO / RLHF Alignment + +```python +from trl import DPOTrainer, DPOConfig +from datasets import load_dataset + +# Dataset format: {"prompt": ..., "chosen": ..., "rejected": ...} +dataset = load_dataset("your-org/preference-data") + +trainer = DPOTrainer( + model=model, + ref_model=None, # None = implicit reference with peft + args=DPOConfig( + output_dir="./dpo-output", + beta=0.1, # KL divergence weight + num_train_epochs=1, + per_device_train_batch_size=1, + gradient_accumulation_steps=16, + learning_rate=5e-7, + bf16=True, + ), + train_dataset=dataset["train"], + peft_config=peft_config, + processing_class=tokenizer, +) +trainer.train() +``` + +## Merging LoRA Adapters for Deployment + +```python +from peft import PeftModel +from transformers import AutoModelForCausalLM + +# Load base model in full precision +base_model = AutoModelForCausalLM.from_pretrained( + "meta-llama/Llama-3.1-8B-Instruct", + torch_dtype=torch.bfloat16, + device_map="cpu", +) + +# Load and merge LoRA adapter +model = PeftModel.from_pretrained(base_model, "./fine-tuned-model") +merged_model = model.merge_and_unload() + +# Save merged model (ready for vLLM serving) +merged_model.save_pretrained("./merged-model", safe_serialization=True) +tokenizer.save_pretrained("./merged-model") + +# Push to Hugging Face Hub +merged_model.push_to_hub("your-org/your-fine-tuned-model") +``` + +## Kubernetes Training Job + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: llm-fine-tune +spec: + template: + spec: + restartPolicy: OnFailure + nodeSelector: + nvidia.com/gpu.product: A100-SXM4-80GB + containers: + - name: trainer + image: nvcr.io/nvidia/pytorch:24.05-py3 + command: ["accelerate", "launch", "-m", "axolotl.cli.train", "/config/config.yaml"] + resources: + limits: + nvidia.com/gpu: "4" + memory: "320Gi" + requests: + nvidia.com/gpu: "4" + volumeMounts: + - name: config + mountPath: /config + - name: model-cache + mountPath: /root/.cache/huggingface + - name: output + mountPath: /output + env: + - name: HUGGING_FACE_HUB_TOKEN + valueFrom: + secretKeyRef: + name: hf-token + key: token + - name: WANDB_API_KEY + valueFrom: + secretKeyRef: + name: wandb-token + key: key + volumes: + - name: config + configMap: + name: axolotl-config + - name: model-cache + persistentVolumeClaim: + claimName: model-cache-pvc + - name: output + persistentVolumeClaim: + claimName: training-output-pvc +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| `CUDA out of memory` | Batch too large | Reduce `micro_batch_size`; increase `gradient_accumulation_steps` | +| Training loss NaN | Learning rate too high | Lower LR to `1e-4` or `5e-5`; add warmup | +| Slow training | No Flash Attention | Install `flash-attn`; enable `flash_attention: true` | +| Poor fine-tune quality | Bad data formatting | Validate dataset format; check `sample_packing` compatibility | +| Adapter merge errors | Mixed quantization | Merge in bf16 on CPU, not in 4-bit | + +## Best Practices + +- Use Flash Attention 2 — it's 2–4× faster and uses less memory. +- Monitor training loss/eval loss via W&B or MLflow; overfit = more dropout or less data. +- Validate with a held-out eval set (5–10%); MMLU or custom evals for quality gates. +- Start with LoRA r=16 before increasing — higher rank = more parameters, diminishing returns. +- Use `sample_packing` in Axolotl to maximize GPU utilization on short sequences. + +## Related Skills + +- [vllm-server](../vllm-server/) - Serve fine-tuned models +- [gpu-server-management](../../servers/gpu-server-management/) - GPU setup +- [llm-inference-scaling](../llm-inference-scaling/) - Deploy at scale +- [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Training pipelines diff --git a/infrastructure/local-ai/llm-inference-scaling/SKILL.md b/infrastructure/local-ai/llm-inference-scaling/SKILL.md new file mode 100644 index 0000000..08387c9 --- /dev/null +++ b/infrastructure/local-ai/llm-inference-scaling/SKILL.md @@ -0,0 +1,270 @@ +--- +name: llm-inference-scaling +description: Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# LLM Inference Scaling + +Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies. + +## When to Use This Skill + +Use this skill when: +- LLM API traffic is unpredictable and you need to scale up/down automatically +- Managing a fleet of vLLM or TGI inference pods on Kubernetes +- Reducing inference costs with spot/preemptible GPU instances +- Implementing queue-based autoscaling for batch inference jobs +- Building a multi-model serving platform that shares GPU resources + +## Prerequisites + +- Kubernetes cluster with GPU nodes (NVIDIA operator installed) +- KEDA (Kubernetes Event-Driven Autoscaler) installed +- Prometheus with GPU metrics (`dcgm-exporter` or `gpu-operator`) +- Helm 3+ for chart deployments + +## GPU Node Setup + +```bash +# Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM) +helm repo add nvidia https://helm.ngc.nvidia.com/nvidia +helm repo update + +helm install gpu-operator nvidia/gpu-operator \ + --namespace gpu-operator \ + --create-namespace \ + --set driver.enabled=true \ + --set dcgm.enabled=true \ + --set devicePlugin.enabled=true + +# Verify GPU nodes are recognized +kubectl get nodes -l nvidia.com/gpu.present=true +kubectl describe node | grep nvidia +``` + +## vLLM Deployment with GPU Resources + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-llama-8b + labels: + app: vllm + model: llama-3.1-8b +spec: + replicas: 1 + selector: + matchLabels: + app: vllm + model: llama-3.1-8b + template: + metadata: + labels: + app: vllm + model: llama-3.1-8b + spec: + nodeSelector: + nvidia.com/gpu.present: "true" + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + containers: + - name: vllm + image: vllm/vllm-openai:latest + args: + - "--model" + - "meta-llama/Llama-3.1-8B-Instruct" + - "--tensor-parallel-size" + - "1" + - "--gpu-memory-utilization" + - "0.90" + - "--max-num-seqs" + - "128" + resources: + requests: + nvidia.com/gpu: "1" + memory: "20Gi" + cpu: "4" + limits: + nvidia.com/gpu: "1" + memory: "24Gi" + cpu: "8" + ports: + - containerPort: 8000 + readinessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: 60 + periodSeconds: 10 + env: + - name: HUGGING_FACE_HUB_TOKEN + valueFrom: + secretKeyRef: + name: hf-token + key: token +``` + +## KEDA Autoscaling on Prometheus Metrics + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: vllm-scaledobject +spec: + scaleTargetRef: + name: vllm-llama-8b + minReplicaCount: 1 + maxReplicaCount: 8 + cooldownPeriod: 300 # 5 min before scale-down + pollingInterval: 15 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus-server.monitoring:9090 + metricName: vllm_num_requests_waiting + threshold: "10" # scale up if >10 requests waiting + query: | + sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"}) + - type: prometheus + metadata: + serverAddress: http://prometheus-server.monitoring:9090 + metricName: vllm_gpu_cache_usage + threshold: "0.8" # scale up if KV cache >80% full + query: | + avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"}) +``` + +## Queue-Based Scaling (Redis + KEDA) + +```yaml +# ScaledJob for async batch inference +apiVersion: keda.sh/v1alpha1 +kind: ScaledJob +metadata: + name: llm-batch-inference +spec: + jobTargetRef: + template: + spec: + containers: + - name: inference-worker + image: myapp/inference-worker:latest + env: + - name: REDIS_URL + value: redis://redis:6379 + - name: QUEUE_NAME + value: inference-jobs + restartPolicy: OnFailure + minReplicaCount: 0 + maxReplicaCount: 20 + pollingInterval: 5 + successfulJobsHistoryLimit: 3 + triggers: + - type: redis + metadata: + address: redis:6379 + listName: inference-jobs + listLength: "5" # 1 worker per 5 queued jobs +``` + +## Spot Instance Strategy + +```yaml +# Mixed node pool: on-demand + spot GPUs +apiVersion: v1 +kind: ConfigMap +metadata: + name: cluster-autoscaler-priority-config +data: + priorities: | + 10: # low priority = prefer + - .*spot.* + 50: + - .*on-demand.* +--- +# Node affinity for spot with on-demand fallback +spec: + affinity: + nodeAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 80 + preference: + matchExpressions: + - key: node.kubernetes.io/lifecycle + operator: In + values: [spot] + - weight: 20 + preference: + matchExpressions: + - key: node.kubernetes.io/lifecycle + operator: In + values: [on-demand] +``` + +## Cluster Autoscaler for GPU Nodes + +```bash +# AWS EKS — enable cluster autoscaler for GPU node group +helm install cluster-autoscaler autoscaler/cluster-autoscaler \ + --namespace kube-system \ + --set autoDiscovery.clusterName=my-cluster \ + --set awsRegion=us-east-1 \ + --set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole \ + --set extraArgs.skip-nodes-with-local-storage=false \ + --set extraArgs.expander=least-waste + +# Annotate GPU node group for autoscaler +kubectl annotate node \ + cluster-autoscaler.kubernetes.io/safe-to-evict="false" +``` + +## Scaling Metrics to Monitor + +```bash +# Prometheus queries for scaling decisions +# Requests waiting in vLLM queue +sum(vllm:num_requests_waiting) by (model) + +# GPU KV cache utilization (>80% = bottleneck) +avg(vllm:gpu_cache_usage_perc) by (pod) + +# Tokens per second throughput +sum(rate(vllm:generation_tokens_total[5m])) by (model) + +# P99 time-to-first-token +histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m])) +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| Pods stuck in `Pending` | No GPU nodes available | Check cluster autoscaler logs; verify node group limits | +| Scale-up too slow | Cluster autoscaler delay + model load time | Pre-warm replicas; increase `minReplicaCount` | +| GPU fragmentation | Multiple small models on large GPUs | Use MIG partitioning or consolidate model sizes | +| Spot eviction causes errors | Spot instance reclamation | Add `PodDisruptionBudget`; use graceful shutdown | +| KEDA not scaling | Prometheus query returns no data | Test query in Prometheus UI first | + +## Best Practices + +- Set `minReplicaCount: 1` to avoid cold starts; scale to 0 only for batch jobs. +- Use `PodDisruptionBudget` with `minAvailable: 1` to survive spot evictions. +- Pre-pull model weights into a shared PVC to speed up pod startup by 5–10×. +- Separate model families across node pools (A10G for 7B, A100 for 70B). +- Use Kubernetes VPA for CPU/memory right-sizing alongside KEDA for replica count. + +## Related Skills + +- [vllm-server](../vllm-server/) - vLLM configuration and tuning +- [gpu-server-management](../../servers/gpu-server-management/) - GPU node setup +- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - KServe +- [kubernetes-ops](../../../devops/orchestration/kubernetes-ops/) - Core Kubernetes +- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost strategies diff --git a/infrastructure/local-ai/rag-infrastructure/SKILL.md b/infrastructure/local-ai/rag-infrastructure/SKILL.md new file mode 100644 index 0000000..eb6341f --- /dev/null +++ b/infrastructure/local-ai/rag-infrastructure/SKILL.md @@ -0,0 +1,253 @@ +--- +name: rag-infrastructure +description: Build and operate Retrieval-Augmented Generation (RAG) infrastructure with vector stores, embedding pipelines, and hybrid search. Covers ingestion, chunking strategies, reranking, and production deployment patterns. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# RAG Infrastructure + +Production infrastructure for Retrieval-Augmented Generation: ingest documents, generate embeddings, store in vector databases, and serve grounded LLM responses. + +## When to Use This Skill + +Use this skill when: +- Building a knowledge base Q&A system over internal documents +- Implementing semantic search over large document collections +- Reducing LLM hallucinations with retrieved context +- Setting up embedding pipelines and vector store infrastructure +- Deploying hybrid search (dense + sparse/BM25) + +## Prerequisites + +- Python 3.10+ with `pip` +- A vector database (Qdrant, Weaviate, Pinecone, or pgvector) +- An embedding model (OpenAI, Cohere, or local via `sentence-transformers`) +- An LLM endpoint (OpenAI API or self-hosted vLLM) +- Docker for local vector DB deployment + +## Architecture Overview + +``` +Documents → Chunker → Embedder → Vector Store + ↓ +User Query → Embedder → Vector Store (search) → Reranker → LLM → Answer +``` + +## Embedding Pipeline + +```python +from sentence_transformers import SentenceTransformer +from qdrant_client import QdrantClient +from qdrant_client.models import Distance, VectorParams, PointStruct +import uuid + +# Local embedding model (no API cost) +model = SentenceTransformer("BAAI/bge-large-en-v1.5") + +# Connect to Qdrant +client = QdrantClient("http://localhost:6333") + +# Create collection +client.create_collection( + collection_name="knowledge-base", + vectors_config=VectorParams(size=1024, distance=Distance.COSINE), +) + +def ingest_documents(docs: list[dict]): + """Chunk, embed, and upsert documents.""" + points = [] + for doc in docs: + chunks = chunk_text(doc["text"], chunk_size=512, overlap=50) + embeddings = model.encode(chunks, batch_size=32, show_progress_bar=True) + for chunk, embedding in zip(chunks, embeddings): + points.append(PointStruct( + id=str(uuid.uuid4()), + vector=embedding.tolist(), + payload={"text": chunk, "source": doc["source"], "title": doc["title"]}, + )) + client.upsert(collection_name="knowledge-base", points=points) + print(f"Ingested {len(points)} chunks") +``` + +## Chunking Strategies + +```python +from langchain.text_splitter import RecursiveCharacterTextSplitter + +def chunk_text(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]: + """Recursive character splitter — best general-purpose strategy.""" + splitter = RecursiveCharacterTextSplitter( + chunk_size=chunk_size, + chunk_overlap=overlap, + separators=["\n\n", "\n", ". ", " ", ""], + ) + return splitter.split_text(text) + +# For code/markdown — use language-aware splitter +from langchain.text_splitter import MarkdownHeaderTextSplitter + +headers = [("#", "H1"), ("##", "H2"), ("###", "H3")] +md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers) +``` + +## Hybrid Search (Dense + Sparse) + +```python +from qdrant_client.models import SparseVector, SparseVectorParams, NamedSparseVector +from fastembed import SparseTextEmbedding + +# Qdrant hybrid collection (dense + BM25 sparse) +client.create_collection( + collection_name="hybrid-kb", + vectors_config={"dense": VectorParams(size=1024, distance=Distance.COSINE)}, + sparse_vectors_config={"sparse": SparseVectorParams()}, +) + +sparse_model = SparseTextEmbedding("prithivida/Splade_PP_en_v1") + +def hybrid_search(query: str, top_k: int = 10) -> list[dict]: + dense_vec = model.encode(query).tolist() + sparse_vec = list(sparse_model.embed(query))[0] + + results = client.query_points( + collection_name="hybrid-kb", + prefetch=[ + {"query": dense_vec, "using": "dense", "limit": 20}, + {"query": SparseVector(indices=sparse_vec.indices.tolist(), + values=sparse_vec.values.tolist()), + "using": "sparse", "limit": 20}, + ], + query={"fusion": "rrf"}, # Reciprocal Rank Fusion + limit=top_k, + ) + return [{"text": p.payload["text"], "score": p.score} for p in results.points] +``` + +## Reranking + +```python +import cohere + +co = cohere.Client("your-api-key") + +def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]: + """Rerank retrieved chunks for relevance (improves RAG quality ~20-30%).""" + response = co.rerank( + model="rerank-english-v3.0", + query=query, + documents=candidates, + top_n=top_n, + ) + return [candidates[r.index] for r in response.results] + +# Alternative: local reranker (no API cost) +from sentence_transformers import CrossEncoder +reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2") + +def local_rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]: + pairs = [[query, c] for c in candidates] + scores = reranker.predict(pairs) + ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True) + return [text for text, _ in ranked[:top_n]] +``` + +## RAG Query Pipeline + +```python +from openai import OpenAI + +llm = OpenAI(base_url="http://localhost:8000/v1", api_key="your-key") + +def rag_query(user_question: str) -> str: + # 1. Retrieve + candidates = hybrid_search(user_question, top_k=20) + texts = [c["text"] for c in candidates] + + # 2. Rerank + top_chunks = local_rerank(user_question, texts, top_n=5) + + # 3. Generate + context = "\n\n---\n\n".join(top_chunks) + response = llm.chat.completions.create( + model="meta-llama/Llama-3.1-8B-Instruct", + messages=[ + {"role": "system", "content": ( + "Answer the question using only the provided context. " + "If the answer isn't in the context, say so.\n\nContext:\n" + context + )}, + {"role": "user", "content": user_question}, + ], + temperature=0.1, + max_tokens=1024, + ) + return response.choices[0].message.content +``` + +## Docker Compose: Full RAG Stack + +```yaml +services: + qdrant: + image: qdrant/qdrant:latest + volumes: + - qdrant-data:/qdrant/storage + ports: + - "6333:6333" + restart: unless-stopped + + redis: + image: redis:7-alpine + volumes: + - redis-data:/data + restart: unless-stopped + + ingestion-worker: + build: ./ingestion + environment: + - QDRANT_URL=http://qdrant:6333 + - REDIS_URL=redis://redis:6379 + depends_on: [qdrant, redis] + restart: unless-stopped + + rag-api: + build: ./api + ports: + - "8080:8080" + environment: + - QDRANT_URL=http://qdrant:6333 + - LLM_BASE_URL=http://vllm:8000/v1 + depends_on: [qdrant] + restart: unless-stopped + +volumes: + qdrant-data: + redis-data: +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| Poor retrieval quality | Chunk size too large | Try 256–512 tokens; overlap 10–15% | +| LLM ignores retrieved context | Context too long | Rerank and keep top 3–5 chunks | +| Slow ingestion | Sequential embedding | Use `batch_size=64` and async upserts | +| Stale documents | No re-ingestion pipeline | Track `doc_hash`; re-embed on change | +| High embedding costs | All chunks re-embedded | Cache embeddings with hash-based dedup | + +## Best Practices + +- Use `BAAI/bge-large-en-v1.5` or `nomic-embed-text` for strong free embeddings. +- Always rerank before passing to LLM — 5 precise chunks beat 20 noisy ones. +- Store source metadata (URL, page, section) in vector payloads for citations. +- Use namespace/tenant isolation in the vector store for multi-tenant RAG. +- Evaluate with RAGAS metrics: faithfulness, answer relevancy, context precision. + +## Related Skills + +- [vector-database-ops](../../databases/vector-database-ops/) - Qdrant/Weaviate management +- [vllm-server](../vllm-server/) - Self-hosted LLM endpoint +- [ollama-stack](../ollama-stack/) - Local LLM for development +- [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Ingestion pipelines diff --git a/infrastructure/local-ai/vllm-server/SKILL.md b/infrastructure/local-ai/vllm-server/SKILL.md new file mode 100644 index 0000000..5c195bb --- /dev/null +++ b/infrastructure/local-ai/vllm-server/SKILL.md @@ -0,0 +1,219 @@ +--- +name: vllm-server +description: Deploy and manage vLLM for high-throughput LLM inference. Configure continuous batching, tensor parallelism, quantization, and OpenAI-compatible API endpoints for production LLM serving. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# vLLM Server Management + +Deploy production-grade LLM inference servers with vLLM — the fastest open-source LLM serving engine with PagedAttention and continuous batching. + +## When to Use This Skill + +Use this skill when: +- Serving open-source LLMs (Llama, Mistral, Qwen, Gemma) at scale +- Building an OpenAI-compatible API endpoint for self-hosted models +- Optimizing LLM throughput and latency for production traffic +- Running multi-GPU inference with tensor or pipeline parallelism +- Deploying quantized models to reduce GPU memory requirements + +## Prerequisites + +- NVIDIA GPU(s) with CUDA 12.1+ (A100/H100 recommended for production) +- Docker or Python 3.9+ with pip +- 40GB+ VRAM for 70B models; 8GB+ for 7B models +- `nvidia-container-toolkit` for Docker GPU passthrough + +## Quick Start + +```bash +# Install vLLM +pip install vllm + +# Serve a model (OpenAI-compatible API) +vllm serve meta-llama/Llama-3.1-8B-Instruct \ + --host 0.0.0.0 \ + --port 8000 \ + --api-key your-secret-key + +# Test the endpoint +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer your-secret-key" \ + -d '{ + "model": "meta-llama/Llama-3.1-8B-Instruct", + "messages": [{"role": "user", "content": "Hello!"}] + }' +``` + +## Docker Deployment + +```bash +docker run --runtime nvidia --gpus all \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + -p 8000:8000 \ + --ipc=host \ + vllm/vllm-openai:latest \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --api-key your-secret-key +``` + +## Docker Compose (Production) + +```yaml +services: + vllm: + image: vllm/vllm-openai:latest + runtime: nvidia + environment: + - NVIDIA_VISIBLE_DEVICES=all + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN} + volumes: + - model-cache:/root/.cache/huggingface + ports: + - "8000:8000" + ipc: host + command: > + --model meta-llama/Llama-3.1-70B-Instruct + --tensor-parallel-size 2 + --max-model-len 32768 + --gpu-memory-utilization 0.90 + --api-key ${VLLM_API_KEY} + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + +volumes: + model-cache: +``` + +## Key Configuration Options + +### Multi-GPU Tensor Parallelism + +```bash +# Split one model across 4 GPUs +vllm serve meta-llama/Llama-3.1-70B-Instruct \ + --tensor-parallel-size 4 \ + --gpu-memory-utilization 0.90 +``` + +### Quantization (Lower VRAM) + +```bash +# AWQ quantization (70B on 2x A100 40GB) +vllm serve casperhansen/llama-3-70b-instruct-awq \ + --quantization awq \ + --tensor-parallel-size 2 + +# GPTQ quantization +vllm serve TheBloke/Llama-2-70B-Chat-GPTQ \ + --quantization gptq + +# FP8 (H100 NVL native) +vllm serve meta-llama/Llama-3.1-405B-Instruct \ + --quantization fp8 \ + --tensor-parallel-size 8 +``` + +### Structured Output & Tools + +```bash +vllm serve meta-llama/Llama-3.1-8B-Instruct \ + --enable-auto-tool-choice \ + --tool-call-parser llama3_json \ + --guided-decoding-backend outlines +``` + +### LoRA Adapters + +```bash +vllm serve meta-llama/Llama-3.1-8B-Instruct \ + --enable-lora \ + --lora-modules sql-lora=/path/to/sql-lora \ + code-lora=/path/to/code-lora \ + --max-lora-rank 64 +``` + +## Performance Tuning + +```bash +# Maximize throughput for batch workloads +vllm serve \ + --max-num-seqs 256 \ # max concurrent sequences + --max-num-batched-tokens 8192 \ # tokens per batch + --gpu-memory-utilization 0.95 \ # use 95% VRAM + --swap-space 4 # CPU swap (GiB) + +# Minimize latency for interactive use +vllm serve \ + --max-num-seqs 32 \ + --enforce-eager # disable CUDA graph capture +``` + +## Benchmarking + +```bash +# Install benchmark tool +pip install vllm + +# Run throughput benchmark +python -m vllm.entrypoints.openai.run_batch \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --input-file prompts.jsonl \ + --output-file results.jsonl + +# Benchmark with vllm bench +vllm bench throughput \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --num-prompts 1000 \ + --input-len 512 \ + --output-len 128 +``` + +## Monitoring + +```bash +# Check running server stats +curl http://localhost:8000/metrics # Prometheus metrics + +# Key metrics to watch: +# vllm:num_requests_running - active requests +# vllm:gpu_cache_usage_perc - KV cache utilization +# vllm:generation_tokens_per_s - throughput +# vllm:time_to_first_token_ms - TTFT latency +# vllm:e2e_request_latency_seconds - end-to-end latency +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| `CUDA out of memory` | Model too large for VRAM | Add `--quantization awq` or reduce `--gpu-memory-utilization` | +| Slow cold start | Model not cached | Pre-pull with `huggingface-cli download ` | +| Low throughput | Too few concurrent requests | Increase `--max-num-seqs` | +| KV cache full errors | Context length too long | Set `--max-model-len` lower | +| `tokenizer error` | Tokenizer mismatch | Use `--tokenizer` to specify correct tokenizer | + +## Best Practices + +- Use `--gpu-memory-utilization 0.90` to leave headroom for CUDA kernels. +- Pin model versions with `--revision` for reproducible deployments. +- Set `HF_HUB_OFFLINE=1` in production to prevent unexpected downloads. +- Use AWQ or GPTQ quantization before tensor parallelism — lower VRAM first. +- Enable `--enable-chunked-prefill` for long-context workloads. +- Monitor `gpu_cache_usage_perc` — above 95% causes queuing. + +## Related Skills + +- [llm-inference-scaling](../llm-inference-scaling/) - Auto-scaling vLLM deployments +- [gpu-server-management](../../servers/gpu-server-management/) - GPU driver setup +- [llm-gateway](../../networking/llm-gateway/) - Load balancing across vLLM instances +- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost management +- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - K8s deployment diff --git a/infrastructure/networking/llm-gateway/SKILL.md b/infrastructure/networking/llm-gateway/SKILL.md new file mode 100644 index 0000000..d743402 --- /dev/null +++ b/infrastructure/networking/llm-gateway/SKILL.md @@ -0,0 +1,266 @@ +--- +name: llm-gateway +description: Deploy an API gateway for LLM traffic with load balancing, rate limiting, key management, semantic caching, fallback routing, and cost tracking. Covers LiteLLM Proxy, OpenRouter-compatible setup, and custom Nginx/Traefik patterns. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# LLM Gateway + +A unified API gateway that routes LLM requests across providers and self-hosted models — with rate limiting, cost tracking, caching, and failover. + +## When to Use This Skill + +Use this skill when: +- Running multiple LLM backends (OpenAI, Anthropic, vLLM, Ollama) behind a single endpoint +- Enforcing per-team or per-user rate limits and spend budgets +- Implementing automatic fallback when a provider is down +- Adding semantic caching to reduce API costs by 20–50% +- Centralizing API key management instead of distributing keys to every app + +## Prerequisites + +- Docker and Docker Compose +- A PostgreSQL or SQLite database (for LiteLLM state) +- LLM API keys (OpenAI, Anthropic, etc.) or self-hosted vLLM endpoints +- Optional: Redis for caching and rate limiting + +## LiteLLM Proxy — Quick Start + +LiteLLM is the de facto open-source LLM gateway with OpenAI-compatible API. + +```bash +# Run with Docker +docker run -d \ + --name litellm-proxy \ + -p 4000:4000 \ + -e OPENAI_API_KEY=$OPENAI_API_KEY \ + -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \ + -v $(pwd)/litellm-config.yaml:/app/config.yaml \ + ghcr.io/berriai/litellm:main-latest \ + --config /app/config.yaml \ + --detailed_debug +``` + +## LiteLLM Configuration + +```yaml +# litellm-config.yaml +model_list: + # OpenAI models + - model_name: gpt-4o + litellm_params: + model: openai/gpt-4o + api_key: os.environ/OPENAI_API_KEY + rpm: 10000 + tpm: 2000000 + + - model_name: gpt-4o-mini + litellm_params: + model: openai/gpt-4o-mini + api_key: os.environ/OPENAI_API_KEY + + # Anthropic + - model_name: claude-sonnet-4-6 + litellm_params: + model: anthropic/claude-sonnet-4-6 + api_key: os.environ/ANTHROPIC_API_KEY + + # Self-hosted vLLM instances (load balanced) + - model_name: llama-3.1-8b + litellm_params: + model: openai/meta-llama/Llama-3.1-8B-Instruct + api_base: http://vllm-1:8000/v1 + api_key: fake # vLLM key + - model_name: llama-3.1-8b + litellm_params: + model: openai/meta-llama/Llama-3.1-8B-Instruct + api_base: http://vllm-2:8000/v1 # second replica — auto load balanced + api_key: fake + + # Fallback: cheap model if primary fails + - model_name: gpt-4o + litellm_params: + model: openai/gpt-4o-mini # fallback to cheaper model + api_key: os.environ/OPENAI_API_KEY + +router_settings: + routing_strategy: least-busy # or: latency-based, simple-shuffle + num_retries: 3 + retry_after: 5 + allowed_fails: 2 + cooldown_time: 60 + + # Fallback configuration + fallbacks: + - gpt-4o: [claude-sonnet-4-6] + - claude-sonnet-4-6: [gpt-4o] + +litellm_settings: + # Semantic caching + cache: true + cache_params: + type: redis + host: redis + port: 6379 + similarity_threshold: 0.90 # cache if >90% semantic similarity + + # Logging + success_callback: ["langfuse"] + failure_callback: ["langfuse"] + langfuse_public_key: os.environ/LANGFUSE_PUBLIC_KEY + langfuse_secret_key: os.environ/LANGFUSE_SECRET_KEY + +general_settings: + master_key: os.environ/LITELLM_MASTER_KEY + database_url: postgresql://litellm:password@postgres:5432/litellm + store_model_in_db: true +``` + +## Docker Compose: Full Gateway Stack + +```yaml +services: + litellm: + image: ghcr.io/berriai/litellm:main-latest + command: ["--config", "/app/config.yaml", "--port", "4000"] + volumes: + - ./litellm-config.yaml:/app/config.yaml + ports: + - "4000:4000" + environment: + - OPENAI_API_KEY=${OPENAI_API_KEY} + - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} + - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY} + - DATABASE_URL=postgresql://litellm:password@postgres:5432/litellm + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_started + restart: unless-stopped + + postgres: + image: postgres:16-alpine + environment: + POSTGRES_DB: litellm + POSTGRES_USER: litellm + POSTGRES_PASSWORD: password + volumes: + - postgres-data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U litellm"] + interval: 5s + retries: 5 + restart: unless-stopped + + redis: + image: redis:7-alpine + command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru + volumes: + - redis-data:/data + restart: unless-stopped + +volumes: + postgres-data: + redis-data: +``` + +## Virtual Keys & Rate Limiting + +```bash +# Create a virtual API key for a team (via LiteLLM API) +curl -X POST http://localhost:4000/key/generate \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "team_id": "team-backend", + "key_alias": "backend-team-key", + "models": ["gpt-4o-mini", "llama-3.1-8b"], + "max_budget": 100, # USD limit + "budget_duration": "monthly", + "rpm_limit": 100, # requests per minute + "tpm_limit": 500000 # tokens per minute + }' + +# View spend +curl http://localhost:4000/spend/keys \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" +``` + +## Nginx Load Balancer (Alternative/Complement) + +```nginx +# nginx.conf — round-robin across vLLM replicas +upstream vllm_backends { + least_conn; + server vllm-1:8000 max_fails=3 fail_timeout=30s; + server vllm-2:8000 max_fails=3 fail_timeout=30s; + server vllm-3:8000 max_fails=3 fail_timeout=30s; + keepalive 32; +} + +server { + listen 80; + server_name llm-api.internal; + + # Rate limiting + limit_req_zone $http_authorization zone=per_key:10m rate=100r/m; + limit_req zone=per_key burst=20 nodelay; + + location /v1/ { + proxy_pass http://vllm_backends; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_read_timeout 300s; # long timeout for streaming + proxy_buffering off; # required for SSE streaming + proxy_cache_bypass 1; + } +} +``` + +## Monitoring Gateway Health + +```bash +# Check LiteLLM health +curl http://localhost:4000/health + +# Model-level health +curl http://localhost:4000/health/liveliness + +# Spend by model +curl http://localhost:4000/spend/models \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" + +# Active virtual keys +curl http://localhost:4000/key/list \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| `ConnectionRefusedError` to backend | Backend not reachable | Check `api_base` URL; verify backend is healthy | +| Rate limit errors (429) | Budget/RPM exceeded | Increase limits or rotate to fallback model | +| Slow streaming responses | `proxy_buffering` enabled | Set `proxy_buffering off` in Nginx | +| Cache miss rate high | Threshold too strict | Lower `similarity_threshold` to `0.85` | +| Postgres connection errors | DB not ready | Add `depends_on` with `condition: service_healthy` | + +## Best Practices + +- Use virtual keys per team/app — never expose raw provider API keys. +- Enable `cache: true` with Redis for repeated or similar queries; can cut costs 30–50%. +- Set `num_retries: 3` with fallbacks to handle provider outages gracefully. +- Log all requests to Langfuse or OpenTelemetry for cost attribution and debugging. +- Use `least-busy` routing strategy for self-hosted models to avoid GPU saturation. + +## Related Skills + +- [vllm-server](../../local-ai/vllm-server/) - Backend inference server +- [llm-inference-scaling](../../local-ai/llm-inference-scaling/) - Auto-scaling backends +- [llm-caching](../../../devops/ai/llm-caching/) - Semantic cache patterns +- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost management diff --git a/infrastructure/servers/gpu-server-management/SKILL.md b/infrastructure/servers/gpu-server-management/SKILL.md new file mode 100644 index 0000000..c1a27ac --- /dev/null +++ b/infrastructure/servers/gpu-server-management/SKILL.md @@ -0,0 +1,221 @@ +--- +name: gpu-server-management +description: Set up and manage NVIDIA GPU servers for AI workloads — driver installation, CUDA toolkit, container toolkit, MIG partitioning, GPU health monitoring, and multi-GPU configuration for LLM inference and training. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# GPU Server Management + +Provision, configure, and monitor NVIDIA GPU servers for AI inference and training workloads. + +## When to Use This Skill + +Use this skill when: +- Setting up a new GPU server for LLM inference or model training +- Installing or upgrading NVIDIA drivers and CUDA toolkit +- Configuring Docker with NVIDIA Container Toolkit for GPU workloads +- Partitioning A100/H100 GPUs with MIG for multi-tenant workloads +- Troubleshooting GPU errors, driver issues, or thermal throttling + +## Prerequisites + +- Ubuntu 22.04 LTS (recommended) or RHEL 8/9 +- NVIDIA GPU (A10G, A100, H100, RTX 4090, or L40S recommended) +- Root or sudo access +- Internet access for package downloads + +## Driver Installation (Ubuntu) + +```bash +# Remove old drivers +sudo apt purge -y 'nvidia*' 'cuda*' 'libcuda*' +sudo apt autoremove -y + +# Add NVIDIA package repository +distribution=$(. /etc/os-release; echo $ID$VERSION_ID) +curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \ + sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg + +curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \ + sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ + sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list + +sudo apt update + +# Install latest driver (560.x as of 2025) +sudo apt install -y nvidia-driver-560 cuda-toolkit-12-6 + +# Install NVIDIA Container Toolkit (Docker GPU support) +sudo apt install -y nvidia-container-toolkit +sudo nvidia-ctk runtime configure --runtime=docker +sudo systemctl restart docker + +# Verify +nvidia-smi +nvcc --version +docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi +``` + +## Post-Install Configuration + +```bash +# Enable persistence mode (reduces driver initialization latency) +sudo nvidia-smi -pm 1 + +# Set power limit (reduce heat/noise on inference servers) +sudo nvidia-smi -pl 350 # watts; check TDP for your GPU model + +# Disable ECC on inference servers (frees ~6% VRAM, less safe) +sudo nvidia-smi --ecc-config=0 # requires reboot + +# Enable P2P for multi-GPU NVLink training +sudo nvidia-smi topo -m # check NVLink topology +``` + +## GPU Health Monitoring + +```bash +# Real-time monitoring (like htop for GPUs) +watch -n 1 nvidia-smi + +# Detailed stats +nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,\ +utilization.memory,memory.used,memory.free,power.draw,clocks.current.graphics \ +--format=csv --loop=1 + +# DCGM — production monitoring daemon (for clusters) +sudo apt install -y datacenter-gpu-manager +sudo systemctl start dcgm +dcgmi discovery -l # list GPUs +dcgmi diag -r 1 # quick health check +dcgmi diag -r 3 # full diagnostic (takes ~20 min) + +# Check GPU errors (XID errors — important for stability) +sudo dmesg | grep -i "NVRM\|nvidia\|XID" +nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total \ + --format=csv,noheader +``` + +## Prometheus GPU Metrics (DCGM Exporter) + +```bash +# Deploy DCGM Exporter for Prometheus scraping +docker run -d \ + --name dcgm-exporter \ + --gpus all \ + --cap-add SYS_ADMIN \ + -p 9400:9400 \ + --restart unless-stopped \ + nvcr.io/nvidia/k8s/dcgm-exporter:latest + +# Key metrics exposed: +# DCGM_FI_DEV_GPU_UTIL - GPU utilization % +# DCGM_FI_DEV_MEM_COPY_UTIL - Memory bandwidth utilization +# DCGM_FI_DEV_FB_USED - Framebuffer memory used (MB) +# DCGM_FI_DEV_SM_CLOCK - SM clock speed (MHz) +# DCGM_FI_DEV_GPU_TEMP - Temperature (°C) +# DCGM_FI_DEV_POWER_USAGE - Power draw (W) +# DCGM_FI_DEV_XID_ERRORS - XID error count (0 = healthy) +``` + +## MIG Partitioning (A100/H100) + +MIG (Multi-Instance GPU) allows slicing one GPU into isolated smaller GPUs. + +```bash +# Enable MIG mode (requires reboot or restart of all processes) +sudo nvidia-smi -mig 1 +sudo systemctl restart nvidia-persistenced + +# List available MIG profiles (A100 80GB example) +nvidia-smi mig -lgip +# 1g.10gb — 1 slice, 10GB (max 7 instances) +# 2g.20gb — 2 slices, 20GB (max 3 instances) +# 3g.40gb — 3 slices, 40GB (max 2 instances) +# 7g.80gb — full GPU, 80GB (max 1 instance) + +# Create MIG instances (e.g., 3× 2g.20gb + 1× 2g.20gb = multi-tenant) +sudo nvidia-smi mig -cgi 2g.20gb,2g.20gb,2g.20gb,2g.20gb -C + +# List created instances +nvidia-smi mig -lgi +nvidia-smi mig -lcgi + +# Use in Docker +docker run --gpus '"device=MIG-GPU-xxx/0/0"' ... + +# Disable MIG +sudo nvidia-smi mig -i 0 -dci +sudo nvidia-smi mig -i 0 -dgi +sudo nvidia-smi -mig 0 +``` + +## Kernel & OS Tuning for GPU Servers + +```bash +# Increase file descriptor limits +echo '* soft nofile 1048576' | sudo tee -a /etc/security/limits.conf +echo '* hard nofile 1048576' | sudo tee -a /etc/security/limits.conf + +# Disable transparent huge pages (reduces latency jitter) +echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled +echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag + +# Persist via rc.local or systemd unit: +cat <<'EOF' | sudo tee /etc/rc.local +#!/bin/bash +echo never > /sys/kernel/mm/transparent_hugepage/enabled +echo never > /sys/kernel/mm/transparent_hugepage/defrag +nvidia-smi -pm 1 +exit 0 +EOF +sudo chmod +x /etc/rc.local + +# PCIe performance mode +sudo nvidia-smi --auto-boost-default=0 +sudo nvidia-smi --auto-boost-permission=0 +``` + +## Multi-GPU Topology Check + +```bash +# Check NVLink and PCIe topology +nvidia-smi topo -m +# Output shows interconnect type: +# NV4 = NVLink 4.0 (H100 SXM) +# NV2 = NVLink 2.0 (A100 SXM) +# PHB = PCIe bus (slower; avoid for tensor parallel training) +# PIX = same PCIe switch (fast) + +# Bandwidth test between GPUs +/usr/local/cuda/samples/bin/x86_64/linux/release/p2pBandwidthLatencyTest +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| `nvidia-smi: command not found` | Driver not installed | Follow driver installation steps above | +| Driver version mismatch | CUDA/driver incompatibility | Check compatibility matrix at developer.nvidia.com | +| GPU temperature >85°C | Poor airflow or fan failure | Check `nvidia-smi -q -d TEMPERATURE`; reseat cooler | +| XID 79 errors | GPU hardware error | Run `dcgmi diag -r 3`; may need GPU replacement | +| `failed to open device` in container | Container toolkit not configured | Run `nvidia-ctk runtime configure --runtime=docker` | +| Low PCIe bandwidth | Wrong slot or power limit | Check `nvidia-smi -q | grep PCIe`; use x16 slot | + +## Best Practices + +- Always enable persistence mode (`nvidia-smi -pm 1`) — reduces first-request latency. +- Monitor XID errors; persistent XID 79/94 indicates hardware failure. +- For training: use NVLink-connected GPUs; for inference: PCIe is usually fine. +- Set up DCGM alerts on temperature >80°C and power draw near TDP. +- Use MIG for multi-tenant inference to provide GPU isolation between models. + +## Related Skills + +- [vllm-server](../../local-ai/vllm-server/) - LLM inference on GPUs +- [llm-fine-tuning](../../local-ai/llm-fine-tuning/) - GPU training setup +- [linux-hardening](../../../security/hardening/linux-hardening/) - Secure the host OS +- [prometheus-grafana](../../../devops/observability/prometheus-grafana/) - Metrics dashboards diff --git a/security/ai/ai-security-hardening/SKILL.md b/security/ai/ai-security-hardening/SKILL.md new file mode 100644 index 0000000..9634082 --- /dev/null +++ b/security/ai/ai-security-hardening/SKILL.md @@ -0,0 +1,318 @@ +--- +name: ai-security-hardening +description: Harden AI/LLM deployments against prompt injection, data exfiltration, model theft, and supply chain attacks. Covers input validation, output filtering, access control, model API security, and compliance controls for production AI systems. +license: MIT +metadata: + author: devops-skills + version: "1.0" +--- + +# AI Security Hardening + +Secure LLM and AI systems against prompt injection, jailbreaks, data leakage, and supply chain threats in production environments. + +## When to Use This Skill + +Use this skill when: +- Deploying an LLM-powered application handling sensitive user data +- Protecting against prompt injection attacks in AI agents +- Implementing output filtering and content moderation +- Securing model weights and API endpoints from theft +- Achieving SOC2 or ISO 27001 compliance for AI systems + +## AI-Specific Threat Model + +``` +Threat Risk Control +───────────────────────────────────────────────────────────────────── +Prompt injection System prompt override Input sanitization, separate context +Data exfiltration PII in model outputs Output filtering, DLP scanning +Jailbreaking Policy bypass Content moderation, guardrails +Model theft Weight extraction via API Rate limiting, access controls +Training data poisoning Backdoored fine-tuned model Dataset validation, provenance +Supply chain attack Malicious model weights Signature verification, scanning +Insecure output XSS/SQLi from LLM response Output encoding, parameterized queries +``` + +## Prompt Injection Defense + +```python +import re +from typing import Optional + +INJECTION_PATTERNS = [ + r"ignore\s+(all\s+)?(previous|prior|above)\s+instructions", + r"you\s+are\s+now\s+", + r"new\s+instructions?:", + r"system\s+prompt", + r"forget\s+everything", + r"act\s+as\s+", + r"jailbreak", + r"dan\s+mode", + r"<\s*system\s*>", + r"\[INST\]", +] + +def detect_prompt_injection(user_input: str) -> tuple[bool, Optional[str]]: + """Return (is_suspicious, matched_pattern).""" + normalized = user_input.lower().strip() + for pattern in INJECTION_PATTERNS: + if re.search(pattern, normalized, re.IGNORECASE): + return True, pattern + return False, None + +def sanitize_user_input(user_input: str, max_length: int = 4000) -> str: + """Sanitize input before passing to LLM.""" + # Truncate + user_input = user_input[:max_length] + + # Remove null bytes and control characters + user_input = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', '', user_input) + + # Check for injection + suspicious, pattern = detect_prompt_injection(user_input) + if suspicious: + raise ValueError(f"Potential prompt injection detected: {pattern}") + + return user_input +``` + +## Guardrails with NeMo Guardrails + +```python +# guardrails.yaml +from nemoguardrails import RailsConfig, LLMRails + +config = RailsConfig.from_path("./guardrails-config") +rails = LLMRails(config) + +async def safe_llm_call(user_message: str) -> str: + response = await rails.generate_async( + messages=[{"role": "user", "content": user_message}] + ) + return response["content"] +``` + +```yaml +# guardrails-config/config.yml +models: + - type: main + engine: openai + model: gpt-4o-mini + +rails: + input: + flows: + - check jailbreak + - check sensitive data + output: + flows: + - check output for PII + - check output for harmful content +``` + +## Output Filtering & PII Scrubbing + +```python +import re +from presidio_analyzer import AnalyzerEngine +from presidio_anonymizer import AnonymizerEngine + +analyzer = AnalyzerEngine() +anonymizer = AnonymizerEngine() + +PII_ENTITIES = ["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD", + "US_SSN", "IBAN_CODE", "IP_ADDRESS", "LOCATION"] + +def scrub_pii_from_output(text: str) -> str: + """Remove PII from LLM output before returning to user.""" + results = analyzer.analyze(text=text, entities=PII_ENTITIES, language="en") + if not results: + return text + anonymized = anonymizer.anonymize(text=text, analyzer_results=results) + return anonymized.text + +def validate_output_safety(output: str) -> bool: + """Check output doesn't contain prompt injection artifacts.""" + dangerous_patterns = [ + r"<\s*script\s*>", # XSS + r"javascript:", # XSS + r";\s*(DROP|DELETE|INSERT)",# SQLi + r"\$\{.*\}", # template injection + r"`.*`", # command injection in some contexts + ] + for pattern in dangerous_patterns: + if re.search(pattern, output, re.IGNORECASE): + return False + return True +``` + +## API Security for LLM Endpoints + +```python +from fastapi import FastAPI, HTTPException, Depends, Request +from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials +import jwt +import time +from collections import defaultdict + +app = FastAPI() +security = HTTPBearer() + +# Rate limiting (per API key) +request_counts = defaultdict(list) + +def rate_limit(api_key: str, max_requests: int = 100, window_seconds: int = 60): + now = time.time() + requests = request_counts[api_key] + # Remove old requests outside window + request_counts[api_key] = [t for t in requests if now - t < window_seconds] + if len(request_counts[api_key]) >= max_requests: + raise HTTPException(status_code=429, detail="Rate limit exceeded") + request_counts[api_key].append(now) + +async def verify_token( + credentials: HTTPAuthorizationCredentials = Depends(security) +) -> dict: + try: + payload = jwt.decode(credentials.credentials, SECRET_KEY, algorithms=["HS256"]) + rate_limit(payload["sub"]) + return payload + except jwt.ExpiredSignatureError: + raise HTTPException(status_code=401, detail="Token expired") + except jwt.InvalidTokenError: + raise HTTPException(status_code=401, detail="Invalid token") + +@app.post("/v1/chat/completions") +async def chat(request: Request, token: dict = Depends(verify_token)): + body = await request.json() + + # Input validation + user_msg = body.get("messages", [{}])[-1].get("content", "") + try: + safe_input = sanitize_user_input(user_msg) + except ValueError as e: + raise HTTPException(status_code=400, detail=str(e)) + + # Call LLM and scrub output + response = await call_llm(safe_input, token["scope"]) + response["choices"][0]["message"]["content"] = scrub_pii_from_output( + response["choices"][0]["message"]["content"] + ) + return response +``` + +## Model Weight Security + +```bash +# Verify model weights with SHA-256 hash before loading +MODEL_DIR="./models/llama-3.1-8b" +EXPECTED_HASH="sha256:abc123..." + +# Generate hash of downloaded model +actual_hash=$(find "$MODEL_DIR" -name "*.safetensors" | sort | xargs sha256sum | sha256sum) +echo "Model hash: $actual_hash" + +# Compare (automate in CI/CD) +if [ "$actual_hash" != "$EXPECTED_HASH" ]; then + echo "ERROR: Model hash mismatch — possible tampering!" + exit 1 +fi + +# Scan model files for embedded malware (ModelScan) +pip install modelscan +modelscan scan -p "$MODEL_DIR" +``` + +## Network Isolation for AI Services + +```yaml +# Kubernetes NetworkPolicy — isolate LLM API +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: llm-api-isolation + namespace: ai-services +spec: + podSelector: + matchLabels: + app: vllm + policyTypes: + - Ingress + - Egress + ingress: + - from: + - namespaceSelector: + matchLabels: + name: backend # only backend can call LLM + ports: + - protocol: TCP + port: 8000 + egress: + - to: + - namespaceSelector: + matchLabels: + name: monitoring # metrics only + ports: + - protocol: TCP + port: 9090 + # Block egress to internet — prevent data exfiltration + # (allow only internal cluster traffic) +``` + +## Audit Logging + +```python +import structlog +from datetime import datetime, timezone + +audit_log = structlog.get_logger("ai.audit") + +def log_llm_interaction( + user_id: str, + session_id: str, + model: str, + prompt_tokens: int, + completion_tokens: int, + was_filtered: bool, + injection_detected: bool, +): + audit_log.info( + "llm_interaction", + timestamp=datetime.now(timezone.utc).isoformat(), + user_id=user_id, + session_id=session_id, + model=model, + prompt_tokens=prompt_tokens, + completion_tokens=completion_tokens, + was_filtered=was_filtered, + injection_detected=injection_detected, + # DO NOT log prompt/completion content — PII risk + ) +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| False positive injection blocks | Overly broad regex | Tune patterns; use ML-based classifier for high-traffic | +| PII in model outputs | Model trained on PII data | Add Presidio scrubbing to output layer | +| API key leakage | Keys in logs or responses | Mask keys in logging; use vault for key storage | +| Model weight tampering | Unverified downloads | Always verify SHA-256; use `modelscan` | +| Rate limit bypass | Per-IP not per-user | Rate limit on authenticated user ID, not IP | + +## Best Practices + +- Never log raw prompts or completions — they may contain PII or sensitive data. +- Treat LLM output as untrusted input — always encode before rendering in HTML. +- Use network policies to prevent LLM pods from making outbound internet calls. +- Rotate API keys quarterly; use short-lived JWT tokens for service-to-service auth. +- Run `modelscan` on any model downloaded from the internet before serving. + +## Related Skills + +- [hashicorp-vault](../../secrets/hashicorp-vault/) - Secrets management for API keys +- [network-security](../../network/) - Network-level controls +- [linux-hardening](../../hardening/linux-hardening/) - Host hardening +- [agent-observability](../../../devops/ai/agent-observability/) - AI audit logging +- [llm-gateway](../../../infrastructure/networking/llm-gateway/) - Centralized access control