Add 12 AI infrastructure and LLM operations skills

New skills covering hot-topic AI engineering subjects: Local AI Infrastructure: - vllm-server: High-throughput LLM serving with PagedAttention, tensor parallelism, quantization - llm-inference-scaling: KEDA-based GPU autoscaling for LLM inference on Kubernetes - rag-infrastructure: Production RAG with hybrid search, reranking, and embedding pipelines - llm-fine-tuning: QLoRA/LoRA fine-tuning with Axolotl, DeepSpeed ZeRO-3, and DPO alignment Infrastructure: - gpu-server-management: NVIDIA driver setup, MIG partitioning, DCGM monitoring - vector-database-ops: Qdrant, Weaviate, pgvector for production AI search - llm-gateway: LiteLLM-based API gateway with rate limiting, virtual keys, fallback routing DevOps/AI: - llm-cost-optimization: Model right-sizing, prompt/semantic caching, batch API, break-even analysis - llm-caching: Multi-layer exact + semantic + provider caching to cut costs 30-70% - ai-pipeline-orchestration: Prefect/Airflow/Dagster for RAG ingestion and training workflows Orchestration: - model-serving-kubernetes: KServe + Triton with canary deployments and GPU autoscaling Security: - ai-security-hardening: Prompt injection defense, PII scrubbing, model weight verification https://claude.ai/code/session_011MN1C4PrkCeg2Qmi7q1ZUe
2026-08-22 12:49:53 +02:00 · 2026-03-02 01:18:29 +00:00
parent cc3848d963
commit dd77232b16
13 changed files with 3327 additions and 0 deletions
@@ -195,6 +195,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [argocd-gitops](devops/orchestration/argocd-gitops/) | GitOps with ArgoCD |
 | [kustomize](devops/orchestration/kustomize/) | Kubernetes manifest customization |
 | [openshift](devops/orchestration/openshift/) | OpenShift cluster management |
+| [model-serving-kubernetes](devops/orchestration/model-serving-kubernetes/) | KServe and Triton model serving with canary deployments and GPU autoscaling |

 ### Observability
 | Skill | Description |
@@ -212,6 +213,9 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 |-------|-------------|
 | [agent-observability](devops/ai/agent-observability/) | Tracing, latency, token, and cost telemetry for agents |
 | [agent-evals](devops/ai/agent-evals/) | Automated regression and safety eval suites for agents |
+| [llm-cost-optimization](devops/ai/llm-cost-optimization/) | Cut LLM API costs with caching, batching, model routing, and self-hosting |
+| [llm-caching](devops/ai/llm-caching/) | Exact and semantic caching layers to reduce API calls by 30–70% |
+| [ai-pipeline-orchestration](devops/ai/ai-pipeline-orchestration/) | Orchestrate RAG ingestion, training, and batch inference with Prefect/Airflow |

 ### Release Management
 | Skill | Description |
@@ -276,6 +280,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 |-------|-------------|
 | [ai-agent-security](security/ai/ai-agent-security/) | Defend agents against injection, tool abuse, and exfiltration |
 | [llm-app-security](security/ai/llm-app-security/) | Harden LLM app inputs, outputs, and tenant isolation |
+| [ai-security-hardening](security/ai/ai-security-hardening/) | Harden LLM deployments against prompt injection, model theft, and data exfiltration |

 </details>

@@ -334,6 +339,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [user-management](infrastructure/servers/user-management/) | Users, groups, sudo |
 | [systemd-services](infrastructure/servers/systemd-services/) | Services and timers |
 | [performance-tuning](infrastructure/servers/performance-tuning/) | System optimization |
+| [gpu-server-management](infrastructure/servers/gpu-server-management/) | NVIDIA GPU driver setup, MIG partitioning, DCGM monitoring for AI workloads |

 ### Networking
 | Skill | Description |
@@ -343,6 +349,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [cdn-setup](infrastructure/networking/cdn-setup/) | CloudFront, Cloudflare |
 | [reverse-proxy](infrastructure/networking/reverse-proxy/) | nginx, Traefik |
 | [service-mesh](infrastructure/networking/service-mesh/) | Istio, Linkerd |
+| [llm-gateway](infrastructure/networking/llm-gateway/) | Unified LLM API gateway with routing, rate limiting, virtual keys, and semantic caching |

 ### Databases
 | Skill | Description |
@@ -353,6 +360,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [mongodb](infrastructure/databases/mongodb/) | MongoDB clusters |
 | [redis](infrastructure/databases/redis/) | Redis caching |
 | [database-backups](infrastructure/databases/database-backups/) | Backup strategies |
+| [vector-database-ops](infrastructure/databases/vector-database-ops/) | Qdrant, Weaviate, and pgvector for production AI search and RAG workloads |

 ### Storage
 | Skill | Description |
@@ -375,6 +383,10 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [ollama-stack](infrastructure/local-ai/ollama-stack/) | Private local inference stack with Ollama |
 | [mac-mini-llm-lab](infrastructure/local-ai/mac-mini-llm-lab/) | Mac mini setup for always-on local LLM serving |
 | [openclaw-local-mac-mini](infrastructure/local-ai/openclaw-local-mac-mini/) | OpenClaw setup for local development and Mac mini hosting |
+| [vllm-server](infrastructure/local-ai/vllm-server/) | High-throughput LLM serving with vLLM — PagedAttention, tensor parallelism, OpenAI API |
+| [llm-inference-scaling](infrastructure/local-ai/llm-inference-scaling/) | Auto-scale LLM inference clusters on Kubernetes with KEDA and GPU-aware scheduling |
+| [rag-infrastructure](infrastructure/local-ai/rag-infrastructure/) | Production RAG with vector stores, hybrid search, embedding pipelines, and reranking |
+| [llm-fine-tuning](infrastructure/local-ai/llm-fine-tuning/) | QLoRA and full fine-tuning with Axolotl, DeepSpeed, and DPO alignment on GPU clusters |

 ### IT Operations
 | Skill | Description |
@@ -0,0 +1,262 @@
+---
+name: ai-pipeline-orchestration
+description: Orchestrate AI/ML pipelines for data ingestion, model training, batch inference, and RAG indexing using Prefect, Airflow, or Dagster. Build reliable, observable, and retriable workflows for production AI systems.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# AI Pipeline Orchestration
+
+Build reliable, observable AI workflows — from document ingestion to batch inference to model training pipelines.
+
+## When to Use This Skill
+
+Use this skill when:
+- Scheduling recurring RAG document ingestion and re-indexing
+- Orchestrating multi-step batch LLM processing workflows
+- Running nightly model evaluation and fine-tuning jobs
+- Building ETL pipelines that feed into AI models
+- Managing dependencies between data preparation and model serving
+
+## Tool Selection
+
+| Tool | Best For | Complexity | GPU Jobs |
+|------|----------|------------|----------|
+| **Prefect** | Modern Python-first; easy to adopt | Low | Good |
+| **Airflow** | Complex DAGs; large teams; existing usage | High | Good |
+| **Dagster** | Asset-centric; strong data lineage | Medium | Excellent |
+| **Temporal** | Long-running workflows; reliability-first | Medium | Good |
+
+## Prefect — Quick Start
+
+```bash
+pip install prefect prefect-kubernetes
+
+# Start Prefect server (or use Prefect Cloud)
+prefect server start
+
+# In another terminal
+prefect worker start --pool default-agent-pool
+```
+
+## Prefect: RAG Ingestion Pipeline
+
+```python
+from prefect import flow, task, get_run_logger
+from prefect.tasks import task_input_hash
+from datetime import timedelta
+import hashlib
+
+@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=24))
+def fetch_documents(source_url: str) -> list[dict]:
+    """Fetch documents from source; cached to avoid re-fetching."""
+    logger = get_run_logger()
+    logger.info(f"Fetching from {source_url}")
+    # ... fetch logic
+    return documents
+
+@task(retries=3, retry_delay_seconds=30)
+def chunk_and_embed(documents: list[dict]) -> list[dict]:
+    """Chunk documents and generate embeddings with retry on failure."""
+    from sentence_transformers import SentenceTransformer
+    model = SentenceTransformer("BAAI/bge-large-en-v1.5")
+    chunks = []
+    for doc in documents:
+        doc_chunks = chunk_text(doc["content"])
+        embeddings = model.encode(doc_chunks, batch_size=64)
+        for chunk, emb in zip(doc_chunks, embeddings):
+            chunks.append({"text": chunk, "embedding": emb.tolist(),
+                           "source": doc["url"], "doc_hash": doc["hash"]})
+    return chunks
+
+@task(retries=2)
+def upsert_to_vector_store(chunks: list[dict]) -> int:
+    """Upsert embeddings to Qdrant, skip unchanged documents."""
+    from qdrant_client import QdrantClient
+    client = QdrantClient("http://qdrant:6333")
+    client.upsert(collection_name="knowledge-base", points=[...])
+    return len(chunks)
+
+@flow(name="rag-ingestion", log_prints=True)
+def rag_ingestion_pipeline(sources: list[str]):
+    """Full RAG ingestion flow — runs daily."""
+    logger = get_run_logger()
+    total = 0
+    for source in sources:
+        docs = fetch_documents(source)
+        chunks = chunk_and_embed(docs)
+        count = upsert_to_vector_store(chunks)
+        total += count
+        logger.info(f"Ingested {count} chunks from {source}")
+    logger.info(f"Pipeline complete: {total} total chunks indexed")
+
+if __name__ == "__main__":
+    rag_ingestion_pipeline.serve(
+        name="daily-rag-ingestion",
+        cron="0 2 * * *",          # 2 AM daily
+        parameters={"sources": ["https://docs.myapp.com", "https://api.myapp.com/kb"]},
+    )
+```
+
+## Prefect: Batch LLM Inference Pipeline
+
+```python
+from prefect import flow, task
+from prefect.concurrency.sync import concurrency
+import asyncio
+from openai import AsyncOpenAI
+
+@task(retries=3, retry_delay_seconds=60)
+async def process_batch(items: list[dict], model: str = "gpt-4o-mini") -> list[dict]:
+    """Process a batch of items through LLM with rate limit protection."""
+    client = AsyncOpenAI()
+    async with concurrency("openai-api", occupy=len(items)):  # rate limit
+        tasks = [
+            client.chat.completions.create(
+                model=model,
+                messages=[{"role": "user", "content": item["prompt"]}],
+                max_tokens=256,
+            )
+            for item in items
+        ]
+        responses = await asyncio.gather(*tasks, return_exceptions=True)
+
+    results = []
+    for item, response in zip(items, responses):
+        if isinstance(response, Exception):
+            results.append({**item, "error": str(response), "output": None})
+        else:
+            results.append({**item, "output": response.choices[0].message.content})
+    return results
+
+@flow(name="batch-llm-inference")
+async def batch_inference_flow(input_file: str, output_file: str, batch_size: int = 50):
+    import json
+    items = [json.loads(line) for line in open(input_file)]
+    batches = [items[i:i+batch_size] for i in range(0, len(items), batch_size)]
+
+    all_results = []
+    for batch in batches:
+        results = await process_batch(batch)
+        all_results.extend(results)
+
+    with open(output_file, "w") as f:
+        for result in all_results:
+            f.write(json.dumps(result) + "\n")
+    return len(all_results)
+```
+
+## Airflow: Model Training DAG
+
+```python
+from airflow.decorators import dag, task
+from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
+from datetime import datetime
+from kubernetes.client import models as k8s
+
+@dag(
+    dag_id="llm_fine_tuning",
+    schedule="@weekly",
+    start_date=datetime(2025, 1, 1),
+    catchup=False,
+    tags=["ai", "training"],
+)
+def llm_fine_tuning_dag():
+
+    @task
+    def prepare_dataset() -> str:
+        """Download and preprocess training data."""
+        # ... data prep logic
+        return "s3://my-bucket/training-data/2025-03-01/"
+
+    train = KubernetesPodOperator(
+        task_id="train_model",
+        name="llm-training-job",
+        namespace="ml",
+        image="nvcr.io/nvidia/pytorch:24.05-py3",
+        cmds=["accelerate", "launch", "-m", "axolotl.cli.train", "/config/config.yaml"],
+        resources=k8s.V1ResourceRequirements(
+            limits={"nvidia.com/gpu": "4", "memory": "320Gi"},
+            requests={"nvidia.com/gpu": "4"},
+        ),
+        node_selector={"nvidia.com/gpu.product": "A100-SXM4-80GB"},
+        volumes=[...],
+        volume_mounts=[...],
+        get_logs=True,
+        is_delete_operator_pod=True,
+    )
+
+    @task
+    def evaluate_model(dataset_path: str) -> dict:
+        """Run evals; fail pipeline if quality drops."""
+        metrics = run_evals()
+        if metrics["accuracy"] < 0.85:
+            raise ValueError(f"Model quality too low: {metrics}")
+        return metrics
+
+    @task
+    def deploy_model(metrics: dict):
+        """Push merged model to HF Hub and update vLLM config."""
+        update_serving_config(new_model="org/fine-tuned-v2")
+
+    dataset = prepare_dataset()
+    train.set_upstream(dataset)
+    eval_result = evaluate_model(dataset)
+    eval_result.set_upstream(train)
+    deploy_model(eval_result)
+
+llm_fine_tuning_dag()
+```
+
+## Dagster: Asset-Based AI Pipeline
+
+```python
+from dagster import asset, AssetExecutionContext, define_asset_job, ScheduleDefinition
+
+@asset(description="Raw documents fetched from knowledge sources")
+def raw_documents(context: AssetExecutionContext) -> list[dict]:
+    context.log.info("Fetching documents...")
+    return fetch_all_documents()
+
+@asset(
+    deps=[raw_documents],
+    description="Chunked and embedded document vectors",
+)
+def document_embeddings(context: AssetExecutionContext, raw_documents) -> int:
+    chunks = process_and_embed(raw_documents)
+    context.log.info(f"Generated {len(chunks)} embeddings")
+    upsert_to_qdrant(chunks)
+    return len(chunks)
+
+@asset(
+    deps=[document_embeddings],
+    description="RAG system quality metrics",
+)
+def rag_quality_metrics(context: AssetExecutionContext) -> dict:
+    metrics = evaluate_rag_system()
+    context.add_output_metadata({"ragas_score": metrics["ragas_score"]})
+    return metrics
+
+# Schedule: refresh embeddings nightly
+nightly_refresh = ScheduleDefinition(
+    job=define_asset_job("rag_refresh_job", [raw_documents, document_embeddings]),
+    cron_schedule="0 1 * * *",
+)
+```
+
+## Best Practices
+
+- Use task-level retries for API calls; use flow-level retries for transient infra failures.
+- Cache expensive steps (embedding generation, data fetching) to speed up reruns.
+- Emit custom metrics from pipelines (chunk count, error rate, cost) to your observability stack.
+- Use `concurrency` limits in Prefect or `pool` slots in Airflow to respect external rate limits.
+- Separate ingestion, training, and deployment pipelines — don't couple them in one giant DAG.
+
+## Related Skills
+
+- [rag-infrastructure](../../infrastructure/local-ai/rag-infrastructure/) - RAG system setup
+- [llm-fine-tuning](../../infrastructure/local-ai/llm-fine-tuning/) - Training jobs
+- [agent-observability](../agent-observability/) - Pipeline monitoring
+- [kubernetes-ops](../orchestration/kubernetes-ops/) - Running pipeline pods on K8s
@@ -0,0 +1,309 @@
+---
+name: llm-caching
+description: Implement multi-layer LLM caching with exact match, semantic similarity, and provider-side prompt caching. Reduce API costs by 30–70%, cut latency, and improve throughput using Redis, GPTCache, and provider caching APIs.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# LLM Caching
+
+Cut LLM costs and latency with exact match, semantic, and provider-side caching layers.
+
+## When to Use This Skill
+
+Use this skill when:
+- The same or similar queries are asked repeatedly (FAQ bots, support tools)
+- LLM API costs are growing and you need immediate savings
+- Serving high request volumes where repeated queries cause bottlenecks
+- Implementing prompt caching for long system prompts (Anthropic/OpenAI)
+- Building offline-capable AI features that need response persistence
+
+## Caching Layers
+
+```
+Request → Exact Cache → Semantic Cache → Provider Cache → LLM API
+             ↓ hit            ↓ hit             ↓ hit
+           instant          ~5ms           50-80% cheaper
+```
+
+## Layer 1: Exact Match Cache (Redis)
+
+```python
+import hashlib
+import json
+import redis
+from openai import OpenAI
+
+r = redis.Redis(host="localhost", port=6379, decode_responses=True)
+client = OpenAI()
+
+def build_cache_key(model: str, messages: list, temperature: float) -> str:
+    """Deterministic key from request parameters."""
+    payload = json.dumps({
+        "model": model,
+        "messages": messages,
+        "temperature": temperature,
+    }, sort_keys=True)
+    return f"llm:exact:{hashlib.sha256(payload.encode()).hexdigest()}"
+
+def cached_completion(model: str, messages: list, temperature: float = 0.0,
+                      ttl: int = 3600) -> dict:
+    key = build_cache_key(model, messages, temperature)
+
+    # Check cache
+    if cached := r.get(key):
+        return json.loads(cached)
+
+    # Call API
+    response = client.chat.completions.create(
+        model=model, messages=messages, temperature=temperature
+    )
+    result = response.model_dump()
+
+    # Cache result (only cache deterministic responses)
+    if temperature == 0.0:
+        r.setex(key, ttl, json.dumps(result))
+
+    return result
+```
+
+## Layer 2: Semantic Cache (GPTCache)
+
+```python
+from gptcache import cache, Config
+from gptcache.adapter import openai
+from gptcache.embedding import Onnx
+from gptcache.manager import CacheBase, VectorBase, get_data_manager
+from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
+
+# Configure GPTCache with Qdrant backend
+def init_gptcache(cache_obj, llm: str):
+    onnx = Onnx()                              # local embedding model
+    data_manager = get_data_manager(
+        CacheBase("redis"),                    # metadata store
+        VectorBase("qdrant",
+                   host="localhost",
+                   port=6333,
+                   collection_name=f"llm-cache-{llm}",
+                   dimension=onnx.dimension),
+    )
+    cache_obj.init(
+        embedding_func=onnx.to_embeddings,
+        data_manager=data_manager,
+        similarity_evaluation=SearchDistanceEvaluation(),
+        config=Config(similarity_threshold=0.80),  # 80% similarity = cache hit
+    )
+
+cache.set_openai_key()
+init_gptcache(cache, "gpt-4o-mini")
+
+# Now openai calls are automatically cached
+response = openai.ChatCompletion.create(
+    model="gpt-4o-mini",
+    messages=[{"role": "user", "content": "What is machine learning?"}],
+)
+# Second call with similar question ("Explain machine learning") → cache hit
+```
+
+## Custom Semantic Cache (Production-Grade)
+
+```python
+from sentence_transformers import SentenceTransformer
+from qdrant_client import QdrantClient
+from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, Range
+import numpy as np
+import uuid
+import time
+
+embed_model = SentenceTransformer("BAAI/bge-small-en-v1.5")  # fast, 33M params
+qdrant = QdrantClient("http://localhost:6333")
+
+CACHE_COLLECTION = "semantic-cache"
+SIMILARITY_THRESHOLD = 0.88
+CACHE_TTL_SECONDS = 86400  # 24h
+
+# Create collection once
+qdrant.create_collection(
+    collection_name=CACHE_COLLECTION,
+    vectors_config=VectorParams(size=384, distance=Distance.COSINE),
+    on_disk_payload=True,
+)
+
+def semantic_cache_lookup(query: str, model: str) -> str | None:
+    embedding = embed_model.encode(query).tolist()
+    results = qdrant.query_points(
+        collection_name=CACHE_COLLECTION,
+        query=embedding,
+        query_filter=Filter(must=[
+            FieldCondition(key="model", match={"value": model}),
+            FieldCondition(key="expires_at", range=Range(gte=time.time())),
+        ]),
+        limit=1,
+        score_threshold=SIMILARITY_THRESHOLD,
+    )
+    if results.points:
+        return results.points[0].payload["response"]
+    return None
+
+def semantic_cache_store(query: str, response: str, model: str):
+    embedding = embed_model.encode(query).tolist()
+    qdrant.upsert(
+        collection_name=CACHE_COLLECTION,
+        points=[PointStruct(
+            id=str(uuid.uuid4()),
+            vector=embedding,
+            payload={
+                "query": query,
+                "response": response,
+                "model": model,
+                "created_at": time.time(),
+                "expires_at": time.time() + CACHE_TTL_SECONDS,
+            },
+        )],
+    )
+
+def smart_llm_call(query: str, model: str = "gpt-4o-mini") -> dict:
+    # 1. Semantic lookup
+    if cached_response := semantic_cache_lookup(query, model):
+        return {"response": cached_response, "source": "semantic_cache", "cost": 0}
+
+    # 2. LLM call
+    response = client.chat.completions.create(
+        model=model,
+        messages=[{"role": "user", "content": query}],
+    )
+    text = response.choices[0].message.content
+    cost = litellm.completion_cost(response)
+
+    # 3. Store in cache
+    semantic_cache_store(query, text, model)
+
+    return {"response": text, "source": "llm_api", "cost": cost}
+```
+
+## Layer 3: Provider-Side Prompt Caching
+
+```python
+# Anthropic — cache long system prompts (saves 90% on cached input tokens)
+import anthropic
+
+client = anthropic.Anthropic()
+
+# Long system prompt — mark for caching
+SYSTEM_PROMPT = open("knowledge-base.txt").read()  # e.g., 50k tokens
+
+def call_with_prompt_cache(user_question: str) -> str:
+    response = client.messages.create(
+        model="claude-sonnet-4-6",
+        max_tokens=1024,
+        system=[
+            {"type": "text", "text": "You are a helpful assistant."},
+            {
+                "type": "text",
+                "text": SYSTEM_PROMPT,
+                "cache_control": {"type": "ephemeral"},  # cache this block
+            }
+        ],
+        messages=[{"role": "user", "content": user_question}],
+    )
+    # Log cache efficiency
+    usage = response.usage
+    cache_savings = usage.cache_read_input_tokens * 0.9  # 90% discount on cached
+    print(f"Cache hits: {usage.cache_read_input_tokens} tokens "
+          f"(saved ~${cache_savings * 3.0 / 1_000_000:.4f})")
+    return response.content[0].text
+
+# OpenAI — automatic for repeated prefixes (≥1,024 tokens)
+# No code change needed; cached tokens appear in usage.prompt_tokens_details
+response = client.chat.completions.create(
+    model="gpt-4o-mini",
+    messages=[
+        {"role": "system", "content": LONG_SYSTEM_PROMPT},  # auto-cached
+        {"role": "user", "content": user_question},
+    ]
+)
+cached = response.usage.prompt_tokens_details.cached_tokens
+print(f"OpenAI cached {cached} tokens")
+```
+
+## Cache Warming
+
+```python
+async def warm_cache(common_queries: list[str], model: str):
+    """Pre-populate cache with known frequent queries."""
+    import asyncio
+    from openai import AsyncOpenAI
+
+    aclient = AsyncOpenAI()
+
+    async def warm_single(query: str):
+        if not semantic_cache_lookup(query, model):
+            response = await aclient.chat.completions.create(
+                model=model,
+                messages=[{"role": "user", "content": query}],
+            )
+            text = response.choices[0].message.content
+            semantic_cache_store(query, text, model)
+            print(f"Warmed: {query[:50]}...")
+
+    await asyncio.gather(*[warm_single(q) for q in common_queries])
+
+# Warm on startup
+import asyncio
+asyncio.run(warm_cache(FREQUENT_QUERIES, "gpt-4o-mini"))
+```
+
+## Cache Metrics
+
+```python
+from prometheus_client import Counter, Histogram
+
+cache_hits = Counter("llm_cache_hits_total", "Cache hits", ["cache_layer", "model"])
+cache_misses = Counter("llm_cache_misses_total", "Cache misses", ["model"])
+cache_savings_usd = Counter("llm_cache_savings_usd_total", "USD saved by cache", ["model"])
+
+# Use in your smart_llm_call function
+if source == "semantic_cache":
+    cache_hits.labels(cache_layer="semantic", model=model).inc()
+    cache_savings_usd.labels(model=model).inc(estimated_cost)
+else:
+    cache_misses.labels(model=model).inc()
+```
+
+## Redis Configuration for LLM Caching
+
+```bash
+# redis.conf tuning for LLM cache workload
+maxmemory 8gb
+maxmemory-policy allkeys-lru    # evict least-recently-used when full
+save ""                          # disable persistence (cache is ephemeral)
+appendonly no
+tcp-keepalive 60
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| Low cache hit rate | Threshold too strict | Lower `SIMILARITY_THRESHOLD` to 0.82–0.85 |
+| Stale cached responses | Long TTL | Use topic-specific TTLs; invalidate on data updates |
+| Cache serving wrong answers | Threshold too loose | Raise threshold or add model-name filtering |
+| Redis OOM | No eviction policy | Set `maxmemory` + `allkeys-lru` |
+| Slow semantic lookup | Large cache collection | Add payload index on `model` + `expires_at` |
+
+## Best Practices
+
+- Start with exact cache — zero cost, instant wins for identical queries.
+- Semantic threshold of 0.88–0.92 balances hit rate vs. accuracy; tune with your data.
+- Set per-model TTLs: longer for stable knowledge (1 week), shorter for news/events (1 hour).
+- Always filter by model name in semantic cache — different models give different answers.
+- Log cache hit rate as a KPI; target 30%+ for FAQ-style applications.
+
+## Related Skills
+
+- [llm-cost-optimization](../llm-cost-optimization/) - Full cost strategy
+- [llm-gateway](../../infrastructure/networking/llm-gateway/) - Gateway-level caching
+- [vector-database-ops](../../infrastructure/databases/vector-database-ops/) - Qdrant setup
+- [agent-observability](../agent-observability/) - Cache metrics dashboards
@@ -0,0 +1,286 @@
+---
+name: llm-cost-optimization
+description: Reduce LLM API and infrastructure costs through model selection, prompt caching, batching, caching, quantization, and self-hosting strategies. Track spend by team and model, set budgets, and implement cost-aware routing.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# LLM Cost Optimization
+
+Cut LLM costs by 50–90% with the right combination of caching, model selection, prompt optimization, and self-hosting.
+
+## When to Use This Skill
+
+Use this skill when:
+- LLM API spend is growing faster than revenue
+- You need to attribute AI costs to teams, products, or customers
+- Implementing caching to avoid redundant LLM calls
+- Deciding when to switch from API providers to self-hosted models
+- Optimizing prompt length without sacrificing quality
+
+## Cost Levers by Impact
+
+| Strategy | Typical Savings | Effort |
+|----------|-----------------|--------|
+| Semantic caching | 20–50% | Low |
+| Model right-sizing | 30–70% | Low |
+| Prompt compression | 10–30% | Medium |
+| Provider caching (prompt cache) | 10–25% | Low |
+| Batching offline workloads | 50% (Batch API) | Medium |
+| Self-hosting 7–8B models | 80–95% at scale | High |
+| Quantization | 30–50% VRAM cost | Medium |
+
+## Track Costs First
+
+```python
+# Use LiteLLM's cost tracking (automatic per-model pricing)
+import litellm
+
+response = litellm.completion(
+    model="gpt-4o-mini",
+    messages=[{"role": "user", "content": "Hello"}],
+)
+cost = litellm.completion_cost(response)
+print(f"Cost: ${cost:.6f}")
+
+# Add custom cost callbacks
+def log_cost(kwargs, completion_response, start_time, end_time):
+    cost = kwargs.get("response_cost", 0)
+    model = kwargs.get("model")
+    user = kwargs.get("user")
+    # Send to your analytics DB
+    db.record_cost(user=user, model=model, cost=cost)
+
+litellm.success_callback = [log_cost]
+```
+
+## Model Right-Sizing
+
+```python
+# Route by task complexity — don't use GPT-4o for everything
+def get_model_for_task(task_type: str) -> str:
+    routing = {
+        "classification":     "gpt-4o-mini",      # ~30× cheaper than gpt-4o
+        "summarization":      "gpt-4o-mini",
+        "extraction":         "gpt-4o-mini",
+        "simple_qa":          "gpt-4o-mini",
+        "complex_reasoning":  "gpt-4o",
+        "code_generation":    "claude-sonnet-4-6",
+        "creative_writing":   "claude-opus-4-6",
+    }
+    return routing.get(task_type, "gpt-4o-mini")
+
+# Cost comparison (per 1M tokens, 2025 approx.)
+# gpt-4o-mini:          input $0.15 / output $0.60
+# gpt-4o:               input $2.50 / output $10.00
+# claude-sonnet-4-6:    input $3.00 / output $15.00
+# llama-3.1-8b (self):  ~$0.05–0.10 all-in (GPU amortized)
+```
+
+## Prompt Caching (Provider-Side)
+
+```python
+# Anthropic — cache long system prompts (saves 90% on cached tokens)
+import anthropic
+
+client = anthropic.Anthropic()
+
+response = client.messages.create(
+    model="claude-sonnet-4-6",
+    max_tokens=1024,
+    system=[
+        {
+            "type": "text",
+            "text": "You are a helpful assistant.",
+        },
+        {
+            "type": "text",
+            "text": open("large-context.txt").read(),  # large doc
+            "cache_control": {"type": "ephemeral"},     # cache this!
+        }
+    ],
+    messages=[{"role": "user", "content": "Summarize the key points."}],
+)
+# First call: full price. Subsequent calls: 90% discount on cached part.
+print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")
+
+# OpenAI — prompt caching is automatic for repeated prefixes >1024 tokens
+# No code change needed; check usage.prompt_tokens_details.cached_tokens
+```
+
+## Batching with OpenAI Batch API (50% Discount)
+
+```python
+import json
+from openai import OpenAI
+
+client = OpenAI()
+
+# Prepare batch requests
+requests = [
+    {
+        "custom_id": f"task-{i}",
+        "method": "POST",
+        "url": "/v1/chat/completions",
+        "body": {
+            "model": "gpt-4o-mini",
+            "messages": [{"role": "user", "content": f"Classify: {text}"}],
+            "max_tokens": 50,
+        }
+    }
+    for i, text in enumerate(texts)
+]
+
+# Write JSONL file
+with open("batch.jsonl", "w") as f:
+    for req in requests:
+        f.write(json.dumps(req) + "\n")
+
+# Upload and create batch
+batch_file = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
+batch = client.batches.create(
+    input_file_id=batch_file.id,
+    endpoint="/v1/chat/completions",
+    completion_window="24h",
+)
+print(f"Batch ID: {batch.id}")  # poll status with client.batches.retrieve(batch.id)
+```
+
+## Semantic Caching
+
+```python
+import hashlib
+import json
+import redis
+import numpy as np
+from sentence_transformers import SentenceTransformer
+
+r = redis.Redis(host="localhost", port=6379)
+embed_model = SentenceTransformer("BAAI/bge-small-en-v1.5")
+
+SIMILARITY_THRESHOLD = 0.92
+CACHE_TTL = 3600 * 24  # 24 hours
+
+def cached_llm_call(prompt: str, llm_fn) -> str:
+    # 1. Exact match (free)
+    exact_key = f"exact:{hashlib.sha256(prompt.encode()).hexdigest()}"
+    if cached := r.get(exact_key):
+        return cached.decode()
+
+    # 2. Semantic match
+    query_vec = embed_model.encode(prompt)
+    cached_keys = r.keys("sem:*")
+    for key in cached_keys:
+        data = json.loads(r.get(key))
+        similarity = np.dot(query_vec, data["embedding"]) / (
+            np.linalg.norm(query_vec) * np.linalg.norm(data["embedding"])
+        )
+        if similarity >= SIMILARITY_THRESHOLD:
+            return data["response"]
+
+    # 3. Cache miss — call LLM
+    response = llm_fn(prompt)
+
+    # Store exact match
+    r.setex(exact_key, CACHE_TTL, response)
+
+    # Store semantic embedding
+    sem_key = f"sem:{hashlib.sha256(prompt.encode()).hexdigest()}"
+    r.setex(sem_key, CACHE_TTL, json.dumps({
+        "embedding": query_vec.tolist(),
+        "response": response,
+        "prompt": prompt,
+    }))
+    return response
+```
+
+## Prompt Compression
+
+```python
+# LLMLingua — compress long prompts by 3–20× with minimal quality loss
+from llmlingua import PromptCompressor
+
+compressor = PromptCompressor(
+    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
+    device_map="cpu",
+)
+
+compressed = compressor.compress_prompt(
+    long_context,
+    ratio=0.5,       # keep 50% of tokens
+    rank_method="longllmlingua",
+)
+print(f"Original: {len(long_context.split())} words")
+print(f"Compressed: {len(compressed['compressed_prompt'].split())} words")
+print(f"Savings: {compressed['saving']}")
+```
+
+## Self-Hosting Break-Even Calculator
+
+```python
+def break_even_analysis(
+    monthly_api_spend_usd: float,
+    gpu_cost_per_hour_usd: float = 2.50,   # e.g., A10G on AWS
+    utilization: float = 0.70,             # 70% GPU utilization
+) -> dict:
+    monthly_gpu_cost = gpu_cost_per_hour_usd * 24 * 30 * utilization
+    break_even = monthly_gpu_cost / monthly_api_spend_usd
+    recommendation = (
+        "Self-host now — strong ROI" if break_even < 0.5 else
+        "Self-host if traffic grows 2×" if break_even < 0.8 else
+        "Stick with API — not enough scale yet"
+    )
+    return {
+        "monthly_gpu_cost": f"${monthly_gpu_cost:.0f}",
+        "monthly_api_spend": f"${monthly_api_spend_usd:.0f}",
+        "gpu_as_pct_of_api": f"{break_even*100:.0f}%",
+        "recommendation": recommendation,
+    }
+
+# Example: $5k/month on OpenAI, $2.50/hr A10G
+print(break_even_analysis(5000))
+# → gpu_cost ~$1,260/mo = 25% of API spend → self-host now
+```
+
+## Cost Dashboard (Grafana)
+
+```python
+# Emit cost metrics to Prometheus
+from prometheus_client import Counter, Histogram
+
+llm_cost_total = Counter(
+    "llm_cost_usd_total",
+    "Total LLM spend in USD",
+    ["model", "team", "task_type"],
+)
+llm_tokens_total = Counter(
+    "llm_tokens_total",
+    "Total tokens used",
+    ["model", "token_type"],  # token_type: prompt, completion, cached
+)
+
+def track_call(model, team, task_type, response):
+    cost = calculate_cost(model, response.usage)
+    llm_cost_total.labels(model=model, team=team, task_type=task_type).inc(cost)
+    llm_tokens_total.labels(model=model, token_type="prompt").inc(
+        response.usage.prompt_tokens)
+    llm_tokens_total.labels(model=model, token_type="completion").inc(
+        response.usage.completion_tokens)
+```
+
+## Best Practices
+
+- Use `gpt-4o-mini` or `claude-haiku` for 80% of tasks — they're 10–30× cheaper.
+- Enable prompt caching for system prompts >1,024 tokens (Anthropic) or >1,024 tokens (OpenAI).
+- Audit your top 5 prompts by token count — compress or cache them.
+- Set hard budget limits with LiteLLM virtual keys before costs spiral.
+- Self-host 7B–8B models when monthly API spend exceeds $2k/month.
+
+## Related Skills
+
+- [llm-gateway](../../infrastructure/networking/llm-gateway/) - Centralized cost control
+- [llm-caching](../llm-caching/) - Semantic caching patterns
+- [vllm-server](../../infrastructure/local-ai/vllm-server/) - Self-hosted inference
+- [agent-observability](../agent-observability/) - Token and cost telemetry
@@ -0,0 +1,314 @@
+---
+name: model-serving-kubernetes
+description: Deploy ML models on Kubernetes with KServe (formerly KFServing) and NVIDIA Triton Inference Server. Includes canary deployments, autoscaling, model versioning, A/B testing, and GPU resource management for production model serving.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# Model Serving on Kubernetes
+
+Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling.
+
+## When to Use This Skill
+
+Use this skill when:
+- Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale
+- Implementing canary deployments and A/B testing for ML models
+- Autoscaling inference pods based on request rate or GPU metrics
+- Deploying LLMs with Triton or KServe on Kubernetes
+- Managing multiple model versions with traffic splitting
+
+## Prerequisites
+
+- Kubernetes 1.28+ with GPU nodes
+- KServe installed (or Triton standalone)
+- `kubectl` and `helm` configured
+- NVIDIA GPU Operator installed on cluster
+
+## KServe Installation
+
+```bash
+# Install KServe with Helm
+helm repo add kserve https://kserve.github.io/helm-charts
+helm repo update
+
+helm install kserve kserve/kserve \
+  --namespace kserve \
+  --create-namespace \
+  --set kserve.controller.gateway.ingressGateway.className=nginx
+
+# Verify
+kubectl get pods -n kserve
+kubectl get crd | grep kserve
+```
+
+## Basic InferenceService (KServe)
+
+```yaml
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  name: sklearn-iris
+  namespace: models
+spec:
+  predictor:
+    sklearn:
+      storageUri: gs://kfserving-examples/models/sklearn/1.0/model
+      resources:
+        requests:
+          cpu: "1"
+          memory: 2Gi
+        limits:
+          cpu: "2"
+          memory: 4Gi
+```
+
+```bash
+kubectl apply -f inference-service.yaml
+
+# Get inference service URL
+kubectl get inferenceservice sklearn-iris -n models
+# NAME           URL                                          READY   ...
+# sklearn-iris   http://sklearn-iris.models.example.com       True
+
+# Test prediction
+curl -X POST http://sklearn-iris.models.example.com/v1/models/sklearn-iris:predict \
+  -H "Content-Type: application/json" \
+  -d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}'
+```
+
+## GPU-Enabled LLM InferenceService
+
+```yaml
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  name: llama-3-8b
+  namespace: models
+  annotations:
+    serving.kserve.io/enable-prometheus-scraping: "true"
+spec:
+  predictor:
+    containers:
+    - name: vllm-container
+      image: vllm/vllm-openai:latest
+      args:
+      - "--model"
+      - "meta-llama/Llama-3.1-8B-Instruct"
+      - "--tensor-parallel-size"
+      - "1"
+      - "--gpu-memory-utilization"
+      - "0.90"
+      ports:
+      - containerPort: 8080
+        protocol: TCP
+      resources:
+        requests:
+          nvidia.com/gpu: "1"
+          memory: "20Gi"
+          cpu: "4"
+        limits:
+          nvidia.com/gpu: "1"
+          memory: "24Gi"
+          cpu: "8"
+      readinessProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        initialDelaySeconds: 60
+        periodSeconds: 10
+      env:
+      - name: HUGGING_FACE_HUB_TOKEN
+        valueFrom:
+          secretKeyRef:
+            name: hf-token
+            key: token
+    nodeSelector:
+      nvidia.com/gpu.present: "true"
+  transformer:
+    containers:
+    - name: kserve-container
+      image: kserve/kserve-transformer:latest
+```
+
+## Canary Deployment (Traffic Splitting)
+
+```yaml
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  name: llama-3-8b
+  namespace: models
+spec:
+  predictor:
+    canaryTrafficPercent: 20    # 20% to new version, 80% to stable
+    containers:
+    - name: vllm-container
+      image: vllm/vllm-openai:latest
+      args:
+      - "--model"
+      - "meta-llama/Llama-3.1-8B-Instruct-v2"  # new model version
+      resources:
+        limits:
+          nvidia.com/gpu: "1"
+```
+
+```bash
+# Gradually increase canary traffic
+kubectl patch inferenceservice llama-3-8b -n models \
+  --type='json' \
+  -p='[{"op":"replace","path":"/spec/predictor/canaryTrafficPercent","value":50}]'
+
+# Promote canary to stable
+kubectl patch inferenceservice llama-3-8b -n models \
+  --type='json' \
+  -p='[{"op":"remove","path":"/spec/predictor/canaryTrafficPercent"}]'
+```
+
+## Autoscaling with KEDA
+
+```yaml
+apiVersion: keda.sh/v1alpha1
+kind: ScaledObject
+metadata:
+  name: llama-scaler
+  namespace: models
+spec:
+  scaleTargetRef:
+    apiVersion: serving.kserve.io/v1beta1
+    kind: InferenceService
+    name: llama-3-8b
+  minReplicaCount: 1
+  maxReplicaCount: 5
+  triggers:
+  - type: prometheus
+    metadata:
+      serverAddress: http://prometheus-server.monitoring:9090
+      metricName: kserve_request_count
+      threshold: "10"
+      query: |
+        sum(rate(kserve_request_count_total{namespace="models",
+            service="llama-3-8b"}[1m]))
+```
+
+## NVIDIA Triton Inference Server
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: triton-server
+  namespace: models
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: triton
+  template:
+    metadata:
+      labels:
+        app: triton
+    spec:
+      containers:
+      - name: triton
+        image: nvcr.io/nvidia/tritonserver:24.05-py3
+        args:
+        - "tritonserver"
+        - "--model-store=s3://my-model-store/models"
+        - "--model-control-mode=poll"        # auto-load new model versions
+        - "--repository-poll-secs=30"
+        - "--metrics-port=8002"
+        ports:
+        - containerPort: 8000   # HTTP
+        - containerPort: 8001   # gRPC
+        - containerPort: 8002   # Metrics
+        resources:
+          limits:
+            nvidia.com/gpu: "1"
+        readinessProbe:
+          httpGet:
+            path: /v2/health/ready
+            port: 8000
+          initialDelaySeconds: 30
+```
+
+## Triton Model Repository Structure
+
+```
+s3://my-model-store/models/
+├── text-classifier/
+│   ├── config.pbtxt
+│   ├── 1/
+│   │   └── model.onnx
+│   └── 2/
+│       └── model.onnx          # new version; auto-loaded
+├── embedding-model/
+│   ├── config.pbtxt
+│   └── 1/
+│       └── model.onnx
+```
+
+```protobuf
+# config.pbtxt for ONNX model
+name: "text-classifier"
+backend: "onnxruntime"
+max_batch_size: 64
+dynamic_batching {
+  preferred_batch_size: [16, 32]
+  max_queue_delay_microseconds: 1000
+}
+input [
+  { name: "input_ids" data_type: TYPE_INT64 dims: [-1] }
+  { name: "attention_mask" data_type: TYPE_INT64 dims: [-1] }
+]
+output [
+  { name: "logits" data_type: TYPE_FP32 dims: [-1] }
+]
+instance_group [
+  { kind: KIND_GPU count: 2 }   # 2 model instances on GPU
+]
+```
+
+## Model Management Commands
+
+```bash
+# List loaded models (Triton)
+curl http://triton:8000/v2/models
+
+# Load a new model version
+curl -X POST http://triton:8000/v2/repository/models/text-classifier/load
+
+# Unload a model
+curl -X POST http://triton:8000/v2/repository/models/text-classifier/unload
+
+# KServe — watch rollout status
+kubectl rollout status deployment/llama-3-8b-predictor -n models
+kubectl get inferenceservice llama-3-8b -n models -w
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `InferenceService not ready` | Model loading or OOM | Check predictor pod logs; increase memory limits |
+| Canary stuck at 0% | KNative routing issue | Check `kubectl get ksvc -n models` |
+| Triton missing model | S3 permissions or path | Verify IAM role; check `--model-store` path |
+| Low GPU utilization | Dynamic batching off | Enable `dynamic_batching` in Triton config |
+| Autoscaler not triggering | Prometheus query wrong | Test query in Prometheus UI |
+
+## Best Practices
+
+- Use canary deployments for all model updates — roll back in seconds if metrics degrade.
+- Enable Triton dynamic batching — it can increase GPU throughput 5–10× for small models.
+- Store models in S3/GCS with versioned paths (`s3://bucket/model/v1/`, `v2/`).
+- Pin GPU node selectors to prevent model pods landing on CPU-only nodes.
+- Monitor p99 latency and error rates per model version during canary rollouts.
+
+## Related Skills
+
+- [vllm-server](../../infrastructure/local-ai/vllm-server/) - vLLM for LLM serving
+- [llm-inference-scaling](../../infrastructure/local-ai/llm-inference-scaling/) - KEDA autoscaling
+- [kubernetes-ops](./kubernetes-ops/) - Core Kubernetes operations
+- [gpu-server-management](../../infrastructure/servers/gpu-server-management/) - GPU nodes
@@ -0,0 +1,285 @@
+---
+name: vector-database-ops
+description: Deploy, manage, and optimize vector databases for AI applications. Covers Qdrant, Weaviate, pgvector, and Pinecone — collection management, indexing strategies, backup, and performance tuning for production RAG and semantic search workloads.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# Vector Database Operations
+
+Run production vector databases for AI-powered search, RAG, and recommendation systems.
+
+## When to Use This Skill
+
+Use this skill when:
+- Setting up a vector database for a RAG or semantic search application
+- Choosing between Qdrant, Weaviate, pgvector, or Pinecone
+- Managing collections, indexes, and data migrations
+- Optimizing query performance and indexing for production loads
+- Implementing multi-tenant vector search with namespace isolation
+
+## Vector Database Comparison
+
+| Database | Best For | Hosting | Filtering | Scale |
+|----------|----------|---------|-----------|-------|
+| **Qdrant** | High-performance, rich filtering, self-hosted | Self / Cloud | Excellent | Very High |
+| **Weaviate** | Schema-first, hybrid search, multi-modal | Self / Cloud | Good | High |
+| **pgvector** | Already on Postgres, simple use cases | Self | Good | Medium |
+| **Pinecone** | Zero-ops managed, serverless | Managed only | Good | Very High |
+| **Chroma** | Local dev, prototyping | Self only | Basic | Low-Medium |
+
+## Qdrant — Production Deployment
+
+```bash
+# Docker (single node)
+docker run -d \
+  --name qdrant \
+  -p 6333:6333 \
+  -p 6334:6334 \
+  -v $(pwd)/qdrant-data:/qdrant/storage \
+  qdrant/qdrant:latest
+
+# With custom config
+docker run -d \
+  --name qdrant \
+  -p 6333:6333 \
+  -v $(pwd)/qdrant-data:/qdrant/storage \
+  -v $(pwd)/qdrant-config.yaml:/qdrant/config/production.yaml \
+  qdrant/qdrant:latest
+```
+
+```yaml
+# qdrant-config.yaml
+storage:
+  storage_path: /qdrant/storage
+  on_disk_payload: true          # store payload on disk (saves RAM)
+
+service:
+  max_request_size_mb: 32
+
+hnsw_index:
+  m: 16                          # graph connections per node
+  ef_construct: 100              # accuracy vs build time trade-off
+  full_scan_threshold: 10000     # switch to brute force below this
+
+quantization:
+  scalar:
+    type: int8
+    quantile: 0.99
+    always_ram: true             # keep quantized index in RAM
+
+telemetry_disabled: true
+```
+
+## Qdrant Collection Management
+
+```python
+from qdrant_client import QdrantClient
+from qdrant_client.models import (
+    Distance, VectorParams, HnswConfigDiff,
+    ScalarQuantizationConfig, ScalarType, QuantizationConfig
+)
+
+client = QdrantClient("http://localhost:6333")
+
+# Create optimized collection
+client.create_collection(
+    collection_name="documents",
+    vectors_config=VectorParams(
+        size=1536,                         # OpenAI ada-002 / text-embedding-3-small
+        distance=Distance.COSINE,
+        on_disk=True,                      # save RAM — vectors stored on disk
+    ),
+    hnsw_config=HnswConfigDiff(
+        m=32,                              # higher = better recall, more RAM
+        ef_construct=200,
+        on_disk=False,                     # keep HNSW graph in RAM for speed
+    ),
+    quantization_config=QuantizationConfig(
+        scalar=ScalarQuantizationConfig(
+            type=ScalarType.INT8,
+            quantile=0.99,
+            always_ram=True,
+        )
+    ),
+)
+
+# Create payload index for fast filtering
+client.create_payload_index(
+    collection_name="documents",
+    field_name="tenant_id",
+    field_schema="keyword",
+)
+client.create_payload_index(
+    collection_name="documents",
+    field_name="created_at",
+    field_schema="datetime",
+)
+
+# Collection info
+info = client.get_collection("documents")
+print(f"Vectors: {info.vectors_count}, Status: {info.status}")
+```
+
+## Qdrant Filtered Search
+
+```python
+from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
+
+# Tenant-isolated search (multi-tenant RAG)
+results = client.query_points(
+    collection_name="documents",
+    query=query_embedding,
+    query_filter=Filter(
+        must=[
+            FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")),
+            FieldCondition(key="doc_type", match=MatchValue(value="contract")),
+        ],
+        should=[
+            FieldCondition(key="created_at", range=Range(gte="2024-01-01")),
+        ],
+    ),
+    limit=10,
+    with_payload=True,
+)
+```
+
+## pgvector — PostgreSQL Extension
+
+```sql
+-- Enable extension
+CREATE EXTENSION IF NOT EXISTS vector;
+
+-- Create table with vector column
+CREATE TABLE documents (
+    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    content     TEXT NOT NULL,
+    embedding   VECTOR(1536),
+    metadata    JSONB DEFAULT '{}',
+    tenant_id   TEXT NOT NULL,
+    created_at  TIMESTAMPTZ DEFAULT NOW()
+);
+
+-- Create HNSW index (faster queries, more memory)
+CREATE INDEX ON documents
+USING hnsw (embedding vector_cosine_ops)
+WITH (m = 16, ef_construction = 64);
+
+-- Create IVFFlat index (less memory, slower build)
+-- CREATE INDEX ON documents
+-- USING ivfflat (embedding vector_cosine_ops)
+-- WITH (lists = 100);
+
+-- Semantic search with metadata filtering
+SELECT id, content, metadata,
+       1 - (embedding <=> $1::vector) AS similarity
+FROM documents
+WHERE tenant_id = 'acme-corp'
+  AND metadata->>'doc_type' = 'contract'
+ORDER BY embedding <=> $1::vector
+LIMIT 10;
+```
+
+```bash
+# Deploy pgvector via Docker
+docker run -d \
+  --name pgvector \
+  -e POSTGRES_PASSWORD=secret \
+  -e POSTGRES_DB=vectordb \
+  -p 5432:5432 \
+  -v pgvector-data:/var/lib/postgresql/data \
+  pgvector/pgvector:pg16
+```
+
+## Weaviate Deployment
+
+```yaml
+# docker-compose for Weaviate
+services:
+  weaviate:
+    image: semitechnologies/weaviate:latest
+    ports:
+      - "8080:8080"
+      - "50051:50051"
+    environment:
+      QUERY_DEFAULTS_LIMIT: 25
+      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "false"
+      AUTHENTICATION_APIKEY_ENABLED: "true"
+      AUTHENTICATION_APIKEY_ALLOWED_KEYS: "${WEAVIATE_API_KEY}"
+      AUTHENTICATION_APIKEY_USERS: "admin"
+      PERSISTENCE_DATA_PATH: /var/lib/weaviate
+      ENABLE_MODULES: text2vec-openai,generative-openai
+      OPENAI_APIKEY: "${OPENAI_API_KEY}"
+      CLUSTER_HOSTNAME: node1
+    volumes:
+      - weaviate-data:/var/lib/weaviate
+    restart: unless-stopped
+
+volumes:
+  weaviate-data:
+```
+
+## Backup and Restore
+
+```bash
+# Qdrant — snapshot backup
+curl -X POST "http://localhost:6333/collections/documents/snapshots"
+# Download snapshot
+curl -O "http://localhost:6333/collections/documents/snapshots/documents-snapshot.snapshot"
+# Restore
+curl -X POST "http://localhost:6333/collections/documents/snapshots/recover" \
+  -H "Content-Type: application/json" \
+  -d '{"location": "/qdrant/snapshots/documents-snapshot.snapshot"}'
+
+# pgvector — standard pg_dump
+pg_dump -h localhost -U postgres -d vectordb \
+  --table=documents --format=custom > documents-backup.dump
+
+# Restore
+pg_restore -h localhost -U postgres -d vectordb documents-backup.dump
+```
+
+## Performance Tuning
+
+```python
+# Qdrant — optimize collection after bulk load
+client.update_collection(
+    collection_name="documents",
+    optimizer_config={"indexing_threshold": 0},  # force indexing now
+)
+
+# Wait for optimization to complete
+import time
+while True:
+    info = client.get_collection("documents")
+    if info.status.value == "green":
+        break
+    time.sleep(5)
+    print(f"Optimizing... segments: {info.segments_count}")
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| Slow queries | No HNSW index built yet | Wait for indexing; check `status == green` |
+| High RAM usage | Vectors in memory | Enable `on_disk=True` for vectors |
+| Poor recall | Low `ef` search param | Increase `ef` in search request (at query time) |
+| pgvector slow | Using IVFFlat without vacuum | Run `VACUUM ANALYZE documents` |
+| Weaviate OOM | Too many objects | Enable async indexing; increase heap |
+
+## Best Practices
+
+- Use cosine distance for normalized embeddings; dot product for unnormalized.
+- Always create payload indexes on filter fields (`tenant_id`, `doc_type`).
+- For datasets >10M vectors, use `on_disk` vectors + `always_ram` quantization.
+- Benchmark with your actual query patterns before choosing IVFFlat vs HNSW.
+- Snapshot before any bulk delete or migration operation.
+
+## Related Skills
+
+- [rag-infrastructure](../../local-ai/rag-infrastructure/) - Full RAG pipeline
+- [databases](../databases/) - General database management
+- [postgresql](../postgresql/) - pgvector host database ops
@@ -0,0 +1,312 @@
+---
+name: llm-fine-tuning
+description: Set up infrastructure for fine-tuning LLMs with QLoRA, LoRA, and full fine-tuning using Hugging Face TRL, Axolotl, and distributed training with DeepSpeed or FSDP. Covers dataset prep, training runs, and model export.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# LLM Fine-Tuning Infrastructure
+
+Train and fine-tune open-source LLMs efficiently — from LoRA on a single GPU to distributed full fine-tuning across multi-node clusters.
+
+## When to Use This Skill
+
+Use this skill when:
+- Fine-tuning an LLM on domain-specific data (legal, medical, code, support)
+- Running QLoRA to fine-tune 70B models on consumer GPUs
+- Setting up distributed training with DeepSpeed or FSDP
+- Exporting fine-tuned adapters for production serving
+- Implementing RLHF, DPO, or instruction tuning pipelines
+
+## Prerequisites
+
+- NVIDIA GPU(s) with 24GB+ VRAM (RTX 4090 / A100 / H100)
+- CUDA 12.1+ and `nvidia-smi` working
+- Python 3.10+ with `pip`
+- Hugging Face account and `HF_TOKEN` for gated models
+- 500GB+ disk for model weights and training data
+
+## Quick Start: QLoRA Fine-Tuning
+
+```bash
+pip install transformers datasets trl peft bitsandbytes accelerate
+
+python - <<'EOF'
+from datasets import load_dataset
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+from peft import LoraConfig, get_peft_model
+from trl import SFTTrainer, SFTConfig
+import torch
+
+model_id = "meta-llama/Llama-3.1-8B-Instruct"
+
+# 4-bit quantization (QLoRA)
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16,
+    bnb_4bit_use_double_quant=True,
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, quantization_config=bnb_config, device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+# LoRA configuration
+peft_config = LoraConfig(
+    r=16,                    # rank
+    lora_alpha=32,
+    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
+                    "gate_proj", "up_proj", "down_proj"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+
+dataset = load_dataset("your-org/your-dataset", split="train")
+
+trainer = SFTTrainer(
+    model=model,
+    args=SFTConfig(
+        output_dir="./output",
+        num_train_epochs=3,
+        per_device_train_batch_size=2,
+        gradient_accumulation_steps=8,
+        learning_rate=2e-4,
+        bf16=True,
+        logging_steps=10,
+        save_strategy="epoch",
+        report_to="wandb",
+    ),
+    train_dataset=dataset,
+    peft_config=peft_config,
+    processing_class=tokenizer,
+)
+trainer.train()
+trainer.save_model("./fine-tuned-model")
+EOF
+```
+
+## Axolotl (Production Fine-Tuning Framework)
+
+```yaml
+# config.yaml — Axolotl QLoRA config for Llama 3.1
+base_model: meta-llama/Llama-3.1-8B-Instruct
+model_type: LlamaForCausalLM
+tokenizer_type: PreTrainedTokenizerFast
+
+load_in_4bit: true
+adapter: qlora
+lora_r: 32
+lora_alpha: 64
+lora_dropout: 0.05
+lora_target_modules:
+  - q_proj
+  - k_proj
+  - v_proj
+  - o_proj
+  - gate_proj
+  - up_proj
+  - down_proj
+
+datasets:
+  - path: your-org/your-dataset
+    type: alpaca              # or sharegpt, chat_template, etc.
+
+dataset_prepared_path: ./prepared-data
+val_set_size: 0.05
+output_dir: ./output
+
+sequence_len: 4096
+sample_packing: true         # pack multiple short samples for efficiency
+
+micro_batch_size: 2
+gradient_accumulation_steps: 8
+num_epochs: 3
+learning_rate: 2e-4
+optimizer: adamw_bnb_8bit
+lr_scheduler: cosine
+warmup_ratio: 0.05
+
+bf16: true
+flash_attention: true
+
+logging_steps: 10
+eval_steps: 100
+save_steps: 200
+wandb_project: my-fine-tune
+```
+
+```bash
+# Run with Axolotl
+pip install axolotl[flash-attn,deepspeed]
+accelerate launch -m axolotl.cli.train config.yaml
+```
+
+## Distributed Training with DeepSpeed
+
+```json
+// deepspeed_zero3.json — ZeRO Stage 3 (split optimizer + gradients + params)
+{
+  "zero_optimization": {
+    "stage": 3,
+    "offload_optimizer": {"device": "cpu", "pin_memory": true},
+    "offload_param": {"device": "cpu", "pin_memory": true},
+    "overlap_comm": true,
+    "contiguous_gradients": true,
+    "sub_group_size": 1e9,
+    "reduce_bucket_size": "auto",
+    "stage3_prefetch_bucket_size": "auto",
+    "stage3_param_persistence_threshold": "auto",
+    "stage3_max_live_parameters": 1e9,
+    "stage3_max_reuse_distance": 1e9,
+    "gather_16bit_weights_on_model_save": true
+  },
+  "bf16": {"enabled": true},
+  "gradient_clipping": 1.0,
+  "train_batch_size": "auto",
+  "train_micro_batch_size_per_gpu": "auto"
+}
+```
+
+```bash
+# Launch 4-GPU DeepSpeed training
+deepspeed --num_gpus=4 train.py \
+  --deepspeed deepspeed_zero3.json \
+  --model_name meta-llama/Llama-3.1-70B-Instruct \
+  --output_dir ./output
+```
+
+## DPO / RLHF Alignment
+
+```python
+from trl import DPOTrainer, DPOConfig
+from datasets import load_dataset
+
+# Dataset format: {"prompt": ..., "chosen": ..., "rejected": ...}
+dataset = load_dataset("your-org/preference-data")
+
+trainer = DPOTrainer(
+    model=model,
+    ref_model=None,           # None = implicit reference with peft
+    args=DPOConfig(
+        output_dir="./dpo-output",
+        beta=0.1,             # KL divergence weight
+        num_train_epochs=1,
+        per_device_train_batch_size=1,
+        gradient_accumulation_steps=16,
+        learning_rate=5e-7,
+        bf16=True,
+    ),
+    train_dataset=dataset["train"],
+    peft_config=peft_config,
+    processing_class=tokenizer,
+)
+trainer.train()
+```
+
+## Merging LoRA Adapters for Deployment
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM
+
+# Load base model in full precision
+base_model = AutoModelForCausalLM.from_pretrained(
+    "meta-llama/Llama-3.1-8B-Instruct",
+    torch_dtype=torch.bfloat16,
+    device_map="cpu",
+)
+
+# Load and merge LoRA adapter
+model = PeftModel.from_pretrained(base_model, "./fine-tuned-model")
+merged_model = model.merge_and_unload()
+
+# Save merged model (ready for vLLM serving)
+merged_model.save_pretrained("./merged-model", safe_serialization=True)
+tokenizer.save_pretrained("./merged-model")
+
+# Push to Hugging Face Hub
+merged_model.push_to_hub("your-org/your-fine-tuned-model")
+```
+
+## Kubernetes Training Job
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: llm-fine-tune
+spec:
+  template:
+    spec:
+      restartPolicy: OnFailure
+      nodeSelector:
+        nvidia.com/gpu.product: A100-SXM4-80GB
+      containers:
+      - name: trainer
+        image: nvcr.io/nvidia/pytorch:24.05-py3
+        command: ["accelerate", "launch", "-m", "axolotl.cli.train", "/config/config.yaml"]
+        resources:
+          limits:
+            nvidia.com/gpu: "4"
+            memory: "320Gi"
+          requests:
+            nvidia.com/gpu: "4"
+        volumeMounts:
+        - name: config
+          mountPath: /config
+        - name: model-cache
+          mountPath: /root/.cache/huggingface
+        - name: output
+          mountPath: /output
+        env:
+        - name: HUGGING_FACE_HUB_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: hf-token
+              key: token
+        - name: WANDB_API_KEY
+          valueFrom:
+            secretKeyRef:
+              name: wandb-token
+              key: key
+      volumes:
+      - name: config
+        configMap:
+          name: axolotl-config
+      - name: model-cache
+        persistentVolumeClaim:
+          claimName: model-cache-pvc
+      - name: output
+        persistentVolumeClaim:
+          claimName: training-output-pvc
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `CUDA out of memory` | Batch too large | Reduce `micro_batch_size`; increase `gradient_accumulation_steps` |
+| Training loss NaN | Learning rate too high | Lower LR to `1e-4` or `5e-5`; add warmup |
+| Slow training | No Flash Attention | Install `flash-attn`; enable `flash_attention: true` |
+| Poor fine-tune quality | Bad data formatting | Validate dataset format; check `sample_packing` compatibility |
+| Adapter merge errors | Mixed quantization | Merge in bf16 on CPU, not in 4-bit |
+
+## Best Practices
+
+- Use Flash Attention 2 — it's 2–4× faster and uses less memory.
+- Monitor training loss/eval loss via W&B or MLflow; overfit = more dropout or less data.
+- Validate with a held-out eval set (5–10%); MMLU or custom evals for quality gates.
+- Start with LoRA r=16 before increasing — higher rank = more parameters, diminishing returns.
+- Use `sample_packing` in Axolotl to maximize GPU utilization on short sequences.
+
+## Related Skills
+
+- [vllm-server](../vllm-server/) - Serve fine-tuned models
+- [gpu-server-management](../../servers/gpu-server-management/) - GPU setup
+- [llm-inference-scaling](../llm-inference-scaling/) - Deploy at scale
+- [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Training pipelines
@@ -0,0 +1,270 @@
+---
+name: llm-inference-scaling
+description: Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# LLM Inference Scaling
+
+Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.
+
+## When to Use This Skill
+
+Use this skill when:
+- LLM API traffic is unpredictable and you need to scale up/down automatically
+- Managing a fleet of vLLM or TGI inference pods on Kubernetes
+- Reducing inference costs with spot/preemptible GPU instances
+- Implementing queue-based autoscaling for batch inference jobs
+- Building a multi-model serving platform that shares GPU resources
+
+## Prerequisites
+
+- Kubernetes cluster with GPU nodes (NVIDIA operator installed)
+- KEDA (Kubernetes Event-Driven Autoscaler) installed
+- Prometheus with GPU metrics (`dcgm-exporter` or `gpu-operator`)
+- Helm 3+ for chart deployments
+
+## GPU Node Setup
+
+```bash
+# Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)
+helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
+helm repo update
+
+helm install gpu-operator nvidia/gpu-operator \
+  --namespace gpu-operator \
+  --create-namespace \
+  --set driver.enabled=true \
+  --set dcgm.enabled=true \
+  --set devicePlugin.enabled=true
+
+# Verify GPU nodes are recognized
+kubectl get nodes -l nvidia.com/gpu.present=true
+kubectl describe node <gpu-node> | grep nvidia
+```
+
+## vLLM Deployment with GPU Resources
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: vllm-llama-8b
+  labels:
+    app: vllm
+    model: llama-3.1-8b
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: vllm
+      model: llama-3.1-8b
+  template:
+    metadata:
+      labels:
+        app: vllm
+        model: llama-3.1-8b
+    spec:
+      nodeSelector:
+        nvidia.com/gpu.present: "true"
+      tolerations:
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
+      containers:
+      - name: vllm
+        image: vllm/vllm-openai:latest
+        args:
+        - "--model"
+        - "meta-llama/Llama-3.1-8B-Instruct"
+        - "--tensor-parallel-size"
+        - "1"
+        - "--gpu-memory-utilization"
+        - "0.90"
+        - "--max-num-seqs"
+        - "128"
+        resources:
+          requests:
+            nvidia.com/gpu: "1"
+            memory: "20Gi"
+            cpu: "4"
+          limits:
+            nvidia.com/gpu: "1"
+            memory: "24Gi"
+            cpu: "8"
+        ports:
+        - containerPort: 8000
+        readinessProbe:
+          httpGet:
+            path: /health
+            port: 8000
+          initialDelaySeconds: 60
+          periodSeconds: 10
+        env:
+        - name: HUGGING_FACE_HUB_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: hf-token
+              key: token
+```
+
+## KEDA Autoscaling on Prometheus Metrics
+
+```yaml
+apiVersion: keda.sh/v1alpha1
+kind: ScaledObject
+metadata:
+  name: vllm-scaledobject
+spec:
+  scaleTargetRef:
+    name: vllm-llama-8b
+  minReplicaCount: 1
+  maxReplicaCount: 8
+  cooldownPeriod: 300          # 5 min before scale-down
+  pollingInterval: 15
+  triggers:
+  - type: prometheus
+    metadata:
+      serverAddress: http://prometheus-server.monitoring:9090
+      metricName: vllm_num_requests_waiting
+      threshold: "10"           # scale up if >10 requests waiting
+      query: |
+        sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
+  - type: prometheus
+    metadata:
+      serverAddress: http://prometheus-server.monitoring:9090
+      metricName: vllm_gpu_cache_usage
+      threshold: "0.8"          # scale up if KV cache >80% full
+      query: |
+        avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})
+```
+
+## Queue-Based Scaling (Redis + KEDA)
+
+```yaml
+# ScaledJob for async batch inference
+apiVersion: keda.sh/v1alpha1
+kind: ScaledJob
+metadata:
+  name: llm-batch-inference
+spec:
+  jobTargetRef:
+    template:
+      spec:
+        containers:
+        - name: inference-worker
+          image: myapp/inference-worker:latest
+          env:
+          - name: REDIS_URL
+            value: redis://redis:6379
+          - name: QUEUE_NAME
+            value: inference-jobs
+        restartPolicy: OnFailure
+  minReplicaCount: 0
+  maxReplicaCount: 20
+  pollingInterval: 5
+  successfulJobsHistoryLimit: 3
+  triggers:
+  - type: redis
+    metadata:
+      address: redis:6379
+      listName: inference-jobs
+      listLength: "5"           # 1 worker per 5 queued jobs
+```
+
+## Spot Instance Strategy
+
+```yaml
+# Mixed node pool: on-demand + spot GPUs
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cluster-autoscaler-priority-config
+data:
+  priorities: |
+    10:  # low priority = prefer
+    - .*spot.*
+    50:
+    - .*on-demand.*
+---
+# Node affinity for spot with on-demand fallback
+spec:
+  affinity:
+    nodeAffinity:
+      preferredDuringSchedulingIgnoredDuringExecution:
+      - weight: 80
+        preference:
+          matchExpressions:
+          - key: node.kubernetes.io/lifecycle
+            operator: In
+            values: [spot]
+      - weight: 20
+        preference:
+          matchExpressions:
+          - key: node.kubernetes.io/lifecycle
+            operator: In
+            values: [on-demand]
+```
+
+## Cluster Autoscaler for GPU Nodes
+
+```bash
+# AWS EKS — enable cluster autoscaler for GPU node group
+helm install cluster-autoscaler autoscaler/cluster-autoscaler \
+  --namespace kube-system \
+  --set autoDiscovery.clusterName=my-cluster \
+  --set awsRegion=us-east-1 \
+  --set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole \
+  --set extraArgs.skip-nodes-with-local-storage=false \
+  --set extraArgs.expander=least-waste
+
+# Annotate GPU node group for autoscaler
+kubectl annotate node <node> \
+  cluster-autoscaler.kubernetes.io/safe-to-evict="false"
+```
+
+## Scaling Metrics to Monitor
+
+```bash
+# Prometheus queries for scaling decisions
+# Requests waiting in vLLM queue
+sum(vllm:num_requests_waiting) by (model)
+
+# GPU KV cache utilization (>80% = bottleneck)
+avg(vllm:gpu_cache_usage_perc) by (pod)
+
+# Tokens per second throughput
+sum(rate(vllm:generation_tokens_total[5m])) by (model)
+
+# P99 time-to-first-token
+histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| Pods stuck in `Pending` | No GPU nodes available | Check cluster autoscaler logs; verify node group limits |
+| Scale-up too slow | Cluster autoscaler delay + model load time | Pre-warm replicas; increase `minReplicaCount` |
+| GPU fragmentation | Multiple small models on large GPUs | Use MIG partitioning or consolidate model sizes |
+| Spot eviction causes errors | Spot instance reclamation | Add `PodDisruptionBudget`; use graceful shutdown |
+| KEDA not scaling | Prometheus query returns no data | Test query in Prometheus UI first |
+
+## Best Practices
+
+- Set `minReplicaCount: 1` to avoid cold starts; scale to 0 only for batch jobs.
+- Use `PodDisruptionBudget` with `minAvailable: 1` to survive spot evictions.
+- Pre-pull model weights into a shared PVC to speed up pod startup by 5–10×.
+- Separate model families across node pools (A10G for 7B, A100 for 70B).
+- Use Kubernetes VPA for CPU/memory right-sizing alongside KEDA for replica count.
+
+## Related Skills
+
+- [vllm-server](../vllm-server/) - vLLM configuration and tuning
+- [gpu-server-management](../../servers/gpu-server-management/) - GPU node setup
+- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - KServe
+- [kubernetes-ops](../../../devops/orchestration/kubernetes-ops/) - Core Kubernetes
+- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost strategies
@@ -0,0 +1,253 @@
+---
+name: rag-infrastructure
+description: Build and operate Retrieval-Augmented Generation (RAG) infrastructure with vector stores, embedding pipelines, and hybrid search. Covers ingestion, chunking strategies, reranking, and production deployment patterns.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# RAG Infrastructure
+
+Production infrastructure for Retrieval-Augmented Generation: ingest documents, generate embeddings, store in vector databases, and serve grounded LLM responses.
+
+## When to Use This Skill
+
+Use this skill when:
+- Building a knowledge base Q&A system over internal documents
+- Implementing semantic search over large document collections
+- Reducing LLM hallucinations with retrieved context
+- Setting up embedding pipelines and vector store infrastructure
+- Deploying hybrid search (dense + sparse/BM25)
+
+## Prerequisites
+
+- Python 3.10+ with `pip`
+- A vector database (Qdrant, Weaviate, Pinecone, or pgvector)
+- An embedding model (OpenAI, Cohere, or local via `sentence-transformers`)
+- An LLM endpoint (OpenAI API or self-hosted vLLM)
+- Docker for local vector DB deployment
+
+## Architecture Overview
+
+```
+Documents → Chunker → Embedder → Vector Store
+                                      ↓
+User Query → Embedder → Vector Store (search) → Reranker → LLM → Answer
+```
+
+## Embedding Pipeline
+
+```python
+from sentence_transformers import SentenceTransformer
+from qdrant_client import QdrantClient
+from qdrant_client.models import Distance, VectorParams, PointStruct
+import uuid
+
+# Local embedding model (no API cost)
+model = SentenceTransformer("BAAI/bge-large-en-v1.5")
+
+# Connect to Qdrant
+client = QdrantClient("http://localhost:6333")
+
+# Create collection
+client.create_collection(
+    collection_name="knowledge-base",
+    vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
+)
+
+def ingest_documents(docs: list[dict]):
+    """Chunk, embed, and upsert documents."""
+    points = []
+    for doc in docs:
+        chunks = chunk_text(doc["text"], chunk_size=512, overlap=50)
+        embeddings = model.encode(chunks, batch_size=32, show_progress_bar=True)
+        for chunk, embedding in zip(chunks, embeddings):
+            points.append(PointStruct(
+                id=str(uuid.uuid4()),
+                vector=embedding.tolist(),
+                payload={"text": chunk, "source": doc["source"], "title": doc["title"]},
+            ))
+    client.upsert(collection_name="knowledge-base", points=points)
+    print(f"Ingested {len(points)} chunks")
+```
+
+## Chunking Strategies
+
+```python
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+def chunk_text(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]:
+    """Recursive character splitter — best general-purpose strategy."""
+    splitter = RecursiveCharacterTextSplitter(
+        chunk_size=chunk_size,
+        chunk_overlap=overlap,
+        separators=["\n\n", "\n", ". ", " ", ""],
+    )
+    return splitter.split_text(text)
+
+# For code/markdown — use language-aware splitter
+from langchain.text_splitter import MarkdownHeaderTextSplitter
+
+headers = [("#", "H1"), ("##", "H2"), ("###", "H3")]
+md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
+```
+
+## Hybrid Search (Dense + Sparse)
+
+```python
+from qdrant_client.models import SparseVector, SparseVectorParams, NamedSparseVector
+from fastembed import SparseTextEmbedding
+
+# Qdrant hybrid collection (dense + BM25 sparse)
+client.create_collection(
+    collection_name="hybrid-kb",
+    vectors_config={"dense": VectorParams(size=1024, distance=Distance.COSINE)},
+    sparse_vectors_config={"sparse": SparseVectorParams()},
+)
+
+sparse_model = SparseTextEmbedding("prithivida/Splade_PP_en_v1")
+
+def hybrid_search(query: str, top_k: int = 10) -> list[dict]:
+    dense_vec = model.encode(query).tolist()
+    sparse_vec = list(sparse_model.embed(query))[0]
+
+    results = client.query_points(
+        collection_name="hybrid-kb",
+        prefetch=[
+            {"query": dense_vec, "using": "dense", "limit": 20},
+            {"query": SparseVector(indices=sparse_vec.indices.tolist(),
+                                   values=sparse_vec.values.tolist()),
+             "using": "sparse", "limit": 20},
+        ],
+        query={"fusion": "rrf"},   # Reciprocal Rank Fusion
+        limit=top_k,
+    )
+    return [{"text": p.payload["text"], "score": p.score} for p in results.points]
+```
+
+## Reranking
+
+```python
+import cohere
+
+co = cohere.Client("your-api-key")
+
+def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
+    """Rerank retrieved chunks for relevance (improves RAG quality ~20-30%)."""
+    response = co.rerank(
+        model="rerank-english-v3.0",
+        query=query,
+        documents=candidates,
+        top_n=top_n,
+    )
+    return [candidates[r.index] for r in response.results]
+
+# Alternative: local reranker (no API cost)
+from sentence_transformers import CrossEncoder
+reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
+
+def local_rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
+    pairs = [[query, c] for c in candidates]
+    scores = reranker.predict(pairs)
+    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
+    return [text for text, _ in ranked[:top_n]]
+```
+
+## RAG Query Pipeline
+
+```python
+from openai import OpenAI
+
+llm = OpenAI(base_url="http://localhost:8000/v1", api_key="your-key")
+
+def rag_query(user_question: str) -> str:
+    # 1. Retrieve
+    candidates = hybrid_search(user_question, top_k=20)
+    texts = [c["text"] for c in candidates]
+
+    # 2. Rerank
+    top_chunks = local_rerank(user_question, texts, top_n=5)
+
+    # 3. Generate
+    context = "\n\n---\n\n".join(top_chunks)
+    response = llm.chat.completions.create(
+        model="meta-llama/Llama-3.1-8B-Instruct",
+        messages=[
+            {"role": "system", "content": (
+                "Answer the question using only the provided context. "
+                "If the answer isn't in the context, say so.\n\nContext:\n" + context
+            )},
+            {"role": "user", "content": user_question},
+        ],
+        temperature=0.1,
+        max_tokens=1024,
+    )
+    return response.choices[0].message.content
+```
+
+## Docker Compose: Full RAG Stack
+
+```yaml
+services:
+  qdrant:
+    image: qdrant/qdrant:latest
+    volumes:
+      - qdrant-data:/qdrant/storage
+    ports:
+      - "6333:6333"
+    restart: unless-stopped
+
+  redis:
+    image: redis:7-alpine
+    volumes:
+      - redis-data:/data
+    restart: unless-stopped
+
+  ingestion-worker:
+    build: ./ingestion
+    environment:
+      - QDRANT_URL=http://qdrant:6333
+      - REDIS_URL=redis://redis:6379
+    depends_on: [qdrant, redis]
+    restart: unless-stopped
+
+  rag-api:
+    build: ./api
+    ports:
+      - "8080:8080"
+    environment:
+      - QDRANT_URL=http://qdrant:6333
+      - LLM_BASE_URL=http://vllm:8000/v1
+    depends_on: [qdrant]
+    restart: unless-stopped
+
+volumes:
+  qdrant-data:
+  redis-data:
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| Poor retrieval quality | Chunk size too large | Try 256–512 tokens; overlap 10–15% |
+| LLM ignores retrieved context | Context too long | Rerank and keep top 3–5 chunks |
+| Slow ingestion | Sequential embedding | Use `batch_size=64` and async upserts |
+| Stale documents | No re-ingestion pipeline | Track `doc_hash`; re-embed on change |
+| High embedding costs | All chunks re-embedded | Cache embeddings with hash-based dedup |
+
+## Best Practices
+
+- Use `BAAI/bge-large-en-v1.5` or `nomic-embed-text` for strong free embeddings.
+- Always rerank before passing to LLM — 5 precise chunks beat 20 noisy ones.
+- Store source metadata (URL, page, section) in vector payloads for citations.
+- Use namespace/tenant isolation in the vector store for multi-tenant RAG.
+- Evaluate with RAGAS metrics: faithfulness, answer relevancy, context precision.
+
+## Related Skills
+
+- [vector-database-ops](../../databases/vector-database-ops/) - Qdrant/Weaviate management
+- [vllm-server](../vllm-server/) - Self-hosted LLM endpoint
+- [ollama-stack](../ollama-stack/) - Local LLM for development
+- [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Ingestion pipelines
@@ -0,0 +1,219 @@
+---
+name: vllm-server
+description: Deploy and manage vLLM for high-throughput LLM inference. Configure continuous batching, tensor parallelism, quantization, and OpenAI-compatible API endpoints for production LLM serving.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# vLLM Server Management
+
+Deploy production-grade LLM inference servers with vLLM — the fastest open-source LLM serving engine with PagedAttention and continuous batching.
+
+## When to Use This Skill
+
+Use this skill when:
+- Serving open-source LLMs (Llama, Mistral, Qwen, Gemma) at scale
+- Building an OpenAI-compatible API endpoint for self-hosted models
+- Optimizing LLM throughput and latency for production traffic
+- Running multi-GPU inference with tensor or pipeline parallelism
+- Deploying quantized models to reduce GPU memory requirements
+
+## Prerequisites
+
+- NVIDIA GPU(s) with CUDA 12.1+ (A100/H100 recommended for production)
+- Docker or Python 3.9+ with pip
+- 40GB+ VRAM for 70B models; 8GB+ for 7B models
+- `nvidia-container-toolkit` for Docker GPU passthrough
+
+## Quick Start
+
+```bash
+# Install vLLM
+pip install vllm
+
+# Serve a model (OpenAI-compatible API)
+vllm serve meta-llama/Llama-3.1-8B-Instruct \
+  --host 0.0.0.0 \
+  --port 8000 \
+  --api-key your-secret-key
+
+# Test the endpoint
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer your-secret-key" \
+  -d '{
+    "model": "meta-llama/Llama-3.1-8B-Instruct",
+    "messages": [{"role": "user", "content": "Hello!"}]
+  }'
+```
+
+## Docker Deployment
+
+```bash
+docker run --runtime nvidia --gpus all \
+  -v ~/.cache/huggingface:/root/.cache/huggingface \
+  -p 8000:8000 \
+  --ipc=host \
+  vllm/vllm-openai:latest \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --api-key your-secret-key
+```
+
+## Docker Compose (Production)
+
+```yaml
+services:
+  vllm:
+    image: vllm/vllm-openai:latest
+    runtime: nvidia
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=all
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
+    volumes:
+      - model-cache:/root/.cache/huggingface
+    ports:
+      - "8000:8000"
+    ipc: host
+    command: >
+      --model meta-llama/Llama-3.1-70B-Instruct
+      --tensor-parallel-size 2
+      --max-model-len 32768
+      --gpu-memory-utilization 0.90
+      --api-key ${VLLM_API_KEY}
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+
+volumes:
+  model-cache:
+```
+
+## Key Configuration Options
+
+### Multi-GPU Tensor Parallelism
+
+```bash
+# Split one model across 4 GPUs
+vllm serve meta-llama/Llama-3.1-70B-Instruct \
+  --tensor-parallel-size 4 \
+  --gpu-memory-utilization 0.90
+```
+
+### Quantization (Lower VRAM)
+
+```bash
+# AWQ quantization (70B on 2x A100 40GB)
+vllm serve casperhansen/llama-3-70b-instruct-awq \
+  --quantization awq \
+  --tensor-parallel-size 2
+
+# GPTQ quantization
+vllm serve TheBloke/Llama-2-70B-Chat-GPTQ \
+  --quantization gptq
+
+# FP8 (H100 NVL native)
+vllm serve meta-llama/Llama-3.1-405B-Instruct \
+  --quantization fp8 \
+  --tensor-parallel-size 8
+```
+
+### Structured Output & Tools
+
+```bash
+vllm serve meta-llama/Llama-3.1-8B-Instruct \
+  --enable-auto-tool-choice \
+  --tool-call-parser llama3_json \
+  --guided-decoding-backend outlines
+```
+
+### LoRA Adapters
+
+```bash
+vllm serve meta-llama/Llama-3.1-8B-Instruct \
+  --enable-lora \
+  --lora-modules sql-lora=/path/to/sql-lora \
+                 code-lora=/path/to/code-lora \
+  --max-lora-rank 64
+```
+
+## Performance Tuning
+
+```bash
+# Maximize throughput for batch workloads
+vllm serve <model> \
+  --max-num-seqs 256 \          # max concurrent sequences
+  --max-num-batched-tokens 8192 \ # tokens per batch
+  --gpu-memory-utilization 0.95 \ # use 95% VRAM
+  --swap-space 4                  # CPU swap (GiB)
+
+# Minimize latency for interactive use
+vllm serve <model> \
+  --max-num-seqs 32 \
+  --enforce-eager              # disable CUDA graph capture
+```
+
+## Benchmarking
+
+```bash
+# Install benchmark tool
+pip install vllm
+
+# Run throughput benchmark
+python -m vllm.entrypoints.openai.run_batch \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --input-file prompts.jsonl \
+  --output-file results.jsonl
+
+# Benchmark with vllm bench
+vllm bench throughput \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --num-prompts 1000 \
+  --input-len 512 \
+  --output-len 128
+```
+
+## Monitoring
+
+```bash
+# Check running server stats
+curl http://localhost:8000/metrics  # Prometheus metrics
+
+# Key metrics to watch:
+# vllm:num_requests_running       - active requests
+# vllm:gpu_cache_usage_perc       - KV cache utilization
+# vllm:generation_tokens_per_s    - throughput
+# vllm:time_to_first_token_ms     - TTFT latency
+# vllm:e2e_request_latency_seconds - end-to-end latency
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `CUDA out of memory` | Model too large for VRAM | Add `--quantization awq` or reduce `--gpu-memory-utilization` |
+| Slow cold start | Model not cached | Pre-pull with `huggingface-cli download <model>` |
+| Low throughput | Too few concurrent requests | Increase `--max-num-seqs` |
+| KV cache full errors | Context length too long | Set `--max-model-len` lower |
+| `tokenizer error` | Tokenizer mismatch | Use `--tokenizer` to specify correct tokenizer |
+
+## Best Practices
+
+- Use `--gpu-memory-utilization 0.90` to leave headroom for CUDA kernels.
+- Pin model versions with `--revision` for reproducible deployments.
+- Set `HF_HUB_OFFLINE=1` in production to prevent unexpected downloads.
+- Use AWQ or GPTQ quantization before tensor parallelism — lower VRAM first.
+- Enable `--enable-chunked-prefill` for long-context workloads.
+- Monitor `gpu_cache_usage_perc` — above 95% causes queuing.
+
+## Related Skills
+
+- [llm-inference-scaling](../llm-inference-scaling/) - Auto-scaling vLLM deployments
+- [gpu-server-management](../../servers/gpu-server-management/) - GPU driver setup
+- [llm-gateway](../../networking/llm-gateway/) - Load balancing across vLLM instances
+- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost management
+- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - K8s deployment
@@ -0,0 +1,266 @@
+---
+name: llm-gateway
+description: Deploy an API gateway for LLM traffic with load balancing, rate limiting, key management, semantic caching, fallback routing, and cost tracking. Covers LiteLLM Proxy, OpenRouter-compatible setup, and custom Nginx/Traefik patterns.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# LLM Gateway
+
+A unified API gateway that routes LLM requests across providers and self-hosted models — with rate limiting, cost tracking, caching, and failover.
+
+## When to Use This Skill
+
+Use this skill when:
+- Running multiple LLM backends (OpenAI, Anthropic, vLLM, Ollama) behind a single endpoint
+- Enforcing per-team or per-user rate limits and spend budgets
+- Implementing automatic fallback when a provider is down
+- Adding semantic caching to reduce API costs by 20–50%
+- Centralizing API key management instead of distributing keys to every app
+
+## Prerequisites
+
+- Docker and Docker Compose
+- A PostgreSQL or SQLite database (for LiteLLM state)
+- LLM API keys (OpenAI, Anthropic, etc.) or self-hosted vLLM endpoints
+- Optional: Redis for caching and rate limiting
+
+## LiteLLM Proxy — Quick Start
+
+LiteLLM is the de facto open-source LLM gateway with OpenAI-compatible API.
+
+```bash
+# Run with Docker
+docker run -d \
+  --name litellm-proxy \
+  -p 4000:4000 \
+  -e OPENAI_API_KEY=$OPENAI_API_KEY \
+  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
+  -v $(pwd)/litellm-config.yaml:/app/config.yaml \
+  ghcr.io/berriai/litellm:main-latest \
+  --config /app/config.yaml \
+  --detailed_debug
+```
+
+## LiteLLM Configuration
+
+```yaml
+# litellm-config.yaml
+model_list:
+  # OpenAI models
+  - model_name: gpt-4o
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+      rpm: 10000
+      tpm: 2000000
+
+  - model_name: gpt-4o-mini
+    litellm_params:
+      model: openai/gpt-4o-mini
+      api_key: os.environ/OPENAI_API_KEY
+
+  # Anthropic
+  - model_name: claude-sonnet-4-6
+    litellm_params:
+      model: anthropic/claude-sonnet-4-6
+      api_key: os.environ/ANTHROPIC_API_KEY
+
+  # Self-hosted vLLM instances (load balanced)
+  - model_name: llama-3.1-8b
+    litellm_params:
+      model: openai/meta-llama/Llama-3.1-8B-Instruct
+      api_base: http://vllm-1:8000/v1
+      api_key: fake                    # vLLM key
+  - model_name: llama-3.1-8b
+    litellm_params:
+      model: openai/meta-llama/Llama-3.1-8B-Instruct
+      api_base: http://vllm-2:8000/v1  # second replica — auto load balanced
+      api_key: fake
+
+  # Fallback: cheap model if primary fails
+  - model_name: gpt-4o
+    litellm_params:
+      model: openai/gpt-4o-mini        # fallback to cheaper model
+      api_key: os.environ/OPENAI_API_KEY
+
+router_settings:
+  routing_strategy: least-busy         # or: latency-based, simple-shuffle
+  num_retries: 3
+  retry_after: 5
+  allowed_fails: 2
+  cooldown_time: 60
+
+  # Fallback configuration
+  fallbacks:
+    - gpt-4o: [claude-sonnet-4-6]
+    - claude-sonnet-4-6: [gpt-4o]
+
+litellm_settings:
+  # Semantic caching
+  cache: true
+  cache_params:
+    type: redis
+    host: redis
+    port: 6379
+    similarity_threshold: 0.90        # cache if >90% semantic similarity
+
+  # Logging
+  success_callback: ["langfuse"]
+  failure_callback: ["langfuse"]
+  langfuse_public_key: os.environ/LANGFUSE_PUBLIC_KEY
+  langfuse_secret_key: os.environ/LANGFUSE_SECRET_KEY
+
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+  database_url: postgresql://litellm:password@postgres:5432/litellm
+  store_model_in_db: true
+```
+
+## Docker Compose: Full Gateway Stack
+
+```yaml
+services:
+  litellm:
+    image: ghcr.io/berriai/litellm:main-latest
+    command: ["--config", "/app/config.yaml", "--port", "4000"]
+    volumes:
+      - ./litellm-config.yaml:/app/config.yaml
+    ports:
+      - "4000:4000"
+    environment:
+      - OPENAI_API_KEY=${OPENAI_API_KEY}
+      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
+      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
+      - DATABASE_URL=postgresql://litellm:password@postgres:5432/litellm
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_started
+    restart: unless-stopped
+
+  postgres:
+    image: postgres:16-alpine
+    environment:
+      POSTGRES_DB: litellm
+      POSTGRES_USER: litellm
+      POSTGRES_PASSWORD: password
+    volumes:
+      - postgres-data:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      retries: 5
+    restart: unless-stopped
+
+  redis:
+    image: redis:7-alpine
+    command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
+    volumes:
+      - redis-data:/data
+    restart: unless-stopped
+
+volumes:
+  postgres-data:
+  redis-data:
+```
+
+## Virtual Keys & Rate Limiting
+
+```bash
+# Create a virtual API key for a team (via LiteLLM API)
+curl -X POST http://localhost:4000/key/generate \
+  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "team_id": "team-backend",
+    "key_alias": "backend-team-key",
+    "models": ["gpt-4o-mini", "llama-3.1-8b"],
+    "max_budget": 100,              # USD limit
+    "budget_duration": "monthly",
+    "rpm_limit": 100,               # requests per minute
+    "tpm_limit": 500000             # tokens per minute
+  }'
+
+# View spend
+curl http://localhost:4000/spend/keys \
+  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
+```
+
+## Nginx Load Balancer (Alternative/Complement)
+
+```nginx
+# nginx.conf — round-robin across vLLM replicas
+upstream vllm_backends {
+    least_conn;
+    server vllm-1:8000 max_fails=3 fail_timeout=30s;
+    server vllm-2:8000 max_fails=3 fail_timeout=30s;
+    server vllm-3:8000 max_fails=3 fail_timeout=30s;
+    keepalive 32;
+}
+
+server {
+    listen 80;
+    server_name llm-api.internal;
+
+    # Rate limiting
+    limit_req_zone $http_authorization zone=per_key:10m rate=100r/m;
+    limit_req zone=per_key burst=20 nodelay;
+
+    location /v1/ {
+        proxy_pass http://vllm_backends;
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_read_timeout 300s;        # long timeout for streaming
+        proxy_buffering off;            # required for SSE streaming
+        proxy_cache_bypass 1;
+    }
+}
+```
+
+## Monitoring Gateway Health
+
+```bash
+# Check LiteLLM health
+curl http://localhost:4000/health
+
+# Model-level health
+curl http://localhost:4000/health/liveliness
+
+# Spend by model
+curl http://localhost:4000/spend/models \
+  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
+
+# Active virtual keys
+curl http://localhost:4000/key/list \
+  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `ConnectionRefusedError` to backend | Backend not reachable | Check `api_base` URL; verify backend is healthy |
+| Rate limit errors (429) | Budget/RPM exceeded | Increase limits or rotate to fallback model |
+| Slow streaming responses | `proxy_buffering` enabled | Set `proxy_buffering off` in Nginx |
+| Cache miss rate high | Threshold too strict | Lower `similarity_threshold` to `0.85` |
+| Postgres connection errors | DB not ready | Add `depends_on` with `condition: service_healthy` |
+
+## Best Practices
+
+- Use virtual keys per team/app — never expose raw provider API keys.
+- Enable `cache: true` with Redis for repeated or similar queries; can cut costs 30–50%.
+- Set `num_retries: 3` with fallbacks to handle provider outages gracefully.
+- Log all requests to Langfuse or OpenTelemetry for cost attribution and debugging.
+- Use `least-busy` routing strategy for self-hosted models to avoid GPU saturation.
+
+## Related Skills
+
+- [vllm-server](../../local-ai/vllm-server/) - Backend inference server
+- [llm-inference-scaling](../../local-ai/llm-inference-scaling/) - Auto-scaling backends
+- [llm-caching](../../../devops/ai/llm-caching/) - Semantic cache patterns
+- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost management
@@ -0,0 +1,221 @@
+---
+name: gpu-server-management
+description: Set up and manage NVIDIA GPU servers for AI workloads — driver installation, CUDA toolkit, container toolkit, MIG partitioning, GPU health monitoring, and multi-GPU configuration for LLM inference and training.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# GPU Server Management
+
+Provision, configure, and monitor NVIDIA GPU servers for AI inference and training workloads.
+
+## When to Use This Skill
+
+Use this skill when:
+- Setting up a new GPU server for LLM inference or model training
+- Installing or upgrading NVIDIA drivers and CUDA toolkit
+- Configuring Docker with NVIDIA Container Toolkit for GPU workloads
+- Partitioning A100/H100 GPUs with MIG for multi-tenant workloads
+- Troubleshooting GPU errors, driver issues, or thermal throttling
+
+## Prerequisites
+
+- Ubuntu 22.04 LTS (recommended) or RHEL 8/9
+- NVIDIA GPU (A10G, A100, H100, RTX 4090, or L40S recommended)
+- Root or sudo access
+- Internet access for package downloads
+
+## Driver Installation (Ubuntu)
+
+```bash
+# Remove old drivers
+sudo apt purge -y 'nvidia*' 'cuda*' 'libcuda*'
+sudo apt autoremove -y
+
+# Add NVIDIA package repository
+distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
+curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
+  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
+
+curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
+  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
+  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
+
+sudo apt update
+
+# Install latest driver (560.x as of 2025)
+sudo apt install -y nvidia-driver-560 cuda-toolkit-12-6
+
+# Install NVIDIA Container Toolkit (Docker GPU support)
+sudo apt install -y nvidia-container-toolkit
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+
+# Verify
+nvidia-smi
+nvcc --version
+docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi
+```
+
+## Post-Install Configuration
+
+```bash
+# Enable persistence mode (reduces driver initialization latency)
+sudo nvidia-smi -pm 1
+
+# Set power limit (reduce heat/noise on inference servers)
+sudo nvidia-smi -pl 350          # watts; check TDP for your GPU model
+
+# Disable ECC on inference servers (frees ~6% VRAM, less safe)
+sudo nvidia-smi --ecc-config=0   # requires reboot
+
+# Enable P2P for multi-GPU NVLink training
+sudo nvidia-smi topo -m          # check NVLink topology
+```
+
+## GPU Health Monitoring
+
+```bash
+# Real-time monitoring (like htop for GPUs)
+watch -n 1 nvidia-smi
+
+# Detailed stats
+nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,\
+utilization.memory,memory.used,memory.free,power.draw,clocks.current.graphics \
+--format=csv --loop=1
+
+# DCGM — production monitoring daemon (for clusters)
+sudo apt install -y datacenter-gpu-manager
+sudo systemctl start dcgm
+dcgmi discovery -l                # list GPUs
+dcgmi diag -r 1                  # quick health check
+dcgmi diag -r 3                  # full diagnostic (takes ~20 min)
+
+# Check GPU errors (XID errors — important for stability)
+sudo dmesg | grep -i "NVRM\|nvidia\|XID"
+nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total \
+  --format=csv,noheader
+```
+
+## Prometheus GPU Metrics (DCGM Exporter)
+
+```bash
+# Deploy DCGM Exporter for Prometheus scraping
+docker run -d \
+  --name dcgm-exporter \
+  --gpus all \
+  --cap-add SYS_ADMIN \
+  -p 9400:9400 \
+  --restart unless-stopped \
+  nvcr.io/nvidia/k8s/dcgm-exporter:latest
+
+# Key metrics exposed:
+# DCGM_FI_DEV_GPU_UTIL          - GPU utilization %
+# DCGM_FI_DEV_MEM_COPY_UTIL     - Memory bandwidth utilization
+# DCGM_FI_DEV_FB_USED           - Framebuffer memory used (MB)
+# DCGM_FI_DEV_SM_CLOCK          - SM clock speed (MHz)
+# DCGM_FI_DEV_GPU_TEMP          - Temperature (°C)
+# DCGM_FI_DEV_POWER_USAGE       - Power draw (W)
+# DCGM_FI_DEV_XID_ERRORS        - XID error count (0 = healthy)
+```
+
+## MIG Partitioning (A100/H100)
+
+MIG (Multi-Instance GPU) allows slicing one GPU into isolated smaller GPUs.
+
+```bash
+# Enable MIG mode (requires reboot or restart of all processes)
+sudo nvidia-smi -mig 1
+sudo systemctl restart nvidia-persistenced
+
+# List available MIG profiles (A100 80GB example)
+nvidia-smi mig -lgip
+# 1g.10gb   — 1 slice,  10GB (max 7 instances)
+# 2g.20gb   — 2 slices, 20GB (max 3 instances)
+# 3g.40gb   — 3 slices, 40GB (max 2 instances)
+# 7g.80gb   — full GPU, 80GB (max 1 instance)
+
+# Create MIG instances (e.g., 3× 2g.20gb + 1× 2g.20gb = multi-tenant)
+sudo nvidia-smi mig -cgi 2g.20gb,2g.20gb,2g.20gb,2g.20gb -C
+
+# List created instances
+nvidia-smi mig -lgi
+nvidia-smi mig -lcgi
+
+# Use in Docker
+docker run --gpus '"device=MIG-GPU-xxx/0/0"' ...
+
+# Disable MIG
+sudo nvidia-smi mig -i 0 -dci
+sudo nvidia-smi mig -i 0 -dgi
+sudo nvidia-smi -mig 0
+```
+
+## Kernel & OS Tuning for GPU Servers
+
+```bash
+# Increase file descriptor limits
+echo '* soft nofile 1048576' | sudo tee -a /etc/security/limits.conf
+echo '* hard nofile 1048576' | sudo tee -a /etc/security/limits.conf
+
+# Disable transparent huge pages (reduces latency jitter)
+echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
+echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
+
+# Persist via rc.local or systemd unit:
+cat <<'EOF' | sudo tee /etc/rc.local
+#!/bin/bash
+echo never > /sys/kernel/mm/transparent_hugepage/enabled
+echo never > /sys/kernel/mm/transparent_hugepage/defrag
+nvidia-smi -pm 1
+exit 0
+EOF
+sudo chmod +x /etc/rc.local
+
+# PCIe performance mode
+sudo nvidia-smi --auto-boost-default=0
+sudo nvidia-smi --auto-boost-permission=0
+```
+
+## Multi-GPU Topology Check
+
+```bash
+# Check NVLink and PCIe topology
+nvidia-smi topo -m
+# Output shows interconnect type:
+# NV4 = NVLink 4.0 (H100 SXM)
+# NV2 = NVLink 2.0 (A100 SXM)
+# PHB = PCIe bus (slower; avoid for tensor parallel training)
+# PIX = same PCIe switch (fast)
+
+# Bandwidth test between GPUs
+/usr/local/cuda/samples/bin/x86_64/linux/release/p2pBandwidthLatencyTest
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `nvidia-smi: command not found` | Driver not installed | Follow driver installation steps above |
+| Driver version mismatch | CUDA/driver incompatibility | Check compatibility matrix at developer.nvidia.com |
+| GPU temperature >85°C | Poor airflow or fan failure | Check `nvidia-smi -q -d TEMPERATURE`; reseat cooler |
+| XID 79 errors | GPU hardware error | Run `dcgmi diag -r 3`; may need GPU replacement |
+| `failed to open device` in container | Container toolkit not configured | Run `nvidia-ctk runtime configure --runtime=docker` |
+| Low PCIe bandwidth | Wrong slot or power limit | Check `nvidia-smi -q | grep PCIe`; use x16 slot |
+
+## Best Practices
+
+- Always enable persistence mode (`nvidia-smi -pm 1`) — reduces first-request latency.
+- Monitor XID errors; persistent XID 79/94 indicates hardware failure.
+- For training: use NVLink-connected GPUs; for inference: PCIe is usually fine.
+- Set up DCGM alerts on temperature >80°C and power draw near TDP.
+- Use MIG for multi-tenant inference to provide GPU isolation between models.
+
+## Related Skills
+
+- [vllm-server](../../local-ai/vllm-server/) - LLM inference on GPUs
+- [llm-fine-tuning](../../local-ai/llm-fine-tuning/) - GPU training setup
+- [linux-hardening](../../../security/hardening/linux-hardening/) - Secure the host OS
+- [prometheus-grafana](../../../devops/observability/prometheus-grafana/) - Metrics dashboards
@@ -0,0 +1,318 @@
+---
+name: ai-security-hardening
+description: Harden AI/LLM deployments against prompt injection, data exfiltration, model theft, and supply chain attacks. Covers input validation, output filtering, access control, model API security, and compliance controls for production AI systems.
+license: MIT
+metadata:
+  author: devops-skills
+  version: "1.0"
+---
+
+# AI Security Hardening
+
+Secure LLM and AI systems against prompt injection, jailbreaks, data leakage, and supply chain threats in production environments.
+
+## When to Use This Skill
+
+Use this skill when:
+- Deploying an LLM-powered application handling sensitive user data
+- Protecting against prompt injection attacks in AI agents
+- Implementing output filtering and content moderation
+- Securing model weights and API endpoints from theft
+- Achieving SOC2 or ISO 27001 compliance for AI systems
+
+## AI-Specific Threat Model
+
+```
+Threat                    Risk                          Control
+─────────────────────────────────────────────────────────────────────
+Prompt injection          System prompt override         Input sanitization, separate context
+Data exfiltration         PII in model outputs           Output filtering, DLP scanning
+Jailbreaking             Policy bypass                  Content moderation, guardrails
+Model theft               Weight extraction via API      Rate limiting, access controls
+Training data poisoning   Backdoored fine-tuned model    Dataset validation, provenance
+Supply chain attack       Malicious model weights        Signature verification, scanning
+Insecure output           XSS/SQLi from LLM response     Output encoding, parameterized queries
+```
+
+## Prompt Injection Defense
+
+```python
+import re
+from typing import Optional
+
+INJECTION_PATTERNS = [
+    r"ignore\s+(all\s+)?(previous|prior|above)\s+instructions",
+    r"you\s+are\s+now\s+",
+    r"new\s+instructions?:",
+    r"system\s+prompt",
+    r"forget\s+everything",
+    r"act\s+as\s+",
+    r"jailbreak",
+    r"dan\s+mode",
+    r"<\s*system\s*>",
+    r"\[INST\]",
+]
+
+def detect_prompt_injection(user_input: str) -> tuple[bool, Optional[str]]:
+    """Return (is_suspicious, matched_pattern)."""
+    normalized = user_input.lower().strip()
+    for pattern in INJECTION_PATTERNS:
+        if re.search(pattern, normalized, re.IGNORECASE):
+            return True, pattern
+    return False, None
+
+def sanitize_user_input(user_input: str, max_length: int = 4000) -> str:
+    """Sanitize input before passing to LLM."""
+    # Truncate
+    user_input = user_input[:max_length]
+
+    # Remove null bytes and control characters
+    user_input = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', '', user_input)
+
+    # Check for injection
+    suspicious, pattern = detect_prompt_injection(user_input)
+    if suspicious:
+        raise ValueError(f"Potential prompt injection detected: {pattern}")
+
+    return user_input
+```
+
+## Guardrails with NeMo Guardrails
+
+```python
+# guardrails.yaml
+from nemoguardrails import RailsConfig, LLMRails
+
+config = RailsConfig.from_path("./guardrails-config")
+rails = LLMRails(config)
+
+async def safe_llm_call(user_message: str) -> str:
+    response = await rails.generate_async(
+        messages=[{"role": "user", "content": user_message}]
+    )
+    return response["content"]
+```
+
+```yaml
+# guardrails-config/config.yml
+models:
+  - type: main
+    engine: openai
+    model: gpt-4o-mini
+
+rails:
+  input:
+    flows:
+      - check jailbreak
+      - check sensitive data
+  output:
+    flows:
+      - check output for PII
+      - check output for harmful content
+```
+
+## Output Filtering & PII Scrubbing
+
+```python
+import re
+from presidio_analyzer import AnalyzerEngine
+from presidio_anonymizer import AnonymizerEngine
+
+analyzer = AnalyzerEngine()
+anonymizer = AnonymizerEngine()
+
+PII_ENTITIES = ["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD",
+                "US_SSN", "IBAN_CODE", "IP_ADDRESS", "LOCATION"]
+
+def scrub_pii_from_output(text: str) -> str:
+    """Remove PII from LLM output before returning to user."""
+    results = analyzer.analyze(text=text, entities=PII_ENTITIES, language="en")
+    if not results:
+        return text
+    anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
+    return anonymized.text
+
+def validate_output_safety(output: str) -> bool:
+    """Check output doesn't contain prompt injection artifacts."""
+    dangerous_patterns = [
+        r"<\s*script\s*>",         # XSS
+        r"javascript:",             # XSS
+        r";\s*(DROP|DELETE|INSERT)",# SQLi
+        r"\$\{.*\}",               # template injection
+        r"`.*`",                   # command injection in some contexts
+    ]
+    for pattern in dangerous_patterns:
+        if re.search(pattern, output, re.IGNORECASE):
+            return False
+    return True
+```
+
+## API Security for LLM Endpoints
+
+```python
+from fastapi import FastAPI, HTTPException, Depends, Request
+from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
+import jwt
+import time
+from collections import defaultdict
+
+app = FastAPI()
+security = HTTPBearer()
+
+# Rate limiting (per API key)
+request_counts = defaultdict(list)
+
+def rate_limit(api_key: str, max_requests: int = 100, window_seconds: int = 60):
+    now = time.time()
+    requests = request_counts[api_key]
+    # Remove old requests outside window
+    request_counts[api_key] = [t for t in requests if now - t < window_seconds]
+    if len(request_counts[api_key]) >= max_requests:
+        raise HTTPException(status_code=429, detail="Rate limit exceeded")
+    request_counts[api_key].append(now)
+
+async def verify_token(
+    credentials: HTTPAuthorizationCredentials = Depends(security)
+) -> dict:
+    try:
+        payload = jwt.decode(credentials.credentials, SECRET_KEY, algorithms=["HS256"])
+        rate_limit(payload["sub"])
+        return payload
+    except jwt.ExpiredSignatureError:
+        raise HTTPException(status_code=401, detail="Token expired")
+    except jwt.InvalidTokenError:
+        raise HTTPException(status_code=401, detail="Invalid token")
+
+@app.post("/v1/chat/completions")
+async def chat(request: Request, token: dict = Depends(verify_token)):
+    body = await request.json()
+
+    # Input validation
+    user_msg = body.get("messages", [{}])[-1].get("content", "")
+    try:
+        safe_input = sanitize_user_input(user_msg)
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+
+    # Call LLM and scrub output
+    response = await call_llm(safe_input, token["scope"])
+    response["choices"][0]["message"]["content"] = scrub_pii_from_output(
+        response["choices"][0]["message"]["content"]
+    )
+    return response
+```
+
+## Model Weight Security
+
+```bash
+# Verify model weights with SHA-256 hash before loading
+MODEL_DIR="./models/llama-3.1-8b"
+EXPECTED_HASH="sha256:abc123..."
+
+# Generate hash of downloaded model
+actual_hash=$(find "$MODEL_DIR" -name "*.safetensors" | sort | xargs sha256sum | sha256sum)
+echo "Model hash: $actual_hash"
+
+# Compare (automate in CI/CD)
+if [ "$actual_hash" != "$EXPECTED_HASH" ]; then
+  echo "ERROR: Model hash mismatch — possible tampering!"
+  exit 1
+fi
+
+# Scan model files for embedded malware (ModelScan)
+pip install modelscan
+modelscan scan -p "$MODEL_DIR"
+```
+
+## Network Isolation for AI Services
+
+```yaml
+# Kubernetes NetworkPolicy — isolate LLM API
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: llm-api-isolation
+  namespace: ai-services
+spec:
+  podSelector:
+    matchLabels:
+      app: vllm
+  policyTypes:
+  - Ingress
+  - Egress
+  ingress:
+  - from:
+    - namespaceSelector:
+        matchLabels:
+          name: backend           # only backend can call LLM
+    ports:
+    - protocol: TCP
+      port: 8000
+  egress:
+  - to:
+    - namespaceSelector:
+        matchLabels:
+          name: monitoring        # metrics only
+    ports:
+    - protocol: TCP
+      port: 9090
+  # Block egress to internet — prevent data exfiltration
+  # (allow only internal cluster traffic)
+```
+
+## Audit Logging
+
+```python
+import structlog
+from datetime import datetime, timezone
+
+audit_log = structlog.get_logger("ai.audit")
+
+def log_llm_interaction(
+    user_id: str,
+    session_id: str,
+    model: str,
+    prompt_tokens: int,
+    completion_tokens: int,
+    was_filtered: bool,
+    injection_detected: bool,
+):
+    audit_log.info(
+        "llm_interaction",
+        timestamp=datetime.now(timezone.utc).isoformat(),
+        user_id=user_id,
+        session_id=session_id,
+        model=model,
+        prompt_tokens=prompt_tokens,
+        completion_tokens=completion_tokens,
+        was_filtered=was_filtered,
+        injection_detected=injection_detected,
+        # DO NOT log prompt/completion content — PII risk
+    )
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| False positive injection blocks | Overly broad regex | Tune patterns; use ML-based classifier for high-traffic |
+| PII in model outputs | Model trained on PII data | Add Presidio scrubbing to output layer |
+| API key leakage | Keys in logs or responses | Mask keys in logging; use vault for key storage |
+| Model weight tampering | Unverified downloads | Always verify SHA-256; use `modelscan` |
+| Rate limit bypass | Per-IP not per-user | Rate limit on authenticated user ID, not IP |
+
+## Best Practices
+
+- Never log raw prompts or completions — they may contain PII or sensitive data.
+- Treat LLM output as untrusted input — always encode before rendering in HTML.
+- Use network policies to prevent LLM pods from making outbound internet calls.
+- Rotate API keys quarterly; use short-lived JWT tokens for service-to-service auth.
+- Run `modelscan` on any model downloaded from the internet before serving.
+
+## Related Skills
+
+- [hashicorp-vault](../../secrets/hashicorp-vault/) - Secrets management for API keys
+- [network-security](../../network/) - Network-level controls
+- [linux-hardening](../../hardening/linux-hardening/) - Host hardening
+- [agent-observability](../../../devops/ai/agent-observability/) - AI audit logging
+- [llm-gateway](../../../infrastructure/networking/llm-gateway/) - Centralized access control