Merge pull request #3 from BagelHole/claude/add-ai-skills-96ZAQ

2026-08-22 12:49:53 +02:00 · 2026-03-02 12:38:44 -05:00
parent cc3848d963 dd77232b16
commit 4401692295
13 changed files with 3327 additions and 0 deletions
@@ -195,6 +195,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [argocd-gitops](devops/orchestration/argocd-gitops/) | GitOps with ArgoCD |
 | [kustomize](devops/orchestration/kustomize/) | Kubernetes manifest customization |
 | [openshift](devops/orchestration/openshift/) | OpenShift cluster management |
 | [model-serving-kubernetes](devops/orchestration/model-serving-kubernetes/) | KServe and Triton model serving with canary deployments and GPU autoscaling |
 ### Observability
 | Skill | Description |
@@ -212,6 +213,9 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 |-------|-------------|
 | [agent-observability](devops/ai/agent-observability/) | Tracing, latency, token, and cost telemetry for agents |
 | [agent-evals](devops/ai/agent-evals/) | Automated regression and safety eval suites for agents |
 | [llm-cost-optimization](devops/ai/llm-cost-optimization/) | Cut LLM API costs with caching, batching, model routing, and self-hosting |
 | [llm-caching](devops/ai/llm-caching/) | Exact and semantic caching layers to reduce API calls by 30–70% |
 | [ai-pipeline-orchestration](devops/ai/ai-pipeline-orchestration/) | Orchestrate RAG ingestion, training, and batch inference with Prefect/Airflow |
 ### Release Management
 | Skill | Description |
@@ -276,6 +280,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 |-------|-------------|
 | [ai-agent-security](security/ai/ai-agent-security/) | Defend agents against injection, tool abuse, and exfiltration |
 | [llm-app-security](security/ai/llm-app-security/) | Harden LLM app inputs, outputs, and tenant isolation |
 | [ai-security-hardening](security/ai/ai-security-hardening/) | Harden LLM deployments against prompt injection, model theft, and data exfiltration |
 </details>
@@ -334,6 +339,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [user-management](infrastructure/servers/user-management/) | Users, groups, sudo |
 | [systemd-services](infrastructure/servers/systemd-services/) | Services and timers |
 | [performance-tuning](infrastructure/servers/performance-tuning/) | System optimization |
 | [gpu-server-management](infrastructure/servers/gpu-server-management/) | NVIDIA GPU driver setup, MIG partitioning, DCGM monitoring for AI workloads |
 ### Networking
 | Skill | Description |
@@ -343,6 +349,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [cdn-setup](infrastructure/networking/cdn-setup/) | CloudFront, Cloudflare |
 | [reverse-proxy](infrastructure/networking/reverse-proxy/) | nginx, Traefik |
 | [service-mesh](infrastructure/networking/service-mesh/) | Istio, Linkerd |
 | [llm-gateway](infrastructure/networking/llm-gateway/) | Unified LLM API gateway with routing, rate limiting, virtual keys, and semantic caching |
 ### Databases
 | Skill | Description |
@@ -353,6 +360,7 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [mongodb](infrastructure/databases/mongodb/) | MongoDB clusters |
 | [redis](infrastructure/databases/redis/) | Redis caching |
 | [database-backups](infrastructure/databases/database-backups/) | Backup strategies |
 | [vector-database-ops](infrastructure/databases/vector-database-ops/) | Qdrant, Weaviate, and pgvector for production AI search and RAG workloads |
 ### Storage
 | Skill | Description |
@@ -375,6 +383,10 @@ No agent? No problem. Browse the skills, copy the scripts, use the configs. It's
 | [ollama-stack](infrastructure/local-ai/ollama-stack/) | Private local inference stack with Ollama |
 | [mac-mini-llm-lab](infrastructure/local-ai/mac-mini-llm-lab/) | Mac mini setup for always-on local LLM serving |
 | [openclaw-local-mac-mini](infrastructure/local-ai/openclaw-local-mac-mini/) | OpenClaw setup for local development and Mac mini hosting |
 | [vllm-server](infrastructure/local-ai/vllm-server/) | High-throughput LLM serving with vLLM — PagedAttention, tensor parallelism, OpenAI API |
 | [llm-inference-scaling](infrastructure/local-ai/llm-inference-scaling/) | Auto-scale LLM inference clusters on Kubernetes with KEDA and GPU-aware scheduling |
 | [rag-infrastructure](infrastructure/local-ai/rag-infrastructure/) | Production RAG with vector stores, hybrid search, embedding pipelines, and reranking |
 | [llm-fine-tuning](infrastructure/local-ai/llm-fine-tuning/) | QLoRA and full fine-tuning with Axolotl, DeepSpeed, and DPO alignment on GPU clusters |
 ### IT Operations
 | Skill | Description |
@@ -0,0 +1,262 @@
 ---
 name: ai-pipeline-orchestration
 description: Orchestrate AI/ML pipelines for data ingestion, model training, batch inference, and RAG indexing using Prefect, Airflow, or Dagster. Build reliable, observable, and retriable workflows for production AI systems.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # AI Pipeline Orchestration
 Build reliable, observable AI workflows — from document ingestion to batch inference to model training pipelines.
 ## When to Use This Skill
 Use this skill when:
 - Scheduling recurring RAG document ingestion and re-indexing
 - Orchestrating multi-step batch LLM processing workflows
 - Running nightly model evaluation and fine-tuning jobs
 - Building ETL pipelines that feed into AI models
 - Managing dependencies between data preparation and model serving
 ## Tool Selection
 | Tool | Best For | Complexity | GPU Jobs |
 |------|----------|------------|----------|
 | **Prefect** | Modern Python-first; easy to adopt | Low | Good |
 | **Airflow** | Complex DAGs; large teams; existing usage | High | Good |
 | **Dagster** | Asset-centric; strong data lineage | Medium | Excellent |
 | **Temporal** | Long-running workflows; reliability-first | Medium | Good |
 ## Prefect — Quick Start
 ```bash
 pip install prefect prefect-kubernetes
 # Start Prefect server (or use Prefect Cloud)
 prefect server start
 # In another terminal
 prefect worker start --pool default-agent-pool
 ```
 ## Prefect: RAG Ingestion Pipeline
 ```python
 from prefect import flow, task, get_run_logger
 from prefect.tasks import task_input_hash
 from datetime import timedelta
 import hashlib
@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=24))
 def fetch_documents(source_url: str) -> list[dict]:
    """Fetch documents from source; cached to avoid re-fetching."""
    logger = get_run_logger()
    logger.info(f"Fetching from {source_url}")
    # ... fetch logic
    return documents
@task(retries=3, retry_delay_seconds=30)
 def chunk_and_embed(documents: list[dict]) -> list[dict]:
    """Chunk documents and generate embeddings with retry on failure."""
    from sentence_transformers import SentenceTransformer
    model = SentenceTransformer("BAAI/bge-large-en-v1.5")
    chunks = []
    for doc in documents:
        doc_chunks = chunk_text(doc["content"])
        embeddings = model.encode(doc_chunks, batch_size=64)
        for chunk, emb in zip(doc_chunks, embeddings):
            chunks.append({"text": chunk, "embedding": emb.tolist(),
                           "source": doc["url"], "doc_hash": doc["hash"]})
    return chunks
@task(retries=2)
 def upsert_to_vector_store(chunks: list[dict]) -> int:
    """Upsert embeddings to Qdrant, skip unchanged documents."""
    from qdrant_client import QdrantClient
    client = QdrantClient("http://qdrant:6333")
    client.upsert(collection_name="knowledge-base", points=[...])
    return len(chunks)
@flow(name="rag-ingestion", log_prints=True)
 def rag_ingestion_pipeline(sources: list[str]):
    """Full RAG ingestion flow — runs daily."""
    logger = get_run_logger()
    total = 0
    for source in sources:
        docs = fetch_documents(source)
        chunks = chunk_and_embed(docs)
        count = upsert_to_vector_store(chunks)
        total += count
        logger.info(f"Ingested {count} chunks from {source}")
    logger.info(f"Pipeline complete: {total} total chunks indexed")
 if __name__ == "__main__":
    rag_ingestion_pipeline.serve(
        name="daily-rag-ingestion",
        cron="0 2 * * *",          # 2 AM daily
        parameters={"sources": ["https://docs.myapp.com", "https://api.myapp.com/kb"]},
    )
 ```
 ## Prefect: Batch LLM Inference Pipeline
 ```python
 from prefect import flow, task
 from prefect.concurrency.sync import concurrency
 import asyncio
 from openai import AsyncOpenAI
@task(retries=3, retry_delay_seconds=60)
 async def process_batch(items: list[dict], model: str = "gpt-4o-mini") -> list[dict]:
    """Process a batch of items through LLM with rate limit protection."""
    client = AsyncOpenAI()
    async with concurrency("openai-api", occupy=len(items)):  # rate limit
        tasks = [
            client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": item["prompt"]}],
                max_tokens=256,
            )
            for item in items
        ]
        responses = await asyncio.gather(*tasks, return_exceptions=True)
    results = []
    for item, response in zip(items, responses):
        if isinstance(response, Exception):
            results.append({**item, "error": str(response), "output": None})
        else:
            results.append({**item, "output": response.choices[0].message.content})
    return results
@flow(name="batch-llm-inference")
 async def batch_inference_flow(input_file: str, output_file: str, batch_size: int = 50):
    import json
    items = [json.loads(line) for line in open(input_file)]
    batches = [items[i:i+batch_size] for i in range(0, len(items), batch_size)]
    all_results = []
    for batch in batches:
        results = await process_batch(batch)
        all_results.extend(results)
    with open(output_file, "w") as f:
        for result in all_results:
            f.write(json.dumps(result) + "\n")
    return len(all_results)
 ```
 ## Airflow: Model Training DAG
 ```python
 from airflow.decorators import dag, task
 from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
 from datetime import datetime
 from kubernetes.client import models as k8s
@dag(
    dag_id="llm_fine_tuning",
    schedule="@weekly",
    start_date=datetime(2025, 1, 1),
    catchup=False,
    tags=["ai", "training"],
 )
 def llm_fine_tuning_dag():
    @task
    def prepare_dataset() -> str:
        """Download and preprocess training data."""
        # ... data prep logic
        return "s3://my-bucket/training-data/2025-03-01/"
    train = KubernetesPodOperator(
        task_id="train_model",
        name="llm-training-job",
        namespace="ml",
        image="nvcr.io/nvidia/pytorch:24.05-py3",
        cmds=["accelerate", "launch", "-m", "axolotl.cli.train", "/config/config.yaml"],
        resources=k8s.V1ResourceRequirements(
            limits={"nvidia.com/gpu": "4", "memory": "320Gi"},
            requests={"nvidia.com/gpu": "4"},
        ),
        node_selector={"nvidia.com/gpu.product": "A100-SXM4-80GB"},
        volumes=[...],
        volume_mounts=[...],
        get_logs=True,
        is_delete_operator_pod=True,
    )
    @task
    def evaluate_model(dataset_path: str) -> dict:
        """Run evals; fail pipeline if quality drops."""
        metrics = run_evals()
        if metrics["accuracy"] < 0.85:
            raise ValueError(f"Model quality too low: {metrics}")
        return metrics
    @task
    def deploy_model(metrics: dict):
        """Push merged model to HF Hub and update vLLM config."""
        update_serving_config(new_model="org/fine-tuned-v2")
    dataset = prepare_dataset()
    train.set_upstream(dataset)
    eval_result = evaluate_model(dataset)
    eval_result.set_upstream(train)
    deploy_model(eval_result)
 llm_fine_tuning_dag()
 ```
 ## Dagster: Asset-Based AI Pipeline
 ```python
 from dagster import asset, AssetExecutionContext, define_asset_job, ScheduleDefinition
@asset(description="Raw documents fetched from knowledge sources")
 def raw_documents(context: AssetExecutionContext) -> list[dict]:
    context.log.info("Fetching documents...")
    return fetch_all_documents()
@asset(
    deps=[raw_documents],
    description="Chunked and embedded document vectors",
 )
 def document_embeddings(context: AssetExecutionContext, raw_documents) -> int:
    chunks = process_and_embed(raw_documents)
    context.log.info(f"Generated {len(chunks)} embeddings")
    upsert_to_qdrant(chunks)
    return len(chunks)
@asset(
    deps=[document_embeddings],
    description="RAG system quality metrics",
 )
 def rag_quality_metrics(context: AssetExecutionContext) -> dict:
    metrics = evaluate_rag_system()
    context.add_output_metadata({"ragas_score": metrics["ragas_score"]})
    return metrics
 # Schedule: refresh embeddings nightly
 nightly_refresh = ScheduleDefinition(
    job=define_asset_job("rag_refresh_job", [raw_documents, document_embeddings]),
    cron_schedule="0 1 * * *",
 )
 ```
 ## Best Practices
 - Use task-level retries for API calls; use flow-level retries for transient infra failures.
 - Cache expensive steps (embedding generation, data fetching) to speed up reruns.
 - Emit custom metrics from pipelines (chunk count, error rate, cost) to your observability stack.
 - Use `concurrency` limits in Prefect or `pool` slots in Airflow to respect external rate limits.
 - Separate ingestion, training, and deployment pipelines — don't couple them in one giant DAG.
 ## Related Skills
 - [rag-infrastructure](../../infrastructure/local-ai/rag-infrastructure/) - RAG system setup
 - [llm-fine-tuning](../../infrastructure/local-ai/llm-fine-tuning/) - Training jobs
 - [agent-observability](../agent-observability/) - Pipeline monitoring
 - [kubernetes-ops](../orchestration/kubernetes-ops/) - Running pipeline pods on K8s
@@ -0,0 +1,309 @@
 ---
 name: llm-caching
 description: Implement multi-layer LLM caching with exact match, semantic similarity, and provider-side prompt caching. Reduce API costs by 30–70%, cut latency, and improve throughput using Redis, GPTCache, and provider caching APIs.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # LLM Caching
 Cut LLM costs and latency with exact match, semantic, and provider-side caching layers.
 ## When to Use This Skill
 Use this skill when:
 - The same or similar queries are asked repeatedly (FAQ bots, support tools)
 - LLM API costs are growing and you need immediate savings
 - Serving high request volumes where repeated queries cause bottlenecks
 - Implementing prompt caching for long system prompts (Anthropic/OpenAI)
 - Building offline-capable AI features that need response persistence
 ## Caching Layers
 ```
 Request → Exact Cache → Semantic Cache → Provider Cache → LLM API
             ↓ hit            ↓ hit             ↓ hit
           instant          ~5ms           50-80% cheaper
 ```
 ## Layer 1: Exact Match Cache (Redis)
 ```python
 import hashlib
 import json
 import redis
 from openai import OpenAI
 r = redis.Redis(host="localhost", port=6379, decode_responses=True)
 client = OpenAI()
 def build_cache_key(model: str, messages: list, temperature: float) -> str:
    """Deterministic key from request parameters."""
    payload = json.dumps({
        "model": model,
        "messages": messages,
        "temperature": temperature,
    }, sort_keys=True)
    return f"llm:exact:{hashlib.sha256(payload.encode()).hexdigest()}"
 def cached_completion(model: str, messages: list, temperature: float = 0.0,
                      ttl: int = 3600) -> dict:
    key = build_cache_key(model, messages, temperature)
    # Check cache
    if cached := r.get(key):
        return json.loads(cached)
    # Call API
    response = client.chat.completions.create(
        model=model, messages=messages, temperature=temperature
    )
    result = response.model_dump()
    # Cache result (only cache deterministic responses)
    if temperature == 0.0:
        r.setex(key, ttl, json.dumps(result))
    return result
 ```
 ## Layer 2: Semantic Cache (GPTCache)
 ```python
 from gptcache import cache, Config
 from gptcache.adapter import openai
 from gptcache.embedding import Onnx
 from gptcache.manager import CacheBase, VectorBase, get_data_manager
 from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
 # Configure GPTCache with Qdrant backend
 def init_gptcache(cache_obj, llm: str):
    onnx = Onnx()                              # local embedding model
    data_manager = get_data_manager(
        CacheBase("redis"),                    # metadata store
        VectorBase("qdrant",
                   host="localhost",
                   port=6333,
                   collection_name=f"llm-cache-{llm}",
                   dimension=onnx.dimension),
    )
    cache_obj.init(
        embedding_func=onnx.to_embeddings,
        data_manager=data_manager,
        similarity_evaluation=SearchDistanceEvaluation(),
        config=Config(similarity_threshold=0.80),  # 80% similarity = cache hit
    )
 cache.set_openai_key()
 init_gptcache(cache, "gpt-4o-mini")
 # Now openai calls are automatically cached
 response = openai.ChatCompletion.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is machine learning?"}],
 )
 # Second call with similar question ("Explain machine learning") → cache hit
 ```
 ## Custom Semantic Cache (Production-Grade)
 ```python
 from sentence_transformers import SentenceTransformer
 from qdrant_client import QdrantClient
 from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, Range
 import numpy as np
 import uuid
 import time
 embed_model = SentenceTransformer("BAAI/bge-small-en-v1.5")  # fast, 33M params
 qdrant = QdrantClient("http://localhost:6333")
 CACHE_COLLECTION = "semantic-cache"
 SIMILARITY_THRESHOLD = 0.88
 CACHE_TTL_SECONDS = 86400  # 24h
 # Create collection once
 qdrant.create_collection(
    collection_name=CACHE_COLLECTION,
    vectors_config=VectorParams(size=384, distance=Distance.COSINE),
    on_disk_payload=True,
 )
 def semantic_cache_lookup(query: str, model: str) -> str | None:
    embedding = embed_model.encode(query).tolist()
    results = qdrant.query_points(
        collection_name=CACHE_COLLECTION,
        query=embedding,
        query_filter=Filter(must=[
            FieldCondition(key="model", match={"value": model}),
            FieldCondition(key="expires_at", range=Range(gte=time.time())),
        ]),
        limit=1,
        score_threshold=SIMILARITY_THRESHOLD,
    )
    if results.points:
        return results.points[0].payload["response"]
    return None
 def semantic_cache_store(query: str, response: str, model: str):
    embedding = embed_model.encode(query).tolist()
    qdrant.upsert(
        collection_name=CACHE_COLLECTION,
        points=[PointStruct(
            id=str(uuid.uuid4()),
            vector=embedding,
            payload={
                "query": query,
                "response": response,
                "model": model,
                "created_at": time.time(),
                "expires_at": time.time() + CACHE_TTL_SECONDS,
            },
        )],
    )
 def smart_llm_call(query: str, model: str = "gpt-4o-mini") -> dict:
    # 1. Semantic lookup
    if cached_response := semantic_cache_lookup(query, model):
        return {"response": cached_response, "source": "semantic_cache", "cost": 0}
    # 2. LLM call
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": query}],
    )
    text = response.choices[0].message.content
    cost = litellm.completion_cost(response)
    # 3. Store in cache
    semantic_cache_store(query, text, model)
    return {"response": text, "source": "llm_api", "cost": cost}
 ```
 ## Layer 3: Provider-Side Prompt Caching
 ```python
 # Anthropic — cache long system prompts (saves 90% on cached input tokens)
 import anthropic
 client = anthropic.Anthropic()
 # Long system prompt — mark for caching
 SYSTEM_PROMPT = open("knowledge-base.txt").read()  # e.g., 50k tokens
 def call_with_prompt_cache(user_question: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        system=[
            {"type": "text", "text": "You are a helpful assistant."},
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"},  # cache this block
            }
        ],
        messages=[{"role": "user", "content": user_question}],
    )
    # Log cache efficiency
    usage = response.usage
    cache_savings = usage.cache_read_input_tokens * 0.9  # 90% discount on cached
    print(f"Cache hits: {usage.cache_read_input_tokens} tokens "
          f"(saved ~${cache_savings * 3.0 / 1_000_000:.4f})")
    return response.content[0].text
 # OpenAI — automatic for repeated prefixes (≥1,024 tokens)
 # No code change needed; cached tokens appear in usage.prompt_tokens_details
 response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": LONG_SYSTEM_PROMPT},  # auto-cached
        {"role": "user", "content": user_question},
    ]
 )
 cached = response.usage.prompt_tokens_details.cached_tokens
 print(f"OpenAI cached {cached} tokens")
 ```
 ## Cache Warming
 ```python
 async def warm_cache(common_queries: list[str], model: str):
    """Pre-populate cache with known frequent queries."""
    import asyncio
    from openai import AsyncOpenAI
    aclient = AsyncOpenAI()
    async def warm_single(query: str):
        if not semantic_cache_lookup(query, model):
            response = await aclient.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": query}],
            )
            text = response.choices[0].message.content
            semantic_cache_store(query, text, model)
            print(f"Warmed: {query[:50]}...")
    await asyncio.gather(*[warm_single(q) for q in common_queries])
 # Warm on startup
 import asyncio
 asyncio.run(warm_cache(FREQUENT_QUERIES, "gpt-4o-mini"))
 ```
 ## Cache Metrics
 ```python
 from prometheus_client import Counter, Histogram
 cache_hits = Counter("llm_cache_hits_total", "Cache hits", ["cache_layer", "model"])
 cache_misses = Counter("llm_cache_misses_total", "Cache misses", ["model"])
 cache_savings_usd = Counter("llm_cache_savings_usd_total", "USD saved by cache", ["model"])
 # Use in your smart_llm_call function
 if source == "semantic_cache":
    cache_hits.labels(cache_layer="semantic", model=model).inc()
    cache_savings_usd.labels(model=model).inc(estimated_cost)
 else:
    cache_misses.labels(model=model).inc()
 ```
 ## Redis Configuration for LLM Caching
 ```bash
 # redis.conf tuning for LLM cache workload
 maxmemory 8gb
 maxmemory-policy allkeys-lru    # evict least-recently-used when full
 save ""                          # disable persistence (cache is ephemeral)
 appendonly no
 tcp-keepalive 60
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | Low cache hit rate | Threshold too strict | Lower `SIMILARITY_THRESHOLD` to 0.82–0.85 |
 | Stale cached responses | Long TTL | Use topic-specific TTLs; invalidate on data updates |
 | Cache serving wrong answers | Threshold too loose | Raise threshold or add model-name filtering |
 | Redis OOM | No eviction policy | Set `maxmemory` + `allkeys-lru` |
 | Slow semantic lookup | Large cache collection | Add payload index on `model` + `expires_at` |
 ## Best Practices
 - Start with exact cache — zero cost, instant wins for identical queries.
 - Semantic threshold of 0.88–0.92 balances hit rate vs. accuracy; tune with your data.
 - Set per-model TTLs: longer for stable knowledge (1 week), shorter for news/events (1 hour).
 - Always filter by model name in semantic cache — different models give different answers.
 - Log cache hit rate as a KPI; target 30%+ for FAQ-style applications.
 ## Related Skills
 - [llm-cost-optimization](../llm-cost-optimization/) - Full cost strategy
 - [llm-gateway](../../infrastructure/networking/llm-gateway/) - Gateway-level caching
 - [vector-database-ops](../../infrastructure/databases/vector-database-ops/) - Qdrant setup
 - [agent-observability](../agent-observability/) - Cache metrics dashboards
@@ -0,0 +1,286 @@
 ---
 name: llm-cost-optimization
 description: Reduce LLM API and infrastructure costs through model selection, prompt caching, batching, caching, quantization, and self-hosting strategies. Track spend by team and model, set budgets, and implement cost-aware routing.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # LLM Cost Optimization
 Cut LLM costs by 50–90% with the right combination of caching, model selection, prompt optimization, and self-hosting.
 ## When to Use This Skill
 Use this skill when:
 - LLM API spend is growing faster than revenue
 - You need to attribute AI costs to teams, products, or customers
 - Implementing caching to avoid redundant LLM calls
 - Deciding when to switch from API providers to self-hosted models
 - Optimizing prompt length without sacrificing quality
 ## Cost Levers by Impact
 | Strategy | Typical Savings | Effort |
 |----------|-----------------|--------|
 | Semantic caching | 20–50% | Low |
 | Model right-sizing | 30–70% | Low |
 | Prompt compression | 10–30% | Medium |
 | Provider caching (prompt cache) | 10–25% | Low |
 | Batching offline workloads | 50% (Batch API) | Medium |
 | Self-hosting 7–8B models | 80–95% at scale | High |
 | Quantization | 30–50% VRAM cost | Medium |
 ## Track Costs First
 ```python
 # Use LiteLLM's cost tracking (automatic per-model pricing)
 import litellm
 response = litellm.completion(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
 )
 cost = litellm.completion_cost(response)
 print(f"Cost: ${cost:.6f}")
 # Add custom cost callbacks
 def log_cost(kwargs, completion_response, start_time, end_time):
    cost = kwargs.get("response_cost", 0)
    model = kwargs.get("model")
    user = kwargs.get("user")
    # Send to your analytics DB
    db.record_cost(user=user, model=model, cost=cost)
 litellm.success_callback = [log_cost]
 ```
 ## Model Right-Sizing
 ```python
 # Route by task complexity — don't use GPT-4o for everything
 def get_model_for_task(task_type: str) -> str:
    routing = {
        "classification":     "gpt-4o-mini",      # ~30× cheaper than gpt-4o
        "summarization":      "gpt-4o-mini",
        "extraction":         "gpt-4o-mini",
        "simple_qa":          "gpt-4o-mini",
        "complex_reasoning":  "gpt-4o",
        "code_generation":    "claude-sonnet-4-6",
        "creative_writing":   "claude-opus-4-6",
    }
    return routing.get(task_type, "gpt-4o-mini")
 # Cost comparison (per 1M tokens, 2025 approx.)
 # gpt-4o-mini:          input $0.15 / output $0.60
 # gpt-4o:               input $2.50 / output $10.00
 # claude-sonnet-4-6:    input $3.00 / output $15.00
 # llama-3.1-8b (self):  ~$0.05–0.10 all-in (GPU amortized)
 ```
 ## Prompt Caching (Provider-Side)
 ```python
 # Anthropic — cache long system prompts (saves 90% on cached tokens)
 import anthropic
 client = anthropic.Anthropic()
 response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a helpful assistant.",
        },
        {
            "type": "text",
            "text": open("large-context.txt").read(),  # large doc
            "cache_control": {"type": "ephemeral"},     # cache this!
        }
    ],
    messages=[{"role": "user", "content": "Summarize the key points."}],
 )
 # First call: full price. Subsequent calls: 90% discount on cached part.
 print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")
 # OpenAI — prompt caching is automatic for repeated prefixes >1024 tokens
 # No code change needed; check usage.prompt_tokens_details.cached_tokens
 ```
 ## Batching with OpenAI Batch API (50% Discount)
 ```python
 import json
 from openai import OpenAI
 client = OpenAI()
 # Prepare batch requests
 requests = [
    {
        "custom_id": f"task-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [{"role": "user", "content": f"Classify: {text}"}],
            "max_tokens": 50,
        }
    }
    for i, text in enumerate(texts)
 ]
 # Write JSONL file
 with open("batch.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")
 # Upload and create batch
 batch_file = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
 batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
 )
 print(f"Batch ID: {batch.id}")  # poll status with client.batches.retrieve(batch.id)
 ```
 ## Semantic Caching
 ```python
 import hashlib
 import json
 import redis
 import numpy as np
 from sentence_transformers import SentenceTransformer
 r = redis.Redis(host="localhost", port=6379)
 embed_model = SentenceTransformer("BAAI/bge-small-en-v1.5")
 SIMILARITY_THRESHOLD = 0.92
 CACHE_TTL = 3600 * 24  # 24 hours
 def cached_llm_call(prompt: str, llm_fn) -> str:
    # 1. Exact match (free)
    exact_key = f"exact:{hashlib.sha256(prompt.encode()).hexdigest()}"
    if cached := r.get(exact_key):
        return cached.decode()
    # 2. Semantic match
    query_vec = embed_model.encode(prompt)
    cached_keys = r.keys("sem:*")
    for key in cached_keys:
        data = json.loads(r.get(key))
        similarity = np.dot(query_vec, data["embedding"]) / (
            np.linalg.norm(query_vec) * np.linalg.norm(data["embedding"])
        )
        if similarity >= SIMILARITY_THRESHOLD:
            return data["response"]
    # 3. Cache miss — call LLM
    response = llm_fn(prompt)
    # Store exact match
    r.setex(exact_key, CACHE_TTL, response)
    # Store semantic embedding
    sem_key = f"sem:{hashlib.sha256(prompt.encode()).hexdigest()}"
    r.setex(sem_key, CACHE_TTL, json.dumps({
        "embedding": query_vec.tolist(),
        "response": response,
        "prompt": prompt,
    }))
    return response
 ```
 ## Prompt Compression
 ```python
 # LLMLingua — compress long prompts by 3–20× with minimal quality loss
 from llmlingua import PromptCompressor
 compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    device_map="cpu",
 )
 compressed = compressor.compress_prompt(
    long_context,
    ratio=0.5,       # keep 50% of tokens
    rank_method="longllmlingua",
 )
 print(f"Original: {len(long_context.split())} words")
 print(f"Compressed: {len(compressed['compressed_prompt'].split())} words")
 print(f"Savings: {compressed['saving']}")
 ```
 ## Self-Hosting Break-Even Calculator
 ```python
 def break_even_analysis(
    monthly_api_spend_usd: float,
    gpu_cost_per_hour_usd: float = 2.50,   # e.g., A10G on AWS
    utilization: float = 0.70,             # 70% GPU utilization
 ) -> dict:
    monthly_gpu_cost = gpu_cost_per_hour_usd * 24 * 30 * utilization
    break_even = monthly_gpu_cost / monthly_api_spend_usd
    recommendation = (
        "Self-host now — strong ROI" if break_even < 0.5 else
        "Self-host if traffic grows 2×" if break_even < 0.8 else
        "Stick with API — not enough scale yet"
    )
    return {
        "monthly_gpu_cost": f"${monthly_gpu_cost:.0f}",
        "monthly_api_spend": f"${monthly_api_spend_usd:.0f}",
        "gpu_as_pct_of_api": f"{break_even*100:.0f}%",
        "recommendation": recommendation,
    }
 # Example: $5k/month on OpenAI, $2.50/hr A10G
 print(break_even_analysis(5000))
 # → gpu_cost ~$1,260/mo = 25% of API spend → self-host now
 ```
 ## Cost Dashboard (Grafana)
 ```python
 # Emit cost metrics to Prometheus
 from prometheus_client import Counter, Histogram
 llm_cost_total = Counter(
    "llm_cost_usd_total",
    "Total LLM spend in USD",
    ["model", "team", "task_type"],
 )
 llm_tokens_total = Counter(
    "llm_tokens_total",
    "Total tokens used",
    ["model", "token_type"],  # token_type: prompt, completion, cached
 )
 def track_call(model, team, task_type, response):
    cost = calculate_cost(model, response.usage)
    llm_cost_total.labels(model=model, team=team, task_type=task_type).inc(cost)
    llm_tokens_total.labels(model=model, token_type="prompt").inc(
        response.usage.prompt_tokens)
    llm_tokens_total.labels(model=model, token_type="completion").inc(
        response.usage.completion_tokens)
 ```
 ## Best Practices
 - Use `gpt-4o-mini` or `claude-haiku` for 80% of tasks — they're 10–30× cheaper.
 - Enable prompt caching for system prompts >1,024 tokens (Anthropic) or >1,024 tokens (OpenAI).
 - Audit your top 5 prompts by token count — compress or cache them.
 - Set hard budget limits with LiteLLM virtual keys before costs spiral.
 - Self-host 7B–8B models when monthly API spend exceeds $2k/month.
 ## Related Skills
 - [llm-gateway](../../infrastructure/networking/llm-gateway/) - Centralized cost control
 - [llm-caching](../llm-caching/) - Semantic caching patterns
 - [vllm-server](../../infrastructure/local-ai/vllm-server/) - Self-hosted inference
 - [agent-observability](../agent-observability/) - Token and cost telemetry
@@ -0,0 +1,314 @@
 ---
 name: model-serving-kubernetes
 description: Deploy ML models on Kubernetes with KServe (formerly KFServing) and NVIDIA Triton Inference Server. Includes canary deployments, autoscaling, model versioning, A/B testing, and GPU resource management for production model serving.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # Model Serving on Kubernetes
 Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling.
 ## When to Use This Skill
 Use this skill when:
 - Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale
 - Implementing canary deployments and A/B testing for ML models
 - Autoscaling inference pods based on request rate or GPU metrics
 - Deploying LLMs with Triton or KServe on Kubernetes
 - Managing multiple model versions with traffic splitting
 ## Prerequisites
 - Kubernetes 1.28+ with GPU nodes
 - KServe installed (or Triton standalone)
 - `kubectl` and `helm` configured
 - NVIDIA GPU Operator installed on cluster
 ## KServe Installation
 ```bash
 # Install KServe with Helm
 helm repo add kserve https://kserve.github.io/helm-charts
 helm repo update
 helm install kserve kserve/kserve \
  --namespace kserve \
  --create-namespace \
  --set kserve.controller.gateway.ingressGateway.className=nginx
 # Verify
 kubectl get pods -n kserve
 kubectl get crd | grep kserve
 ```
 ## Basic InferenceService (KServe)
 ```yaml
 apiVersion: serving.kserve.io/v1beta1
 kind: InferenceService
 metadata:
  name: sklearn-iris
  namespace: models
 spec:
  predictor:
    sklearn:
      storageUri: gs://kfserving-examples/models/sklearn/1.0/model
      resources:
        requests:
          cpu: "1"
          memory: 2Gi
        limits:
          cpu: "2"
          memory: 4Gi
 ```
 ```bash
 kubectl apply -f inference-service.yaml
 # Get inference service URL
 kubectl get inferenceservice sklearn-iris -n models
 # NAME           URL                                          READY   ...
 # sklearn-iris   http://sklearn-iris.models.example.com       True
 # Test prediction
 curl -X POST http://sklearn-iris.models.example.com/v1/models/sklearn-iris:predict \
  -H "Content-Type: application/json" \
  -d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}'
 ```
 ## GPU-Enabled LLM InferenceService
 ```yaml
 apiVersion: serving.kserve.io/v1beta1
 kind: InferenceService
 metadata:
  name: llama-3-8b
  namespace: models
  annotations:
    serving.kserve.io/enable-prometheus-scraping: "true"
 spec:
  predictor:
    containers:
    - name: vllm-container
      image: vllm/vllm-openai:latest
      args:
      - "--model"
      - "meta-llama/Llama-3.1-8B-Instruct"
      - "--tensor-parallel-size"
      - "1"
      - "--gpu-memory-utilization"
      - "0.90"
      ports:
      - containerPort: 8080
        protocol: TCP
      resources:
        requests:
          nvidia.com/gpu: "1"
          memory: "20Gi"
          cpu: "4"
        limits:
          nvidia.com/gpu: "1"
          memory: "24Gi"
          cpu: "8"
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        initialDelaySeconds: 60
        periodSeconds: 10
      env:
      - name: HUGGING_FACE_HUB_TOKEN
        valueFrom:
          secretKeyRef:
            name: hf-token
            key: token
    nodeSelector:
      nvidia.com/gpu.present: "true"
  transformer:
    containers:
    - name: kserve-container
      image: kserve/kserve-transformer:latest
 ```
 ## Canary Deployment (Traffic Splitting)
 ```yaml
 apiVersion: serving.kserve.io/v1beta1
 kind: InferenceService
 metadata:
  name: llama-3-8b
  namespace: models
 spec:
  predictor:
    canaryTrafficPercent: 20    # 20% to new version, 80% to stable
    containers:
    - name: vllm-container
      image: vllm/vllm-openai:latest
      args:
      - "--model"
      - "meta-llama/Llama-3.1-8B-Instruct-v2"  # new model version
      resources:
        limits:
          nvidia.com/gpu: "1"
 ```
 ```bash
 # Gradually increase canary traffic
 kubectl patch inferenceservice llama-3-8b -n models \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/predictor/canaryTrafficPercent","value":50}]'
 # Promote canary to stable
 kubectl patch inferenceservice llama-3-8b -n models \
  --type='json' \
  -p='[{"op":"remove","path":"/spec/predictor/canaryTrafficPercent"}]'
 ```
 ## Autoscaling with KEDA
 ```yaml
 apiVersion: keda.sh/v1alpha1
 kind: ScaledObject
 metadata:
  name: llama-scaler
  namespace: models
 spec:
  scaleTargetRef:
    apiVersion: serving.kserve.io/v1beta1
    kind: InferenceService
    name: llama-3-8b
  minReplicaCount: 1
  maxReplicaCount: 5
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: kserve_request_count
      threshold: "10"
      query: |
        sum(rate(kserve_request_count_total{namespace="models",
            service="llama-3-8b"}[1m]))
 ```
 ## NVIDIA Triton Inference Server
 ```yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: triton-server
  namespace: models
 spec:
  replicas: 2
  selector:
    matchLabels:
      app: triton
  template:
    metadata:
      labels:
        app: triton
    spec:
      containers:
      - name: triton
        image: nvcr.io/nvidia/tritonserver:24.05-py3
        args:
        - "tritonserver"
        - "--model-store=s3://my-model-store/models"
        - "--model-control-mode=poll"        # auto-load new model versions
        - "--repository-poll-secs=30"
        - "--metrics-port=8002"
        ports:
        - containerPort: 8000   # HTTP
        - containerPort: 8001   # gRPC
        - containerPort: 8002   # Metrics
        resources:
          limits:
            nvidia.com/gpu: "1"
        readinessProbe:
          httpGet:
            path: /v2/health/ready
            port: 8000
          initialDelaySeconds: 30
 ```
 ## Triton Model Repository Structure
 ```
 s3://my-model-store/models/
 ├── text-classifier/
 │   ├── config.pbtxt
 │   ├── 1/
 │   │   └── model.onnx
 │   └── 2/
 │       └── model.onnx          # new version; auto-loaded
 ├── embedding-model/
 │   ├── config.pbtxt
 │   └── 1/
 │       └── model.onnx
 ```
 ```protobuf
 # config.pbtxt for ONNX model
 name: "text-classifier"
 backend: "onnxruntime"
 max_batch_size: 64
 dynamic_batching {
  preferred_batch_size: [16, 32]
  max_queue_delay_microseconds: 1000
 }
 input [
  { name: "input_ids" data_type: TYPE_INT64 dims: [-1] }
  { name: "attention_mask" data_type: TYPE_INT64 dims: [-1] }
 ]
 output [
  { name: "logits" data_type: TYPE_FP32 dims: [-1] }
 ]
 instance_group [
  { kind: KIND_GPU count: 2 }   # 2 model instances on GPU
 ]
 ```
 ## Model Management Commands
 ```bash
 # List loaded models (Triton)
 curl http://triton:8000/v2/models
 # Load a new model version
 curl -X POST http://triton:8000/v2/repository/models/text-classifier/load
 # Unload a model
 curl -X POST http://triton:8000/v2/repository/models/text-classifier/unload
 # KServe — watch rollout status
 kubectl rollout status deployment/llama-3-8b-predictor -n models
 kubectl get inferenceservice llama-3-8b -n models -w
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | `InferenceService not ready` | Model loading or OOM | Check predictor pod logs; increase memory limits |
 | Canary stuck at 0% | KNative routing issue | Check `kubectl get ksvc -n models` |
 | Triton missing model | S3 permissions or path | Verify IAM role; check `--model-store` path |
 | Low GPU utilization | Dynamic batching off | Enable `dynamic_batching` in Triton config |
 | Autoscaler not triggering | Prometheus query wrong | Test query in Prometheus UI |
 ## Best Practices
 - Use canary deployments for all model updates — roll back in seconds if metrics degrade.
 - Enable Triton dynamic batching — it can increase GPU throughput 5–10× for small models.
 - Store models in S3/GCS with versioned paths (`s3://bucket/model/v1/`, `v2/`).
 - Pin GPU node selectors to prevent model pods landing on CPU-only nodes.
 - Monitor p99 latency and error rates per model version during canary rollouts.
 ## Related Skills
 - [vllm-server](../../infrastructure/local-ai/vllm-server/) - vLLM for LLM serving
 - [llm-inference-scaling](../../infrastructure/local-ai/llm-inference-scaling/) - KEDA autoscaling
 - [kubernetes-ops](./kubernetes-ops/) - Core Kubernetes operations
 - [gpu-server-management](../../infrastructure/servers/gpu-server-management/) - GPU nodes
@@ -0,0 +1,285 @@
 ---
 name: vector-database-ops
 description: Deploy, manage, and optimize vector databases for AI applications. Covers Qdrant, Weaviate, pgvector, and Pinecone — collection management, indexing strategies, backup, and performance tuning for production RAG and semantic search workloads.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # Vector Database Operations
 Run production vector databases for AI-powered search, RAG, and recommendation systems.
 ## When to Use This Skill
 Use this skill when:
 - Setting up a vector database for a RAG or semantic search application
 - Choosing between Qdrant, Weaviate, pgvector, or Pinecone
 - Managing collections, indexes, and data migrations
 - Optimizing query performance and indexing for production loads
 - Implementing multi-tenant vector search with namespace isolation
 ## Vector Database Comparison
 | Database | Best For | Hosting | Filtering | Scale |
 |----------|----------|---------|-----------|-------|
 | **Qdrant** | High-performance, rich filtering, self-hosted | Self / Cloud | Excellent | Very High |
 | **Weaviate** | Schema-first, hybrid search, multi-modal | Self / Cloud | Good | High |
 | **pgvector** | Already on Postgres, simple use cases | Self | Good | Medium |
 | **Pinecone** | Zero-ops managed, serverless | Managed only | Good | Very High |
 | **Chroma** | Local dev, prototyping | Self only | Basic | Low-Medium |
 ## Qdrant — Production Deployment
 ```bash
 # Docker (single node)
 docker run -d \
  --name qdrant \
  -p 6333:6333 \
  -p 6334:6334 \
  -v $(pwd)/qdrant-data:/qdrant/storage \
  qdrant/qdrant:latest
 # With custom config
 docker run -d \
  --name qdrant \
  -p 6333:6333 \
  -v $(pwd)/qdrant-data:/qdrant/storage \
  -v $(pwd)/qdrant-config.yaml:/qdrant/config/production.yaml \
  qdrant/qdrant:latest
 ```
 ```yaml
 # qdrant-config.yaml
 storage:
  storage_path: /qdrant/storage
  on_disk_payload: true          # store payload on disk (saves RAM)
 service:
  max_request_size_mb: 32
 hnsw_index:
  m: 16                          # graph connections per node
  ef_construct: 100              # accuracy vs build time trade-off
  full_scan_threshold: 10000     # switch to brute force below this
 quantization:
  scalar:
    type: int8
    quantile: 0.99
    always_ram: true             # keep quantized index in RAM
 telemetry_disabled: true
 ```
 ## Qdrant Collection Management
 ```python
 from qdrant_client import QdrantClient
 from qdrant_client.models import (
    Distance, VectorParams, HnswConfigDiff,
    ScalarQuantizationConfig, ScalarType, QuantizationConfig
 )
 client = QdrantClient("http://localhost:6333")
 # Create optimized collection
 client.create_collection(
    collection_name="documents",
    vectors_config=VectorParams(
        size=1536,                         # OpenAI ada-002 / text-embedding-3-small
        distance=Distance.COSINE,
        on_disk=True,                      # save RAM — vectors stored on disk
    ),
    hnsw_config=HnswConfigDiff(
        m=32,                              # higher = better recall, more RAM
        ef_construct=200,
        on_disk=False,                     # keep HNSW graph in RAM for speed
    ),
    quantization_config=QuantizationConfig(
        scalar=ScalarQuantizationConfig(
            type=ScalarType.INT8,
            quantile=0.99,
            always_ram=True,
        )
    ),
 )
 # Create payload index for fast filtering
 client.create_payload_index(
    collection_name="documents",
    field_name="tenant_id",
    field_schema="keyword",
 )
 client.create_payload_index(
    collection_name="documents",
    field_name="created_at",
    field_schema="datetime",
 )
 # Collection info
 info = client.get_collection("documents")
 print(f"Vectors: {info.vectors_count}, Status: {info.status}")
 ```
 ## Qdrant Filtered Search
 ```python
 from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
 # Tenant-isolated search (multi-tenant RAG)
 results = client.query_points(
    collection_name="documents",
    query=query_embedding,
    query_filter=Filter(
        must=[
            FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")),
            FieldCondition(key="doc_type", match=MatchValue(value="contract")),
        ],
        should=[
            FieldCondition(key="created_at", range=Range(gte="2024-01-01")),
        ],
    ),
    limit=10,
    with_payload=True,
 )
 ```
 ## pgvector — PostgreSQL Extension
 ```sql
 -- Enable extension
 CREATE EXTENSION IF NOT EXISTS vector;
 -- Create table with vector column
 CREATE TABLE documents (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content     TEXT NOT NULL,
    embedding   VECTOR(1536),
    metadata    JSONB DEFAULT '{}',
    tenant_id   TEXT NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT NOW()
 );
 -- Create HNSW index (faster queries, more memory)
 CREATE INDEX ON documents
 USING hnsw (embedding vector_cosine_ops)
 WITH (m = 16, ef_construction = 64);
 -- Create IVFFlat index (less memory, slower build)
 -- CREATE INDEX ON documents
 -- USING ivfflat (embedding vector_cosine_ops)
 -- WITH (lists = 100);
 -- Semantic search with metadata filtering
 SELECT id, content, metadata,
       1 - (embedding <=> $1::vector) AS similarity
 FROM documents
 WHERE tenant_id = 'acme-corp'
  AND metadata->>'doc_type' = 'contract'
 ORDER BY embedding <=> $1::vector
 LIMIT 10;
 ```
 ```bash
 # Deploy pgvector via Docker
 docker run -d \
  --name pgvector \
  -e POSTGRES_PASSWORD=secret \
  -e POSTGRES_DB=vectordb \
  -p 5432:5432 \
  -v pgvector-data:/var/lib/postgresql/data \
  pgvector/pgvector:pg16
 ```
 ## Weaviate Deployment
 ```yaml
 # docker-compose for Weaviate
 services:
  weaviate:
    image: semitechnologies/weaviate:latest
    ports:
      - "8080:8080"
      - "50051:50051"
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "false"
      AUTHENTICATION_APIKEY_ENABLED: "true"
      AUTHENTICATION_APIKEY_ALLOWED_KEYS: "${WEAVIATE_API_KEY}"
      AUTHENTICATION_APIKEY_USERS: "admin"
      PERSISTENCE_DATA_PATH: /var/lib/weaviate
      ENABLE_MODULES: text2vec-openai,generative-openai
      OPENAI_APIKEY: "${OPENAI_API_KEY}"
      CLUSTER_HOSTNAME: node1
    volumes:
      - weaviate-data:/var/lib/weaviate
    restart: unless-stopped
 volumes:
  weaviate-data:
 ```
 ## Backup and Restore
 ```bash
 # Qdrant — snapshot backup
 curl -X POST "http://localhost:6333/collections/documents/snapshots"
 # Download snapshot
 curl -O "http://localhost:6333/collections/documents/snapshots/documents-snapshot.snapshot"
 # Restore
 curl -X POST "http://localhost:6333/collections/documents/snapshots/recover" \
  -H "Content-Type: application/json" \
  -d '{"location": "/qdrant/snapshots/documents-snapshot.snapshot"}'
 # pgvector — standard pg_dump
 pg_dump -h localhost -U postgres -d vectordb \
  --table=documents --format=custom > documents-backup.dump
 # Restore
 pg_restore -h localhost -U postgres -d vectordb documents-backup.dump
 ```
 ## Performance Tuning
 ```python
 # Qdrant — optimize collection after bulk load
 client.update_collection(
    collection_name="documents",
    optimizer_config={"indexing_threshold": 0},  # force indexing now
 )
 # Wait for optimization to complete
 import time
 while True:
    info = client.get_collection("documents")
    if info.status.value == "green":
        break
    time.sleep(5)
    print(f"Optimizing... segments: {info.segments_count}")
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | Slow queries | No HNSW index built yet | Wait for indexing; check `status == green` |
 | High RAM usage | Vectors in memory | Enable `on_disk=True` for vectors |
 | Poor recall | Low `ef` search param | Increase `ef` in search request (at query time) |
 | pgvector slow | Using IVFFlat without vacuum | Run `VACUUM ANALYZE documents` |
 | Weaviate OOM | Too many objects | Enable async indexing; increase heap |
 ## Best Practices
 - Use cosine distance for normalized embeddings; dot product for unnormalized.
 - Always create payload indexes on filter fields (`tenant_id`, `doc_type`).
 - For datasets >10M vectors, use `on_disk` vectors + `always_ram` quantization.
 - Benchmark with your actual query patterns before choosing IVFFlat vs HNSW.
 - Snapshot before any bulk delete or migration operation.
 ## Related Skills
 - [rag-infrastructure](../../local-ai/rag-infrastructure/) - Full RAG pipeline
 - [databases](../databases/) - General database management
 - [postgresql](../postgresql/) - pgvector host database ops
@@ -0,0 +1,312 @@
 ---
 name: llm-fine-tuning
 description: Set up infrastructure for fine-tuning LLMs with QLoRA, LoRA, and full fine-tuning using Hugging Face TRL, Axolotl, and distributed training with DeepSpeed or FSDP. Covers dataset prep, training runs, and model export.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # LLM Fine-Tuning Infrastructure
 Train and fine-tune open-source LLMs efficiently — from LoRA on a single GPU to distributed full fine-tuning across multi-node clusters.
 ## When to Use This Skill
 Use this skill when:
 - Fine-tuning an LLM on domain-specific data (legal, medical, code, support)
 - Running QLoRA to fine-tune 70B models on consumer GPUs
 - Setting up distributed training with DeepSpeed or FSDP
 - Exporting fine-tuned adapters for production serving
 - Implementing RLHF, DPO, or instruction tuning pipelines
 ## Prerequisites
 - NVIDIA GPU(s) with 24GB+ VRAM (RTX 4090 / A100 / H100)
 - CUDA 12.1+ and `nvidia-smi` working
 - Python 3.10+ with `pip`
 - Hugging Face account and `HF_TOKEN` for gated models
 - 500GB+ disk for model weights and training data
 ## Quick Start: QLoRA Fine-Tuning
 ```bash
 pip install transformers datasets trl peft bitsandbytes accelerate
 python - <<'EOF'
 from datasets import load_dataset
 from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
 from peft import LoraConfig, get_peft_model
 from trl import SFTTrainer, SFTConfig
 import torch
 model_id = "meta-llama/Llama-3.1-8B-Instruct"
 # 4-bit quantization (QLoRA)
 bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
 )
 model = AutoModelForCausalLM.from_pretrained(
    model_id, quantization_config=bnb_config, device_map="auto"
 )
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 # LoRA configuration
 peft_config = LoraConfig(
    r=16,                    # rank
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
 )
 dataset = load_dataset("your-org/your-dataset", split="train")
 trainer = SFTTrainer(
    model=model,
    args=SFTConfig(
        output_dir="./output",
        num_train_epochs=3,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
        save_strategy="epoch",
        report_to="wandb",
    ),
    train_dataset=dataset,
    peft_config=peft_config,
    processing_class=tokenizer,
 )
 trainer.train()
 trainer.save_model("./fine-tuned-model")
 EOF
 ```
 ## Axolotl (Production Fine-Tuning Framework)
 ```yaml
 # config.yaml — Axolotl QLoRA config for Llama 3.1
 base_model: meta-llama/Llama-3.1-8B-Instruct
 model_type: LlamaForCausalLM
 tokenizer_type: PreTrainedTokenizerFast
 load_in_4bit: true
 adapter: qlora
 lora_r: 32
 lora_alpha: 64
 lora_dropout: 0.05
 lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
 datasets:
  - path: your-org/your-dataset
    type: alpaca              # or sharegpt, chat_template, etc.
 dataset_prepared_path: ./prepared-data
 val_set_size: 0.05
 output_dir: ./output
 sequence_len: 4096
 sample_packing: true         # pack multiple short samples for efficiency
 micro_batch_size: 2
 gradient_accumulation_steps: 8
 num_epochs: 3
 learning_rate: 2e-4
 optimizer: adamw_bnb_8bit
 lr_scheduler: cosine
 warmup_ratio: 0.05
 bf16: true
 flash_attention: true
 logging_steps: 10
 eval_steps: 100
 save_steps: 200
 wandb_project: my-fine-tune
 ```
 ```bash
 # Run with Axolotl
 pip install axolotl[flash-attn,deepspeed]
 accelerate launch -m axolotl.cli.train config.yaml
 ```
 ## Distributed Training with DeepSpeed
 ```json
 // deepspeed_zero3.json — ZeRO Stage 3 (split optimizer + gradients + params)
 {
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {"device": "cpu", "pin_memory": true},
    "offload_param": {"device": "cpu", "pin_memory": true},
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 1e9,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 1e9,
    "stage3_max_reuse_distance": 1e9,
    "gather_16bit_weights_on_model_save": true
  },
  "bf16": {"enabled": true},
  "gradient_clipping": 1.0,
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto"
 }
 ```
 ```bash
 # Launch 4-GPU DeepSpeed training
 deepspeed --num_gpus=4 train.py \
  --deepspeed deepspeed_zero3.json \
  --model_name meta-llama/Llama-3.1-70B-Instruct \
  --output_dir ./output
 ```
 ## DPO / RLHF Alignment
 ```python
 from trl import DPOTrainer, DPOConfig
 from datasets import load_dataset
 # Dataset format: {"prompt": ..., "chosen": ..., "rejected": ...}
 dataset = load_dataset("your-org/preference-data")
 trainer = DPOTrainer(
    model=model,
    ref_model=None,           # None = implicit reference with peft
    args=DPOConfig(
        output_dir="./dpo-output",
        beta=0.1,             # KL divergence weight
        num_train_epochs=1,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=16,
        learning_rate=5e-7,
        bf16=True,
    ),
    train_dataset=dataset["train"],
    peft_config=peft_config,
    processing_class=tokenizer,
 )
 trainer.train()
 ```
 ## Merging LoRA Adapters for Deployment
 ```python
 from peft import PeftModel
 from transformers import AutoModelForCausalLM
 # Load base model in full precision
 base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="cpu",
 )
 # Load and merge LoRA adapter
 model = PeftModel.from_pretrained(base_model, "./fine-tuned-model")
 merged_model = model.merge_and_unload()
 # Save merged model (ready for vLLM serving)
 merged_model.save_pretrained("./merged-model", safe_serialization=True)
 tokenizer.save_pretrained("./merged-model")
 # Push to Hugging Face Hub
 merged_model.push_to_hub("your-org/your-fine-tuned-model")
 ```
 ## Kubernetes Training Job
 ```yaml
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: llm-fine-tune
 spec:
  template:
    spec:
      restartPolicy: OnFailure
      nodeSelector:
        nvidia.com/gpu.product: A100-SXM4-80GB
      containers:
      - name: trainer
        image: nvcr.io/nvidia/pytorch:24.05-py3
        command: ["accelerate", "launch", "-m", "axolotl.cli.train", "/config/config.yaml"]
        resources:
          limits:
            nvidia.com/gpu: "4"
            memory: "320Gi"
          requests:
            nvidia.com/gpu: "4"
        volumeMounts:
        - name: config
          mountPath: /config
        - name: model-cache
          mountPath: /root/.cache/huggingface
        - name: output
          mountPath: /output
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token
        - name: WANDB_API_KEY
          valueFrom:
            secretKeyRef:
              name: wandb-token
              key: key
      volumes:
      - name: config
        configMap:
          name: axolotl-config
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-cache-pvc
      - name: output
        persistentVolumeClaim:
          claimName: training-output-pvc
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | `CUDA out of memory` | Batch too large | Reduce `micro_batch_size`; increase `gradient_accumulation_steps` |
 | Training loss NaN | Learning rate too high | Lower LR to `1e-4` or `5e-5`; add warmup |
 | Slow training | No Flash Attention | Install `flash-attn`; enable `flash_attention: true` |
 | Poor fine-tune quality | Bad data formatting | Validate dataset format; check `sample_packing` compatibility |
 | Adapter merge errors | Mixed quantization | Merge in bf16 on CPU, not in 4-bit |
 ## Best Practices
 - Use Flash Attention 2 — it's 2–4× faster and uses less memory.
 - Monitor training loss/eval loss via W&B or MLflow; overfit = more dropout or less data.
 - Validate with a held-out eval set (5–10%); MMLU or custom evals for quality gates.
 - Start with LoRA r=16 before increasing — higher rank = more parameters, diminishing returns.
 - Use `sample_packing` in Axolotl to maximize GPU utilization on short sequences.
 ## Related Skills
 - [vllm-server](../vllm-server/) - Serve fine-tuned models
 - [gpu-server-management](../../servers/gpu-server-management/) - GPU setup
 - [llm-inference-scaling](../llm-inference-scaling/) - Deploy at scale
 - [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Training pipelines
@@ -0,0 +1,270 @@
 ---
 name: llm-inference-scaling
 description: Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # LLM Inference Scaling
 Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.
 ## When to Use This Skill
 Use this skill when:
 - LLM API traffic is unpredictable and you need to scale up/down automatically
 - Managing a fleet of vLLM or TGI inference pods on Kubernetes
 - Reducing inference costs with spot/preemptible GPU instances
 - Implementing queue-based autoscaling for batch inference jobs
 - Building a multi-model serving platform that shares GPU resources
 ## Prerequisites
 - Kubernetes cluster with GPU nodes (NVIDIA operator installed)
 - KEDA (Kubernetes Event-Driven Autoscaler) installed
 - Prometheus with GPU metrics (`dcgm-exporter` or `gpu-operator`)
 - Helm 3+ for chart deployments
 ## GPU Node Setup
 ```bash
 # Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)
 helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
 helm repo update
 helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=true \
  --set dcgm.enabled=true \
  --set devicePlugin.enabled=true
 # Verify GPU nodes are recognized
 kubectl get nodes -l nvidia.com/gpu.present=true
 kubectl describe node <gpu-node> | grep nvidia
 ```
 ## vLLM Deployment with GPU Resources
 ```yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: vllm-llama-8b
  labels:
    app: vllm
    model: llama-3.1-8b
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
      model: llama-3.1-8b
  template:
    metadata:
      labels:
        app: vllm
        model: llama-3.1-8b
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model"
        - "meta-llama/Llama-3.1-8B-Instruct"
        - "--tensor-parallel-size"
        - "1"
        - "--gpu-memory-utilization"
        - "0.90"
        - "--max-num-seqs"
        - "128"
        resources:
          requests:
            nvidia.com/gpu: "1"
            memory: "20Gi"
            cpu: "4"
          limits:
            nvidia.com/gpu: "1"
            memory: "24Gi"
            cpu: "8"
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token
 ```
 ## KEDA Autoscaling on Prometheus Metrics
 ```yaml
 apiVersion: keda.sh/v1alpha1
 kind: ScaledObject
 metadata:
  name: vllm-scaledobject
 spec:
  scaleTargetRef:
    name: vllm-llama-8b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300          # 5 min before scale-down
  pollingInterval: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_num_requests_waiting
      threshold: "10"           # scale up if >10 requests waiting
      query: |
        sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_gpu_cache_usage
      threshold: "0.8"          # scale up if KV cache >80% full
      query: |
        avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})
 ```
 ## Queue-Based Scaling (Redis + KEDA)
 ```yaml
 # ScaledJob for async batch inference
 apiVersion: keda.sh/v1alpha1
 kind: ScaledJob
 metadata:
  name: llm-batch-inference
 spec:
  jobTargetRef:
    template:
      spec:
        containers:
        - name: inference-worker
          image: myapp/inference-worker:latest
          env:
          - name: REDIS_URL
            value: redis://redis:6379
          - name: QUEUE_NAME
            value: inference-jobs
        restartPolicy: OnFailure
  minReplicaCount: 0
  maxReplicaCount: 20
  pollingInterval: 5
  successfulJobsHistoryLimit: 3
  triggers:
  - type: redis
    metadata:
      address: redis:6379
      listName: inference-jobs
      listLength: "5"           # 1 worker per 5 queued jobs
 ```
 ## Spot Instance Strategy
 ```yaml
 # Mixed node pool: on-demand + spot GPUs
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: cluster-autoscaler-priority-config
 data:
  priorities: |
    10:  # low priority = prefer
    - .*spot.*
    50:
    - .*on-demand.*
 ---
 # Node affinity for spot with on-demand fallback
 spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
          - key: node.kubernetes.io/lifecycle
            operator: In
            values: [spot]
      - weight: 20
        preference:
          matchExpressions:
          - key: node.kubernetes.io/lifecycle
            operator: In
            values: [on-demand]
 ```
 ## Cluster Autoscaler for GPU Nodes
 ```bash
 # AWS EKS — enable cluster autoscaler for GPU node group
 helm install cluster-autoscaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --set autoDiscovery.clusterName=my-cluster \
  --set awsRegion=us-east-1 \
  --set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole \
  --set extraArgs.skip-nodes-with-local-storage=false \
  --set extraArgs.expander=least-waste
 # Annotate GPU node group for autoscaler
 kubectl annotate node <node> \
  cluster-autoscaler.kubernetes.io/safe-to-evict="false"
 ```
 ## Scaling Metrics to Monitor
 ```bash
 # Prometheus queries for scaling decisions
 # Requests waiting in vLLM queue
 sum(vllm:num_requests_waiting) by (model)
 # GPU KV cache utilization (>80% = bottleneck)
 avg(vllm:gpu_cache_usage_perc) by (pod)
 # Tokens per second throughput
 sum(rate(vllm:generation_tokens_total[5m])) by (model)
 # P99 time-to-first-token
 histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | Pods stuck in `Pending` | No GPU nodes available | Check cluster autoscaler logs; verify node group limits |
 | Scale-up too slow | Cluster autoscaler delay + model load time | Pre-warm replicas; increase `minReplicaCount` |
 | GPU fragmentation | Multiple small models on large GPUs | Use MIG partitioning or consolidate model sizes |
 | Spot eviction causes errors | Spot instance reclamation | Add `PodDisruptionBudget`; use graceful shutdown |
 | KEDA not scaling | Prometheus query returns no data | Test query in Prometheus UI first |
 ## Best Practices
 - Set `minReplicaCount: 1` to avoid cold starts; scale to 0 only for batch jobs.
 - Use `PodDisruptionBudget` with `minAvailable: 1` to survive spot evictions.
 - Pre-pull model weights into a shared PVC to speed up pod startup by 5–10×.
 - Separate model families across node pools (A10G for 7B, A100 for 70B).
 - Use Kubernetes VPA for CPU/memory right-sizing alongside KEDA for replica count.
 ## Related Skills
 - [vllm-server](../vllm-server/) - vLLM configuration and tuning
 - [gpu-server-management](../../servers/gpu-server-management/) - GPU node setup
 - [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - KServe
 - [kubernetes-ops](../../../devops/orchestration/kubernetes-ops/) - Core Kubernetes
 - [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost strategies
@@ -0,0 +1,253 @@
 ---
 name: rag-infrastructure
 description: Build and operate Retrieval-Augmented Generation (RAG) infrastructure with vector stores, embedding pipelines, and hybrid search. Covers ingestion, chunking strategies, reranking, and production deployment patterns.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # RAG Infrastructure
 Production infrastructure for Retrieval-Augmented Generation: ingest documents, generate embeddings, store in vector databases, and serve grounded LLM responses.
 ## When to Use This Skill
 Use this skill when:
 - Building a knowledge base Q&A system over internal documents
 - Implementing semantic search over large document collections
 - Reducing LLM hallucinations with retrieved context
 - Setting up embedding pipelines and vector store infrastructure
 - Deploying hybrid search (dense + sparse/BM25)
 ## Prerequisites
 - Python 3.10+ with `pip`
 - A vector database (Qdrant, Weaviate, Pinecone, or pgvector)
 - An embedding model (OpenAI, Cohere, or local via `sentence-transformers`)
 - An LLM endpoint (OpenAI API or self-hosted vLLM)
 - Docker for local vector DB deployment
 ## Architecture Overview
 ```
 Documents → Chunker → Embedder → Vector Store
                                      ↓
 User Query → Embedder → Vector Store (search) → Reranker → LLM → Answer
 ```
 ## Embedding Pipeline
 ```python
 from sentence_transformers import SentenceTransformer
 from qdrant_client import QdrantClient
 from qdrant_client.models import Distance, VectorParams, PointStruct
 import uuid
 # Local embedding model (no API cost)
 model = SentenceTransformer("BAAI/bge-large-en-v1.5")
 # Connect to Qdrant
 client = QdrantClient("http://localhost:6333")
 # Create collection
 client.create_collection(
    collection_name="knowledge-base",
    vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
 )
 def ingest_documents(docs: list[dict]):
    """Chunk, embed, and upsert documents."""
    points = []
    for doc in docs:
        chunks = chunk_text(doc["text"], chunk_size=512, overlap=50)
        embeddings = model.encode(chunks, batch_size=32, show_progress_bar=True)
        for chunk, embedding in zip(chunks, embeddings):
            points.append(PointStruct(
                id=str(uuid.uuid4()),
                vector=embedding.tolist(),
                payload={"text": chunk, "source": doc["source"], "title": doc["title"]},
            ))
    client.upsert(collection_name="knowledge-base", points=points)
    print(f"Ingested {len(points)} chunks")
 ```
 ## Chunking Strategies
 ```python
 from langchain.text_splitter import RecursiveCharacterTextSplitter
 def chunk_text(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]:
    """Recursive character splitter — best general-purpose strategy."""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=overlap,
        separators=["\n\n", "\n", ". ", " ", ""],
    )
    return splitter.split_text(text)
 # For code/markdown — use language-aware splitter
 from langchain.text_splitter import MarkdownHeaderTextSplitter
 headers = [("#", "H1"), ("##", "H2"), ("###", "H3")]
 md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
 ```
 ## Hybrid Search (Dense + Sparse)
 ```python
 from qdrant_client.models import SparseVector, SparseVectorParams, NamedSparseVector
 from fastembed import SparseTextEmbedding
 # Qdrant hybrid collection (dense + BM25 sparse)
 client.create_collection(
    collection_name="hybrid-kb",
    vectors_config={"dense": VectorParams(size=1024, distance=Distance.COSINE)},
    sparse_vectors_config={"sparse": SparseVectorParams()},
 )
 sparse_model = SparseTextEmbedding("prithivida/Splade_PP_en_v1")
 def hybrid_search(query: str, top_k: int = 10) -> list[dict]:
    dense_vec = model.encode(query).tolist()
    sparse_vec = list(sparse_model.embed(query))[0]
    results = client.query_points(
        collection_name="hybrid-kb",
        prefetch=[
            {"query": dense_vec, "using": "dense", "limit": 20},
            {"query": SparseVector(indices=sparse_vec.indices.tolist(),
                                   values=sparse_vec.values.tolist()),
             "using": "sparse", "limit": 20},
        ],
        query={"fusion": "rrf"},   # Reciprocal Rank Fusion
        limit=top_k,
    )
    return [{"text": p.payload["text"], "score": p.score} for p in results.points]
 ```
 ## Reranking
 ```python
 import cohere
 co = cohere.Client("your-api-key")
 def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
    """Rerank retrieved chunks for relevance (improves RAG quality ~20-30%)."""
    response = co.rerank(
        model="rerank-english-v3.0",
        query=query,
        documents=candidates,
        top_n=top_n,
    )
    return [candidates[r.index] for r in response.results]
 # Alternative: local reranker (no API cost)
 from sentence_transformers import CrossEncoder
 reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 def local_rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
    pairs = [[query, c] for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [text for text, _ in ranked[:top_n]]
 ```
 ## RAG Query Pipeline
 ```python
 from openai import OpenAI
 llm = OpenAI(base_url="http://localhost:8000/v1", api_key="your-key")
 def rag_query(user_question: str) -> str:
    # 1. Retrieve
    candidates = hybrid_search(user_question, top_k=20)
    texts = [c["text"] for c in candidates]
    # 2. Rerank
    top_chunks = local_rerank(user_question, texts, top_n=5)
    # 3. Generate
    context = "\n\n---\n\n".join(top_chunks)
    response = llm.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": (
                "Answer the question using only the provided context. "
                "If the answer isn't in the context, say so.\n\nContext:\n" + context
            )},
            {"role": "user", "content": user_question},
        ],
        temperature=0.1,
        max_tokens=1024,
    )
    return response.choices[0].message.content
 ```
 ## Docker Compose: Full RAG Stack
 ```yaml
 services:
  qdrant:
    image: qdrant/qdrant:latest
    volumes:
      - qdrant-data:/qdrant/storage
    ports:
      - "6333:6333"
    restart: unless-stopped
  redis:
    image: redis:7-alpine
    volumes:
      - redis-data:/data
    restart: unless-stopped
  ingestion-worker:
    build: ./ingestion
    environment:
      - QDRANT_URL=http://qdrant:6333
      - REDIS_URL=redis://redis:6379
    depends_on: [qdrant, redis]
    restart: unless-stopped
  rag-api:
    build: ./api
    ports:
      - "8080:8080"
    environment:
      - QDRANT_URL=http://qdrant:6333
      - LLM_BASE_URL=http://vllm:8000/v1
    depends_on: [qdrant]
    restart: unless-stopped
 volumes:
  qdrant-data:
  redis-data:
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | Poor retrieval quality | Chunk size too large | Try 256–512 tokens; overlap 10–15% |
 | LLM ignores retrieved context | Context too long | Rerank and keep top 3–5 chunks |
 | Slow ingestion | Sequential embedding | Use `batch_size=64` and async upserts |
 | Stale documents | No re-ingestion pipeline | Track `doc_hash`; re-embed on change |
 | High embedding costs | All chunks re-embedded | Cache embeddings with hash-based dedup |
 ## Best Practices
 - Use `BAAI/bge-large-en-v1.5` or `nomic-embed-text` for strong free embeddings.
 - Always rerank before passing to LLM — 5 precise chunks beat 20 noisy ones.
 - Store source metadata (URL, page, section) in vector payloads for citations.
 - Use namespace/tenant isolation in the vector store for multi-tenant RAG.
 - Evaluate with RAGAS metrics: faithfulness, answer relevancy, context precision.
 ## Related Skills
 - [vector-database-ops](../../databases/vector-database-ops/) - Qdrant/Weaviate management
 - [vllm-server](../vllm-server/) - Self-hosted LLM endpoint
 - [ollama-stack](../ollama-stack/) - Local LLM for development
 - [ai-pipeline-orchestration](../../../devops/ai/ai-pipeline-orchestration/) - Ingestion pipelines
@@ -0,0 +1,219 @@
 ---
 name: vllm-server
 description: Deploy and manage vLLM for high-throughput LLM inference. Configure continuous batching, tensor parallelism, quantization, and OpenAI-compatible API endpoints for production LLM serving.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # vLLM Server Management
 Deploy production-grade LLM inference servers with vLLM — the fastest open-source LLM serving engine with PagedAttention and continuous batching.
 ## When to Use This Skill
 Use this skill when:
 - Serving open-source LLMs (Llama, Mistral, Qwen, Gemma) at scale
 - Building an OpenAI-compatible API endpoint for self-hosted models
 - Optimizing LLM throughput and latency for production traffic
 - Running multi-GPU inference with tensor or pipeline parallelism
 - Deploying quantized models to reduce GPU memory requirements
 ## Prerequisites
 - NVIDIA GPU(s) with CUDA 12.1+ (A100/H100 recommended for production)
 - Docker or Python 3.9+ with pip
 - 40GB+ VRAM for 70B models; 8GB+ for 7B models
 - `nvidia-container-toolkit` for Docker GPU passthrough
 ## Quick Start
 ```bash
 # Install vLLM
 pip install vllm
 # Serve a model (OpenAI-compatible API)
 vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --api-key your-secret-key
 # Test the endpoint
 curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
 ```
 ## Docker Deployment
 ```bash
 docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --api-key your-secret-key
 ```
 ## Docker Compose (Production)
 ```yaml
 services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    volumes:
      - model-cache:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model meta-llama/Llama-3.1-70B-Instruct
      --tensor-parallel-size 2
      --max-model-len 32768
      --gpu-memory-utilization 0.90
      --api-key ${VLLM_API_KEY}
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
 volumes:
  model-cache:
 ```
 ## Key Configuration Options
 ### Multi-GPU Tensor Parallelism
 ```bash
 # Split one model across 4 GPUs
 vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90
 ```
 ### Quantization (Lower VRAM)
 ```bash
 # AWQ quantization (70B on 2x A100 40GB)
 vllm serve casperhansen/llama-3-70b-instruct-awq \
  --quantization awq \
  --tensor-parallel-size 2
 # GPTQ quantization
 vllm serve TheBloke/Llama-2-70B-Chat-GPTQ \
  --quantization gptq
 # FP8 (H100 NVL native)
 vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --quantization fp8 \
  --tensor-parallel-size 8
 ```
 ### Structured Output & Tools
 ```bash
 vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser llama3_json \
  --guided-decoding-backend outlines
 ```
 ### LoRA Adapters
 ```bash
 vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules sql-lora=/path/to/sql-lora \
                 code-lora=/path/to/code-lora \
  --max-lora-rank 64
 ```
 ## Performance Tuning
 ```bash
 # Maximize throughput for batch workloads
 vllm serve <model> \
  --max-num-seqs 256 \          # max concurrent sequences
  --max-num-batched-tokens 8192 \ # tokens per batch
  --gpu-memory-utilization 0.95 \ # use 95% VRAM
  --swap-space 4                  # CPU swap (GiB)
 # Minimize latency for interactive use
 vllm serve <model> \
  --max-num-seqs 32 \
  --enforce-eager              # disable CUDA graph capture
 ```
 ## Benchmarking
 ```bash
 # Install benchmark tool
 pip install vllm
 # Run throughput benchmark
 python -m vllm.entrypoints.openai.run_batch \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-file prompts.jsonl \
  --output-file results.jsonl
 # Benchmark with vllm bench
 vllm bench throughput \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 1000 \
  --input-len 512 \
  --output-len 128
 ```
 ## Monitoring
 ```bash
 # Check running server stats
 curl http://localhost:8000/metrics  # Prometheus metrics
 # Key metrics to watch:
 # vllm:num_requests_running       - active requests
 # vllm:gpu_cache_usage_perc       - KV cache utilization
 # vllm:generation_tokens_per_s    - throughput
 # vllm:time_to_first_token_ms     - TTFT latency
 # vllm:e2e_request_latency_seconds - end-to-end latency
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | `CUDA out of memory` | Model too large for VRAM | Add `--quantization awq` or reduce `--gpu-memory-utilization` |
 | Slow cold start | Model not cached | Pre-pull with `huggingface-cli download <model>` |
 | Low throughput | Too few concurrent requests | Increase `--max-num-seqs` |
 | KV cache full errors | Context length too long | Set `--max-model-len` lower |
 | `tokenizer error` | Tokenizer mismatch | Use `--tokenizer` to specify correct tokenizer |
 ## Best Practices
 - Use `--gpu-memory-utilization 0.90` to leave headroom for CUDA kernels.
 - Pin model versions with `--revision` for reproducible deployments.
 - Set `HF_HUB_OFFLINE=1` in production to prevent unexpected downloads.
 - Use AWQ or GPTQ quantization before tensor parallelism — lower VRAM first.
 - Enable `--enable-chunked-prefill` for long-context workloads.
 - Monitor `gpu_cache_usage_perc` — above 95% causes queuing.
 ## Related Skills
 - [llm-inference-scaling](../llm-inference-scaling/) - Auto-scaling vLLM deployments
 - [gpu-server-management](../../servers/gpu-server-management/) - GPU driver setup
 - [llm-gateway](../../networking/llm-gateway/) - Load balancing across vLLM instances
 - [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost management
 - [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - K8s deployment
@@ -0,0 +1,266 @@
 ---
 name: llm-gateway
 description: Deploy an API gateway for LLM traffic with load balancing, rate limiting, key management, semantic caching, fallback routing, and cost tracking. Covers LiteLLM Proxy, OpenRouter-compatible setup, and custom Nginx/Traefik patterns.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # LLM Gateway
 A unified API gateway that routes LLM requests across providers and self-hosted models — with rate limiting, cost tracking, caching, and failover.
 ## When to Use This Skill
 Use this skill when:
 - Running multiple LLM backends (OpenAI, Anthropic, vLLM, Ollama) behind a single endpoint
 - Enforcing per-team or per-user rate limits and spend budgets
 - Implementing automatic fallback when a provider is down
 - Adding semantic caching to reduce API costs by 20–50%
 - Centralizing API key management instead of distributing keys to every app
 ## Prerequisites
 - Docker and Docker Compose
 - A PostgreSQL or SQLite database (for LiteLLM state)
 - LLM API keys (OpenAI, Anthropic, etc.) or self-hosted vLLM endpoints
 - Optional: Redis for caching and rate limiting
 ## LiteLLM Proxy — Quick Start
 LiteLLM is the de facto open-source LLM gateway with OpenAI-compatible API.
 ```bash
 # Run with Docker
 docker run -d \
  --name litellm-proxy \
  -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/litellm-config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml \
  --detailed_debug
 ```
 ## LiteLLM Configuration
 ```yaml
 # litellm-config.yaml
 model_list:
  # OpenAI models
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 10000
      tpm: 2000000
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  # Anthropic
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY
  # Self-hosted vLLM instances (load balanced)
  - model_name: llama-3.1-8b
    litellm_params:
      model: openai/meta-llama/Llama-3.1-8B-Instruct
      api_base: http://vllm-1:8000/v1
      api_key: fake                    # vLLM key
  - model_name: llama-3.1-8b
    litellm_params:
      model: openai/meta-llama/Llama-3.1-8B-Instruct
      api_base: http://vllm-2:8000/v1  # second replica — auto load balanced
      api_key: fake
  # Fallback: cheap model if primary fails
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o-mini        # fallback to cheaper model
      api_key: os.environ/OPENAI_API_KEY
 router_settings:
  routing_strategy: least-busy         # or: latency-based, simple-shuffle
  num_retries: 3
  retry_after: 5
  allowed_fails: 2
  cooldown_time: 60
  # Fallback configuration
  fallbacks:
    - gpt-4o: [claude-sonnet-4-6]
    - claude-sonnet-4-6: [gpt-4o]
 litellm_settings:
  # Semantic caching
  cache: true
  cache_params:
    type: redis
    host: redis
    port: 6379
    similarity_threshold: 0.90        # cache if >90% semantic similarity
  # Logging
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]
  langfuse_public_key: os.environ/LANGFUSE_PUBLIC_KEY
  langfuse_secret_key: os.environ/LANGFUSE_SECRET_KEY
 general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: postgresql://litellm:password@postgres:5432/litellm
  store_model_in_db: true
 ```
 ## Docker Compose: Full Gateway Stack
 ```yaml
 services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    volumes:
      - ./litellm-config.yaml:/app/config.yaml
    ports:
      - "4000:4000"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - DATABASE_URL=postgresql://litellm:password@postgres:5432/litellm
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    restart: unless-stopped
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: password
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      retries: 5
    restart: unless-stopped
  redis:
    image: redis:7-alpine
    command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
    volumes:
      - redis-data:/data
    restart: unless-stopped
 volumes:
  postgres-data:
  redis-data:
 ```
 ## Virtual Keys & Rate Limiting
 ```bash
 # Create a virtual API key for a team (via LiteLLM API)
 curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "team_id": "team-backend",
    "key_alias": "backend-team-key",
    "models": ["gpt-4o-mini", "llama-3.1-8b"],
    "max_budget": 100,              # USD limit
    "budget_duration": "monthly",
    "rpm_limit": 100,               # requests per minute
    "tpm_limit": 500000             # tokens per minute
  }'
 # View spend
 curl http://localhost:4000/spend/keys \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
 ```
 ## Nginx Load Balancer (Alternative/Complement)
 ```nginx
 # nginx.conf — round-robin across vLLM replicas
 upstream vllm_backends {
    least_conn;
    server vllm-1:8000 max_fails=3 fail_timeout=30s;
    server vllm-2:8000 max_fails=3 fail_timeout=30s;
    server vllm-3:8000 max_fails=3 fail_timeout=30s;
    keepalive 32;
 }
 server {
    listen 80;
    server_name llm-api.internal;
    # Rate limiting
    limit_req_zone $http_authorization zone=per_key:10m rate=100r/m;
    limit_req zone=per_key burst=20 nodelay;
    location /v1/ {
        proxy_pass http://vllm_backends;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_read_timeout 300s;        # long timeout for streaming
        proxy_buffering off;            # required for SSE streaming
        proxy_cache_bypass 1;
    }
 }
 ```
 ## Monitoring Gateway Health
 ```bash
 # Check LiteLLM health
 curl http://localhost:4000/health
 # Model-level health
 curl http://localhost:4000/health/liveliness
 # Spend by model
 curl http://localhost:4000/spend/models \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
 # Active virtual keys
 curl http://localhost:4000/key/list \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | `ConnectionRefusedError` to backend | Backend not reachable | Check `api_base` URL; verify backend is healthy |
 | Rate limit errors (429) | Budget/RPM exceeded | Increase limits or rotate to fallback model |
 | Slow streaming responses | `proxy_buffering` enabled | Set `proxy_buffering off` in Nginx |
 | Cache miss rate high | Threshold too strict | Lower `similarity_threshold` to `0.85` |
 | Postgres connection errors | DB not ready | Add `depends_on` with `condition: service_healthy` |
 ## Best Practices
 - Use virtual keys per team/app — never expose raw provider API keys.
 - Enable `cache: true` with Redis for repeated or similar queries; can cut costs 30–50%.
 - Set `num_retries: 3` with fallbacks to handle provider outages gracefully.
 - Log all requests to Langfuse or OpenTelemetry for cost attribution and debugging.
 - Use `least-busy` routing strategy for self-hosted models to avoid GPU saturation.
 ## Related Skills
 - [vllm-server](../../local-ai/vllm-server/) - Backend inference server
 - [llm-inference-scaling](../../local-ai/llm-inference-scaling/) - Auto-scaling backends
 - [llm-caching](../../../devops/ai/llm-caching/) - Semantic cache patterns
 - [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost management
@@ -0,0 +1,221 @@
 ---
 name: gpu-server-management
 description: Set up and manage NVIDIA GPU servers for AI workloads — driver installation, CUDA toolkit, container toolkit, MIG partitioning, GPU health monitoring, and multi-GPU configuration for LLM inference and training.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # GPU Server Management
 Provision, configure, and monitor NVIDIA GPU servers for AI inference and training workloads.
 ## When to Use This Skill
 Use this skill when:
 - Setting up a new GPU server for LLM inference or model training
 - Installing or upgrading NVIDIA drivers and CUDA toolkit
 - Configuring Docker with NVIDIA Container Toolkit for GPU workloads
 - Partitioning A100/H100 GPUs with MIG for multi-tenant workloads
 - Troubleshooting GPU errors, driver issues, or thermal throttling
 ## Prerequisites
 - Ubuntu 22.04 LTS (recommended) or RHEL 8/9
 - NVIDIA GPU (A10G, A100, H100, RTX 4090, or L40S recommended)
 - Root or sudo access
 - Internet access for package downloads
 ## Driver Installation (Ubuntu)
 ```bash
 # Remove old drivers
 sudo apt purge -y 'nvidia*' 'cuda*' 'libcuda*'
 sudo apt autoremove -y
 # Add NVIDIA package repository
 distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
 curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
 curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
 sudo apt update
 # Install latest driver (560.x as of 2025)
 sudo apt install -y nvidia-driver-560 cuda-toolkit-12-6
 # Install NVIDIA Container Toolkit (Docker GPU support)
 sudo apt install -y nvidia-container-toolkit
 sudo nvidia-ctk runtime configure --runtime=docker
 sudo systemctl restart docker
 # Verify
 nvidia-smi
 nvcc --version
 docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi
 ```
 ## Post-Install Configuration
 ```bash
 # Enable persistence mode (reduces driver initialization latency)
 sudo nvidia-smi -pm 1
 # Set power limit (reduce heat/noise on inference servers)
 sudo nvidia-smi -pl 350          # watts; check TDP for your GPU model
 # Disable ECC on inference servers (frees ~6% VRAM, less safe)
 sudo nvidia-smi --ecc-config=0   # requires reboot
 # Enable P2P for multi-GPU NVLink training
 sudo nvidia-smi topo -m          # check NVLink topology
 ```
 ## GPU Health Monitoring
 ```bash
 # Real-time monitoring (like htop for GPUs)
 watch -n 1 nvidia-smi
 # Detailed stats
 nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,\
 utilization.memory,memory.used,memory.free,power.draw,clocks.current.graphics \
 --format=csv --loop=1
 # DCGM — production monitoring daemon (for clusters)
 sudo apt install -y datacenter-gpu-manager
 sudo systemctl start dcgm
 dcgmi discovery -l                # list GPUs
 dcgmi diag -r 1                  # quick health check
 dcgmi diag -r 3                  # full diagnostic (takes ~20 min)
 # Check GPU errors (XID errors — important for stability)
 sudo dmesg | grep -i "NVRM\|nvidia\|XID"
 nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total \
  --format=csv,noheader
 ```
 ## Prometheus GPU Metrics (DCGM Exporter)
 ```bash
 # Deploy DCGM Exporter for Prometheus scraping
 docker run -d \
  --name dcgm-exporter \
  --gpus all \
  --cap-add SYS_ADMIN \
  -p 9400:9400 \
  --restart unless-stopped \
  nvcr.io/nvidia/k8s/dcgm-exporter:latest
 # Key metrics exposed:
 # DCGM_FI_DEV_GPU_UTIL          - GPU utilization %
 # DCGM_FI_DEV_MEM_COPY_UTIL     - Memory bandwidth utilization
 # DCGM_FI_DEV_FB_USED           - Framebuffer memory used (MB)
 # DCGM_FI_DEV_SM_CLOCK          - SM clock speed (MHz)
 # DCGM_FI_DEV_GPU_TEMP          - Temperature (°C)
 # DCGM_FI_DEV_POWER_USAGE       - Power draw (W)
 # DCGM_FI_DEV_XID_ERRORS        - XID error count (0 = healthy)
 ```
 ## MIG Partitioning (A100/H100)
 MIG (Multi-Instance GPU) allows slicing one GPU into isolated smaller GPUs.
 ```bash
 # Enable MIG mode (requires reboot or restart of all processes)
 sudo nvidia-smi -mig 1
 sudo systemctl restart nvidia-persistenced
 # List available MIG profiles (A100 80GB example)
 nvidia-smi mig -lgip
 # 1g.10gb   — 1 slice,  10GB (max 7 instances)
 # 2g.20gb   — 2 slices, 20GB (max 3 instances)
 # 3g.40gb   — 3 slices, 40GB (max 2 instances)
 # 7g.80gb   — full GPU, 80GB (max 1 instance)
 # Create MIG instances (e.g., 3× 2g.20gb + 1× 2g.20gb = multi-tenant)
 sudo nvidia-smi mig -cgi 2g.20gb,2g.20gb,2g.20gb,2g.20gb -C
 # List created instances
 nvidia-smi mig -lgi
 nvidia-smi mig -lcgi
 # Use in Docker
 docker run --gpus '"device=MIG-GPU-xxx/0/0"' ...
 # Disable MIG
 sudo nvidia-smi mig -i 0 -dci
 sudo nvidia-smi mig -i 0 -dgi
 sudo nvidia-smi -mig 0
 ```
 ## Kernel & OS Tuning for GPU Servers
 ```bash
 # Increase file descriptor limits
 echo '* soft nofile 1048576' | sudo tee -a /etc/security/limits.conf
 echo '* hard nofile 1048576' | sudo tee -a /etc/security/limits.conf
 # Disable transparent huge pages (reduces latency jitter)
 echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
 echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
 # Persist via rc.local or systemd unit:
 cat <<'EOF' | sudo tee /etc/rc.local
 #!/bin/bash
 echo never > /sys/kernel/mm/transparent_hugepage/enabled
 echo never > /sys/kernel/mm/transparent_hugepage/defrag
 nvidia-smi -pm 1
 exit 0
 EOF
 sudo chmod +x /etc/rc.local
 # PCIe performance mode
 sudo nvidia-smi --auto-boost-default=0
 sudo nvidia-smi --auto-boost-permission=0
 ```
 ## Multi-GPU Topology Check
 ```bash
 # Check NVLink and PCIe topology
 nvidia-smi topo -m
 # Output shows interconnect type:
 # NV4 = NVLink 4.0 (H100 SXM)
 # NV2 = NVLink 2.0 (A100 SXM)
 # PHB = PCIe bus (slower; avoid for tensor parallel training)
 # PIX = same PCIe switch (fast)
 # Bandwidth test between GPUs
 /usr/local/cuda/samples/bin/x86_64/linux/release/p2pBandwidthLatencyTest
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | `nvidia-smi: command not found` | Driver not installed | Follow driver installation steps above |
 | Driver version mismatch | CUDA/driver incompatibility | Check compatibility matrix at developer.nvidia.com |
 | GPU temperature >85°C | Poor airflow or fan failure | Check `nvidia-smi -q -d TEMPERATURE`; reseat cooler |
 | XID 79 errors | GPU hardware error | Run `dcgmi diag -r 3`; may need GPU replacement |
 | `failed to open device` in container | Container toolkit not configured | Run `nvidia-ctk runtime configure --runtime=docker` |
 | Low PCIe bandwidth | Wrong slot or power limit | Check `nvidia-smi -q | grep PCIe`; use x16 slot |
 ## Best Practices
 - Always enable persistence mode (`nvidia-smi -pm 1`) — reduces first-request latency.
 - Monitor XID errors; persistent XID 79/94 indicates hardware failure.
 - For training: use NVLink-connected GPUs; for inference: PCIe is usually fine.
 - Set up DCGM alerts on temperature >80°C and power draw near TDP.
 - Use MIG for multi-tenant inference to provide GPU isolation between models.
 ## Related Skills
 - [vllm-server](../../local-ai/vllm-server/) - LLM inference on GPUs
 - [llm-fine-tuning](../../local-ai/llm-fine-tuning/) - GPU training setup
 - [linux-hardening](../../../security/hardening/linux-hardening/) - Secure the host OS
 - [prometheus-grafana](../../../devops/observability/prometheus-grafana/) - Metrics dashboards
@@ -0,0 +1,318 @@
 ---
 name: ai-security-hardening
 description: Harden AI/LLM deployments against prompt injection, data exfiltration, model theft, and supply chain attacks. Covers input validation, output filtering, access control, model API security, and compliance controls for production AI systems.
 license: MIT
 metadata:
  author: devops-skills
  version: "1.0"
 ---
 # AI Security Hardening
 Secure LLM and AI systems against prompt injection, jailbreaks, data leakage, and supply chain threats in production environments.
 ## When to Use This Skill
 Use this skill when:
 - Deploying an LLM-powered application handling sensitive user data
 - Protecting against prompt injection attacks in AI agents
 - Implementing output filtering and content moderation
 - Securing model weights and API endpoints from theft
 - Achieving SOC2 or ISO 27001 compliance for AI systems
 ## AI-Specific Threat Model
 ```
 Threat                    Risk                          Control
 ─────────────────────────────────────────────────────────────────────
 Prompt injection          System prompt override         Input sanitization, separate context
 Data exfiltration         PII in model outputs           Output filtering, DLP scanning
 Jailbreaking             Policy bypass                  Content moderation, guardrails
 Model theft               Weight extraction via API      Rate limiting, access controls
 Training data poisoning   Backdoored fine-tuned model    Dataset validation, provenance
 Supply chain attack       Malicious model weights        Signature verification, scanning
 Insecure output           XSS/SQLi from LLM response     Output encoding, parameterized queries
 ```
 ## Prompt Injection Defense
 ```python
 import re
 from typing import Optional
 INJECTION_PATTERNS = [
    r"ignore\s+(all\s+)?(previous|prior|above)\s+instructions",
    r"you\s+are\s+now\s+",
    r"new\s+instructions?:",
    r"system\s+prompt",
    r"forget\s+everything",
    r"act\s+as\s+",
    r"jailbreak",
    r"dan\s+mode",
    r"<\s*system\s*>",
    r"\[INST\]",
 ]
 def detect_prompt_injection(user_input: str) -> tuple[bool, Optional[str]]:
    """Return (is_suspicious, matched_pattern)."""
    normalized = user_input.lower().strip()
    for pattern in INJECTION_PATTERNS:
        if re.search(pattern, normalized, re.IGNORECASE):
            return True, pattern
    return False, None
 def sanitize_user_input(user_input: str, max_length: int = 4000) -> str:
    """Sanitize input before passing to LLM."""
    # Truncate
    user_input = user_input[:max_length]
    # Remove null bytes and control characters
    user_input = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', '', user_input)
    # Check for injection
    suspicious, pattern = detect_prompt_injection(user_input)
    if suspicious:
        raise ValueError(f"Potential prompt injection detected: {pattern}")
    return user_input
 ```
 ## Guardrails with NeMo Guardrails
 ```python
 # guardrails.yaml
 from nemoguardrails import RailsConfig, LLMRails
 config = RailsConfig.from_path("./guardrails-config")
 rails = LLMRails(config)
 async def safe_llm_call(user_message: str) -> str:
    response = await rails.generate_async(
        messages=[{"role": "user", "content": user_message}]
    )
    return response["content"]
 ```
 ```yaml
 # guardrails-config/config.yml
 models:
  - type: main
    engine: openai
    model: gpt-4o-mini
 rails:
  input:
    flows:
      - check jailbreak
      - check sensitive data
  output:
    flows:
      - check output for PII
      - check output for harmful content
 ```
 ## Output Filtering & PII Scrubbing
 ```python
 import re
 from presidio_analyzer import AnalyzerEngine
 from presidio_anonymizer import AnonymizerEngine
 analyzer = AnalyzerEngine()
 anonymizer = AnonymizerEngine()
 PII_ENTITIES = ["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD",
                "US_SSN", "IBAN_CODE", "IP_ADDRESS", "LOCATION"]
 def scrub_pii_from_output(text: str) -> str:
    """Remove PII from LLM output before returning to user."""
    results = analyzer.analyze(text=text, entities=PII_ENTITIES, language="en")
    if not results:
        return text
    anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
    return anonymized.text
 def validate_output_safety(output: str) -> bool:
    """Check output doesn't contain prompt injection artifacts."""
    dangerous_patterns = [
        r"<\s*script\s*>",         # XSS
        r"javascript:",             # XSS
        r";\s*(DROP|DELETE|INSERT)",# SQLi
        r"\$\{.*\}",               # template injection
        r"`.*`",                   # command injection in some contexts
    ]
    for pattern in dangerous_patterns:
        if re.search(pattern, output, re.IGNORECASE):
            return False
    return True
 ```
 ## API Security for LLM Endpoints
 ```python
 from fastapi import FastAPI, HTTPException, Depends, Request
 from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
 import jwt
 import time
 from collections import defaultdict
 app = FastAPI()
 security = HTTPBearer()
 # Rate limiting (per API key)
 request_counts = defaultdict(list)
 def rate_limit(api_key: str, max_requests: int = 100, window_seconds: int = 60):
    now = time.time()
    requests = request_counts[api_key]
    # Remove old requests outside window
    request_counts[api_key] = [t for t in requests if now - t < window_seconds]
    if len(request_counts[api_key]) >= max_requests:
        raise HTTPException(status_code=429, detail="Rate limit exceeded")
    request_counts[api_key].append(now)
 async def verify_token(
    credentials: HTTPAuthorizationCredentials = Depends(security)
 ) -> dict:
    try:
        payload = jwt.decode(credentials.credentials, SECRET_KEY, algorithms=["HS256"])
        rate_limit(payload["sub"])
        return payload
    except jwt.ExpiredSignatureError:
        raise HTTPException(status_code=401, detail="Token expired")
    except jwt.InvalidTokenError:
        raise HTTPException(status_code=401, detail="Invalid token")
@app.post("/v1/chat/completions")
 async def chat(request: Request, token: dict = Depends(verify_token)):
    body = await request.json()
    # Input validation
    user_msg = body.get("messages", [{}])[-1].get("content", "")
    try:
        safe_input = sanitize_user_input(user_msg)
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e))
    # Call LLM and scrub output
    response = await call_llm(safe_input, token["scope"])
    response["choices"][0]["message"]["content"] = scrub_pii_from_output(
        response["choices"][0]["message"]["content"]
    )
    return response
 ```
 ## Model Weight Security
 ```bash
 # Verify model weights with SHA-256 hash before loading
 MODEL_DIR="./models/llama-3.1-8b"
 EXPECTED_HASH="sha256:abc123..."
 # Generate hash of downloaded model
 actual_hash=$(find "$MODEL_DIR" -name "*.safetensors" | sort | xargs sha256sum | sha256sum)
 echo "Model hash: $actual_hash"
 # Compare (automate in CI/CD)
 if [ "$actual_hash" != "$EXPECTED_HASH" ]; then
  echo "ERROR: Model hash mismatch — possible tampering!"
  exit 1
 fi
 # Scan model files for embedded malware (ModelScan)
 pip install modelscan
 modelscan scan -p "$MODEL_DIR"
 ```
 ## Network Isolation for AI Services
 ```yaml
 # Kubernetes NetworkPolicy — isolate LLM API
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
  name: llm-api-isolation
  namespace: ai-services
 spec:
  podSelector:
    matchLabels:
      app: vllm
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: backend           # only backend can call LLM
    ports:
    - protocol: TCP
      port: 8000
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: monitoring        # metrics only
    ports:
    - protocol: TCP
      port: 9090
  # Block egress to internet — prevent data exfiltration
  # (allow only internal cluster traffic)
 ```
 ## Audit Logging
 ```python
 import structlog
 from datetime import datetime, timezone
 audit_log = structlog.get_logger("ai.audit")
 def log_llm_interaction(
    user_id: str,
    session_id: str,
    model: str,
    prompt_tokens: int,
    completion_tokens: int,
    was_filtered: bool,
    injection_detected: bool,
 ):
    audit_log.info(
        "llm_interaction",
        timestamp=datetime.now(timezone.utc).isoformat(),
        user_id=user_id,
        session_id=session_id,
        model=model,
        prompt_tokens=prompt_tokens,
        completion_tokens=completion_tokens,
        was_filtered=was_filtered,
        injection_detected=injection_detected,
        # DO NOT log prompt/completion content — PII risk
    )
 ```
 ## Common Issues
 | Issue | Cause | Fix |
 |-------|-------|-----|
 | False positive injection blocks | Overly broad regex | Tune patterns; use ML-based classifier for high-traffic |
 | PII in model outputs | Model trained on PII data | Add Presidio scrubbing to output layer |
 | API key leakage | Keys in logs or responses | Mask keys in logging; use vault for key storage |
 | Model weight tampering | Unverified downloads | Always verify SHA-256; use `modelscan` |
 | Rate limit bypass | Per-IP not per-user | Rate limit on authenticated user ID, not IP |
 ## Best Practices
 - Never log raw prompts or completions — they may contain PII or sensitive data.
 - Treat LLM output as untrusted input — always encode before rendering in HTML.
 - Use network policies to prevent LLM pods from making outbound internet calls.
 - Rotate API keys quarterly; use short-lived JWT tokens for service-to-service auth.
 - Run `modelscan` on any model downloaded from the internet before serving.
 ## Related Skills
 - [hashicorp-vault](../../secrets/hashicorp-vault/) - Secrets management for API keys
 - [network-security](../../network/) - Network-level controls
 - [linux-hardening](../../hardening/linux-hardening/) - Host hardening
 - [agent-observability](../../../devops/ai/agent-observability/) - AI audit logging
 - [llm-gateway](../../../infrastructure/networking/llm-gateway/) - Centralized access control