mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
V2
This commit is contained in:
Toby
@@ -11,19 +11,410 @@ metadata:
|
||||
|
||||
Run resilient and cost-efficient GPU clusters for production AI workloads.
|
||||
|
||||
## Key Capabilities
|
||||
## When to Use This Skill
|
||||
|
||||
- NVIDIA device plugin and GPU operator lifecycle
|
||||
- MIG partitioning for multi-workload efficiency
|
||||
- GPU-aware autoscaling (KEDA/cluster autoscaler)
|
||||
- Node health checks and proactive remediation
|
||||
- Setting up GPU node pools in Kubernetes for AI inference or training
|
||||
- Configuring NVIDIA device plugin and GPU operator
|
||||
- Implementing MIG partitioning to share GPUs across workloads
|
||||
- Building GPU-aware autoscaling policies
|
||||
- Monitoring GPU health with DCGM and Prometheus
|
||||
- Troubleshooting GPU scheduling, driver, or OOM issues
|
||||
|
||||
## Cluster Baseline
|
||||
## Prerequisites
|
||||
|
||||
- Dedicated GPU node pools with taints and tolerations
|
||||
- Runtime class and driver/toolkit compatibility checks
|
||||
- Local SSD or high-throughput network storage for model weights
|
||||
- DCGM metrics exported to Prometheus
|
||||
- Kubernetes 1.28+ cluster with GPU-capable nodes
|
||||
- NVIDIA GPUs (A10, L4, A100, H100, or similar)
|
||||
- NVIDIA drivers installed on nodes (535+ recommended)
|
||||
- Helm 3 for operator and plugin installation
|
||||
- Prometheus stack for metrics collection
|
||||
|
||||
## NVIDIA GPU Operator Installation
|
||||
|
||||
The GPU Operator automates driver, toolkit, device plugin, and DCGM deployment.
|
||||
|
||||
```bash
|
||||
# Add NVIDIA Helm repo
|
||||
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
|
||||
helm repo update
|
||||
|
||||
# Install GPU Operator
|
||||
helm install gpu-operator nvidia/gpu-operator \
|
||||
--namespace gpu-operator \
|
||||
--create-namespace \
|
||||
--set driver.enabled=true \
|
||||
--set toolkit.enabled=true \
|
||||
--set devicePlugin.enabled=true \
|
||||
--set dcgmExporter.enabled=true \
|
||||
--set migManager.enabled=true \
|
||||
--set nodeStatusExporter.enabled=true \
|
||||
--version v24.3.0
|
||||
|
||||
# Verify installation
|
||||
kubectl get pods -n gpu-operator
|
||||
kubectl get nodes -o json | jq '.items[].status.allocatable["nvidia.com/gpu"]'
|
||||
```
|
||||
|
||||
## NVIDIA Device Plugin (Standalone)
|
||||
|
||||
If not using the GPU Operator, deploy the device plugin directly.
|
||||
|
||||
```yaml
|
||||
# nvidia-device-plugin.yaml
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: nvidia-device-plugin
|
||||
namespace: kube-system
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
name: nvidia-device-plugin
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
name: nvidia-device-plugin
|
||||
spec:
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
priorityClassName: system-node-critical
|
||||
containers:
|
||||
- name: nvidia-device-plugin
|
||||
image: nvcr.io/nvidia/k8s-device-plugin:v0.15.0
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
- name: FAIL_ON_INIT_ERROR
|
||||
value: "false"
|
||||
- name: DEVICE_SPLIT_COUNT
|
||||
value: "1"
|
||||
- name: DEVICE_LIST_STRATEGY
|
||||
value: "envvar"
|
||||
volumeMounts:
|
||||
- name: device-plugin
|
||||
mountPath: /var/lib/kubelet/device-plugins
|
||||
volumes:
|
||||
- name: device-plugin
|
||||
hostPath:
|
||||
path: /var/lib/kubelet/device-plugins
|
||||
```
|
||||
|
||||
## MIG (Multi-Instance GPU) Partitioning
|
||||
|
||||
MIG allows a single A100 or H100 to be split into isolated GPU instances.
|
||||
|
||||
```yaml
|
||||
# mig-config.yaml - ConfigMap for MIG Manager
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: mig-parted-config
|
||||
namespace: gpu-operator
|
||||
data:
|
||||
config.yaml: |
|
||||
version: v1
|
||||
mig-configs:
|
||||
# 7 small instances for inference microservices
|
||||
all-1g.10gb:
|
||||
- devices: all
|
||||
mig-enabled: true
|
||||
mig-devices:
|
||||
"1g.10gb": 7
|
||||
|
||||
# 3 medium instances for mid-size models
|
||||
all-2g.20gb:
|
||||
- devices: all
|
||||
mig-enabled: true
|
||||
mig-devices:
|
||||
"2g.20gb": 3
|
||||
|
||||
# Mixed: 1 large + 2 small
|
||||
mixed-inference:
|
||||
- devices: all
|
||||
mig-enabled: true
|
||||
mig-devices:
|
||||
"3g.40gb": 1
|
||||
"1g.10gb": 4
|
||||
|
||||
# Full GPU for training (no partitioning)
|
||||
all-disabled:
|
||||
- devices: all
|
||||
mig-enabled: false
|
||||
```
|
||||
|
||||
```bash
|
||||
# Apply MIG profile to a node
|
||||
kubectl label nodes gpu-node-01 nvidia.com/mig.config=all-1g.10gb --overwrite
|
||||
|
||||
# Verify MIG instances
|
||||
kubectl exec -it nvidia-device-plugin-xxxxx -n kube-system -- nvidia-smi mig -lgi
|
||||
|
||||
# Check available MIG resources
|
||||
kubectl get nodes gpu-node-01 -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com")))'
|
||||
```
|
||||
|
||||
### Requesting MIG Slices in Pods
|
||||
|
||||
```yaml
|
||||
# pod-with-mig.yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: inference-small
|
||||
spec:
|
||||
containers:
|
||||
- name: model
|
||||
image: registry.internal/vllm-server:latest
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/mig-1g.10gb: 1
|
||||
# For medium slice:
|
||||
# nvidia.com/mig-2g.20gb: 1
|
||||
# For large slice:
|
||||
# nvidia.com/mig-3g.40gb: 1
|
||||
```
|
||||
|
||||
## GPU Time-Slicing
|
||||
|
||||
For GPUs that do not support MIG (A10, L4), use time-slicing to share a GPU.
|
||||
|
||||
```yaml
|
||||
# time-slicing-config.yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: time-slicing-config
|
||||
namespace: gpu-operator
|
||||
data:
|
||||
any: |-
|
||||
version: v1
|
||||
flags:
|
||||
migStrategy: none
|
||||
sharing:
|
||||
timeSlicing:
|
||||
renameByDefault: false
|
||||
failRequestsGreaterThanOne: false
|
||||
resources:
|
||||
- name: nvidia.com/gpu
|
||||
replicas: 4
|
||||
```
|
||||
|
||||
```bash
|
||||
# Apply time-slicing config
|
||||
kubectl patch clusterpolicy/cluster-policy \
|
||||
--type merge \
|
||||
-p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"any"}}}}'
|
||||
|
||||
# After applying, each physical GPU appears as 4 virtual GPUs
|
||||
kubectl get nodes -o json | jq '.items[].status.allocatable["nvidia.com/gpu"]'
|
||||
# Output: "4" per physical GPU
|
||||
```
|
||||
|
||||
## DCGM Monitoring
|
||||
|
||||
```yaml
|
||||
# dcgm-servicemonitor.yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: dcgm-exporter
|
||||
namespace: gpu-operator
|
||||
labels:
|
||||
release: prometheus
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nvidia-dcgm-exporter
|
||||
endpoints:
|
||||
- port: gpu-metrics
|
||||
interval: 15s
|
||||
path: /metrics
|
||||
```
|
||||
|
||||
### Key DCGM Metrics and Alert Rules
|
||||
|
||||
```yaml
|
||||
# gpu-alerts.yaml
|
||||
groups:
|
||||
- name: gpu-health
|
||||
rules:
|
||||
- alert: GPUHighTemperature
|
||||
expr: DCGM_FI_DEV_GPU_TEMP > 85
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "GPU {{ $labels.gpu }} temperature above 85C on {{ $labels.node }}"
|
||||
|
||||
- alert: GPUMemoryPressure
|
||||
expr: (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE) > 0.90
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "GPU memory above 90% on {{ $labels.node }} GPU {{ $labels.gpu }}"
|
||||
|
||||
- alert: GPUECCErrors
|
||||
expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h]) > 0
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Double-bit ECC errors detected on {{ $labels.node }} GPU {{ $labels.gpu }}"
|
||||
|
||||
- alert: GPUXidErrors
|
||||
expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Xid error on {{ $labels.node }} GPU {{ $labels.gpu }}: {{ $labels.xid }}"
|
||||
|
||||
- alert: GPULowUtilization
|
||||
expr: DCGM_FI_DEV_GPU_UTIL < 10 and on(pod) kube_pod_status_phase{phase="Running"} == 1
|
||||
for: 30m
|
||||
labels:
|
||||
severity: info
|
||||
annotations:
|
||||
summary: "GPU underutilized on {{ $labels.node }} - consider rightsizing"
|
||||
|
||||
- alert: GPUDriverMismatch
|
||||
expr: count(count by (driver_version)(DCGM_FI_DRIVER_VERSION)) > 1
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Multiple GPU driver versions detected across cluster"
|
||||
```
|
||||
|
||||
## GPU Node Pool Configuration
|
||||
|
||||
```yaml
|
||||
# gpu-nodepool.yaml
|
||||
apiVersion: v1
|
||||
kind: Node
|
||||
metadata:
|
||||
labels:
|
||||
gpu-type: a100
|
||||
gpu-memory: "80gb"
|
||||
gpu-mig-capable: "true"
|
||||
node-role: gpu-inference
|
||||
spec:
|
||||
taints:
|
||||
- key: nvidia.com/gpu
|
||||
value: "true"
|
||||
effect: NoSchedule
|
||||
---
|
||||
# Inference deployment with GPU scheduling
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: llm-inference
|
||||
namespace: ai-serving
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: llm-inference
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: llm-inference
|
||||
spec:
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
nodeSelector:
|
||||
gpu-type: a100
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 100
|
||||
podAffinityTerm:
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: llm-inference
|
||||
topologyKey: kubernetes.io/hostname
|
||||
containers:
|
||||
- name: vllm
|
||||
image: registry.internal/vllm-server:0.4.1
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: 1
|
||||
cpu: "4"
|
||||
memory: "32Gi"
|
||||
limits:
|
||||
nvidia.com/gpu: 1
|
||||
cpu: "8"
|
||||
memory: "64Gi"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: "all"
|
||||
```
|
||||
|
||||
## GPU Autoscaling
|
||||
|
||||
```yaml
|
||||
# gpu-hpa.yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: llm-inference-hpa
|
||||
namespace: ai-serving
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: llm-inference
|
||||
minReplicas: 2
|
||||
maxReplicas: 8
|
||||
metrics:
|
||||
- type: Pods
|
||||
pods:
|
||||
metric:
|
||||
name: DCGM_FI_DEV_GPU_UTIL
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "75"
|
||||
- type: Pods
|
||||
pods:
|
||||
metric:
|
||||
name: inference_queue_depth
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "10"
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 60
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 2
|
||||
periodSeconds: 120
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 1
|
||||
periodSeconds: 300
|
||||
---
|
||||
# Cluster Autoscaler config for GPU node pools
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: cluster-autoscaler-config
|
||||
namespace: kube-system
|
||||
data:
|
||||
config: |
|
||||
expander: priority
|
||||
scale-down-delay-after-add: 10m
|
||||
scale-down-unneeded-time: 10m
|
||||
skip-nodes-with-local-storage: false
|
||||
balance-similar-node-groups: true
|
||||
expendable-pods-priority-cutoff: -10
|
||||
gpu-total:
|
||||
- min: 2
|
||||
max: 16
|
||||
gpu: nvidia.com/gpu
|
||||
```
|
||||
|
||||
## Scheduling Patterns
|
||||
|
||||
@@ -32,27 +423,30 @@ Run resilient and cost-efficient GPU clusters for production AI workloads.
|
||||
- Pin model replicas with anti-affinity for availability.
|
||||
- Reserve headroom for failover and rolling updates.
|
||||
|
||||
## Autoscaling Strategy
|
||||
|
||||
- Scale on queue depth + GPU utilization, not CPU alone.
|
||||
- Warm spare replicas for large model cold-start mitigation.
|
||||
- Cap burst scaling to avoid quota exhaustion.
|
||||
|
||||
## Reliability Checks
|
||||
|
||||
- ECC error and Xid monitoring
|
||||
- GPU memory pressure alerts
|
||||
- Driver mismatch detection during upgrades
|
||||
- Pod preemption impact analysis
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
- Prefer MIG slices for smaller inference services.
|
||||
- Schedule batch jobs in off-peak windows.
|
||||
- Route low-priority traffic to cheaper model tiers.
|
||||
- Use spot/preemptible instances for training workloads.
|
||||
- Monitor GPU utilization and rightsize deployments.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Check | Fix |
|
||||
|---------|-------|-----|
|
||||
| Pod stuck in Pending | `kubectl describe pod` for GPU resource events | Verify node has allocatable GPUs, check taints/tolerations |
|
||||
| CUDA OOM during inference | Model too large for GPU memory | Reduce batch size, use quantization, or use MIG slice |
|
||||
| DCGM metrics missing | ServiceMonitor labels matching | Verify DCGM exporter pod is running and scrape config |
|
||||
| Driver mismatch after upgrade | `nvidia-smi` on each node | Cordon node, drain, upgrade driver, uncordon |
|
||||
| GPU not detected | Device plugin pod logs | Restart device plugin, check NVIDIA container toolkit |
|
||||
| Time-slicing not working | ConfigMap applied but no extra GPUs | Restart device plugin pods after config change |
|
||||
| ECC errors increasing | `nvidia-smi -q -d ECC` | Schedule node drain and hardware replacement |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [llm-inference-scaling](../llm-inference-scaling/) - Autoscale inference workloads
|
||||
- [model-serving-kubernetes](../../../devops/orchestration/model-serving-kubernetes/) - Production model serving patterns
|
||||
- [gpu-server-management](../../servers/gpu-server-management/) - Host-level GPU management fundamentals
|
||||
- [multi-tenant-llm-hosting](../multi-tenant-llm-hosting/) - Multi-tenant GPU sharing
|
||||
- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost optimization strategies
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: mac-mini-llm-lab
|
||||
description: Configure a Mac mini as a reliable local LLM server with remote access, observability, and power-safe operation.
|
||||
description: Configure a Mac mini as a reliable local LLM server with remote access, observability, and power-safe operation. Use when building an always-on private AI inference server on Apple Silicon.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
@@ -11,27 +11,323 @@ metadata:
|
||||
|
||||
Turn a Mac mini into a low-noise, always-on local AI appliance.
|
||||
|
||||
## System Setup
|
||||
## When to Use This Skill
|
||||
|
||||
1. Update macOS and Xcode command line tools.
|
||||
2. Install Homebrew and core packages (`tmux`, `htop`, `ollama`).
|
||||
3. Enable automatic login and restart-after-power-failure.
|
||||
4. Configure Tailscale or WireGuard for remote access.
|
||||
Use this skill when:
|
||||
- Setting up a dedicated local LLM inference server
|
||||
- Building a private AI development environment
|
||||
- Need always-on model serving without cloud costs
|
||||
- Running models that require Apple Silicon unified memory (32-192GB)
|
||||
- Creating a home lab AI server for a small team
|
||||
|
||||
## Reliability Checklist
|
||||
## Prerequisites
|
||||
|
||||
- Keep device on wired Ethernet.
|
||||
- Use UPS for power protection.
|
||||
- Schedule weekly reboot window.
|
||||
- Add launchd service for Ollama auto-start.
|
||||
- Mac mini with Apple Silicon (M2/M3/M4, 16GB+ unified memory recommended)
|
||||
- macOS Sonoma 14+ or Sequoia 15+
|
||||
- Ethernet connection (recommended over Wi-Fi)
|
||||
- UPS for power protection (optional but recommended)
|
||||
|
||||
## Initial System Setup
|
||||
|
||||
```bash
|
||||
# Update macOS
|
||||
softwareupdate --install --all
|
||||
|
||||
# Install Xcode command-line tools
|
||||
xcode-select --install
|
||||
|
||||
# Install Homebrew
|
||||
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
|
||||
|
||||
# Core packages
|
||||
brew install tmux htop btop wget jq git neovim
|
||||
|
||||
# Python environment (for MLX and custom scripts)
|
||||
brew install python@3.12 uv
|
||||
|
||||
# Monitoring
|
||||
brew install prometheus node_exporter
|
||||
```
|
||||
|
||||
## Ollama Setup
|
||||
|
||||
```bash
|
||||
# Install Ollama
|
||||
brew install ollama
|
||||
|
||||
# Pull models based on your RAM
|
||||
# 16GB Mac mini:
|
||||
ollama pull llama3.1:8b
|
||||
ollama pull nomic-embed-text
|
||||
ollama pull codellama:7b
|
||||
|
||||
# 32GB Mac mini:
|
||||
ollama pull llama3.1:8b
|
||||
ollama pull qwen2.5:14b
|
||||
ollama pull deepseek-coder-v2:16b
|
||||
ollama pull nomic-embed-text
|
||||
|
||||
# 64GB+ Mac mini:
|
||||
ollama pull llama3.1:70b
|
||||
ollama pull qwen2.5:32b
|
||||
ollama pull codellama:34b
|
||||
|
||||
# Verify Metal acceleration
|
||||
ollama run llama3.1:8b --verbose
|
||||
# Look for: "metal" in output
|
||||
```
|
||||
|
||||
## MLX Framework (Apple Silicon Native)
|
||||
|
||||
MLX runs models natively on Apple Silicon with excellent performance:
|
||||
|
||||
```bash
|
||||
# Install MLX
|
||||
uv pip install mlx mlx-lm
|
||||
|
||||
# Run a model
|
||||
python3 -c "
|
||||
from mlx_lm import load, generate
|
||||
model, tokenizer = load('mlx-community/Llama-3.1-8B-Instruct-4bit')
|
||||
response = generate(model, tokenizer, prompt='Explain Docker in 3 sentences', max_tokens=200)
|
||||
print(response)
|
||||
"
|
||||
|
||||
# MLX server (OpenAI-compatible API)
|
||||
uv pip install mlx-lm[server]
|
||||
mlx_lm.server --model mlx-community/Llama-3.1-8B-Instruct-4bit --port 8080
|
||||
```
|
||||
|
||||
## Auto-Start with launchd
|
||||
|
||||
```xml
|
||||
<!-- ~/Library/LaunchAgents/com.ollama.serve.plist -->
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>com.ollama.serve</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/opt/homebrew/bin/ollama</string>
|
||||
<string>serve</string>
|
||||
</array>
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>OLLAMA_HOST</key>
|
||||
<string>0.0.0.0</string>
|
||||
<key>OLLAMA_NUM_PARALLEL</key>
|
||||
<string>4</string>
|
||||
<key>OLLAMA_MAX_LOADED_MODELS</key>
|
||||
<string>2</string>
|
||||
<key>OLLAMA_FLASH_ATTENTION</key>
|
||||
<string>1</string>
|
||||
</dict>
|
||||
<key>RunAtLoad</key>
|
||||
<true/>
|
||||
<key>KeepAlive</key>
|
||||
<true/>
|
||||
<key>StandardOutPath</key>
|
||||
<string>/tmp/ollama.log</string>
|
||||
<key>StandardErrorPath</key>
|
||||
<string>/tmp/ollama.err</string>
|
||||
</dict>
|
||||
</plist>
|
||||
```
|
||||
|
||||
```bash
|
||||
# Load the service
|
||||
launchctl load ~/Library/LaunchAgents/com.ollama.serve.plist
|
||||
|
||||
# Check status
|
||||
launchctl list | grep ollama
|
||||
|
||||
# Unload if needed
|
||||
launchctl unload ~/Library/LaunchAgents/com.ollama.serve.plist
|
||||
```
|
||||
|
||||
## Power & Reliability
|
||||
|
||||
```bash
|
||||
# Prevent sleep (keeps running with lid closed on Mac mini)
|
||||
sudo pmset -a disablesleep 1
|
||||
sudo pmset -a sleep 0
|
||||
|
||||
# Auto-restart after power failure
|
||||
sudo pmset -a autorestart 1
|
||||
|
||||
# Schedule weekly reboot (Sunday 4 AM)
|
||||
sudo pmset repeat shutdown MTWRFSU 03:55:00
|
||||
sudo pmset repeat poweron MTWRFSU 04:00:00
|
||||
|
||||
# Check power settings
|
||||
pmset -g
|
||||
```
|
||||
|
||||
## Remote Access
|
||||
|
||||
### Tailscale (Recommended)
|
||||
|
||||
```bash
|
||||
# Install Tailscale for easy secure remote access
|
||||
brew install --cask tailscale
|
||||
|
||||
# Enable from menu bar, authenticate
|
||||
# Access your Mac mini from anywhere: http://mac-mini:11434
|
||||
```
|
||||
|
||||
### SSH Hardening
|
||||
|
||||
```bash
|
||||
# Enable remote login
|
||||
sudo systemsetup -setremotelogin on
|
||||
|
||||
# Edit SSH config
|
||||
sudo nano /etc/ssh/sshd_config
|
||||
# Add:
|
||||
# PasswordAuthentication no
|
||||
# PubkeyAuthentication yes
|
||||
# PermitRootLogin no
|
||||
# AllowUsers yourusername
|
||||
|
||||
# Restart SSH
|
||||
sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist
|
||||
sudo launchctl load /System/Library/LaunchDaemons/ssh.plist
|
||||
```
|
||||
|
||||
### Reverse Proxy with Caddy
|
||||
|
||||
```bash
|
||||
brew install caddy
|
||||
|
||||
# Caddyfile
|
||||
cat > /opt/homebrew/etc/Caddyfile << 'EOF'
|
||||
llm.local:443 {
|
||||
tls internal
|
||||
reverse_proxy localhost:11434
|
||||
|
||||
@api path /v1/*
|
||||
handle @api {
|
||||
reverse_proxy localhost:11434
|
||||
}
|
||||
}
|
||||
|
||||
webui.local:443 {
|
||||
tls internal
|
||||
reverse_proxy localhost:3000
|
||||
}
|
||||
EOF
|
||||
|
||||
brew services start caddy
|
||||
```
|
||||
|
||||
## Open WebUI Setup
|
||||
|
||||
```bash
|
||||
# Run Open WebUI via Docker
|
||||
docker run -d \
|
||||
--name open-webui \
|
||||
-p 3000:8080 \
|
||||
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
|
||||
-e WEBUI_AUTH=true \
|
||||
-v open-webui:/app/backend/data \
|
||||
--restart unless-stopped \
|
||||
ghcr.io/open-webui/open-webui:main
|
||||
|
||||
# Or install Docker first if not available
|
||||
brew install --cask docker
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Health check script
|
||||
cat > ~/scripts/llm-health.sh << 'SCRIPT'
|
||||
#!/bin/bash
|
||||
# Check Ollama
|
||||
if curl -sf http://localhost:11434/api/tags > /dev/null; then
|
||||
echo "$(date): Ollama OK"
|
||||
curl -s http://localhost:11434/api/ps | python3 -m json.tool
|
||||
else
|
||||
echo "$(date): Ollama DOWN"
|
||||
# Restart
|
||||
launchctl kickstart -k gui/$(id -u)/com.ollama.serve
|
||||
fi
|
||||
|
||||
# System stats
|
||||
echo "CPU: $(top -l 1 -n 0 | grep 'CPU usage')"
|
||||
echo "Memory: $(vm_stat | head -5)"
|
||||
echo "Disk: $(df -h / | tail -1)"
|
||||
echo "Thermal: $(sudo powermetrics --samplers smc -n 1 2>/dev/null | grep 'CPU die' || echo 'N/A')"
|
||||
SCRIPT
|
||||
chmod +x ~/scripts/llm-health.sh
|
||||
|
||||
# Schedule health check every 5 minutes
|
||||
# Add to crontab: crontab -e
|
||||
# */5 * * * * ~/scripts/llm-health.sh >> ~/logs/llm-health.log 2>&1
|
||||
```
|
||||
|
||||
### Memory Usage by Model
|
||||
|
||||
| Model | RAM Required | Tokens/sec (M2) | Tokens/sec (M4) |
|
||||
|-------|-------------|-----------------|-----------------|
|
||||
| llama3.1:8b (Q4) | ~5 GB | ~25 t/s | ~45 t/s |
|
||||
| qwen2.5:14b (Q4) | ~9 GB | ~15 t/s | ~30 t/s |
|
||||
| llama3.1:70b (Q4) | ~40 GB | ~5 t/s | ~10 t/s |
|
||||
| nomic-embed-text | ~300 MB | N/A | N/A |
|
||||
| codellama:13b | ~8 GB | ~18 t/s | ~35 t/s |
|
||||
|
||||
## Security Checklist
|
||||
|
||||
- Disable unnecessary sharing services.
|
||||
- Enforce FileVault and strong local admin password.
|
||||
- Restrict SSH to key-based auth only.
|
||||
```bash
|
||||
# Enable FileVault disk encryption
|
||||
sudo fdesetup enable
|
||||
|
||||
# Enable firewall
|
||||
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setglobalstate on
|
||||
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setstealthmode on
|
||||
|
||||
# Disable unnecessary sharing services
|
||||
sudo launchctl disable system/com.apple.screensharing
|
||||
sudo launchctl disable system/com.apple.AirPlayXPCHelper
|
||||
|
||||
# Set strong admin password
|
||||
# System Settings > Users & Groups
|
||||
|
||||
# Restrict Ollama to local network only (if not using Tailscale)
|
||||
# Set OLLAMA_HOST=127.0.0.1 in launchd plist
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
```bash
|
||||
# Increase file descriptor limits for concurrent requests
|
||||
sudo launchctl limit maxfiles 65536 200000
|
||||
|
||||
# Check unified memory pressure
|
||||
memory_pressure
|
||||
|
||||
# Monitor GPU usage (Metal)
|
||||
sudo powermetrics --samplers gpu_power -n 1
|
||||
|
||||
# Optimize for inference (disable Spotlight indexing on model dirs)
|
||||
mdutil -i off ~/.ollama
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|---------|
|
||||
| Model loading slow | First load caches to memory; subsequent loads are fast |
|
||||
| Out of memory | Use smaller quantization (Q4_K_M), reduce `OLLAMA_MAX_LOADED_MODELS` |
|
||||
| Mac sleeping | Run `sudo pmset -a disablesleep 1` |
|
||||
| Ollama not starting | Check `launchctl list | grep ollama`, view `/tmp/ollama.err` |
|
||||
| Slow over Wi-Fi | Use Ethernet; Wi-Fi adds latency to streaming responses |
|
||||
| Thermal throttling | Ensure adequate ventilation, check `powermetrics` |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [ollama-stack](../ollama-stack/) - Local inference software stack
|
||||
- [ssh-configuration](../../servers/ssh-configuration/) - Secure remote shell access
|
||||
- [ollama-stack](../ollama-stack/) — Software stack with Docker Compose and LiteLLM
|
||||
- [ssh-configuration](../../servers/ssh-configuration/) — Secure remote access
|
||||
- [vpn-setup](../../../security/network/vpn-setup/) — Remote access via WireGuard/Tailscale
|
||||
|
||||
@@ -11,6 +11,22 @@ metadata:
|
||||
|
||||
Host many teams/customers on shared inference infrastructure without sacrificing security, performance, or cost governance.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Building an internal LLM platform shared by multiple teams
|
||||
- Hosting LLM inference for external customers with isolation requirements
|
||||
- Implementing per-tenant quotas, billing, and rate limiting
|
||||
- Designing request routing for multi-model, multi-tenant environments
|
||||
- Preventing noisy-neighbor issues on shared GPU infrastructure
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes cluster with GPU node pools
|
||||
- API gateway or LLM gateway (LiteLLM, Envoy, Kong)
|
||||
- Prometheus + Grafana for per-tenant observability
|
||||
- Redis or equivalent for rate limiting state
|
||||
- Billing system or cost attribution database
|
||||
|
||||
## Isolation Model
|
||||
|
||||
- Strong tenant identity on every request
|
||||
@@ -18,6 +34,473 @@ Host many teams/customers on shared inference infrastructure without sacrificing
|
||||
- Namespace or workload isolation for high-risk tenants
|
||||
- Strict data retention and log partitioning controls
|
||||
|
||||
## vLLM Multi-Model Serving
|
||||
|
||||
```yaml
|
||||
# vllm-deployment.yaml - Multi-model serving with vLLM
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: vllm-gpt4o-equivalent
|
||||
namespace: llm-serving
|
||||
labels:
|
||||
app: vllm
|
||||
model-tier: premium
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: vllm
|
||||
model-tier: premium
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: vllm
|
||||
model-tier: premium
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "8080"
|
||||
spec:
|
||||
containers:
|
||||
- name: vllm
|
||||
image: vllm/vllm-openai:v0.4.1
|
||||
args:
|
||||
- "--model=/models/llama-3.1-70b"
|
||||
- "--tensor-parallel-size=2"
|
||||
- "--max-model-len=8192"
|
||||
- "--gpu-memory-utilization=0.90"
|
||||
- "--max-num-seqs=128"
|
||||
- "--enable-prefix-caching"
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
name: inference
|
||||
- containerPort: 8080
|
||||
name: metrics
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: 2
|
||||
cpu: "8"
|
||||
memory: "64Gi"
|
||||
limits:
|
||||
nvidia.com/gpu: 2
|
||||
cpu: "16"
|
||||
memory: "128Gi"
|
||||
volumeMounts:
|
||||
- name: model-weights
|
||||
mountPath: /models
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: model-weights
|
||||
persistentVolumeClaim:
|
||||
claimName: premium-model-weights
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
nodeSelector:
|
||||
gpu-type: a100
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: vllm-economy
|
||||
namespace: llm-serving
|
||||
labels:
|
||||
app: vllm
|
||||
model-tier: economy
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: vllm
|
||||
model-tier: economy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: vllm
|
||||
model-tier: economy
|
||||
spec:
|
||||
containers:
|
||||
- name: vllm
|
||||
image: vllm/vllm-openai:v0.4.1
|
||||
args:
|
||||
- "--model=/models/llama-3.1-8b"
|
||||
- "--max-model-len=4096"
|
||||
- "--gpu-memory-utilization=0.85"
|
||||
- "--max-num-seqs=256"
|
||||
- "--enable-prefix-caching"
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
name: inference
|
||||
- containerPort: 8080
|
||||
name: metrics
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: 1
|
||||
cpu: "4"
|
||||
memory: "32Gi"
|
||||
limits:
|
||||
nvidia.com/gpu: 1
|
||||
cpu: "8"
|
||||
memory: "64Gi"
|
||||
volumeMounts:
|
||||
- name: model-weights
|
||||
mountPath: /models
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: model-weights
|
||||
persistentVolumeClaim:
|
||||
claimName: economy-model-weights
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
```
|
||||
|
||||
## Per-Tenant Quota Configuration
|
||||
|
||||
```yaml
|
||||
# tenant-quotas-configmap.yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: tenant-quotas
|
||||
namespace: llm-serving
|
||||
data:
|
||||
quotas.yaml: |
|
||||
tenants:
|
||||
acme-corp:
|
||||
tier: enterprise
|
||||
models_allowed:
|
||||
- llama-3.1-70b
|
||||
- llama-3.1-8b
|
||||
- nomic-embed-text
|
||||
rate_limits:
|
||||
requests_per_minute: 300
|
||||
tokens_per_minute: 500000
|
||||
concurrent_requests: 50
|
||||
budget:
|
||||
daily_limit_usd: 500.00
|
||||
monthly_limit_usd: 10000.00
|
||||
alert_threshold_percent: 80
|
||||
priority: high
|
||||
|
||||
startup-xyz:
|
||||
tier: standard
|
||||
models_allowed:
|
||||
- llama-3.1-8b
|
||||
- nomic-embed-text
|
||||
rate_limits:
|
||||
requests_per_minute: 60
|
||||
tokens_per_minute: 100000
|
||||
concurrent_requests: 10
|
||||
budget:
|
||||
daily_limit_usd: 50.00
|
||||
monthly_limit_usd: 1000.00
|
||||
alert_threshold_percent: 80
|
||||
priority: medium
|
||||
|
||||
internal-dev:
|
||||
tier: free
|
||||
models_allowed:
|
||||
- llama-3.1-8b
|
||||
rate_limits:
|
||||
requests_per_minute: 20
|
||||
tokens_per_minute: 50000
|
||||
concurrent_requests: 5
|
||||
budget:
|
||||
daily_limit_usd: 10.00
|
||||
monthly_limit_usd: 200.00
|
||||
alert_threshold_percent: 90
|
||||
priority: low
|
||||
```
|
||||
|
||||
## Namespace Isolation for High-Risk Tenants
|
||||
|
||||
```yaml
|
||||
# tenant-namespace.yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: tenant-acme-corp
|
||||
labels:
|
||||
tenant: acme-corp
|
||||
isolation: strict
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: tenant-isolation
|
||||
namespace: tenant-acme-corp
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: llm-gateway
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: llm-serving
|
||||
ports:
|
||||
- port: 8000
|
||||
protocol: TCP
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: kube-dns
|
||||
ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ResourceQuota
|
||||
metadata:
|
||||
name: tenant-quota
|
||||
namespace: tenant-acme-corp
|
||||
spec:
|
||||
hard:
|
||||
requests.cpu: "16"
|
||||
requests.memory: "64Gi"
|
||||
limits.cpu: "32"
|
||||
limits.memory: "128Gi"
|
||||
requests.nvidia.com/gpu: "4"
|
||||
pods: "20"
|
||||
```
|
||||
|
||||
## Request Routing and Rate Limiting
|
||||
|
||||
```python
|
||||
# gateway_router.py
|
||||
"""Multi-tenant request router with rate limiting and model routing."""
|
||||
import time
|
||||
import json
|
||||
import redis
|
||||
from fastapi import FastAPI, HTTPException, Header, Request
|
||||
from typing import Optional
|
||||
import httpx
|
||||
import yaml
|
||||
|
||||
app = FastAPI()
|
||||
redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)
|
||||
|
||||
# Load tenant config
|
||||
with open("/etc/config/quotas.yaml") as f:
|
||||
TENANT_CONFIG = yaml.safe_load(f)["tenants"]
|
||||
|
||||
MODEL_ENDPOINTS = {
|
||||
"llama-3.1-70b": "http://vllm-gpt4o-equivalent:8000",
|
||||
"llama-3.1-8b": "http://vllm-economy:8000",
|
||||
"nomic-embed-text": "http://embedding-service:8000",
|
||||
}
|
||||
|
||||
def check_rate_limit(tenant_id: str, config: dict) -> bool:
|
||||
"""Check and update rate limit for a tenant."""
|
||||
key = f"ratelimit:{tenant_id}:{int(time.time() // 60)}"
|
||||
current = redis_client.incr(key)
|
||||
if current == 1:
|
||||
redis_client.expire(key, 120)
|
||||
return current <= config["rate_limits"]["requests_per_minute"]
|
||||
|
||||
def check_concurrent(tenant_id: str, config: dict) -> bool:
|
||||
"""Check concurrent request limit."""
|
||||
key = f"concurrent:{tenant_id}"
|
||||
current = int(redis_client.get(key) or 0)
|
||||
return current < config["rate_limits"]["concurrent_requests"]
|
||||
|
||||
def check_budget(tenant_id: str, config: dict) -> bool:
|
||||
"""Check if tenant is within daily budget."""
|
||||
key = f"spend:{tenant_id}:{time.strftime('%Y-%m-%d')}"
|
||||
current_spend = float(redis_client.get(key) or 0)
|
||||
return current_spend < config["budget"]["daily_limit_usd"]
|
||||
|
||||
def record_usage(tenant_id: str, model: str, prompt_tokens: int, completion_tokens: int):
|
||||
"""Record token usage and cost for billing."""
|
||||
# Cost rates per 1K tokens
|
||||
rates = {
|
||||
"llama-3.1-70b": {"prompt": 0.004, "completion": 0.012},
|
||||
"llama-3.1-8b": {"prompt": 0.0005, "completion": 0.0015},
|
||||
"nomic-embed-text": {"prompt": 0.0001, "completion": 0.0},
|
||||
}
|
||||
rate = rates.get(model, {"prompt": 0.001, "completion": 0.003})
|
||||
cost = (prompt_tokens * rate["prompt"] + completion_tokens * rate["completion"]) / 1000
|
||||
|
||||
# Update daily spend
|
||||
spend_key = f"spend:{tenant_id}:{time.strftime('%Y-%m-%d')}"
|
||||
redis_client.incrbyfloat(spend_key, cost)
|
||||
redis_client.expire(spend_key, 172800)
|
||||
|
||||
# Record for billing export
|
||||
billing_key = f"billing:{tenant_id}:{time.strftime('%Y-%m')}"
|
||||
redis_client.rpush(billing_key, json.dumps({
|
||||
"timestamp": time.time(),
|
||||
"model": model,
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"completion_tokens": completion_tokens,
|
||||
"cost_usd": cost,
|
||||
}))
|
||||
|
||||
@app.post("/v1/chat/completions")
|
||||
async def chat_completions(
|
||||
request: Request,
|
||||
x_tenant_id: str = Header(...),
|
||||
x_api_key: str = Header(...),
|
||||
):
|
||||
"""Route chat completion request with tenant controls."""
|
||||
if x_tenant_id not in TENANT_CONFIG:
|
||||
raise HTTPException(status_code=403, detail="Unknown tenant")
|
||||
|
||||
config = TENANT_CONFIG[x_tenant_id]
|
||||
body = await request.json()
|
||||
model = body.get("model", "llama-3.1-8b")
|
||||
|
||||
# Check model access
|
||||
if model not in config["models_allowed"]:
|
||||
raise HTTPException(status_code=403, detail=f"Model {model} not allowed for tenant")
|
||||
|
||||
# Check rate limit
|
||||
if not check_rate_limit(x_tenant_id, config):
|
||||
raise HTTPException(status_code=429, detail="Rate limit exceeded")
|
||||
|
||||
# Check concurrent requests
|
||||
if not check_concurrent(x_tenant_id, config):
|
||||
raise HTTPException(status_code=429, detail="Concurrent request limit exceeded")
|
||||
|
||||
# Check budget
|
||||
if not check_budget(x_tenant_id, config):
|
||||
raise HTTPException(status_code=402, detail="Daily budget exceeded")
|
||||
|
||||
# Route to model endpoint
|
||||
endpoint = MODEL_ENDPOINTS.get(model)
|
||||
if not endpoint:
|
||||
raise HTTPException(status_code=404, detail=f"Model {model} not available")
|
||||
|
||||
# Track concurrent requests
|
||||
concurrent_key = f"concurrent:{x_tenant_id}"
|
||||
redis_client.incr(concurrent_key)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
response = await client.post(
|
||||
f"{endpoint}/v1/chat/completions",
|
||||
json=body,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
result = response.json()
|
||||
|
||||
# Record usage
|
||||
usage = result.get("usage", {})
|
||||
record_usage(
|
||||
x_tenant_id, model,
|
||||
usage.get("prompt_tokens", 0),
|
||||
usage.get("completion_tokens", 0),
|
||||
)
|
||||
|
||||
return result
|
||||
finally:
|
||||
redis_client.decr(concurrent_key)
|
||||
```
|
||||
|
||||
## Rate Limiting with Envoy
|
||||
|
||||
```yaml
|
||||
# envoy-ratelimit.yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: envoy-ratelimit-config
|
||||
namespace: llm-serving
|
||||
data:
|
||||
config.yaml: |
|
||||
domain: llm-gateway
|
||||
descriptors:
|
||||
# Per-tenant rate limits
|
||||
- key: tenant_id
|
||||
value: acme-corp
|
||||
rate_limit:
|
||||
unit: minute
|
||||
requests_per_unit: 300
|
||||
- key: tenant_id
|
||||
value: startup-xyz
|
||||
rate_limit:
|
||||
unit: minute
|
||||
requests_per_unit: 60
|
||||
- key: tenant_id
|
||||
value: internal-dev
|
||||
rate_limit:
|
||||
unit: minute
|
||||
requests_per_unit: 20
|
||||
|
||||
# Global rate limit as safety net
|
||||
- key: global
|
||||
rate_limit:
|
||||
unit: second
|
||||
requests_per_unit: 100
|
||||
```
|
||||
|
||||
## Billing Integration
|
||||
|
||||
```python
|
||||
# billing_export.py
|
||||
"""Export tenant usage data for billing systems."""
|
||||
import redis
|
||||
import json
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, List
|
||||
|
||||
redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)
|
||||
|
||||
def generate_tenant_invoice(tenant_id: str, month: str) -> Dict:
|
||||
"""Generate monthly invoice for a tenant."""
|
||||
billing_key = f"billing:{tenant_id}:{month}"
|
||||
records = redis_client.lrange(billing_key, 0, -1)
|
||||
|
||||
usage_by_model = {}
|
||||
total_cost = 0.0
|
||||
total_requests = 0
|
||||
|
||||
for record_json in records:
|
||||
record = json.loads(record_json)
|
||||
model = record["model"]
|
||||
|
||||
if model not in usage_by_model:
|
||||
usage_by_model[model] = {
|
||||
"requests": 0,
|
||||
"prompt_tokens": 0,
|
||||
"completion_tokens": 0,
|
||||
"cost_usd": 0.0,
|
||||
}
|
||||
|
||||
usage_by_model[model]["requests"] += 1
|
||||
usage_by_model[model]["prompt_tokens"] += record["prompt_tokens"]
|
||||
usage_by_model[model]["completion_tokens"] += record["completion_tokens"]
|
||||
usage_by_model[model]["cost_usd"] += record["cost_usd"]
|
||||
|
||||
total_cost += record["cost_usd"]
|
||||
total_requests += 1
|
||||
|
||||
return {
|
||||
"tenant_id": tenant_id,
|
||||
"billing_period": month,
|
||||
"generated_at": datetime.utcnow().isoformat(),
|
||||
"summary": {
|
||||
"total_requests": total_requests,
|
||||
"total_cost_usd": round(total_cost, 4),
|
||||
},
|
||||
"usage_by_model": usage_by_model,
|
||||
}
|
||||
|
||||
def get_tenant_spend_today(tenant_id: str) -> float:
|
||||
"""Get current day spend for budget alerts."""
|
||||
key = f"spend:{tenant_id}:{datetime.utcnow().strftime('%Y-%m-%d')}"
|
||||
return float(redis_client.get(key) or 0)
|
||||
```
|
||||
|
||||
## Noisy-Neighbor Controls
|
||||
|
||||
- Per-tenant RPM/TPM limits
|
||||
@@ -25,13 +508,69 @@ Host many teams/customers on shared inference infrastructure without sacrificing
|
||||
- Fair scheduling with weighted priority classes
|
||||
- Backpressure and graceful degradation policies
|
||||
|
||||
## Billing and Chargeback
|
||||
```yaml
|
||||
# priority-classes.yaml
|
||||
apiVersion: scheduling.k8s.io/v1
|
||||
kind: PriorityClass
|
||||
metadata:
|
||||
name: tenant-enterprise
|
||||
value: 1000
|
||||
globalDefault: false
|
||||
description: "Enterprise tenant workloads"
|
||||
---
|
||||
apiVersion: scheduling.k8s.io/v1
|
||||
kind: PriorityClass
|
||||
metadata:
|
||||
name: tenant-standard
|
||||
value: 500
|
||||
globalDefault: false
|
||||
description: "Standard tenant workloads"
|
||||
---
|
||||
apiVersion: scheduling.k8s.io/v1
|
||||
kind: PriorityClass
|
||||
metadata:
|
||||
name: tenant-free
|
||||
value: 100
|
||||
globalDefault: false
|
||||
description: "Free tier tenant workloads"
|
||||
```
|
||||
|
||||
Track per-tenant:
|
||||
- prompt/completion/cached tokens,
|
||||
- model type and route,
|
||||
- latency and success rate,
|
||||
- cost with markup or internal transfer pricing.
|
||||
## Per-Tenant Monitoring
|
||||
|
||||
```yaml
|
||||
# tenant-alerts.yaml
|
||||
groups:
|
||||
- name: tenant-alerts
|
||||
rules:
|
||||
- alert: TenantBudgetWarning
|
||||
expr: |
|
||||
llm_tenant_daily_spend_usd
|
||||
/ llm_tenant_daily_budget_usd > 0.80
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Tenant {{ $labels.tenant }} at 80% of daily budget"
|
||||
|
||||
- alert: TenantRateLimitHitting
|
||||
expr: |
|
||||
rate(llm_rate_limit_rejections_total[5m]) > 1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: info
|
||||
annotations:
|
||||
summary: "Tenant {{ $labels.tenant }} hitting rate limits"
|
||||
|
||||
- alert: TenantErrorRateHigh
|
||||
expr: |
|
||||
rate(llm_tenant_errors_total[5m])
|
||||
/ rate(llm_tenant_requests_total[5m]) > 0.10
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Tenant {{ $labels.tenant }} error rate above 10%"
|
||||
```
|
||||
|
||||
## Security Baseline
|
||||
|
||||
@@ -48,8 +587,21 @@ Track per-tenant:
|
||||
4. Run tenant-specific load/safety tests.
|
||||
5. Enable production traffic with canary limits.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Check | Fix |
|
||||
|---------|-------|-----|
|
||||
| Tenant getting 429 errors | Rate limit counters in Redis | Increase RPM/TPM limits or upgrade tier |
|
||||
| One tenant slowing others | Concurrent request counts per tenant | Reduce concurrency cap for offending tenant |
|
||||
| Billing data missing | Redis billing keys and export job logs | Check billing export CronJob and Redis connectivity |
|
||||
| Tenant cannot access model | Tenant config in ConfigMap | Add model to `models_allowed` list |
|
||||
| Cross-tenant data leakage | Cache key prefixes and namespace isolation | Ensure cache keys include tenant_id prefix |
|
||||
| Budget alerts not firing | Prometheus scrape targets and alert rules | Verify metric export and Alertmanager config |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [llm-gateway](../../networking/llm-gateway/) - Key management and traffic routing
|
||||
- [llm-cost-optimization](../../../devops/ai/llm-cost-optimization/) - Cost controls and optimization tactics
|
||||
- [zero-trust](../../../security/network/zero-trust/) - Identity-centric network and access patterns
|
||||
- [gpu-kubernetes-operations](../gpu-kubernetes-operations/) - GPU cluster management
|
||||
- [llm-inference-scaling](../llm-inference-scaling/) - Autoscaling inference workloads
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: ollama-stack
|
||||
description: Run local LLM workloads with Ollama, Open WebUI, and GPU-aware tuning for private development environments.
|
||||
description: Run local LLM workloads with Ollama, Open WebUI, and GPU-aware tuning for private development environments. Use when setting up private inference, local AI dev environments, or air-gapped LLM deployments.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
@@ -11,28 +11,353 @@ metadata:
|
||||
|
||||
Deploy a local LLM stack for offline and privacy-first workflows.
|
||||
|
||||
## Minimal Setup
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Setting up private/local LLM inference for development
|
||||
- Building air-gapped AI environments
|
||||
- Running models on personal hardware (Mac, Linux, Windows with GPU)
|
||||
- Creating team-shared inference endpoints
|
||||
- Prototyping before committing to cloud LLM APIs
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- 8 GB+ RAM (16 GB+ recommended for 7B+ models)
|
||||
- For GPU acceleration: NVIDIA GPU with 6 GB+ VRAM, or Apple Silicon Mac
|
||||
- Docker (for containerized deployment)
|
||||
- 20 GB+ disk for model storage
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Install Ollama
|
||||
curl -fsSL https://ollama.com/install.sh | sh
|
||||
|
||||
# Start the server
|
||||
ollama serve
|
||||
|
||||
# Pull and run a model
|
||||
ollama pull llama3.1:8b
|
||||
ollama run llama3.1:8b
|
||||
ollama run llama3.1:8b "Explain Kubernetes pods in one paragraph"
|
||||
|
||||
# List available models
|
||||
ollama list
|
||||
|
||||
# Pull specific quantization
|
||||
ollama pull llama3.1:8b-instruct-q4_K_M
|
||||
```
|
||||
|
||||
## Docker Compose Pattern
|
||||
## Model Selection Guide
|
||||
|
||||
- Ollama container with persistent model volume
|
||||
- Open WebUI for chat interface
|
||||
- Optional LiteLLM proxy for unified API routing
|
||||
| Model | Size | VRAM | Best For |
|
||||
|-------|------|------|----------|
|
||||
| `llama3.1:8b` | 4.7 GB | 6 GB | General chat, coding |
|
||||
| `llama3.1:70b` | 40 GB | 48 GB | Complex reasoning |
|
||||
| `codellama:13b` | 7.4 GB | 10 GB | Code generation |
|
||||
| `mistral:7b` | 4.1 GB | 6 GB | Fast general tasks |
|
||||
| `mixtral:8x7b` | 26 GB | 32 GB | High-quality MoE |
|
||||
| `nomic-embed-text` | 274 MB | 1 GB | Embeddings for RAG |
|
||||
| `llava:13b` | 8 GB | 10 GB | Vision + text |
|
||||
| `deepseek-coder-v2:16b` | 9 GB | 12 GB | Code generation |
|
||||
| `qwen2.5:14b` | 9 GB | 12 GB | Multilingual, reasoning |
|
||||
|
||||
## Best Practices
|
||||
## Docker Compose — Full Stack
|
||||
|
||||
- Pin model versions for reproducibility.
|
||||
- Monitor VRAM, RAM, and swap utilization.
|
||||
- Restrict network exposure to trusted subnets.
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
services:
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
container_name: ollama
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "11434:11434"
|
||||
volumes:
|
||||
- ollama_data:/root/.ollama
|
||||
environment:
|
||||
- OLLAMA_HOST=0.0.0.0
|
||||
- OLLAMA_NUM_PARALLEL=4
|
||||
- OLLAMA_MAX_LOADED_MODELS=2
|
||||
- OLLAMA_FLASH_ATTENTION=1
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
open-webui:
|
||||
image: ghcr.io/open-webui/open-webui:main
|
||||
container_name: open-webui
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "3000:8080"
|
||||
volumes:
|
||||
- webui_data:/app/backend/data
|
||||
environment:
|
||||
- OLLAMA_BASE_URL=http://ollama:11434
|
||||
- WEBUI_AUTH=true
|
||||
- WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY:-change-me-in-production}
|
||||
- DEFAULT_MODELS=llama3.1:8b
|
||||
depends_on:
|
||||
ollama:
|
||||
condition: service_healthy
|
||||
|
||||
litellm:
|
||||
image: ghcr.io/berriai/litellm:main-latest
|
||||
container_name: litellm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "4000:4000"
|
||||
volumes:
|
||||
- ./litellm-config.yaml:/app/config.yaml
|
||||
command: ["--config", "/app/config.yaml"]
|
||||
depends_on:
|
||||
ollama:
|
||||
condition: service_healthy
|
||||
|
||||
volumes:
|
||||
ollama_data:
|
||||
webui_data:
|
||||
```
|
||||
|
||||
### LiteLLM Proxy Config
|
||||
|
||||
```yaml
|
||||
# litellm-config.yaml
|
||||
model_list:
|
||||
- model_name: llama3
|
||||
litellm_params:
|
||||
model: ollama/llama3.1:8b
|
||||
api_base: http://ollama:11434
|
||||
- model_name: codellama
|
||||
litellm_params:
|
||||
model: ollama/codellama:13b
|
||||
api_base: http://ollama:11434
|
||||
- model_name: embeddings
|
||||
litellm_params:
|
||||
model: ollama/nomic-embed-text
|
||||
api_base: http://ollama:11434
|
||||
|
||||
general_settings:
|
||||
master_key: sk-local-dev-key
|
||||
max_budget: 0 # unlimited for local
|
||||
```
|
||||
|
||||
## API Usage
|
||||
|
||||
Ollama exposes an OpenAI-compatible API:
|
||||
|
||||
```bash
|
||||
# Chat completion
|
||||
curl http://localhost:11434/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "llama3.1:8b",
|
||||
"messages": [{"role": "user", "content": "Hello"}],
|
||||
"stream": false
|
||||
}'
|
||||
|
||||
# Embeddings
|
||||
curl http://localhost:11434/v1/embeddings \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "nomic-embed-text",
|
||||
"input": "The quick brown fox"
|
||||
}'
|
||||
|
||||
# List models
|
||||
curl http://localhost:11434/api/tags
|
||||
```
|
||||
|
||||
### Python Client
|
||||
|
||||
```python
|
||||
# pip install ollama
|
||||
import ollama
|
||||
|
||||
# Chat
|
||||
response = ollama.chat(
|
||||
model="llama3.1:8b",
|
||||
messages=[{"role": "user", "content": "Explain Docker in 3 sentences"}],
|
||||
)
|
||||
print(response["message"]["content"])
|
||||
|
||||
# Streaming
|
||||
for chunk in ollama.chat(
|
||||
model="llama3.1:8b",
|
||||
messages=[{"role": "user", "content": "Write a haiku about containers"}],
|
||||
stream=True,
|
||||
):
|
||||
print(chunk["message"]["content"], end="", flush=True)
|
||||
|
||||
# Embeddings
|
||||
result = ollama.embed(model="nomic-embed-text", input="Hello world")
|
||||
print(f"Embedding dimensions: {len(result['embeddings'][0])}")
|
||||
```
|
||||
|
||||
### OpenAI SDK Compatibility
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="llama3.1:8b",
|
||||
messages=[{"role": "user", "content": "Hello"}],
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Custom Modelfiles
|
||||
|
||||
Create specialized models with custom system prompts and parameters:
|
||||
|
||||
```dockerfile
|
||||
# Modelfile.devops-assistant
|
||||
FROM llama3.1:8b
|
||||
|
||||
SYSTEM """You are a DevOps expert assistant. You provide concise, production-ready
|
||||
advice about infrastructure, CI/CD, containers, and cloud services.
|
||||
Always include relevant commands and config examples."""
|
||||
|
||||
PARAMETER temperature 0.3
|
||||
PARAMETER top_p 0.9
|
||||
PARAMETER num_ctx 8192
|
||||
PARAMETER repeat_penalty 1.1
|
||||
```
|
||||
|
||||
```bash
|
||||
# Build and use custom model
|
||||
ollama create devops-assistant -f Modelfile.devops-assistant
|
||||
ollama run devops-assistant "Set up a GitHub Actions workflow for Docker builds"
|
||||
```
|
||||
|
||||
## GPU Configuration
|
||||
|
||||
### NVIDIA
|
||||
|
||||
```bash
|
||||
# Verify GPU access
|
||||
nvidia-smi
|
||||
ollama run llama3.1:8b --verbose # Shows GPU layers loaded
|
||||
|
||||
# Environment tuning
|
||||
export OLLAMA_NUM_PARALLEL=4 # Concurrent requests
|
||||
export OLLAMA_MAX_LOADED_MODELS=2 # Models in VRAM
|
||||
export OLLAMA_FLASH_ATTENTION=1 # Faster attention
|
||||
export CUDA_VISIBLE_DEVICES=0,1 # Multi-GPU
|
||||
```
|
||||
|
||||
### Apple Silicon
|
||||
|
||||
```bash
|
||||
# Metal acceleration is automatic on macOS
|
||||
# Verify with:
|
||||
ollama run llama3.1:8b --verbose
|
||||
# Look for: "metal" in the output
|
||||
|
||||
# Optimize for unified memory
|
||||
export OLLAMA_NUM_PARALLEL=2 # Keep memory headroom
|
||||
export OLLAMA_MAX_LOADED_MODELS=1 # One model at a time on 16GB
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Check running models and memory usage
|
||||
curl http://localhost:11434/api/ps
|
||||
|
||||
# Prometheus metrics (if enabled)
|
||||
curl http://localhost:11434/metrics
|
||||
|
||||
# Quick health check script
|
||||
#!/bin/bash
|
||||
response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:11434/api/tags)
|
||||
if [ "$response" = "200" ]; then
|
||||
echo "Ollama is healthy"
|
||||
curl -s http://localhost:11434/api/ps | python3 -m json.tool
|
||||
else
|
||||
echo "Ollama is down (HTTP $response)"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
## Systemd Service
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/ollama.service
|
||||
[Unit]
|
||||
Description=Ollama LLM Server
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/ollama serve
|
||||
User=ollama
|
||||
Group=ollama
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
Environment="OLLAMA_HOST=0.0.0.0"
|
||||
Environment="OLLAMA_NUM_PARALLEL=4"
|
||||
Environment="OLLAMA_FLASH_ATTENTION=1"
|
||||
LimitNOFILE=65535
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo useradd -r -s /bin/false -m -d /usr/share/ollama ollama
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now ollama
|
||||
sudo systemctl status ollama
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
- Bind to `127.0.0.1` in production (default), use reverse proxy for remote access
|
||||
- Set `WEBUI_AUTH=true` on Open WebUI
|
||||
- Use nginx with TLS for remote access:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name llm.internal.example.com;
|
||||
ssl_certificate /etc/ssl/certs/llm.pem;
|
||||
ssl_certificate_key /etc/ssl/private/llm.key;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:11434;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off; # Required for streaming
|
||||
proxy_read_timeout 600s; # Long model responses
|
||||
allow 10.0.0.0/8;
|
||||
deny all;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|---------|
|
||||
| Model too slow | Use smaller quantization (`q4_K_M`), enable flash attention |
|
||||
| Out of memory | Reduce `num_ctx`, use smaller model, set `OLLAMA_MAX_LOADED_MODELS=1` |
|
||||
| GPU not detected | Check `nvidia-smi`, reinstall CUDA drivers, verify Docker GPU runtime |
|
||||
| Connection refused | Check `OLLAMA_HOST` setting, verify firewall rules |
|
||||
| Model download fails | Check disk space, retry with `ollama pull --insecure` for self-signed registries |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [mac-mini-llm-lab](../mac-mini-llm-lab/) - Apple Silicon optimization
|
||||
- [docker-compose](../../../devops/containers/docker-compose/) - Service orchestration
|
||||
- [mac-mini-llm-lab](../mac-mini-llm-lab/) — Apple Silicon optimization
|
||||
- [docker-compose](../../../devops/containers/docker-compose/) — Service orchestration
|
||||
- [vllm-server](../vllm-server/) — High-throughput production inference
|
||||
- [llm-gateway](../../../infrastructure/networking/llm-gateway/) — Unified API routing
|
||||
|
||||
@@ -9,63 +9,603 @@ metadata:
|
||||
|
||||
# OpenClaw Local + Mac mini Setup
|
||||
|
||||
Use this skill when you want to run [OpenClaw](https://github.com/openclaw/openclaw) on a developer laptop or promote it to a stable Mac mini host.
|
||||
Use this skill when you want to run [OpenClaw](https://github.com/openclaw/openclaw) on a developer laptop or promote it to a stable Mac mini host. Covers cloning and bootstrapping, Docker Compose configuration, Mac mini hardware optimization, networking, monitoring, and production-grade launchd services.
|
||||
|
||||
## Local Setup (any modern dev machine)
|
||||
## When to Use
|
||||
|
||||
1. Clone and enter repository.
|
||||
2. Follow upstream prerequisites from OpenClaw README (runtime, package manager, model/provider requirements).
|
||||
3. Create a local environment file from the example and configure keys/endpoints.
|
||||
4. Install dependencies and run the development command.
|
||||
5. Validate startup by loading the local UI/API health endpoint.
|
||||
- Running OpenClaw as a private, always-on local AI agent
|
||||
- Setting up a dedicated Mac mini as a home-lab AI server
|
||||
- Deploying OpenClaw with Docker Compose for reproducible environments
|
||||
- Optimizing macOS for headless server operation
|
||||
- Monitoring a local AI service for uptime and performance
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- macOS 13 (Ventura) or later on Apple Silicon (M1/M2/M4 Mac mini recommended)
|
||||
- Docker Desktop for Mac or OrbStack installed
|
||||
- Git, Node.js (v18+), and a package manager (npm or pnpm)
|
||||
- API keys for your chosen LLM provider (OpenAI, Anthropic, or local Ollama)
|
||||
- At least 16 GB RAM (32 GB recommended for local model serving)
|
||||
|
||||
## Local Setup (Any Dev Machine)
|
||||
|
||||
### Clone and Bootstrap
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/openclaw/openclaw.git
|
||||
cd openclaw
|
||||
# Follow upstream bootstrap steps in repo docs
|
||||
# cp .env.example .env
|
||||
# <install deps>
|
||||
# <run dev server>
|
||||
|
||||
# Review the upstream README for current prerequisites
|
||||
cat README.md
|
||||
|
||||
# Copy the example environment file
|
||||
cp .env.example .env
|
||||
|
||||
# Edit .env with your provider keys and configuration
|
||||
# At minimum, set the model provider and API key
|
||||
cat > .env << 'ENV'
|
||||
# LLM Provider Configuration
|
||||
OPENAI_API_KEY=sk-your-openai-key-here
|
||||
# Or for Anthropic:
|
||||
# ANTHROPIC_API_KEY=sk-ant-your-key-here
|
||||
# Or for local Ollama:
|
||||
# OLLAMA_BASE_URL=http://localhost:11434
|
||||
|
||||
# Application settings
|
||||
NODE_ENV=development
|
||||
PORT=3000
|
||||
HOST=0.0.0.0
|
||||
LOG_LEVEL=info
|
||||
|
||||
# Database (if applicable)
|
||||
DATABASE_URL=sqlite:./data/openclaw.db
|
||||
ENV
|
||||
```
|
||||
|
||||
## Mac mini Production-ish Setup
|
||||
|
||||
### Host baseline
|
||||
|
||||
- Keep macOS updated and enable automatic security updates.
|
||||
- Use wired Ethernet and a UPS for stability.
|
||||
- Enable FileVault and lock down local admin access.
|
||||
- Configure Tailscale or WireGuard for secure remote admin.
|
||||
|
||||
### Service operation
|
||||
|
||||
- Run OpenClaw in a dedicated user account.
|
||||
- Store secrets in macOS Keychain or a managed secret store (avoid plain-text files in shared folders).
|
||||
- Use `tmux` for manual operation or `launchd` for auto-start on reboot.
|
||||
- Keep logs rotated and monitor disk usage.
|
||||
|
||||
### launchd pattern (example)
|
||||
|
||||
Create `/Library/LaunchDaemons/com.openclaw.service.plist` to run startup command from the OpenClaw directory, then:
|
||||
### Install Dependencies and Run
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
npm install
|
||||
# Or with pnpm:
|
||||
# pnpm install
|
||||
|
||||
# Run database migrations if needed
|
||||
npm run db:migrate
|
||||
|
||||
# Start the development server
|
||||
npm run dev
|
||||
|
||||
# Verify startup
|
||||
curl -s http://localhost:3000/api/health | jq .
|
||||
# Expected: {"status":"ok","version":"..."}
|
||||
```
|
||||
|
||||
### Validate the Setup
|
||||
|
||||
```bash
|
||||
# Check the API health endpoint
|
||||
curl -f http://localhost:3000/api/health
|
||||
|
||||
# Check the UI loads
|
||||
curl -s -o /dev/null -w '%{http_code}' http://localhost:3000/
|
||||
# Expected: 200
|
||||
|
||||
# Run built-in tests if available
|
||||
npm test
|
||||
```
|
||||
|
||||
## Docker Compose Setup
|
||||
|
||||
### docker-compose.yml
|
||||
|
||||
```yaml
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
openclaw:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
image: openclaw:latest
|
||||
container_name: openclaw
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "3000:3000"
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
- NODE_ENV=production
|
||||
- HOST=0.0.0.0
|
||||
- PORT=3000
|
||||
volumes:
|
||||
- openclaw-data:/app/data
|
||||
- ./config:/app/config:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 4G
|
||||
reservations:
|
||||
memory: 1G
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "50m"
|
||||
max-file: "5"
|
||||
|
||||
# Optional: Redis for caching/queues
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: openclaw-redis
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
# Optional: Ollama for local model serving
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
container_name: openclaw-ollama
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "11434:11434"
|
||||
volumes:
|
||||
- ollama-models:/root/.ollama
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 16G
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
openclaw-data:
|
||||
redis-data:
|
||||
ollama-models:
|
||||
```
|
||||
|
||||
### Running with Docker Compose
|
||||
|
||||
```bash
|
||||
# Build and start all services
|
||||
docker compose up -d --build
|
||||
|
||||
# Check service status
|
||||
docker compose ps
|
||||
|
||||
# View logs
|
||||
docker compose logs -f openclaw
|
||||
docker compose logs -f --tail=100 ollama
|
||||
|
||||
# Pull a model into Ollama (if using local models)
|
||||
docker exec openclaw-ollama ollama pull llama3:8b
|
||||
docker exec openclaw-ollama ollama list
|
||||
|
||||
# Restart a single service
|
||||
docker compose restart openclaw
|
||||
|
||||
# Stop everything
|
||||
docker compose down
|
||||
|
||||
# Stop and remove volumes (full reset)
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
## Mac mini Production Setup
|
||||
|
||||
### macOS Hardening and Baseline
|
||||
|
||||
```bash
|
||||
# Enable automatic security updates
|
||||
sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate AutomaticCheckEnabled -bool true
|
||||
sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate AutomaticDownload -bool true
|
||||
sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate CriticalUpdateInstall -bool true
|
||||
|
||||
# Enable FileVault disk encryption
|
||||
sudo fdesetup enable
|
||||
|
||||
# Disable sleep (headless server should never sleep)
|
||||
sudo pmset -a sleep 0
|
||||
sudo pmset -a disksleep 0
|
||||
sudo pmset -a displaysleep 0
|
||||
|
||||
# Enable auto-restart after power failure
|
||||
sudo pmset -a autorestart 1
|
||||
|
||||
# Disable screen saver
|
||||
defaults -currentHost write com.apple.screensaver idleTime 0
|
||||
|
||||
# Set hostname
|
||||
sudo scutil --set ComputerName "openclaw-mini"
|
||||
sudo scutil --set HostName "openclaw-mini"
|
||||
sudo scutil --set LocalHostName "openclaw-mini"
|
||||
|
||||
# Verify power settings
|
||||
pmset -g
|
||||
```
|
||||
|
||||
### Dedicated User Account
|
||||
|
||||
```bash
|
||||
# Create a dedicated service user
|
||||
sudo sysadminctl -addUser openclaw -fullName "OpenClaw Service" -password "temp-change-me" -admin
|
||||
|
||||
# Switch to the service user for setup
|
||||
su - openclaw
|
||||
|
||||
# Clone and configure OpenClaw in the user's home
|
||||
cd ~
|
||||
git clone https://github.com/openclaw/openclaw.git
|
||||
cd openclaw
|
||||
cp .env.example .env
|
||||
# Edit .env with production values
|
||||
```
|
||||
|
||||
### Secrets Management
|
||||
|
||||
```bash
|
||||
# Store API keys in macOS Keychain instead of plaintext .env
|
||||
security add-generic-password -a openclaw -s "OPENAI_API_KEY" -w "sk-your-key-here"
|
||||
security add-generic-password -a openclaw -s "ANTHROPIC_API_KEY" -w "sk-ant-your-key-here"
|
||||
|
||||
# Retrieve a secret from Keychain in scripts
|
||||
OPENAI_API_KEY=$(security find-generic-password -a openclaw -s "OPENAI_API_KEY" -w)
|
||||
export OPENAI_API_KEY
|
||||
|
||||
# Helper script to load secrets from Keychain
|
||||
cat > /Users/openclaw/openclaw/load-secrets.sh << 'SCRIPT'
|
||||
#!/usr/bin/env bash
|
||||
export OPENAI_API_KEY=$(security find-generic-password -a openclaw -s "OPENAI_API_KEY" -w 2>/dev/null)
|
||||
export ANTHROPIC_API_KEY=$(security find-generic-password -a openclaw -s "ANTHROPIC_API_KEY" -w 2>/dev/null)
|
||||
SCRIPT
|
||||
chmod 700 /Users/openclaw/openclaw/load-secrets.sh
|
||||
```
|
||||
|
||||
### launchd Service Configuration
|
||||
|
||||
```xml
|
||||
<!-- /Library/LaunchDaemons/com.openclaw.service.plist -->
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
|
||||
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>com.openclaw.service</string>
|
||||
|
||||
<key>UserName</key>
|
||||
<string>openclaw</string>
|
||||
|
||||
<key>WorkingDirectory</key>
|
||||
<string>/Users/openclaw/openclaw</string>
|
||||
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/bin/bash</string>
|
||||
<string>-c</string>
|
||||
<string>source ./load-secrets.sh && /usr/local/bin/node ./dist/server.js</string>
|
||||
</array>
|
||||
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>NODE_ENV</key>
|
||||
<string>production</string>
|
||||
<key>PORT</key>
|
||||
<string>3000</string>
|
||||
<key>HOST</key>
|
||||
<string>0.0.0.0</string>
|
||||
<key>PATH</key>
|
||||
<string>/usr/local/bin:/usr/bin:/bin</string>
|
||||
</dict>
|
||||
|
||||
<key>RunAtLoad</key>
|
||||
<true/>
|
||||
|
||||
<key>KeepAlive</key>
|
||||
<dict>
|
||||
<key>SuccessfulExit</key>
|
||||
<false/>
|
||||
</dict>
|
||||
|
||||
<key>ThrottleInterval</key>
|
||||
<integer>10</integer>
|
||||
|
||||
<key>StandardOutPath</key>
|
||||
<string>/var/log/openclaw/stdout.log</string>
|
||||
|
||||
<key>StandardErrorPath</key>
|
||||
<string>/var/log/openclaw/stderr.log</string>
|
||||
|
||||
<key>SoftResourceLimits</key>
|
||||
<dict>
|
||||
<key>NumberOfFiles</key>
|
||||
<integer>65536</integer>
|
||||
</dict>
|
||||
</dict>
|
||||
</plist>
|
||||
```
|
||||
|
||||
```bash
|
||||
# Create log directory
|
||||
sudo mkdir -p /var/log/openclaw
|
||||
sudo chown openclaw:staff /var/log/openclaw
|
||||
|
||||
# Load the service
|
||||
sudo launchctl load -w /Library/LaunchDaemons/com.openclaw.service.plist
|
||||
sudo launchctl list | rg openclaw
|
||||
|
||||
# Verify it is running
|
||||
sudo launchctl list | grep openclaw
|
||||
curl -f http://localhost:3000/api/health
|
||||
|
||||
# Stop/start/restart the service
|
||||
sudo launchctl stop com.openclaw.service
|
||||
sudo launchctl start com.openclaw.service
|
||||
|
||||
# Unload the service (disable)
|
||||
sudo launchctl unload /Library/LaunchDaemons/com.openclaw.service.plist
|
||||
|
||||
# View logs
|
||||
tail -f /var/log/openclaw/stdout.log
|
||||
tail -f /var/log/openclaw/stderr.log
|
||||
```
|
||||
|
||||
### Docker Compose via launchd
|
||||
|
||||
```xml
|
||||
<!-- /Library/LaunchDaemons/com.openclaw.docker.plist -->
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
|
||||
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>com.openclaw.docker</string>
|
||||
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/usr/local/bin/docker</string>
|
||||
<string>compose</string>
|
||||
<string>-f</string>
|
||||
<string>/Users/openclaw/openclaw/docker-compose.yml</string>
|
||||
<string>up</string>
|
||||
</array>
|
||||
|
||||
<key>RunAtLoad</key>
|
||||
<true/>
|
||||
|
||||
<key>KeepAlive</key>
|
||||
<true/>
|
||||
|
||||
<key>StandardOutPath</key>
|
||||
<string>/var/log/openclaw/docker-stdout.log</string>
|
||||
|
||||
<key>StandardErrorPath</key>
|
||||
<string>/var/log/openclaw/docker-stderr.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
```
|
||||
|
||||
## Networking
|
||||
|
||||
### Tailscale for Secure Remote Access
|
||||
|
||||
```bash
|
||||
# Install Tailscale on the Mac mini
|
||||
brew install --cask tailscale
|
||||
|
||||
# Authenticate and connect
|
||||
open /Applications/Tailscale.app
|
||||
# Or via CLI:
|
||||
tailscale up --authkey tskey-auth-your-key-here
|
||||
|
||||
# Verify Tailscale IP
|
||||
tailscale ip -4
|
||||
# e.g., 100.64.x.x
|
||||
|
||||
# Access OpenClaw from any Tailscale device
|
||||
curl http://100.64.x.x:3000/api/health
|
||||
|
||||
# Enable MagicDNS for friendly names
|
||||
# Access via: http://openclaw-mini:3000
|
||||
```
|
||||
|
||||
### Nginx Reverse Proxy (Optional)
|
||||
|
||||
```bash
|
||||
# Install nginx via Homebrew
|
||||
brew install nginx
|
||||
|
||||
# Configure reverse proxy
|
||||
cat > /opt/homebrew/etc/nginx/servers/openclaw.conf << 'NGINX'
|
||||
server {
|
||||
listen 80;
|
||||
server_name openclaw-mini openclaw-mini.local;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:3000;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_read_timeout 300s;
|
||||
proxy_send_timeout 300s;
|
||||
}
|
||||
|
||||
# Rate limiting for API endpoints
|
||||
location /api/ {
|
||||
limit_req zone=api burst=20 nodelay;
|
||||
proxy_pass http://127.0.0.1:3000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
NGINX
|
||||
|
||||
# Test and reload nginx
|
||||
nginx -t
|
||||
brew services restart nginx
|
||||
```
|
||||
|
||||
### macOS Firewall
|
||||
|
||||
```bash
|
||||
# Enable the application firewall
|
||||
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setglobalstate on
|
||||
|
||||
# Allow specific apps
|
||||
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/local/bin/node
|
||||
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /opt/homebrew/bin/nginx
|
||||
|
||||
# Block all incoming except allowed
|
||||
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setblockall on
|
||||
|
||||
# Verify
|
||||
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Check Script
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# /Users/openclaw/openclaw/healthcheck.sh
|
||||
set -euo pipefail
|
||||
|
||||
ENDPOINT="http://localhost:3000/api/health"
|
||||
LOGFILE="/var/log/openclaw/healthcheck.log"
|
||||
ALERT_EMAIL="admin@example.com"
|
||||
MAX_FAILURES=3
|
||||
FAILURE_COUNT_FILE="/tmp/openclaw-failures"
|
||||
|
||||
timestamp() { date '+%Y-%m-%d %H:%M:%S'; }
|
||||
|
||||
# Initialize failure counter
|
||||
if [ ! -f "$FAILURE_COUNT_FILE" ]; then
|
||||
echo 0 > "$FAILURE_COUNT_FILE"
|
||||
fi
|
||||
|
||||
if curl -sf --max-time 10 "$ENDPOINT" > /dev/null 2>&1; then
|
||||
echo "$(timestamp) OK" >> "$LOGFILE"
|
||||
echo 0 > "$FAILURE_COUNT_FILE"
|
||||
else
|
||||
FAILURES=$(cat "$FAILURE_COUNT_FILE")
|
||||
FAILURES=$((FAILURES + 1))
|
||||
echo "$FAILURES" > "$FAILURE_COUNT_FILE"
|
||||
echo "$(timestamp) FAIL (count: $FAILURES)" >> "$LOGFILE"
|
||||
|
||||
if [ "$FAILURES" -ge "$MAX_FAILURES" ]; then
|
||||
echo "$(timestamp) ALERT: OpenClaw down for $FAILURES checks" >> "$LOGFILE"
|
||||
# Attempt restart
|
||||
sudo launchctl stop com.openclaw.service
|
||||
sleep 2
|
||||
sudo launchctl start com.openclaw.service
|
||||
echo "$(timestamp) Service restarted" >> "$LOGFILE"
|
||||
echo 0 > "$FAILURE_COUNT_FILE"
|
||||
fi
|
||||
fi
|
||||
```
|
||||
|
||||
```bash
|
||||
# Schedule health checks every 5 minutes via cron
|
||||
crontab -e
|
||||
# Add:
|
||||
# */5 * * * * /Users/openclaw/openclaw/healthcheck.sh
|
||||
```
|
||||
|
||||
### Resource Monitoring
|
||||
|
||||
```bash
|
||||
# Monitor CPU and memory usage of OpenClaw
|
||||
ps aux | grep -E 'node|docker' | grep -v grep
|
||||
|
||||
# Continuous monitoring with top (non-interactive)
|
||||
top -l 1 -s 0 | grep -E 'node|docker'
|
||||
|
||||
# Disk usage check
|
||||
df -h /Users/openclaw
|
||||
du -sh /Users/openclaw/openclaw/data/
|
||||
|
||||
# Docker resource usage
|
||||
docker stats --no-stream openclaw openclaw-redis openclaw-ollama
|
||||
|
||||
# macOS Activity Monitor from CLI
|
||||
sudo powermetrics --samplers cpu_power,gpu_power -n 1
|
||||
```
|
||||
|
||||
### Log Rotation
|
||||
|
||||
```bash
|
||||
# /etc/newsyslog.d/openclaw.conf
|
||||
# logfilename [owner:group] mode count size when flags [/pid_file] [sig_num]
|
||||
/var/log/openclaw/stdout.log openclaw:staff 644 10 5120 * JN
|
||||
/var/log/openclaw/stderr.log openclaw:staff 644 10 5120 * JN
|
||||
/var/log/openclaw/healthcheck.log openclaw:staff 644 10 1024 * JN
|
||||
```
|
||||
|
||||
```bash
|
||||
# Force log rotation
|
||||
sudo newsyslog -F
|
||||
|
||||
# Or use a simple cron-based rotation
|
||||
cat > /Users/openclaw/rotate-logs.sh << 'ROTATE'
|
||||
#!/usr/bin/env bash
|
||||
LOGDIR="/var/log/openclaw"
|
||||
for log in "$LOGDIR"/*.log; do
|
||||
if [ -f "$log" ] && [ "$(stat -f%z "$log")" -gt 52428800 ]; then
|
||||
mv "$log" "${log}.$(date +%Y%m%d%H%M%S)"
|
||||
gzip "${log}."*
|
||||
touch "$log"
|
||||
fi
|
||||
done
|
||||
# Keep only last 10 rotated logs
|
||||
ls -t "$LOGDIR"/*.gz 2>/dev/null | tail -n +11 | xargs rm -f
|
||||
ROTATE
|
||||
chmod +x /Users/openclaw/rotate-logs.sh
|
||||
```
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
- App starts after reboot without manual intervention.
|
||||
- Health check succeeds from local network.
|
||||
- Secrets are not committed and not world-readable.
|
||||
- Access to admin interfaces is restricted to trusted users/devices.
|
||||
- App starts after reboot without manual intervention (`launchctl list | grep openclaw`)
|
||||
- Health check succeeds from local network (`curl -f http://<ip>:3000/api/health`)
|
||||
- Health check succeeds via Tailscale (`curl -f http://100.64.x.x:3000/api/health`)
|
||||
- Secrets are not committed and not world-readable (`ls -la .env`, check `.gitignore`)
|
||||
- Access to admin interfaces is restricted to trusted users/devices
|
||||
- Docker volumes persist across container restarts (`docker compose down && docker compose up -d`)
|
||||
- Log rotation is active and disk usage stays bounded
|
||||
- Automatic restart works after crash (kill the process and verify relaunch)
|
||||
|
||||
## Troubleshooting Quick Hits
|
||||
## Troubleshooting
|
||||
|
||||
- Slow responses: verify model backend availability and local RAM/CPU pressure.
|
||||
- Boot failures: inspect launchd logs and working directory paths.
|
||||
- Auth errors: re-check provider keys, scopes, and endpoint URLs.
|
||||
- Random crashes: pin dependency versions and restart with clean environment.
|
||||
| Symptom | Diagnostic | Fix |
|
||||
|---|---|---|
|
||||
| Slow responses | `top -l 1`, check model backend | Verify RAM/CPU pressure; use a smaller model or remote API |
|
||||
| Boot failures | `sudo launchctl list`, check logs | Inspect `/var/log/openclaw/stderr.log`, fix working directory |
|
||||
| Auth errors | Check `.env` or Keychain secrets | Re-check provider keys, scopes, and endpoint URLs |
|
||||
| Random crashes | `log show --predicate 'process == "node"'` | Pin dependency versions, check for OOM in `dmesg` |
|
||||
| Port 3000 in use | `lsof -i :3000` | Kill conflicting process or change PORT in `.env` |
|
||||
| Docker won't start | `docker info`, `docker compose logs` | Ensure Docker Desktop/OrbStack is running |
|
||||
| Ollama model slow | `docker stats openclaw-ollama` | Allocate more RAM to Docker, use quantized model |
|
||||
| Tailscale unreachable | `tailscale status`, `ping 100.64.x.x` | Re-authenticate with `tailscale up`, check firewall |
|
||||
| Disk full | `df -h`, `du -sh ~/openclaw/data/` | Prune Docker images (`docker system prune`), rotate logs |
|
||||
|
||||
## Related Skills
|
||||
|
||||
|
||||
Reference in New Issue
Block a user