External/DevOps-Security-Agent-Skills
1
0
Fork 0
mirror of https://github.com/BagelHole/DevOps-Security-Agent-Skills.git synced 2026-08-22 12:49:53 +02:00
Code Issues Packages Projects Releases Wiki Activity

V2

Browse Source
This commit is contained in:
Toby
2026-03-24 18:02:50 -04:00
parent 2d209b9258
commit ba9e489584
111 changed files with 48382 additions and 3099 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infrastructure/networking/ai-inference-service-mesh/SKILL.md
+392 -17
View File
@@ -17,35 +17,410 @@ Apply Istio/Linkerd mesh controls to secure and optimize east-west AI traffic ac
- Apply fine-grained traffic policies without app code changes
- Run progressive delivery for model-serving backends
- Observe latency hops for retrieval + generation chains
- Route inference requests by model version, tenant, or priority tier
- Protect expensive GPU-backed services from cascading failures
## Prerequisites
```bash
# Install Istio with production profile
istioctl install --set profile=default \
--set meshConfig.accessLogFile=/dev/stdout \
--set meshConfig.defaultConfig.holdApplicationUntilProxyStarts=true
# Label inference namespace for sidecar injection
kubectl create namespace ai-inference
kubectl label namespace ai-inference istio-injection=enabled
# Verify installation
istioctl verify-install
istioctl analyze -n ai-inference
```
## Core Patterns
### Security
- mTLS strict mode cluster-wide
- AuthorizationPolicy per service account
- Egress policies for approved model endpoints only
### mTLS Strict Mode Cluster-Wide
### Traffic Management
- Canary by header or percentage for new model versions
- Retry budgets tuned for long-running streaming requests
- Circuit breakers to protect overloaded inference backends
```yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
---
# Namespace-level override if needed for gradual rollout
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: ai-inference-mtls
namespace: ai-inference
spec:
mtls:
mode: STRICT
portLevelMtls:
# gRPC inference port
8081:
mode: STRICT
# Prometheus metrics port - allow plaintext scraping
9090:
mode: PERMISSIVE
```
### Resilience
- Outlier detection on failing pods
- Locality-aware routing in multi-zone clusters
- Failover to secondary cluster/provider
### AuthorizationPolicy Per Service Account
```yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: model-server-access
namespace: ai-inference
spec:
selector:
matchLabels:
app: model-server
action: ALLOW
rules:
- from:
- source:
principals:
- "cluster.local/ns/ai-inference/sa/api-gateway"
- "cluster.local/ns/ai-inference/sa/orchestrator"
to:
- operation:
methods: ["POST"]
paths: ["/v1/predict", "/v1/embeddings", "/v2/models/*/infer"]
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: deny-external-to-retriever
namespace: ai-inference
spec:
selector:
matchLabels:
app: vector-retriever
action: DENY
rules:
- from:
- source:
notNamespaces: ["ai-inference"]
```
### Egress Policy for Approved Model Endpoints
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: openai-api
namespace: ai-inference
spec:
hosts:
- api.openai.com
ports:
- number: 443
name: https
protocol: TLS
resolution: DNS
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: openai-api-tls
namespace: ai-inference
spec:
host: api.openai.com
trafficPolicy:
tls:
mode: SIMPLE
connectionPool:
http:
h2UpgradePolicy: UPGRADE
tcp:
maxConnections: 50
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: restrict-egress
namespace: ai-inference
spec:
action: ALLOW
rules:
- to:
- operation:
hosts:
- "api.openai.com"
- "models.anthropic.com"
- "*.blob.core.windows.net"
```
## Traffic Management
### VirtualService for A/B Model Testing
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: model-server
namespace: ai-inference
spec:
hosts:
- model-server
http:
# Route by header for explicit model version selection
- match:
- headers:
x-model-version:
exact: "v2-experimental"
route:
- destination:
host: model-server
subset: v2-experimental
timeout: 120s
# Route by header for A/B test cohort
- match:
- headers:
x-ab-cohort:
exact: "treatment"
route:
- destination:
host: model-server
subset: v2-experimental
weight: 100
timeout: 120s
# Default traffic split: 90/10 canary
- route:
- destination:
host: model-server
subset: v1-stable
weight: 90
- destination:
host: model-server
subset: v2-experimental
weight: 10
timeout: 60s
retries:
attempts: 2
perTryTimeout: 30s
retryOn: unavailable,resource-exhausted
```
### DestinationRule with Subsets
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: model-server
namespace: ai-inference
spec:
host: model-server
trafficPolicy:
connectionPool:
http:
h2UpgradePolicy: UPGRADE
maxRequestsPerConnection: 100
tcp:
maxConnections: 200
connectTimeout: 5s
loadBalancer:
simple: LEAST_REQUEST
subsets:
- name: v1-stable
labels:
version: v1
trafficPolicy:
connectionPool:
http:
maxRequestsPerConnection: 50
- name: v2-experimental
labels:
version: v2
trafficPolicy:
connectionPool:
http:
maxRequestsPerConnection: 20
```
### Circuit Breaking for Inference Backends
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: model-server-circuit-breaker
namespace: ai-inference
spec:
host: model-server
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
connectTimeout: 10s
http:
http1MaxPendingRequests: 50
http2MaxRequests: 200
maxRequestsPerConnection: 10
maxRetries: 3
outlierDetection:
consecutive5xxErrors: 3
interval: 15s
baseEjectionTime: 30s
maxEjectionPercent: 50
minHealthPercent: 30
splitExternalLocalOriginErrors: true
---
# Separate circuit breaker for the vector retriever
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: vector-retriever-circuit-breaker
namespace: ai-inference
spec:
host: vector-retriever
trafficPolicy:
connectionPool:
tcp:
maxConnections: 300
http:
http1MaxPendingRequests: 200
http2MaxRequests: 500
outlierDetection:
consecutive5xxErrors: 5
interval: 10s
baseEjectionTime: 15s
maxEjectionPercent: 30
```
### Retry Budget for Streaming Requests
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: streaming-inference
namespace: ai-inference
spec:
hosts:
- model-server
http:
# Streaming endpoint: no retries, long timeout
- match:
- uri:
prefix: /v1/stream
route:
- destination:
host: model-server
subset: v1-stable
timeout: 300s
retries:
attempts: 0
# Embeddings endpoint: safe to retry, short timeout
- match:
- uri:
prefix: /v1/embeddings
route:
- destination:
host: model-server
subset: v1-stable
timeout: 15s
retries:
attempts: 3
perTryTimeout: 5s
retryOn: 5xx,reset,connect-failure,retriable-status-codes
```
## Resilience
### Locality-Aware Routing
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: model-server-locality
namespace: ai-inference
spec:
host: model-server
trafficPolicy:
loadBalancer:
localityLbSetting:
enabled: true
distribute:
- from: "us-east-1/us-east-1a/*"
to:
"us-east-1/us-east-1a/*": 80
"us-east-1/us-east-1b/*": 20
failover:
- from: us-east-1
to: us-west-2
outlierDetection:
consecutive5xxErrors: 3
interval: 10s
baseEjectionTime: 30s
```
## Observability
- Capture distributed traces across the full AI request path
- Emit service-level and route-level p95/p99 latency
- Segment metrics by model and tenant labels
```yaml
# Telemetry resource for custom metrics on inference services
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: inference-telemetry
namespace: ai-inference
spec:
metrics:
- providers:
- name: prometheus
overrides:
- match:
metric: REQUEST_DURATION
mode: CLIENT_AND_SERVER
tagOverrides:
model_name:
operation: UPSERT
value: "request.headers['x-model-name']"
tenant_id:
operation: UPSERT
value: "request.headers['x-tenant-id']"
tracing:
- providers:
- name: zipkin
randomSamplingPercentage: 10.0
```
### Kiali Dashboard Check
```bash
# Port-forward Kiali
kubectl port-forward svc/kiali -n istio-system 20001:20001 &
# Verify mesh health via API
curl -s http://localhost:20001/kiali/api/namespaces/ai-inference/health | jq .
# Check proxy sync status
istioctl proxy-status -n ai-inference
# Debug a specific pod sidecar config
istioctl proxy-config routes deploy/model-server -n ai-inference -o json
istioctl proxy-config cluster deploy/model-server -n ai-inference
```
## Pitfalls to Avoid
- Aggressive timeouts that break streaming responses
- Blanket retries that amplify expensive generation calls
- Aggressive timeouts that break streaming responses -- set 300s+ for generation endpoints
- Blanket retries that amplify expensive generation calls -- disable retries on non-idempotent routes
- Missing identity boundaries between tenant-facing and internal services
- Forgetting to exempt health check and metrics ports from strict mTLS
- Setting outlier ejection too aggressively on small pools (maxEjectionPercent too high)
- Not using `holdApplicationUntilProxyStarts` causing race conditions on startup
## Related Skills
infrastructure/networking/cdn-setup/SKILL.md
+322 -23
View File
@@ -9,51 +9,350 @@ metadata:
# CDN Setup
Configure content delivery networks.
Configure content delivery networks for fast, reliable global asset delivery with proper caching, invalidation, and security.
## When to Use
- Serving static assets (JS, CSS, images, fonts) globally with low latency.
- Offloading traffic from origin servers to reduce compute costs.
- Adding TLS termination and DDoS protection at the edge.
- Implementing geo-based routing or content restrictions.
- Accelerating API responses with edge caching.
## Prerequisites
- Domain with DNS management access.
- Origin server or S3/R2 bucket with content to serve.
- AWS CLI configured (for CloudFront).
- Cloudflare account with zone configured (for Cloudflare CDN).
- Terraform 1.5+ (for infrastructure-as-code examples).
## AWS CloudFront
### Create a Distribution via CLI
```bash
# Create an S3 origin distribution with OAC (Origin Access Control)
aws cloudfront create-distribution --distribution-config '{
"CallerReference": "my-distribution",
"CallerReference": "my-site-'$(date +%s)'",
"Comment": "Production site CDN",
"Enabled": true,
"Origins": {
"Quantity": 1,
"Items": [{
"Id": "myS3Origin",
"DomainName": "mybucket.s3.amazonaws.com",
"S3OriginConfig": {"OriginAccessIdentity": ""}
"Id": "s3-origin",
"DomainName": "my-bucket.s3.us-east-1.amazonaws.com",
"OriginPath": "",
"S3OriginConfig": {
"OriginAccessIdentity": ""
},
"OriginAccessControlId": "E2QWRUHAPOMQZL"
}]
},
"DefaultCacheBehavior": {
"TargetOriginId": "myS3Origin",
"TargetOriginId": "s3-origin",
"ViewerProtocolPolicy": "redirect-to-https",
"CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6"
"AllowedMethods": {
"Quantity": 2,
"Items": ["GET", "HEAD"]
},
"CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6",
"Compress": true
},
"Enabled": true
"DefaultRootObject": "index.html",
"PriceClass": "PriceClass_100",
"ViewerCertificate": {
"ACMCertificateArn": "arn:aws:acm:us-east-1:123456789:certificate/abc-123",
"SSLSupportMethod": "sni-only",
"MinimumProtocolVersion": "TLSv1.2_2021"
},
"Aliases": {
"Quantity": 1,
"Items": ["www.example.com"]
},
"CustomErrorResponses": {
"Quantity": 1,
"Items": [{
"ErrorCode": 404,
"ResponseCode": "200",
"ResponsePagePath": "/index.html",
"ErrorCachingMinTTL": 10
}]
}
}'
```
## Cloudflare
### Cache Invalidation
```bash
# Via API
curl -X POST "https://api.cloudflare.com/client/v4/zones" \
-H "Authorization: Bearer $TOKEN" \
-d '{"name":"example.com","jump_start":true}'
# Invalidate specific paths
aws cloudfront create-invalidation \
--distribution-id E1A2B3C4D5E6F7 \
--paths "/index.html" "/css/*" "/js/*"
# Invalidate everything (costs apply per path)
aws cloudfront create-invalidation \
--distribution-id E1A2B3C4D5E6F7 \
--paths "/*"
# Check invalidation status
aws cloudfront get-invalidation \
--distribution-id E1A2B3C4D5E6F7 \
--id I1A2B3C4D5E6F7
# List recent invalidations
aws cloudfront list-invalidations --distribution-id E1A2B3C4D5E6F7
```
## Cache Headers
### CloudFront Functions (Lightweight Edge Logic)
```nginx
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
expires 30d;
add_header Cache-Control "public, immutable";
```javascript
// URL rewrite function — add index.html to directory requests
function handler(event) {
var request = event.request;
var uri = request.uri;
if (uri.endsWith('/')) {
request.uri += 'index.html';
} else if (!uri.includes('.')) {
request.uri += '/index.html';
}
return request;
}
```
## Best Practices
### CloudFront with Terraform
- Set appropriate cache headers
- Use cache invalidation sparingly
- Implement cache warming
- Monitor cache hit ratios
```hcl
# cloudfront.tf
resource "aws_cloudfront_distribution" "site" {
enabled = true
is_ipv6_enabled = true
default_root_object = "index.html"
aliases = ["www.example.com"]
price_class = "PriceClass_100"
origin {
domain_name = aws_s3_bucket.site.bucket_regional_domain_name
origin_id = "s3-origin"
origin_access_control_id = aws_cloudfront_origin_access_control.oac.id
}
default_cache_behavior {
allowed_methods = ["GET", "HEAD"]
cached_methods = ["GET", "HEAD"]
target_origin_id = "s3-origin"
cache_policy_id = "658327ea-f89d-4fab-a63d-7e88639e58f6" # CachingOptimized
origin_request_policy_id = "88a5eaf4-2fd4-4709-b370-b4c650ea3fcf" # CORS-S3Origin
viewer_protocol_policy = "redirect-to-https"
compress = true
}
# SPA fallback
custom_error_response {
error_code = 404
response_code = 200
response_page_path = "/index.html"
}
# API pass-through (no caching)
ordered_cache_behavior {
path_pattern = "/api/*"
allowed_methods = ["GET", "HEAD", "OPTIONS", "PUT", "POST", "PATCH", "DELETE"]
cached_methods = ["GET", "HEAD"]
target_origin_id = "api-origin"
cache_policy_id = "4135ea2d-6df8-44a3-9df3-4b5a84be39ad" # CachingDisabled
origin_request_policy_id = "b689b0a8-53d0-40ab-baf2-68738e2966ac" # AllViewerExceptHostHeader
viewer_protocol_policy = "https-only"
}
viewer_certificate {
acm_certificate_arn = aws_acm_certificate.cert.arn
ssl_support_method = "sni-only"
minimum_protocol_version = "TLSv1.2_2021"
}
restrictions {
geo_restriction {
restriction_type = "none"
}
}
}
resource "aws_cloudfront_origin_access_control" "oac" {
name = "s3-oac"
origin_access_control_origin_type = "s3"
signing_behavior = "always"
signing_protocol = "sigv4"
}
```
## Cloudflare CDN
### Zone Setup
```bash
# Add a zone
curl -X POST "https://api.cloudflare.com/client/v4/zones" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"example.com","jump_start":true}'
# Get zone ID
ZONE_ID=$(curl -s "https://api.cloudflare.com/client/v4/zones?name=example.com" \
-H "Authorization: Bearer $CF_API_TOKEN" | jq -r '.result[0].id')
```
### Cache Rules (Replacing Page Rules)
```bash
# Create a cache rule for static assets
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/rulesets/phases/http_request_cache_settings/entrypoint" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"rules": [
{
"expression": "(http.request.uri.path.extension in {\"css\" \"js\" \"png\" \"jpg\" \"woff2\" \"svg\"})",
"action": "set_cache_settings",
"action_parameters": {
"cache": true,
"browser_ttl": { "mode": "override_origin", "default": 2592000 },
"edge_ttl": { "mode": "override_origin", "default": 86400 }
},
"description": "Cache static assets aggressively"
}
]
}'
```
### Purge Cache
```bash
# Purge everything
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"purge_everything":true}'
# Purge specific URLs
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"files":["https://example.com/style.css","https://example.com/app.js"]}'
# Purge by prefix
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"prefixes":["https://example.com/images/"]}'
```
## Cache Headers on Origin
### nginx Cache Headers
```nginx
# Immutable hashed assets (fingerprinted filenames)
location ~* \.(js|css)$ {
if ($uri ~* "\.[a-f0-9]{8,}\.(js|css)$") {
expires 1y;
add_header Cache-Control "public, immutable";
}
expires 7d;
add_header Cache-Control "public, must-revalidate";
}
# Images and fonts
location ~* \.(jpg|jpeg|png|gif|ico|svg|webp|woff2|ttf)$ {
expires 30d;
add_header Cache-Control "public, immutable";
}
# HTML — always revalidate
location ~* \.html$ {
expires -1;
add_header Cache-Control "no-cache, must-revalidate";
}
# API responses — no caching
location /api/ {
add_header Cache-Control "no-store, no-cache";
add_header Vary "Authorization, Accept";
}
```
### Cache-Control Cheat Sheet
| Header | Meaning |
|--------|---------|
| `public, max-age=31536000, immutable` | Cache for 1 year, never revalidate (hashed assets) |
| `public, max-age=86400, must-revalidate` | Cache 1 day, check freshness after |
| `private, max-age=600` | Browser cache only, 10 min (user-specific content) |
| `no-cache` | Always revalidate with origin before serving |
| `no-store` | Never cache (sensitive data) |
| `s-maxage=3600` | CDN caches for 1 hour, overrides `max-age` for shared caches |
## Cache Warming
```bash
# Warm cache for critical pages after deployment
#!/bin/bash
URLS=(
"https://www.example.com/"
"https://www.example.com/products"
"https://www.example.com/about"
"https://www.example.com/css/main.abc123.css"
"https://www.example.com/js/app.def456.js"
)
for url in "${URLS[@]}"; do
curl -s -o /dev/null -w "%{http_code} %{time_total}s %{url_effective}\n" "$url"
done
```
## Monitoring Cache Performance
```bash
# Check cache status from response headers
curl -sI https://www.example.com/style.css | grep -i -E "cf-cache|x-cache|age|cache-control"
# Expected headers:
# cf-cache-status: HIT (Cloudflare)
# x-cache: Hit from cloudfront (CloudFront)
# age: 3600 (seconds since cached)
# CloudFront cache hit ratio
aws cloudwatch get-metric-statistics \
--namespace AWS/CloudFront \
--metric-name CacheHitRate \
--dimensions Name=DistributionId,Value=E1A2B3C4D5E6F7 \
--start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
--period 300 \
--statistics Average
```
## Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| `cf-cache-status: DYNAMIC` | No cache rule matches or Cache-Control prevents it | Set `s-maxage` or create a cache rule for the path |
| Cache hit ratio below 50% | Low TTLs or high URL cardinality (query strings) | Increase TTL; strip unnecessary query strings in cache key |
| Stale content after deploy | Old objects still cached at edge | Invalidate; use content-hashed filenames to avoid this entirely |
| CORS errors through CDN | CDN strips or caches wrong `Vary` header | Add `Vary: Origin` and configure origin request policy to forward `Origin` |
| 502 errors from CDN | Origin down or timeout | Check origin health; increase CDN origin timeout settings |
| Mixed content warnings | CDN serves HTTPS but origin links use HTTP | Set `viewer-protocol-policy: redirect-to-https`; fix origin URLs |
| High invalidation costs | Purging `/*` on every deploy | Use fingerprinted filenames; only invalidate `index.html` |
## Related Skills
- [dns-management](../dns-management/) - DNS records for CDN CNAME setup
- [cloudflare-pages](../../cloudflare/cloudflare-pages/) - Cloudflare's built-in CDN for Pages projects
- [reverse-proxy](../reverse-proxy/) - Origin server configuration behind CDN
- [load-balancing](../load-balancing/) - Multi-origin CDN backends
infrastructure/networking/dns-management/SKILL.md
+336 -44
View File
@@ -9,59 +9,351 @@ metadata:
# DNS Management
Configure and manage DNS infrastructure.
Configure and manage DNS zones, records, and resolution for production infrastructure.
## When to Use
- Setting up domains for web applications, APIs, and email.
- Migrating DNS providers or consolidating zones.
- Configuring DNS for CDN, load balancers, and cloud services.
- Troubleshooting resolution failures, propagation delays, or misconfigurations.
- Implementing DNSSEC, SPF, DKIM, and DMARC for email security.
## Prerequisites
- Domain registered with a registrar (Namecheap, Route53, Google Domains, Cloudflare).
- Access to DNS provider dashboard or API.
- AWS CLI configured (for Route53 examples).
- `dig` and `nslookup` available locally (included in most OS installs).
## DNS Record Types Reference
| Type | Purpose | Example Value |
|-------|---------|---------------|
| A | IPv4 address | `93.184.216.34` |
| AAAA | IPv6 address | `2606:2800:220:1:248:1893:25c8:1946` |
| CNAME | Alias to another domain | `www.example.com -> example.com` |
| MX | Mail server with priority | `10 mail.example.com` |
| TXT | Arbitrary text (SPF, DKIM, verification) | `v=spf1 include:_spf.google.com ~all` |
| NS | Authoritative name servers | `ns1.example.com` |
| SRV | Service location (host, port, priority) | `10 5 5060 sip.example.com` |
| CAA | Certificate Authority Authorization | `0 issue "letsencrypt.org"` |
| PTR | Reverse DNS lookup | `34.216.184.93.in-addr.arpa` |
## AWS Route 53
```bash
# Create hosted zone
aws route53 create-hosted-zone --name example.com --caller-reference $(date +%s)
# Create record
aws route53 change-resource-record-sets --hosted-zone-id ZXXXXX --change-batch '{
"Changes": [{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "www.example.com",
"Type": "A",
"TTL": 300,
"ResourceRecords": [{"Value": "1.2.3.4"}]
}
}]
}'
```
## BIND Configuration
### Hosted Zone Management
```bash
# /etc/bind/zones/example.com.db
$TTL 86400
@ IN SOA ns1.example.com. admin.example.com. (
2024010101 ; Serial
3600 ; Refresh
1800 ; Retry
604800 ; Expire
86400 ) ; Minimum TTL
# Create a hosted zone
aws route53 create-hosted-zone \
--name example.com \
--caller-reference "$(date +%s)"
IN NS ns1.example.com.
IN A 1.2.3.4
www IN A 1.2.3.4
# List hosted zones
aws route53 list-hosted-zones
# Get name servers for a zone (update at your registrar)
aws route53 get-hosted-zone --id Z1234567890ABC \
--query 'DelegationSet.NameServers'
```
## Common Records
### Create and Manage Records
```
A - IPv4 address
AAAA - IPv6 address
CNAME - Alias to another domain
MX - Mail server
TXT - Text record (SPF, DKIM)
NS - Name server
```bash
# Create an A record
aws route53 change-resource-record-sets \
--hosted-zone-id Z1234567890ABC \
--change-batch '{
"Changes": [{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "app.example.com",
"Type": "A",
"TTL": 300,
"ResourceRecords": [{"Value": "93.184.216.34"}]
}
}]
}'
# Create a CNAME record
aws route53 change-resource-record-sets \
--hosted-zone-id Z1234567890ABC \
--change-batch '{
"Changes": [{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "www.example.com",
"Type": "CNAME",
"TTL": 300,
"ResourceRecords": [{"Value": "example.com"}]
}
}]
}'
# Create an alias record (no TTL, Route53-specific)
aws route53 change-resource-record-sets \
--hosted-zone-id Z1234567890ABC \
--change-batch '{
"Changes": [{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "example.com",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "Z2FDTNDATAQYW2",
"DNSName": "d1234567890.cloudfront.net",
"EvaluateTargetHealth": false
}
}
}]
}'
# List records in a zone
aws route53 list-resource-record-sets --hosted-zone-id Z1234567890ABC
# Delete a record (Action: DELETE with exact match)
aws route53 change-resource-record-sets \
--hosted-zone-id Z1234567890ABC \
--change-batch '{
"Changes": [{
"Action": "DELETE",
"ResourceRecordSet": {
"Name": "old.example.com",
"Type": "A",
"TTL": 300,
"ResourceRecords": [{"Value": "1.2.3.4"}]
}
}]
}'
```
## Best Practices
### Route 53 Health Checks
- Low TTL during migrations
- Implement DNSSEC
- Use multiple name servers
- Monitor DNS resolution
```bash
# Create a health check
aws route53 create-health-check --caller-reference "$(date +%s)" \
--health-check-config '{
"IPAddress": "93.184.216.34",
"Port": 443,
"Type": "HTTPS",
"ResourcePath": "/health",
"RequestInterval": 30,
"FailureThreshold": 3
}'
```
## Cloudflare DNS
### Manage Records via API
```bash
# Get zone ID
ZONE_ID=$(curl -s "https://api.cloudflare.com/client/v4/zones?name=example.com" \
-H "Authorization: Bearer $CF_API_TOKEN" | jq -r '.result[0].id')
# Create an A record (proxied through Cloudflare)
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"type":"A","name":"app","content":"93.184.216.34","proxied":true,"ttl":1}'
# Create a CNAME record (DNS only, not proxied)
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-d '{"type":"CNAME","name":"docs","content":"docs.readthedocs.io","proxied":false,"ttl":3600}'
# List all records
curl -s "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
-H "Authorization: Bearer $CF_API_TOKEN" | jq '.result[] | {name, type, content, proxied}'
# Delete a record
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
-H "Authorization: Bearer $CF_API_TOKEN"
```
## Terraform DNS Management
### Route 53 with Terraform
```hcl
# dns.tf
resource "aws_route53_zone" "main" {
name = "example.com"
}
resource "aws_route53_record" "app" {
zone_id = aws_route53_zone.main.zone_id
name = "app.example.com"
type = "A"
ttl = 300
records = ["93.184.216.34"]
}
resource "aws_route53_record" "www" {
zone_id = aws_route53_zone.main.zone_id
name = "www.example.com"
type = "CNAME"
ttl = 300
records = ["example.com"]
}
# Alias record for CloudFront
resource "aws_route53_record" "cdn" {
zone_id = aws_route53_zone.main.zone_id
name = "example.com"
type = "A"
alias {
name = aws_cloudfront_distribution.main.domain_name
zone_id = aws_cloudfront_distribution.main.hosted_zone_id
evaluate_target_health = false
}
}
# Email records
resource "aws_route53_record" "mx" {
zone_id = aws_route53_zone.main.zone_id
name = "example.com"
type = "MX"
ttl = 3600
records = [
"1 aspmx.l.google.com",
"5 alt1.aspmx.l.google.com",
"5 alt2.aspmx.l.google.com",
]
}
resource "aws_route53_record" "spf" {
zone_id = aws_route53_zone.main.zone_id
name = "example.com"
type = "TXT"
ttl = 3600
records = ["v=spf1 include:_spf.google.com ~all"]
}
resource "aws_route53_record" "dmarc" {
zone_id = aws_route53_zone.main.zone_id
name = "_dmarc.example.com"
type = "TXT"
ttl = 3600
records = ["v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com; pct=100"]
}
```
### Cloudflare with Terraform
```hcl
resource "cloudflare_record" "app" {
zone_id = var.cloudflare_zone_id
name = "app"
content = "93.184.216.34"
type = "A"
proxied = true
}
resource "cloudflare_record" "mail" {
zone_id = var.cloudflare_zone_id
name = "@"
content = "aspmx.l.google.com"
type = "MX"
priority = 1
}
```
## DNS Troubleshooting Commands
### dig
```bash
# Query A record
dig app.example.com A +short
# Query from a specific DNS server
dig @8.8.8.8 app.example.com A
# Show full answer with TTL
dig app.example.com A +noall +answer
# Query MX records
dig example.com MX +short
# Trace the full resolution path
dig app.example.com +trace
# Check DNSSEC validation
dig example.com +dnssec +short
# Query TXT records (SPF, DKIM)
dig example.com TXT +short
dig default._domainkey.example.com TXT +short
```
### nslookup
```bash
# Basic lookup
nslookup app.example.com
# Specify DNS server
nslookup app.example.com 8.8.8.8
# Query specific record type
nslookup -type=MX example.com
nslookup -type=TXT example.com
```
### Check DNS Propagation
```bash
# Query multiple public resolvers
for dns in 8.8.8.8 1.1.1.1 9.9.9.9 208.67.222.222; do
echo "=== $dns ==="
dig @$dns app.example.com A +short
done
```
## Email Security Records
```bash
# SPF — authorize sending servers
# TXT record on example.com
"v=spf1 include:_spf.google.com include:sendgrid.net -all"
# DKIM — email signing verification
# TXT record on google._domainkey.example.com
# (value provided by your email provider)
# DMARC — policy for failed SPF/DKIM
# TXT record on _dmarc.example.com
"v=DMARC1; p=reject; rua=mailto:dmarc-reports@example.com; pct=100"
```
## TTL Strategies
| Scenario | Recommended TTL | Rationale |
|----------|----------------|-----------|
| Stable production records | 3600-86400 (1h-24h) | Reduce DNS queries, faster resolution |
| Pre-migration warmup | 60-300 (1-5 min) | Lower TTL days before migration |
| During migration/failover | 60 | Fast propagation of changes |
| Post-migration cooldown | Gradually increase to 3600+ | Return to normal after confirming stability |
| Load-balanced records | 60-300 | Allow health-check-driven failover |
## Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| DNS changes not visible | TTL not expired on recursive resolvers | Wait for old TTL to expire; lower TTL before next change |
| `SERVFAIL` response | DNSSEC validation failure or broken delegation | Check NS records at registrar; verify DNSSEC signatures |
| `NXDOMAIN` for valid record | Wrong hosted zone or missing record | Verify record exists with `dig @<authoritative-ns> domain` |
| CNAME at zone apex returns error | CNAME not allowed at zone apex per RFC | Use ALIAS (Route53) or proxied A record (Cloudflare) |
| Email going to spam | Missing or broken SPF/DKIM/DMARC | Verify TXT records with `dig example.com TXT`; test at mail-tester.com |
| Slow resolution | Recursive resolver far from authoritative NS | Use Anycast DNS providers (Cloudflare, Route53) |
| Inconsistent results across resolvers | Partial propagation or cache poisoning | Query authoritative NS directly; check for conflicting records |
## Related Skills
- [cdn-setup](../cdn-setup/) - CDN CNAME and alias record configuration
- [load-balancing](../load-balancing/) - DNS-based load balancing and health checks
- [cloudflare-zero-trust](../../cloudflare/cloudflare-zero-trust/) - Tunnel DNS routing
- [reverse-proxy](../reverse-proxy/) - Connecting domains to backend services
infrastructure/networking/load-balancing/SKILL.md
+347 -24
View File
@@ -9,56 +9,379 @@ metadata:
# Load Balancing
Distribute traffic across application servers.
Distribute traffic across application servers for high availability, scalability, and fault tolerance.
## When to Use
- Distributing HTTP/HTTPS traffic across multiple backend servers.
- Implementing health checks to route around unhealthy instances.
- Terminating TLS at the load balancer for simplified certificate management.
- Enabling blue/green or canary deployments with traffic shifting.
- Scaling horizontally behind a single entry point.
## Prerequisites
- Two or more backend servers running the same application.
- TLS certificate for HTTPS termination (ACM, Let's Encrypt, or self-signed for internal).
- For AWS: VPC with public and private subnets across availability zones.
- For nginx/HAProxy: Linux server with root access.
## nginx Load Balancer
### Basic Round-Robin
```nginx
upstream backend {
least_conn;
server backend1:8080 weight=3;
server backend2:8080;
server backend3:8080 backup;
# /etc/nginx/conf.d/loadbalancer.conf
upstream app_backend {
server 10.0.1.10:8080;
server 10.0.1.11:8080;
server 10.0.1.12:8080;
}
server {
listen 80;
server_name app.example.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
server_name app.example.com;
ssl_certificate /etc/ssl/certs/app.example.com.pem;
ssl_certificate_key /etc/ssl/private/app.example.com-key.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
proxy_pass http://backend;
proxy_pass http://app_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 5s;
proxy_read_timeout 30s;
proxy_send_timeout 30s;
}
location /health {
access_log off;
return 200 "OK";
}
}
```
## HAProxy
### Weighted and Backup Servers
```nginx
upstream app_backend {
least_conn; # Route to server with fewest active connections
server 10.0.1.10:8080 weight=5; # Gets 5x traffic
server 10.0.1.11:8080 weight=3; # Gets 3x traffic
server 10.0.1.12:8080 weight=1; # Gets 1x traffic
server 10.0.1.20:8080 backup; # Only used when others are down
server 10.0.1.21:8080 down; # Temporarily removed from pool
}
```
### Health Checks (nginx Plus / OpenResty)
```nginx
upstream app_backend {
zone backend 64k; # Shared memory zone for health data
server 10.0.1.10:8080;
server 10.0.1.11:8080;
server 10.0.1.12:8080;
}
# Health check (requires nginx Plus or third-party module)
# match healthy {
# status 200;
# body ~ "OK";
# }
# health_check interval=5s fails=3 passes=2 match=healthy;
```
### Sticky Sessions (IP Hash)
```nginx
upstream app_backend {
ip_hash; # Same client IP always goes to the same server
server 10.0.1.10:8080;
server 10.0.1.11:8080;
server 10.0.1.12:8080;
}
```
## HAProxy Configuration
### Full Production Config
```
# /etc/haproxy/haproxy.cfg
global
log /dev/log local0
maxconn 4096
daemon
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
tune.ssl.default-dh-param 2048
defaults
mode http
log global
option httplog
option dontlognull
option forwardfor
timeout connect 5s
timeout client 30s
timeout server 30s
timeout http-request 10s
timeout http-keep-alive 5s
retries 3
frontend http_front
bind *:80
default_backend http_back
redirect scheme https code 301 if !{ ssl_fc }
backend http_back
frontend https_front
bind *:443 ssl crt /etc/ssl/certs/app.example.com.pem
http-request set-header X-Forwarded-Proto https
# Route based on path
acl is_api path_beg /api/
acl is_ws path_beg /ws/
use_backend api_servers if is_api
use_backend ws_servers if is_ws
default_backend web_servers
backend web_servers
balance roundrobin
option httpchk GET /health HTTP/1.1\r\nHost:\ app.example.com
http-check expect status 200
cookie SERVERID insert indirect nocache
server web1 10.0.1.10:8080 check inter 5s fall 3 rise 2 cookie web1
server web2 10.0.1.11:8080 check inter 5s fall 3 rise 2 cookie web2
server web3 10.0.1.12:8080 check inter 5s fall 3 rise 2 cookie web3
backend api_servers
balance leastconn
option httpchk GET /api/health
http-check expect status 200
server api1 10.0.2.10:8080 check inter 5s fall 3 rise 2
server api2 10.0.2.11:8080 check inter 5s fall 3 rise 2
backend ws_servers
balance source
option httpchk GET /health
server web1 10.0.0.1:8080 check
server web2 10.0.0.2:8080 check
timeout tunnel 1h
server ws1 10.0.3.10:8080 check inter 5s fall 3 rise 2
server ws2 10.0.3.11:8080 check inter 5s fall 3 rise 2
listen stats
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
stats admin if LOCALHOST
```
## AWS ALB
### HAProxy Management
```bash
aws elbv2 create-load-balancer \
--name my-alb \
--subnets subnet-xxx subnet-yyy \
--security-groups sg-xxx \
--type application
# Test config before reloading
haproxy -c -f /etc/haproxy/haproxy.cfg
# Reload without dropping connections
sudo systemctl reload haproxy
# View stats from CLI
echo "show stat" | sudo socat stdio /var/run/haproxy/admin.sock
# Drain a server (stop new connections, let existing finish)
echo "set server web_servers/web1 state drain" | sudo socat stdio /var/run/haproxy/admin.sock
# Set server to maintenance
echo "set server web_servers/web1 state maint" | sudo socat stdio /var/run/haproxy/admin.sock
# Re-enable server
echo "set server web_servers/web1 state ready" | sudo socat stdio /var/run/haproxy/admin.sock
```
## Best Practices
## AWS Application Load Balancer (ALB)
- Implement health checks
- Use sticky sessions when needed
- Enable connection draining
- Monitor backend health
### Create ALB via CLI
```bash
# Create the load balancer
aws elbv2 create-load-balancer \
--name my-app-alb \
--subnets subnet-aaa111 subnet-bbb222 \
--security-groups sg-xxx123 \
--type application \
--scheme internet-facing
# Create a target group
aws elbv2 create-target-group \
--name my-app-targets \
--protocol HTTP \
--port 8080 \
--vpc-id vpc-xxx123 \
--health-check-protocol HTTP \
--health-check-path /health \
--health-check-interval-seconds 15 \
--healthy-threshold-count 2 \
--unhealthy-threshold-count 3 \
--target-type instance
# Register targets
aws elbv2 register-targets \
--target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456:targetgroup/my-app-targets/abc123 \
--targets Id=i-0123456789abc Id=i-0987654321def
# Create HTTPS listener
aws elbv2 create-listener \
--load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456:loadbalancer/app/my-app-alb/abc123 \
--protocol HTTPS \
--port 443 \
--certificates CertificateArn=arn:aws:acm:us-east-1:123456:certificate/abc-123 \
--default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:us-east-1:123456:targetgroup/my-app-targets/abc123
# Create HTTP redirect listener
aws elbv2 create-listener \
--load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456:loadbalancer/app/my-app-alb/abc123 \
--protocol HTTP \
--port 80 \
--default-actions Type=redirect,RedirectConfig='{Protocol=HTTPS,Port=443,StatusCode=HTTP_301}'
```
### Check Target Health
```bash
# Check health of registered targets
aws elbv2 describe-target-health \
--target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456:targetgroup/my-app-targets/abc123
```
## AWS Network Load Balancer (NLB)
```bash
# Create NLB (for TCP, UDP, or TLS traffic)
aws elbv2 create-load-balancer \
--name my-tcp-nlb \
--subnets subnet-aaa111 subnet-bbb222 \
--type network \
--scheme internet-facing
# Create TCP target group
aws elbv2 create-target-group \
--name my-tcp-targets \
--protocol TCP \
--port 5432 \
--vpc-id vpc-xxx123 \
--health-check-protocol TCP \
--target-type ip
```
## ALB with Terraform
```hcl
resource "aws_lb" "app" {
name = "my-app-alb"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.alb.id]
subnets = var.public_subnet_ids
enable_deletion_protection = true
}
resource "aws_lb_target_group" "app" {
name = "my-app-tg"
port = 8080
protocol = "HTTP"
vpc_id = var.vpc_id
health_check {
path = "/health"
port = "traffic-port"
healthy_threshold = 2
unhealthy_threshold = 3
timeout = 5
interval = 15
matcher = "200"
}
deregistration_delay = 30
stickiness {
type = "lb_cookie"
cookie_duration = 86400
enabled = true
}
}
resource "aws_lb_listener" "https" {
load_balancer_arn = aws_lb.app.arn
port = 443
protocol = "HTTPS"
ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06"
certificate_arn = aws_acm_certificate.cert.arn
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.app.arn
}
}
resource "aws_lb_listener" "http_redirect" {
load_balancer_arn = aws_lb.app.arn
port = 80
protocol = "HTTP"
default_action {
type = "redirect"
redirect {
port = "443"
protocol = "HTTPS"
status_code = "HTTP_301"
}
}
}
```
## Load Balancing Algorithms
| Algorithm | Use Case | nginx | HAProxy |
|-----------|----------|-------|---------|
| Round Robin | Default, equal servers | `(default)` | `balance roundrobin` |
| Least Connections | Uneven request durations | `least_conn` | `balance leastconn` |
| IP Hash | Session persistence without cookies | `ip_hash` | `balance source` |
| URI Hash | Cache locality per URL | `hash $request_uri` | `balance uri` |
| Random with Two | Large server pools | `random two least_conn` | `balance random(2)` |
## Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| All backends show "unhealthy" | Health check path returns non-200 | Verify `/health` endpoint returns 200; check security groups |
| 502 Bad Gateway | Backend not running or wrong port | Confirm backend is listening on the configured port |
| Uneven traffic distribution | Sticky sessions or weighted config | Check session affinity settings; review server weights |
| Connection timeouts | Backend too slow or timeout too low | Increase `proxy_read_timeout` or HAProxy `timeout server` |
| TLS handshake failures | Certificate mismatch or expired cert | Verify cert matches the domain; renew if expired |
| ALB returns 503 | No healthy targets registered | Check target group health; verify targets are in correct subnets |
| WebSocket disconnects | Proxy not configured for upgrades | Add `proxy_set_header Upgrade` and `Connection "upgrade"` |
## Related Skills
- [reverse-proxy](../reverse-proxy/) - Reverse proxy configuration patterns
- [dns-management](../dns-management/) - DNS records pointing to load balancers
- [cdn-setup](../cdn-setup/) - CDN in front of load balanced origins
- [service-mesh](../service-mesh/) - Service-level load balancing in Kubernetes
infrastructure/networking/reverse-proxy/SKILL.md
+359 -23
View File
@@ -9,60 +9,396 @@ metadata:
# Reverse Proxy
Configure reverse proxies for application routing.
Configure reverse proxies to route traffic, terminate TLS, enforce rate limits, and serve as the gateway between clients and backend services.
## nginx
## When to Use
- Routing traffic from a public domain to one or more backend services.
- Terminating TLS at the edge and forwarding plain HTTP to backends.
- Adding rate limiting, CORS, security headers, and access control.
- Consolidating multiple services under a single domain with path-based routing.
- Handling WebSocket upgrades, gRPC proxying, or HTTP/2 passthrough.
## Prerequisites
- Backend service(s) running on known host:port.
- TLS certificate (Let's Encrypt, ACM, or self-signed for development).
- nginx 1.25+ or Traefik 3.x installed.
- DNS record pointing the domain to the proxy server.
## nginx Reverse Proxy
### Basic HTTPS Proxy with Redirect
```nginx
# /etc/nginx/sites-available/app.example.com
server {
listen 80;
server_name api.example.com;
return 301 https://$server_name$request_uri;
server_name app.example.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
server_name api.example.com;
ssl_certificate /etc/ssl/certs/api.crt;
ssl_certificate_key /etc/ssl/private/api.key;
server_name app.example.com;
# TLS configuration
ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Security headers
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
add_header X-Frame-Options DENY always;
add_header X-Content-Type-Options nosniff always;
add_header Referrer-Policy strict-origin-when-cross-origin always;
# Proxy to backend
location / {
proxy_pass http://backend:8080;
proxy_pass http://127.0.0.1:3000;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /ws {
proxy_pass http://backend:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# Timeouts
proxy_connect_timeout 5s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
# Buffering
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
}
}
```
## Traefik
### Path-Based Routing to Multiple Services
```nginx
server {
listen 443 ssl http2;
server_name app.example.com;
ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
# Frontend SPA
location / {
proxy_pass http://127.0.0.1:3000;
proxy_set_header Host $host;
}
# API backend
location /api/ {
proxy_pass http://127.0.0.1:8080/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 120s;
}
# WebSocket endpoint
location /ws/ {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_read_timeout 86400s; # 24h for long-lived connections
}
# Static assets with caching
location /static/ {
alias /var/www/static/;
expires 30d;
add_header Cache-Control "public, immutable";
}
}
```
### Rate Limiting
```nginx
# Define rate limit zones in http block
http {
# 10 requests/second per IP
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
# 1 request/second for login
limit_req_zone $binary_remote_addr zone=login_limit:10m rate=1r/s;
# Connection limit per IP
limit_conn_zone $binary_remote_addr zone=conn_limit:10m;
}
server {
listen 443 ssl http2;
server_name app.example.com;
# Apply rate limit to API
location /api/ {
limit_req zone=api_limit burst=20 nodelay;
limit_req_status 429;
proxy_pass http://127.0.0.1:8080;
}
# Strict rate limit on auth endpoints
location /api/auth/ {
limit_req zone=login_limit burst=5;
limit_req_status 429;
proxy_pass http://127.0.0.1:8080;
}
# Connection limit
location / {
limit_conn conn_limit 100;
proxy_pass http://127.0.0.1:3000;
}
}
```
### Gzip and Brotli Compression
```nginx
http {
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml image/svg+xml;
gzip_min_length 256;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 5;
# Brotli (requires ngx_brotli module)
# brotli on;
# brotli_types text/plain text/css application/json application/javascript text/xml application/xml image/svg+xml;
# brotli_comp_level 6;
}
```
### Let's Encrypt with Certbot
```bash
# Install certbot with nginx plugin
sudo apt install certbot python3-certbot-nginx
# Obtain and install certificate
sudo certbot --nginx -d app.example.com -d www.example.com
# Auto-renewal is configured via systemd timer
sudo systemctl status certbot.timer
# Manual renewal test
sudo certbot renew --dry-run
```
## Traefik Reverse Proxy
### Static Configuration
```yaml
# traefik.yml
entryPoints:
web:
address: ":80"
http:
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: ":443"
certificatesResolvers:
letsencrypt:
acme:
email: admin@example.com
storage: /letsencrypt/acme.json
httpChallenge:
entryPoint: web
providers:
docker:
exposedByDefault: false
file:
directory: /etc/traefik/dynamic/
api:
dashboard: true
insecure: false
log:
level: INFO
accessLog:
filePath: /var/log/traefik/access.log
```
## Best Practices
### Dynamic Configuration (File Provider)
- Implement SSL termination
- Set proper headers
- Configure timeouts
- Enable gzip compression
```yaml
# /etc/traefik/dynamic/services.yml
http:
routers:
app:
rule: "Host(`app.example.com`)"
entryPoints:
- websecure
service: app
tls:
certResolver: letsencrypt
middlewares:
- security-headers
- rate-limit
api:
rule: "Host(`app.example.com`) && PathPrefix(`/api`)"
entryPoints:
- websecure
service: api
tls:
certResolver: letsencrypt
services:
app:
loadBalancer:
servers:
- url: "http://127.0.0.1:3000"
healthCheck:
path: /health
interval: 10s
timeout: 3s
api:
loadBalancer:
servers:
- url: "http://127.0.0.1:8080"
healthCheck:
path: /api/health
interval: 10s
timeout: 3s
middlewares:
security-headers:
headers:
stsSeconds: 63072000
stsIncludeSubdomains: true
frameDeny: true
contentTypeNosniff: true
browserXssFilter: true
referrerPolicy: strict-origin-when-cross-origin
rate-limit:
rateLimit:
average: 100
burst: 50
period: 1m
```
### Traefik with Docker Labels
```yaml
# docker-compose.yml
version: "3.8"
services:
traefik:
image: traefik:v3.0
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./traefik.yml:/etc/traefik/traefik.yml:ro
- letsencrypt:/letsencrypt
frontend:
image: my-frontend:latest
labels:
- "traefik.enable=true"
- "traefik.http.routers.frontend.rule=Host(`app.example.com`)"
- "traefik.http.routers.frontend.tls.certresolver=letsencrypt"
- "traefik.http.services.frontend.loadbalancer.server.port=3000"
api:
image: my-api:latest
labels:
- "traefik.enable=true"
- "traefik.http.routers.api.rule=Host(`app.example.com`) && PathPrefix(`/api`)"
- "traefik.http.routers.api.tls.certresolver=letsencrypt"
- "traefik.http.services.api.loadbalancer.server.port=8080"
- "traefik.http.routers.api.middlewares=api-ratelimit"
- "traefik.http.middlewares.api-ratelimit.ratelimit.average=50"
- "traefik.http.middlewares.api-ratelimit.ratelimit.burst=25"
volumes:
letsencrypt:
```
## nginx Testing and Management
```bash
# Test configuration syntax
sudo nginx -t
# Reload without downtime
sudo nginx -s reload
# View active connections
sudo nginx -s status
# Check which config file is active
nginx -V 2>&1 | grep -o '\-\-conf-path=[^ ]*'
# Monitor access logs
tail -f /var/log/nginx/access.log
# Monitor error logs
tail -f /var/log/nginx/error.log
```
## IP Allowlisting and Geoblocking
```nginx
# Allow only specific IPs (admin panel)
location /admin/ {
allow 203.0.113.0/24;
allow 198.51.100.5;
deny all;
proxy_pass http://127.0.0.1:3000;
}
# Block by country (requires GeoIP2 module)
# geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
# auto_reload 60m;
# $geoip2_data_country_iso_code country iso_code;
# }
# if ($geoip2_data_country_iso_code = "XX") {
# return 403;
# }
```
## Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| 502 Bad Gateway | Backend not running or unreachable | Verify backend is listening; check `proxy_pass` URL |
| 504 Gateway Timeout | Backend too slow | Increase `proxy_read_timeout`; check backend performance |
| Mixed content warnings | `X-Forwarded-Proto` not set | Add `proxy_set_header X-Forwarded-Proto $scheme` |
| WebSocket disconnects after 60s | Default proxy timeout expires | Set `proxy_read_timeout 86400s` for WebSocket locations |
| Rate limit hits legitimate users | Zone rate too aggressive | Increase `rate` or `burst` values; use different zones per endpoint |
| Let's Encrypt renewal fails | Port 80 blocked or wrong server block | Ensure `.well-known/acme-challenge/` is accessible |
| Traefik shows 404 for all routes | Docker labels not detected | Verify Docker socket is mounted; check `exposedByDefault` setting |
| TLS handshake failure | Certificate chain incomplete | Include intermediate certificates in `ssl_certificate` |
## Related Skills
- [load-balancing](../load-balancing/) - Multi-backend traffic distribution
- [cdn-setup](../cdn-setup/) - CDN in front of reverse proxy
- [dns-management](../dns-management/) - DNS records for proxy domains
- [service-mesh](../service-mesh/) - Service-level routing in Kubernetes
infrastructure/networking/service-mesh/SKILL.md
+366 -30
View File
@@ -9,62 +9,398 @@ metadata:
# Service Mesh
Implement service-to-service communication management.
Implement service-to-service communication management with mTLS, traffic shaping, observability, and policy enforcement using Istio or Linkerd.
## When to Use
- Securing microservice communication with automatic mTLS.
- Implementing canary deployments, traffic splitting, or A/B testing.
- Adding circuit breakers, retries, and timeouts without changing application code.
- Gaining service-level observability (latency, error rates, request volume).
- Enforcing authorization policies between services.
## Prerequisites
- Kubernetes cluster (1.26+) with kubectl configured.
- Helm 3 installed (for some installation methods).
- Sufficient cluster resources (Istio control plane needs ~2 GB RAM).
- For Istio: `istioctl` CLI installed.
- For Linkerd: `linkerd` CLI installed.
## Istio Installation
```bash
istioctl install --set profile=demo
### Install with istioctl
# Enable sidecar injection
```bash
# Download istioctl
curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH
# Install with the production profile
istioctl install --set profile=default -y
# Or use the demo profile (includes all addons, good for learning)
istioctl install --set profile=demo -y
# Verify installation
istioctl verify-install
# Check running components
kubectl get pods -n istio-system
```
### Enable Sidecar Injection
```bash
# Enable automatic sidecar injection for a namespace
kubectl label namespace default istio-injection=enabled
# Verify label
kubectl get namespace default --show-labels
# Restart existing pods to inject sidecars
kubectl rollout restart deployment -n default
# Check sidecar status
kubectl get pods -n default -o jsonpath='{range .items[*]}{.metadata.name}{" containers: "}{range .spec.containers[*]}{.name}{" "}{end}{"\n"}{end}'
```
### Install Observability Addons
```bash
# Install Kiali, Prometheus, Grafana, Jaeger
kubectl apply -f samples/addons/prometheus.yaml
kubectl apply -f samples/addons/grafana.yaml
kubectl apply -f samples/addons/jaeger.yaml
kubectl apply -f samples/addons/kiali.yaml
# Wait for rollout
kubectl rollout status deployment/kiali -n istio-system
# Access dashboards
istioctl dashboard kiali
istioctl dashboard grafana
istioctl dashboard jaeger
```
## Traffic Management
### VirtualService (Routing Rules)
```yaml
apiVersion: networking.istio.io/v1alpha3
# virtualservice.yaml — canary deployment with traffic split
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: myapp
name: my-app
namespace: default
spec:
hosts:
- myapp
- my-app
http:
- match:
- headers:
canary:
exact: "true"
route:
- destination:
host: myapp
subset: canary
- route:
- destination:
host: myapp
subset: stable
weight: 90
- destination:
host: myapp
subset: canary
weight: 10
# Header-based routing (canary testers)
- match:
- headers:
x-canary:
exact: "true"
route:
- destination:
host: my-app
subset: canary
# Percentage-based traffic split
- route:
- destination:
host: my-app
subset: stable
weight: 90
- destination:
host: my-app
subset: canary
weight: 10
timeout: 30s
retries:
attempts: 3
perTryTimeout: 10s
retryOn: gateway-error,connect-failure,refused-stream
```
## mTLS
### DestinationRule (Subsets and Connection Policy)
```yaml
# destinationrule.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: my-app
namespace: default
spec:
host: my-app
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
h2UpgradePolicy: DEFAULT
http1MaxPendingRequests: 100
http2MaxRequests: 1000
maxRequestsPerConnection: 10
outlierDetection:
consecutive5xxErrors: 5
interval: 10s
baseEjectionTime: 30s
maxEjectionPercent: 50
subsets:
- name: stable
labels:
version: v1
- name: canary
labels:
version: v2
```
### Gateway (Ingress Traffic)
```yaml
# gateway.yaml — expose service to external traffic
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: app-gateway
namespace: default
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: app-tls-cert # Kubernetes secret
hosts:
- app.example.com
- port:
number: 80
name: http
protocol: HTTP
hosts:
- app.example.com
tls:
httpsRedirect: true
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: app-external
namespace: default
spec:
hosts:
- app.example.com
gateways:
- app-gateway
http:
- route:
- destination:
host: my-app
port:
number: 8080
```
## mTLS Configuration
### Strict mTLS (Cluster-Wide)
```yaml
# peer-authentication.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system # Applies to entire mesh
spec:
mtls:
mode: STRICT
```
## Best Practices
### Permissive mTLS (Per Namespace)
- Enable strict mTLS
- Implement circuit breakers
- Use traffic shifting for deployments
- Monitor with Kiali and Jaeger
```yaml
# Allow both plaintext and mTLS during migration
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: legacy-apps
spec:
mtls:
mode: PERMISSIVE
```
### Verify mTLS Status
```bash
# Check mTLS status for a namespace
istioctl x describe pod <pod-name> -n default
# View TLS configuration
istioctl proxy-config cluster <pod-name>.default --fqdn my-app.default.svc.cluster.local -o json | grep -A5 "tlsContext"
# Verify with istioctl authn
istioctl authn tls-check <pod-name>.default my-app.default.svc.cluster.local
```
## Authorization Policies
```yaml
# authz-policy.yaml — only allow frontend to call API
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: api-access
namespace: default
spec:
selector:
matchLabels:
app: my-api
action: ALLOW
rules:
- from:
- source:
principals:
- "cluster.local/ns/default/sa/frontend"
to:
- operation:
methods: ["GET", "POST"]
paths: ["/api/*"]
---
# Deny all other traffic to api
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: deny-all
namespace: default
spec:
selector:
matchLabels:
app: my-api
action: DENY
rules:
- from:
- source:
notPrincipals:
- "cluster.local/ns/default/sa/frontend"
```
## Circuit Breaking
```yaml
# circuit-breaker.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: my-api-circuit-breaker
spec:
host: my-api
trafficPolicy:
connectionPool:
tcp:
maxConnections: 50
http:
http1MaxPendingRequests: 50
http2MaxRequests: 100
maxRetries: 3
outlierDetection:
consecutive5xxErrors: 3
interval: 15s
baseEjectionTime: 60s
maxEjectionPercent: 100
```
## Linkerd Installation
```bash
# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh
export PATH=$HOME/.linkerd2/bin:$PATH
# Validate cluster prerequisites
linkerd check --pre
# Install Linkerd CRDs
linkerd install --crds | kubectl apply -f -
# Install Linkerd control plane
linkerd install | kubectl apply -f -
# Verify installation
linkerd check
# Inject sidecar into a namespace
kubectl get deploy -n my-app -o yaml | linkerd inject - | kubectl apply -f -
# Or annotate namespace for auto-injection
kubectl annotate namespace my-app linkerd.io/inject=enabled
# View live traffic dashboard
linkerd viz install | kubectl apply -f -
linkerd viz dashboard
```
### Linkerd Traffic Split (SMI)
```yaml
# traffic-split.yaml
apiVersion: split.smi-spec.io/v1alpha4
kind: TrafficSplit
metadata:
name: my-app-split
namespace: default
spec:
service: my-app
backends:
- service: my-app-stable
weight: 900
- service: my-app-canary
weight: 100
```
## Debugging
```bash
# Istio: check proxy configuration
istioctl proxy-config routes <pod-name>.default
istioctl proxy-config clusters <pod-name>.default
istioctl proxy-config listeners <pod-name>.default
# Istio: analyze configuration for issues
istioctl analyze -n default
# Istio: proxy debug logs
istioctl proxy-config log <pod-name>.default --level debug
# Linkerd: check proxy stats
linkerd viz stat deploy -n default
linkerd viz top deploy/my-app -n default
linkerd viz edges deploy -n default
```
## Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| Sidecar not injected | Missing namespace label | Add `istio-injection=enabled` label; restart pods |
| 503 errors between services | mTLS mismatch (one side plaintext) | Set `PeerAuthentication` to `PERMISSIVE` during migration |
| High latency after mesh install | Sidecar resource limits too low | Increase sidecar CPU/memory limits in mesh config |
| VirtualService not routing | Missing DestinationRule subsets | Create matching DestinationRule with subset labels |
| `upstream connect error` | Circuit breaker tripped | Check outlier detection settings; increase thresholds |
| Authorization policy blocks everything | Default deny without matching allow rule | Add explicit ALLOW rule before DENY-all |
| Kiali shows "Unknown" traffic | Missing sidecar on calling service | Inject sidecar into all communicating services |
## Related Skills
- [load-balancing](../load-balancing/) - Layer 4/7 load balancing outside Kubernetes
- [reverse-proxy](../reverse-proxy/) - Ingress-level proxying
- [ai-inference-service-mesh](../ai-inference-service-mesh/) - Mesh patterns for ML workloads
- [dns-management](../dns-management/) - DNS for mesh ingress gateways