This commit is contained in:
Toby
2026-01-27 17:35:45 -05:00
commit 2639af6531
176 changed files with 27104 additions and 0 deletions
@@ -0,0 +1,187 @@
# Kubernetes Best Practices
## Resource Management
### Always Set Resource Requests and Limits
```yaml
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
```
**Guidelines:**
- Requests = guaranteed resources
- Limits = maximum resources
- Set requests based on normal usage
- Set limits based on peak usage
- Memory limit = 2x request is common
- Avoid CPU limits in most cases (causes throttling)
### Use Horizontal Pod Autoscaler
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: myapp-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
```
## Pod Configuration
### Use Liveness and Readiness Probes
```yaml
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
```
### Configure Pod Disruption Budgets
```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: myapp-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: myapp
```
### Use Anti-Affinity for High Availability
```yaml
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: myapp
topologyKey: kubernetes.io/hostname
```
## Security
### Run as Non-Root
```yaml
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
```
### Read-Only Root Filesystem
```yaml
securityContext:
readOnlyRootFilesystem: true
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}
```
### Drop All Capabilities
```yaml
securityContext:
capabilities:
drop:
- ALL
```
## Networking
### Use Network Policies
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
```
### Service Mesh for mTLS
- Istio, Linkerd, or Consul Connect
- Automatic encryption between services
- Traffic management capabilities
## Configuration Management
### Use ConfigMaps for Configuration
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
LOG_LEVEL: "info"
DATABASE_HOST: "postgres.default.svc"
```
### Use Secrets for Sensitive Data
```yaml
apiVersion: v1
kind: Secret
metadata:
name: app-secrets
type: Opaque
stringData:
DATABASE_PASSWORD: "secret123"
```
### External Secrets for Production
- Use External Secrets Operator
- Integrate with Vault, AWS Secrets Manager, etc.
- Never commit secrets to git
## Observability
### Structured Logging
- Output JSON logs
- Include correlation IDs
- Use consistent field names
### Metrics
- Expose Prometheus metrics
- Use standard naming conventions
- Include SLI metrics
### Distributed Tracing
- Implement OpenTelemetry
- Propagate trace context
- Sample appropriately
@@ -0,0 +1,165 @@
# Kubernetes Troubleshooting Guide
## Common Issues and Solutions
### Pod Issues
#### Pod Stuck in Pending
```bash
# Check events
kubectl describe pod <pod-name> -n <namespace>
# Common causes:
# - Insufficient resources
kubectl describe nodes | grep -A 5 "Allocated resources"
# - No matching nodes (taints/tolerations)
kubectl get nodes -o json | jq '.items[].spec.taints'
# - PVC not bound
kubectl get pvc -n <namespace>
```
#### Pod in CrashLoopBackOff
```bash
# Check logs
kubectl logs <pod-name> -n <namespace> --previous
# Check container exit code
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
# Common exit codes:
# 0 - Success (check livenessProbe)
# 1 - Application error
# 137 - OOMKilled (increase memory)
# 139 - Segmentation fault
# 143 - SIGTERM received
```
#### Pod in ImagePullBackOff
```bash
# Check image name
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[0].image}'
# Verify image exists
docker pull <image>
# Check imagePullSecrets
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.imagePullSecrets}'
kubectl get secret <secret-name> -n <namespace> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
```
### Service Issues
#### Service Not Accessible
```bash
# Verify endpoints exist
kubectl get endpoints <service-name> -n <namespace>
# Check selector matches pod labels
kubectl get svc <service-name> -n <namespace> -o jsonpath='{.spec.selector}'
kubectl get pods -n <namespace> --show-labels
# Test from within cluster
kubectl run debug --rm -it --image=busybox -- wget -qO- http://<service>.<namespace>.svc.cluster.local
```
#### DNS Resolution Issues
```bash
# Test DNS from pod
kubectl run dns-test --rm -it --image=busybox -- nslookup kubernetes.default
# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns
# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns
```
### Node Issues
#### Node NotReady
```bash
# Check node conditions
kubectl describe node <node-name> | grep -A 20 Conditions
# Check kubelet status
systemctl status kubelet
# Check kubelet logs
journalctl -u kubelet -f
# Common causes:
# - Disk pressure
# - Memory pressure
# - Network issues
# - Container runtime issues
```
#### Node Disk Pressure
```bash
# Check disk usage
kubectl describe node <node-name> | grep -A 3 "Allocated resources"
# Cleanup unused images
docker system prune -af
# Check for large logs
du -sh /var/log/containers/*
```
### Networking Issues
#### Pod-to-Pod Communication Fails
```bash
# Test connectivity
kubectl exec <pod-a> -- ping <pod-b-ip>
# Check network policies
kubectl get networkpolicies -n <namespace>
# Verify CNI plugin
kubectl get pods -n kube-system | grep -E "calico|weave|flannel|cilium"
```
### Storage Issues
#### PVC Stuck in Pending
```bash
# Check PVC events
kubectl describe pvc <pvc-name> -n <namespace>
# Verify StorageClass exists
kubectl get storageclass
# Check provisioner pods
kubectl get pods -n kube-system | grep provisioner
```
## Diagnostic Commands Cheat Sheet
```bash
# Cluster overview
kubectl cluster-info
kubectl get componentstatuses
# Resource usage
kubectl top nodes
kubectl top pods -n <namespace>
# Events
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
# Logs
kubectl logs -f <pod> -n <namespace>
kubectl logs -f <pod> -n <namespace> --all-containers
# Exec into pod
kubectl exec -it <pod> -n <namespace> -- /bin/sh
# Port forward
kubectl port-forward <pod> 8080:80 -n <namespace>
# Copy files
kubectl cp <namespace>/<pod>:/path/to/file ./local-file
```