External/DevOps-Security-Agent-Skills
1
0
Fork 0
mirror of https://github.com/BagelHole/DevOps-Security-Agent-Skills.git synced 2026-08-22 12:49:53 +02:00
Code Issues Packages Projects Releases Wiki Activity

V2

Browse Source
This commit is contained in:
Toby
2026-03-24 18:02:50 -04:00
parent 2d209b9258
commit ba9e489584
111 changed files with 48382 additions and 3099 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infrastructure/databases/database-backups/SKILL.md
+376 -38
View File
@@ -9,62 +9,400 @@ metadata:
# Database Backups
Implement comprehensive database backup strategies.
Implement comprehensive, automated database backup strategies with tested recovery procedures.
## When to Use
- You are deploying a new database and need a backup plan from day one.
- You need to automate nightly or hourly backups for PostgreSQL, MySQL, or MongoDB.
- You want to ship backups to S3-compatible object storage with retention policies.
- You are building or verifying disaster recovery runbooks.
## Prerequisites
- Database client tools installed (`pg_dump`, `mysqldump`, `mongodump`).
- AWS CLI or `restic` for remote storage.
- `cron` or systemd timers for scheduling.
- An S3 bucket (or S3-compatible endpoint) for offsite backups.
## Backup Types
```yaml
backup_types:
full:
description: Complete database copy
frequency: Weekly
incremental:
description: Changes since last backup
frequency: Daily
transaction_log:
description: Continuous transaction logging
frequency: Continuous
```
| Type | Description | Frequency | Use Case |
|---|---|---|---|
| Full | Complete database copy | Weekly | Baseline for restores |
| Incremental | Changes since last backup | Daily | Reduce storage and time |
| Transaction log / WAL | Continuous log shipping | Continuous | Point-in-time recovery (PITR) |
| Snapshot | Storage-level snapshot (EBS, ZFS) | Daily | Fast full restores |
## Automated Backup Script
## PostgreSQL Backups
### Logical Backup with pg_dump
```bash
#!/bin/bash
# pg_backup.sh — PostgreSQL logical backup
set -euo pipefail
DB_NAME="mydb"
DB_USER="backup_user"
DB_HOST="localhost"
BACKUP_DIR="/backups/postgres"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups"
FILENAME="${BACKUP_DIR}/${DB_NAME}_${DATE}.dump"
# PostgreSQL
pg_dump -Fc mydb > $BACKUP_DIR/pg_$DATE.dump
mkdir -p "$BACKUP_DIR"
# MySQL
mysqldump -u root -p$MYSQL_PWD mydb | gzip > $BACKUP_DIR/mysql_$DATE.sql.gz
# Custom compressed format (recommended for selective restore)
pg_dump -h "$DB_HOST" -U "$DB_USER" -Fc -Z6 "$DB_NAME" > "$FILENAME"
# Upload to S3
aws s3 cp $BACKUP_DIR/pg_$DATE.dump s3://backups/postgres/
# Cleanup old backups (keep 7 days)
find $BACKUP_DIR -name "*.dump" -mtime +7 -delete
echo "[$(date)] PostgreSQL backup complete: $FILENAME ($(du -h "$FILENAME" | cut -f1))"
```
## Recovery Testing
### Physical Backup with pg_basebackup
```bash
# Create test environment
docker run -d --name restore-test postgres:15
#!/bin/bash
# pg_basebackup.sh — PostgreSQL physical backup for PITR
set -euo pipefail
# Restore backup
pg_restore -d testdb backup.dump
BACKUP_DIR="/backups/postgres/base_$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"
# Verify data integrity
psql testdb -c "SELECT COUNT(*) FROM users;"
pg_basebackup \
-h localhost \
-U replicator \
-D "$BACKUP_DIR" \
--wal-method=stream \
--checkpoint=fast \
--progress \
--verbose
echo "[$(date)] Base backup complete: $BACKUP_DIR"
```
### PostgreSQL Restore
```bash
# Restore from custom-format dump
pg_restore -h localhost -U myapp -d mydb --clean --if-exists /backups/postgres/mydb_20250115_020000.dump
# Restore a single table
pg_restore -h localhost -U myapp -d mydb -t orders /backups/postgres/mydb_20250115_020000.dump
# Restore from plain SQL
psql -h localhost -U myapp -d mydb < /backups/postgres/mydb_20250115.sql
```
## MySQL Backups
### Logical Backup with mysqldump
```bash
#!/bin/bash
# mysql_backup.sh — MySQL logical backup
set -euo pipefail
DB_NAME="mydb"
DB_USER="backup_user"
DB_PASS="${MYSQL_BACKUP_PASSWORD}"
BACKUP_DIR="/backups/mysql"
DATE=$(date +%Y%m%d_%H%M%S)
FILENAME="${BACKUP_DIR}/${DB_NAME}_${DATE}.sql.gz"
mkdir -p "$BACKUP_DIR"
mysqldump -u "$DB_USER" -p"$DB_PASS" \
--single-transaction \
--routines \
--triggers \
--events \
"$DB_NAME" | gzip > "$FILENAME"
echo "[$(date)] MySQL backup complete: $FILENAME ($(du -h "$FILENAME" | cut -f1))"
```
### Physical Backup with Percona XtraBackup
```bash
#!/bin/bash
# xtrabackup.sh — MySQL physical backup
set -euo pipefail
BACKUP_DIR="/backups/mysql/full_$(date +%Y%m%d)"
xtrabackup --backup \
--user=backup_user \
--password="${MYSQL_BACKUP_PASSWORD}" \
--target-dir="$BACKUP_DIR"
xtrabackup --prepare --target-dir="$BACKUP_DIR"
echo "[$(date)] XtraBackup complete: $BACKUP_DIR"
```
### MySQL Restore
```bash
# Restore from compressed mysqldump
gunzip < /backups/mysql/mydb_20250115_020000.sql.gz | mysql -u root -p mydb
# Restore from XtraBackup
sudo systemctl stop mysql
sudo rm -rf /var/lib/mysql/*
xtrabackup --move-back --target-dir=/backups/mysql/full_20250115
sudo chown -R mysql:mysql /var/lib/mysql
sudo systemctl start mysql
```
## MongoDB Backups
### Logical Backup with mongodump
```bash
#!/bin/bash
# mongo_backup.sh — MongoDB backup
set -euo pipefail
MONGO_URI="mongodb://backup_user:${MONGO_BACKUP_PASSWORD}@localhost:27017"
BACKUP_DIR="/backups/mongodb"
DATE=$(date +%Y%m%d_%H%M%S)
TARGET="${BACKUP_DIR}/${DATE}"
mkdir -p "$BACKUP_DIR"
# Full backup with compression
mongodump --uri="$MONGO_URI" --gzip --out="$TARGET"
echo "[$(date)] MongoDB backup complete: $TARGET"
```
### MongoDB Restore
```bash
# Restore all databases
mongorestore --uri="mongodb://admin:secret@localhost:27017" \
--gzip --drop /backups/mongodb/20250115_020000/
# Restore a single database
mongorestore --uri="mongodb://admin:secret@localhost:27017" \
--gzip --drop --db mydb /backups/mongodb/20250115_020000/mydb/
# Restore a single collection
mongorestore --uri="mongodb://admin:secret@localhost:27017" \
--gzip --drop --db mydb --collection users \
/backups/mongodb/20250115_020000/mydb/users.bson.gz
```
## Upload to S3
```bash
#!/bin/bash
# s3_upload.sh — Upload backups to S3
set -euo pipefail
S3_BUCKET="s3://my-backups"
BACKUP_DIR="/backups"
DATE=$(date +%Y%m%d)
# Upload PostgreSQL backup
aws s3 cp "${BACKUP_DIR}/postgres/" "${S3_BUCKET}/postgres/${DATE}/" \
--recursive --storage-class STANDARD_IA \
--sse AES256
# Upload MySQL backup
aws s3 cp "${BACKUP_DIR}/mysql/" "${S3_BUCKET}/mysql/${DATE}/" \
--recursive --storage-class STANDARD_IA \
--sse AES256
# Upload MongoDB backup
aws s3 cp "${BACKUP_DIR}/mongodb/" "${S3_BUCKET}/mongodb/${DATE}/" \
--recursive --storage-class STANDARD_IA \
--sse AES256
echo "[$(date)] S3 upload complete for ${DATE}"
```
### S3 Lifecycle Policy for Retention
```json
{
"Rules": [
{
"ID": "BackupRetention",
"Status": "Enabled",
"Filter": { "Prefix": "" },
"Transitions": [
{ "Days": 30, "StorageClass": "GLACIER" }
],
"Expiration": { "Days": 365 }
}
]
}
```
```bash
aws s3api put-bucket-lifecycle-configuration \
--bucket my-backups \
--lifecycle-configuration file://lifecycle.json
```
## Restic Backup (Encrypted, Deduplicated)
```bash
# Initialize a restic repository on S3
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."
export RESTIC_PASSWORD="strong_encryption_password"
export RESTIC_REPOSITORY="s3:s3.amazonaws.com/my-backups-restic"
restic init
# Backup the local backup directory
restic backup /backups/postgres /backups/mysql /backups/mongodb
# List snapshots
restic snapshots
# Prune old snapshots — keep 7 daily, 4 weekly, 6 monthly
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
# Restore a snapshot
restic restore latest --target /restore/
```
## Cron Schedules
```bash
# /etc/cron.d/db-backups
# PostgreSQL: nightly at 02:00
0 2 * * * backup /opt/scripts/pg_backup.sh >> /var/log/backup-pg.log 2>&1
# MySQL: nightly at 02:30
30 2 * * * backup /opt/scripts/mysql_backup.sh >> /var/log/backup-mysql.log 2>&1
# MongoDB: nightly at 03:00
0 3 * * * backup /opt/scripts/mongo_backup.sh >> /var/log/backup-mongo.log 2>&1
# Upload to S3: daily at 04:00
0 4 * * * backup /opt/scripts/s3_upload.sh >> /var/log/backup-s3.log 2>&1
# Local cleanup: keep 7 days of local backups
0 5 * * * backup find /backups -type f -mtime +7 -delete >> /var/log/backup-cleanup.log 2>&1
# Restic prune: weekly on Sunday at 06:00
0 6 * * 0 backup /opt/scripts/restic_prune.sh >> /var/log/backup-restic.log 2>&1
```
## Automated Recovery Testing
```bash
#!/bin/bash
# verify_backup.sh — weekly restore test
set -euo pipefail
BACKUP_FILE=$(ls -t /backups/postgres/mydb_*.dump | head -1)
echo "[$(date)] Starting backup verification with $BACKUP_FILE"
# Spin up a temporary PostgreSQL container
docker run -d --name pg-restore-test \
-e POSTGRES_USER=testuser \
-e POSTGRES_PASSWORD=testpass \
-e POSTGRES_DB=testdb \
postgres:16-alpine
# Wait for container to be ready
sleep 5
until docker exec pg-restore-test pg_isready -U testuser; do
sleep 2
done
# Copy backup into container and restore
docker cp "$BACKUP_FILE" pg-restore-test:/tmp/backup.dump
docker exec pg-restore-test pg_restore -U testuser -d testdb --clean --if-exists /tmp/backup.dump
# Run verification queries
USERS_COUNT=$(docker exec pg-restore-test psql -U testuser -d testdb -tAc "SELECT COUNT(*) FROM users;")
ORDERS_COUNT=$(docker exec pg-restore-test psql -U testuser -d testdb -tAc "SELECT COUNT(*) FROM orders;")
echo "[$(date)] Verification: users=$USERS_COUNT, orders=$ORDERS_COUNT"
# Cleanup
docker rm -f pg-restore-test
# Alert on failure
if [ "$USERS_COUNT" -lt 1 ]; then
echo "ALERT: Backup verification failed — users table is empty" >&2
exit 1
fi
echo "[$(date)] Backup verification PASSED"
```
## Unified Backup Script
```bash
#!/bin/bash
# backup_all.sh — unified backup orchestrator
set -euo pipefail
LOG="/var/log/backup-all.log"
ALERT_EMAIL="ops@example.com"
ERRORS=0
log() { echo "[$(date '+%F %T')] $*" | tee -a "$LOG"; }
run_backup() {
local name="$1" script="$2"
log "Starting $name backup..."
if bash "$script" >> "$LOG" 2>&1; then
log "$name backup succeeded."
else
log "ERROR: $name backup FAILED."
ERRORS=$((ERRORS + 1))
fi
}
run_backup "PostgreSQL" /opt/scripts/pg_backup.sh
run_backup "MySQL" /opt/scripts/mysql_backup.sh
run_backup "MongoDB" /opt/scripts/mongo_backup.sh
run_backup "S3 Upload" /opt/scripts/s3_upload.sh
if [ "$ERRORS" -gt 0 ]; then
log "Backup run completed with $ERRORS error(s). Sending alert."
mail -s "BACKUP ALERT: $ERRORS failure(s)" "$ALERT_EMAIL" < "$LOG"
exit 1
fi
log "All backups completed successfully."
```
## Best Practices
- 3-2-1 Rule: 3 copies, 2 media types, 1 offsite
- Regular recovery testing
- Encrypt backups at rest
- Monitor backup success
- Document recovery procedures
- **3-2-1 Rule**: Keep 3 copies of data, on 2 different media types, with 1 offsite.
- **Encrypt backups at rest**: Use `restic` (built-in encryption), AWS SSE, or `gpg`.
- **Test restores regularly**: A backup that has never been restored is not a backup.
- **Monitor backup jobs**: Alert immediately on any failure; do not rely on silent cron jobs.
- **Document RTOs and RPOs**: Define Recovery Time Objective and Recovery Point Objective for each database.
- **Version your backup scripts**: Store them in Git alongside your infrastructure code.
- **Use `--single-transaction`**: For MySQL and PostgreSQL logical backups to get a consistent snapshot.
- **Separate backup credentials**: Use a dedicated read-only database user for backups.
## Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| `pg_dump: too many clients` | Backup connection competes with app pool | Schedule during low traffic; increase `max_connections` by 5 for backup user |
| `mysqldump` hangs on large table | Table lock contention | Use `--single-transaction` (InnoDB) or schedule during maintenance window |
| `mongodump` slow on replica | Reading from secondary under load | Use `--readPreference=secondaryPreferred` and schedule off-peak |
| S3 upload fails with timeout | Large file over slow connection | Use `aws s3 cp --expected-size` or multipart with `aws s3api` |
| Restic prune takes hours | Too many snapshots accumulated | Run `restic forget --prune` more frequently; limit snapshot count |
| Restore fails with "role does not exist" | Backup includes role-dependent objects | Create roles first or use `--no-owner --no-privileges` on restore |
## Related Skills
- [postgresql](../postgresql/) - PostgreSQL administration and pg_dump details
- [mysql](../mysql/) - MySQL administration and mysqldump details
- [mongodb](../mongodb/) - MongoDB administration and mongodump details
- [redis](../redis/) - Redis RDB/AOF persistence and backup
infrastructure/databases/mongodb/SKILL.md
+391 -50
View File
@@ -9,71 +9,412 @@ metadata:
# MongoDB
Administer MongoDB NoSQL databases.
Administer, optimize, and secure MongoDB NoSQL databases in development and production environments.
## Installation & Setup
## When to Use
- You need a document-oriented database with flexible schemas.
- Your data is semi-structured or heavily nested (JSON-like documents).
- You need horizontal scaling through sharding.
- Your application benefits from rich querying and aggregation pipelines.
## Prerequisites
- Linux server (Debian/Ubuntu or RHEL-based) or Docker.
- Root or sudo access for package installation.
- MongoDB 7.x recommended for production (6.x still supported).
## Installation and Setup
```bash
# Install
apt install mongodb-org
# Debian / Ubuntu — MongoDB 7
curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \
sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor
echo "deb [ signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] \
https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | \
sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
sudo apt update
sudo apt install -y mongodb-org
# Start service
systemctl start mongod
# Start and enable
sudo systemctl enable --now mongod
# Connect
mongosh
# Verify
mongosh --eval "db.version()"
```
## Initial User Setup
```javascript
// Connect without auth first
// mongosh
# Create user
use admin
// Create admin user
db.createUser({
user: "admin",
pwd: "secret",
roles: ["root"]
})
```
## Basic Operations
```javascript
// Create database and collection
use mydb
db.users.insertOne({ name: "John", email: "john@example.com" })
// Query
db.users.find({ name: "John" })
db.users.find().sort({ name: 1 }).limit(10)
// Index
db.users.createIndex({ email: 1 }, { unique: true })
```
## Replica Set
```javascript
// Initialize replica set
rs.initiate({
_id: "myReplicaSet",
members: [
{ _id: 0, host: "mongo1:27017" },
{ _id: 1, host: "mongo2:27017" },
{ _id: 2, host: "mongo3:27017" }
pwd: "strong_admin_password",
roles: [
{ role: "userAdminAnyDatabase", db: "admin" },
{ role: "readWriteAnyDatabase", db: "admin" },
{ role: "clusterAdmin", db: "admin" }
]
})
// Create an application-scoped user
use mydb
db.createUser({
user: "myapp",
pwd: "strong_app_password",
roles: [{ role: "readWrite", db: "mydb" }]
})
```
## Backup
Enable authentication in `/etc/mongod.conf`:
```yaml
security:
authorization: enabled
```
```bash
# Backup
mongodump --out /backup/
# Restore
mongorestore /backup/
sudo systemctl restart mongod
# Now connect with credentials
mongosh -u myapp -p strong_app_password --authenticationDatabase mydb
```
## Best Practices
## mongosh Commands Reference
- Use replica sets in production
- Implement proper indexing
- Enable authentication
- Regular backups with mongodump
```javascript
// Show databases and collections
show dbs
use mydb
show collections
// Insert documents
db.users.insertOne({ name: "Alice", email: "alice@example.com", age: 30 })
db.users.insertMany([
{ name: "Bob", email: "bob@example.com", age: 25 },
{ name: "Carol", email: "carol@example.com", age: 35 }
])
// Query documents
db.users.find({ age: { $gte: 25 } }).sort({ name: 1 }).limit(10)
db.users.findOne({ email: "alice@example.com" })
db.users.countDocuments({ age: { $gte: 30 } })
// Update
db.users.updateOne(
{ email: "alice@example.com" },
{ $set: { age: 31 }, $currentDate: { updatedAt: true } }
)
db.users.updateMany(
{ age: { $lt: 30 } },
{ $set: { tier: "junior" } }
)
// Delete
db.users.deleteOne({ email: "bob@example.com" })
db.users.deleteMany({ tier: "junior" })
```
## Indexing
```javascript
// Single-field index
db.users.createIndex({ email: 1 }, { unique: true })
// Compound index
db.orders.createIndex({ userId: 1, createdAt: -1 })
// Text index for search
db.articles.createIndex({ title: "text", body: "text" })
db.articles.find({ $text: { $search: "mongodb scaling" } })
// TTL index — auto-delete documents after 30 days
db.sessions.createIndex({ createdAt: 1 }, { expireAfterSeconds: 2592000 })
// List indexes
db.users.getIndexes()
// Drop an index
db.users.dropIndex("email_1")
// Explain a query to verify index usage
db.orders.find({ userId: 42 }).explain("executionStats")
```
## Aggregation Pipeline Examples
```javascript
// Revenue per status
db.orders.aggregate([
{ $group: {
_id: "$status",
totalRevenue: { $sum: "$total" },
count: { $sum: 1 }
}},
{ $sort: { totalRevenue: -1 } }
])
// Top 5 customers by order value (with a join)
db.orders.aggregate([
{ $group: {
_id: "$userId",
spent: { $sum: "$total" },
orderCount: { $sum: 1 }
}},
{ $sort: { spent: -1 } },
{ $limit: 5 },
{ $lookup: {
from: "users",
localField: "_id",
foreignField: "_id",
as: "user"
}},
{ $unwind: "$user" },
{ $project: {
_id: 0,
name: "$user.name",
email: "$user.email",
spent: 1,
orderCount: 1
}}
])
// Daily signup trend
db.users.aggregate([
{ $group: {
_id: { $dateToString: { format: "%Y-%m-%d", date: "$createdAt" } },
signups: { $sum: 1 }
}},
{ $sort: { _id: 1 } },
{ $limit: 30 }
])
```
## Replica Set Setup
A replica set requires a minimum of three members (or two data-bearing nodes plus an arbiter).
### Configuration File for Each Member
```yaml
# /etc/mongod.conf (adjust port and dbPath per member)
storage:
dbPath: /var/lib/mongodb
net:
port: 27017
bindIp: 0.0.0.0
replication:
replSetName: rs0
security:
authorization: enabled
keyFile: /etc/mongodb-keyfile
```
```bash
# Generate a shared keyfile for internal auth
openssl rand -base64 756 > /etc/mongodb-keyfile
chmod 400 /etc/mongodb-keyfile
chown mongodb:mongodb /etc/mongodb-keyfile
# Copy this file to all replica set members
```
### Initialize the Replica Set
```javascript
// Connect to the first member
// mongosh --port 27017
rs.initiate({
_id: "rs0",
members: [
{ _id: 0, host: "mongo1:27017", priority: 2 },
{ _id: 1, host: "mongo2:27017", priority: 1 },
{ _id: 2, host: "mongo3:27017", priority: 1 }
]
})
// Check status
rs.status()
// View replication lag per member
rs.printReplicationInfo()
rs.printSecondaryReplicationInfo()
```
## Backup and Restore
```bash
# Full dump of all databases
mongodump --uri="mongodb://admin:secret@localhost:27017" --out=/backups/full_$(date +%F)
# Single database
mongodump --uri="mongodb://myapp:secret@localhost:27017/mydb" --out=/backups/mydb_$(date +%F)
# Compressed dump
mongodump --uri="mongodb://admin:secret@localhost:27017" --gzip --out=/backups/gz_$(date +%F)
# Restore all databases
mongorestore --uri="mongodb://admin:secret@localhost:27017" /backups/full_2025-01-15/
# Restore a single database, dropping existing data first
mongorestore --uri="mongodb://admin:secret@localhost:27017" \
--drop --db mydb /backups/mydb_2025-01-15/mydb/
# Restore compressed dump
mongorestore --uri="mongodb://admin:secret@localhost:27017" --gzip /backups/gz_2025-01-15/
```
## Docker Compose Setup
```yaml
# docker-compose.yml
version: "3.9"
services:
mongo1:
image: mongo:7
restart: unless-stopped
ports:
- "27017:27017"
environment:
MONGO_INITDB_ROOT_USERNAME: admin
MONGO_INITDB_ROOT_PASSWORD: secret
volumes:
- mongo1_data:/data/db
- ./mongo-keyfile:/etc/mongodb-keyfile:ro
command: >
mongod
--replSet rs0
--keyFile /etc/mongodb-keyfile
--bind_ip_all
healthcheck:
test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
interval: 10s
timeout: 5s
retries: 5
mongo2:
image: mongo:7
restart: unless-stopped
volumes:
- mongo2_data:/data/db
- ./mongo-keyfile:/etc/mongodb-keyfile:ro
command: >
mongod
--replSet rs0
--keyFile /etc/mongodb-keyfile
--bind_ip_all
mongo3:
image: mongo:7
restart: unless-stopped
volumes:
- mongo3_data:/data/db
- ./mongo-keyfile:/etc/mongodb-keyfile:ro
command: >
mongod
--replSet rs0
--keyFile /etc/mongodb-keyfile
--bind_ip_all
mongo-init:
image: mongo:7
restart: "no"
depends_on:
mongo1:
condition: service_healthy
entrypoint: >
mongosh --host mongo1 -u admin -p secret --authenticationDatabase admin --eval '
rs.initiate({
_id: "rs0",
members: [
{ _id: 0, host: "mongo1:27017", priority: 2 },
{ _id: 1, host: "mongo2:27017", priority: 1 },
{ _id: 2, host: "mongo3:27017", priority: 1 }
]
})
'
volumes:
mongo1_data:
mongo2_data:
mongo3_data:
```
```bash
# Generate keyfile before starting
openssl rand -base64 756 > mongo-keyfile
chmod 400 mongo-keyfile
docker compose up -d
# Connect
mongosh "mongodb://admin:secret@127.0.0.1:27017/?replicaSet=rs0&authSource=admin"
```
## Monitoring Queries
```javascript
// Server status summary
db.serverStatus().connections
db.serverStatus().opcounters
// Current operations (look for long-running queries)
db.currentOp({ secs_running: { $gte: 5 } })
// Collection stats
db.orders.stats()
// Index sizes
db.orders.stats().indexSizes
// Profiler — log slow queries (> 100ms)
db.setProfilingLevel(1, { slowms: 100 })
db.system.profile.find().sort({ ts: -1 }).limit(5)
// Replica set lag
rs.printSecondaryReplicationInfo()
```
## Configuration Tuning
```yaml
# /etc/mongod.conf — production recommendations
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
wiredTiger:
engineConfig:
cacheSizeGB: 4 # ~50% of RAM, leave rest for OS cache
collectionConfig:
blockCompressor: snappy
net:
port: 27017
bindIp: 0.0.0.0
maxIncomingConnections: 500
operationProfiling:
mode: slowOp
slowOpThresholdMs: 100
```
## Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| `COLLSCAN` in explain output | Missing index on queried field | Create an appropriate index |
| Replica member stuck in `RECOVERING` | Oplog window exceeded | Resync by removing data and restarting the member |
| `too many open files` | OS file descriptor limit too low | Set `ulimit -n 65535` in service file |
| High memory usage | WiredTiger cache too large | Reduce `cacheSizeGB` in config |
| Slow aggregation pipelines | No index on `$match` stage fields | Add index; place `$match` as early as possible in pipeline |
| Authentication failure | Wrong `authenticationDatabase` | Specify `--authenticationDatabase admin` for admin users |
## Related Skills
- [redis](../redis/) - Caching layer in front of MongoDB
- [database-backups](../database-backups/) - Automated backup strategies
- [postgresql](../postgresql/) - Alternative relational database
infrastructure/databases/mysql/SKILL.md
+339 -44
View File
@@ -9,70 +9,365 @@ metadata:
# MySQL / MariaDB
Administer MySQL and MariaDB databases.
Administer, optimize, and secure MySQL and MariaDB databases in development and production environments.
## Installation & Setup
## When to Use
- You need a mature, widely supported relational database.
- Your stack depends on MySQL-specific features or compatibility (WordPress, Magento, many PHP frameworks).
- You are setting up source-replica replication for read scaling.
- You want to tune InnoDB for high-throughput transactional workloads.
## Prerequisites
- Linux server (Debian/Ubuntu or RHEL-based) or Docker.
- Root or sudo access for package installation.
- Familiarity with SQL fundamentals.
## Installation and Setup
```bash
# Debian / Ubuntu — MySQL 8
sudo apt update
sudo apt install -y mysql-server
# RHEL / Amazon Linux
sudo dnf install -y mysql-server
sudo systemctl enable --now mysqld
# Run the secure installation wizard
sudo mysql_secure_installation
# Prompts: set root password, remove anonymous users, disable remote root, remove test db
# Verify
mysql --version
sudo systemctl status mysql
```
## Initial User and Database Setup
```bash
sudo mysql -u root -p
```
```sql
-- Create a database
CREATE DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- Create an application user with strong auth
CREATE USER 'myapp'@'%' IDENTIFIED BY 'strong_password_here';
GRANT SELECT, INSERT, UPDATE, DELETE ON mydb.* TO 'myapp'@'%';
FLUSH PRIVILEGES;
-- Verify
SHOW GRANTS FOR 'myapp'@'%';
```
## mysql CLI Reference
```bash
# Connect
mysql -u myapp -p -h 127.0.0.1 mydb
# Execute a single statement
mysql -u myapp -p -e "SELECT COUNT(*) FROM orders;" mydb
# Import a SQL file
mysql -u myapp -p mydb < schema.sql
# Export query results to CSV
mysql -u myapp -p -e "SELECT * FROM users" mydb \
| tr '\t' ',' > users.csv
```
```
-- Inside the mysql shell
SHOW DATABASES;
USE mydb;
SHOW TABLES;
DESCRIBE users;
SHOW CREATE TABLE users\G
SHOW PROCESSLIST;
SHOW ENGINE INNODB STATUS\G
```
## Configuration Tuning
Edit `/etc/mysql/mysql.conf.d/mysqld.cnf` (or `/etc/my.cnf` on RHEL).
```ini
[mysqld]
# -- Networking --
bind-address = 0.0.0.0
max_connections = 300
wait_timeout = 600
interactive_timeout = 600
# -- InnoDB (most impactful settings) --
innodb_buffer_pool_size = 4G # ~70% of RAM on a dedicated server
innodb_buffer_pool_instances = 4 # 1 per GB of pool (up to 64)
innodb_log_file_size = 1G
innodb_flush_log_at_trx_commit = 1 # 1 = ACID; 2 = faster, slight risk
innodb_flush_method = O_DIRECT # avoids double buffering on Linux
innodb_io_capacity = 2000 # raise for SSD
innodb_io_capacity_max = 4000
# -- Query cache (disabled in MySQL 8, use ProxySQL or app cache) --
# query_cache_type = 0
# -- Logging --
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 1
log_error = /var/log/mysql/error.log
# -- Binary log (required for replication) --
server-id = 1
log_bin = /var/log/mysql/mysql-bin
binlog_expire_logs_seconds = 604800 # 7 days
sync_binlog = 1
# -- Character set --
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
```
```bash
# Apply changes
sudo systemctl restart mysql
# Verify a setting at runtime
mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';"
```
## Backup and Restore
### Logical Backups with mysqldump
```bash
# Single database
mysqldump -u root -p --single-transaction --routines --triggers \
mydb > /backups/mydb_$(date +%F).sql
# All databases
mysqldump -u root -p --all-databases --single-transaction \
> /backups/all_$(date +%F).sql
# Compressed backup
mysqldump -u root -p --single-transaction mydb \
| gzip > /backups/mydb_$(date +%F).sql.gz
# Restore
mysql -u root -p mydb < /backups/mydb_2025-01-15.sql
# Restore compressed
gunzip < /backups/mydb_2025-01-15.sql.gz | mysql -u root -p mydb
```
### Physical Backups with Percona XtraBackup
```bash
# Install
apt install mysql-server
sudo apt install -y percona-xtrabackup-80
# Secure installation
mysql_secure_installation
# Full backup
xtrabackup --backup --user=root --password=secret \
--target-dir=/backups/full_$(date +%F)
# Access
mysql -u root -p
# Prepare the backup (apply redo logs)
xtrabackup --prepare --target-dir=/backups/full_2025-01-15
# Create database and user
CREATE DATABASE mydb;
CREATE USER 'myapp'@'%' IDENTIFIED BY 'secret';
GRANT ALL PRIVILEGES ON mydb.* TO 'myapp'@'%';
FLUSH PRIVILEGES;
# Restore (stop MySQL first)
sudo systemctl stop mysql
sudo rm -rf /var/lib/mysql/*
xtrabackup --move-back --target-dir=/backups/full_2025-01-15
sudo chown -R mysql:mysql /var/lib/mysql
sudo systemctl start mysql
```
## Configuration
### Incremental Backup with XtraBackup
```bash
# Incremental based on the full backup
xtrabackup --backup --user=root --password=secret \
--target-dir=/backups/inc_$(date +%F) \
--incremental-basedir=/backups/full_2025-01-15
# Prepare: apply full, then incremental
xtrabackup --prepare --apply-log-only --target-dir=/backups/full_2025-01-15
xtrabackup --prepare --target-dir=/backups/full_2025-01-15 \
--incremental-dir=/backups/inc_2025-01-16
```
## Source-Replica Replication
### Source (Primary)
```ini
# /etc/mysql/mysql.conf.d/mysqld.cnf
[mysqld]
innodb_buffer_pool_size = 1G
max_connections = 200
slow_query_log = 1
long_query_time = 2
server-id = 1
log_bin = /var/log/mysql/mysql-bin
binlog_format = ROW
```
## Backup & Restore
```sql
-- Create replication user
CREATE USER 'replicator'@'10.0.0.%' IDENTIFIED BY 'repl_secret';
GRANT REPLICATION SLAVE ON *.* TO 'replicator'@'10.0.0.%';
FLUSH PRIVILEGES;
```bash
# Backup
mysqldump -u root -p mydb > backup.sql
mysqldump -u root -p --all-databases > full_backup.sql
# Restore
mysql -u root -p mydb < backup.sql
-- Get current binary log position
SHOW MASTER STATUS\G
```
## Replication
### Replica
```bash
# Primary
```ini
# /etc/mysql/mysql.conf.d/mysqld.cnf
[mysqld]
server-id = 1
log_bin = mysql-bin
# Replica
CHANGE MASTER TO
MASTER_HOST='primary',
MASTER_USER='replicator',
MASTER_PASSWORD='secret',
MASTER_LOG_FILE='mysql-bin.000001',
MASTER_LOG_POS=0;
START SLAVE;
server-id = 2
relay_log = /var/log/mysql/relay-bin
read_only = ON
```
## Best Practices
```sql
-- Point replica to source (use SHOW MASTER STATUS values)
CHANGE REPLICATION SOURCE TO
SOURCE_HOST = '10.0.0.1',
SOURCE_USER = 'replicator',
SOURCE_PASSWORD = 'repl_secret',
SOURCE_LOG_FILE = 'mysql-bin.000003',
SOURCE_LOG_POS = 154;
- Enable slow query logging
- Use InnoDB storage engine
- Regular backups with mysqldump
- Monitor with SHOW PROCESSLIST
START REPLICA;
-- Verify
SHOW REPLICA STATUS\G
-- Check: Replica_IO_Running = Yes, Replica_SQL_Running = Yes, Seconds_Behind_Source = 0
```
## Monitoring Queries
```sql
-- Connection statistics
SHOW STATUS LIKE 'Threads_connected';
SHOW STATUS LIKE 'Max_used_connections';
-- InnoDB buffer pool hit ratio (should be > 99%)
SELECT
ROUND(100 - (
(SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME = 'Innodb_buffer_pool_reads') /
(SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME = 'Innodb_buffer_pool_read_requests')
) * 100, 2) AS buffer_pool_hit_pct;
-- Top 10 slow queries (requires performance_schema)
SELECT DIGEST_TEXT, COUNT_STAR, AVG_TIMER_WAIT / 1e12 AS avg_sec
FROM performance_schema.events_statements_summary_by_digest
ORDER BY AVG_TIMER_WAIT DESC
LIMIT 10;
-- Table sizes
SELECT table_name,
ROUND(data_length / 1024 / 1024, 2) AS data_mb,
ROUND(index_length / 1024 / 1024, 2) AS index_mb,
table_rows
FROM information_schema.tables
WHERE table_schema = 'mydb'
ORDER BY data_length DESC;
-- Check replication lag
SHOW REPLICA STATUS\G
-- Look at Seconds_Behind_Source
```
## Docker Compose Setup
```yaml
# docker-compose.yml
version: "3.9"
services:
mysql:
image: mysql:8.0
restart: unless-stopped
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: rootpass
MYSQL_DATABASE: mydb
MYSQL_USER: myapp
MYSQL_PASSWORD: secret
volumes:
- mysql_data:/var/lib/mysql
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
command: >
--innodb-buffer-pool-size=512M
--max-connections=200
--slow-query-log=ON
--long-query-time=1
--character-set-server=utf8mb4
--collation-server=utf8mb4_unicode_ci
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-prootpass"]
interval: 10s
timeout: 5s
retries: 5
phpmyadmin:
image: phpmyadmin:latest
restart: unless-stopped
ports:
- "8080:80"
environment:
PMA_HOST: mysql
PMA_USER: root
PMA_PASSWORD: rootpass
depends_on:
mysql:
condition: service_healthy
volumes:
mysql_data:
```
```bash
docker compose up -d
mysql -h 127.0.0.1 -u myapp -psecret mydb
```
## Maintenance Tasks
```bash
# Optimize a fragmented table (locks the table briefly)
mysql -u root -p -e "OPTIMIZE TABLE mydb.orders;"
# Analyze tables to update statistics
mysql -u root -p -e "ANALYZE TABLE mydb.orders;"
# Check and repair a table
mysql -u root -p -e "CHECK TABLE mydb.orders;"
mysql -u root -p -e "REPAIR TABLE mydb.orders;"
# Rotate slow query log
sudo mv /var/log/mysql/slow.log /var/log/mysql/slow.log.old
mysqladmin -u root -p flush-logs
```
## Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| `Too many connections` | Connection limit exceeded | Increase `max_connections`; use connection pooling (ProxySQL) |
| Slow queries across the board | `innodb_buffer_pool_size` too small | Set to ~70% of available RAM and restart |
| Replication stopped (`SQL_Running: No`) | Duplicate key or schema mismatch on replica | Check `SHOW REPLICA STATUS\G` error; skip or fix the row |
| `Table is full` | Disk space exhausted or table limit hit | Free disk space; check `innodb_data_file_path` autoextend |
| `Lock wait timeout exceeded` | Long-running transaction holding row locks | Identify with `SHOW ENGINE INNODB STATUS`; kill the blocking query |
| High IOPS / disk usage | Redo log too small causing frequent flushes | Increase `innodb_log_file_size` (requires restart) |
## Related Skills
- [postgresql](../postgresql/) - Alternative relational database
- [database-backups](../database-backups/) - Automated backup strategies
- [redis](../redis/) - Caching layer to reduce database load
- [planetscale](../planetscale/) - Managed MySQL-compatible with branching
infrastructure/databases/planetscale/SKILL.md
+260 -10
View File
@@ -9,23 +9,273 @@ metadata:
# PlanetScale
Use PlanetScale for serverless MySQL with non-blocking schema change workflows.
Use PlanetScale for serverless MySQL-compatible databases with non-blocking schema change workflows built on Vitess.
## When to Use
- You need a managed MySQL-compatible database with zero-downtime migrations.
- Your team wants Git-like branching for schema development.
- You are building a serverless or edge application that benefits from connection pooling.
- You need horizontal sharding without managing Vitess directly.
## Prerequisites
- A PlanetScale account (free tier available).
- The `pscale` CLI installed locally.
- Node.js 18+ if using Prisma or other ORM integrations.
## Install the pscale CLI
```bash
# macOS
brew install planetscale/tap/pscale
# Linux (deb)
curl -fsSL https://github.com/planetscale/cli/releases/latest/download/pscale_linux_amd64.deb -o pscale.deb
sudo dpkg -i pscale.deb
# Verify installation
pscale version
# Authenticate
pscale auth login
```
## Create and Manage Databases
```bash
# Create a new database
pscale database create my-app --region us-east
# List databases
pscale database list
# Show database info
pscale database show my-app
# Delete a database (destructive)
pscale database delete my-app
```
## Branching Workflow
1. Create a database branch for schema work.
2. Apply migrations to the branch.
3. Open a deploy request and run checks.
4. Merge to production during low-risk windows.
PlanetScale branches work like Git branches for your database schema. The `main` branch is the production branch by default.
## Operational Best Practices
```bash
# Create a development branch from main
pscale branch create my-app add-users-table
- Keep schema changes backward compatible first.
- Use connection pooling for serverless apps.
- Monitor query insights for slow statements.
- Define rollback strategy for every deploy request.
# List all branches
pscale branch list my-app
# Open a shell on the branch to apply schema changes
pscale shell my-app add-users-table
```
### Apply Schema Changes on a Branch
```sql
-- Inside the pscale shell on the development branch
CREATE TABLE users (
id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
email VARCHAR(255) NOT NULL,
name VARCHAR(255) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
UNIQUE KEY idx_users_email (email)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE orders (
id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
user_id BIGINT NOT NULL,
total DECIMAL(10,2) NOT NULL DEFAULT 0.00,
status ENUM('pending','paid','shipped','cancelled') DEFAULT 'pending',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
KEY idx_orders_user_id (user_id),
KEY idx_orders_status (status)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```
> PlanetScale does not enforce foreign keys at the database level. Use application-level constraints or Vitess-level routing rules instead.
## Deploy Requests
Deploy requests are the pull-request equivalent for database schemas. They show a diff, run linting, and merge non-blocking into production.
```bash
# Create a deploy request from branch to main
pscale deploy-request create my-app add-users-table
# List open deploy requests
pscale deploy-request list my-app
# Show diff for a deploy request
pscale deploy-request diff my-app 1
# Deploy (merge) the request
pscale deploy-request deploy my-app 1
# Close without deploying
pscale deploy-request close my-app 1
# Delete the branch after successful deploy
pscale branch delete my-app add-users-table
```
## Connection Strings and Proxying
```bash
# Create a password (connection credential) for a branch
pscale password create my-app main production-creds
# Output includes host, username, and password for the connection string:
# mysql://USERNAME:PASSWORD@HOST/my-app?sslmode=verify_identity
# Proxy a branch to localhost for local development (no password needed)
pscale connect my-app add-users-table --port 3306
```
### Environment Variable Pattern
```bash
# .env (local development using pscale connect)
DATABASE_URL="mysql://root@127.0.0.1:3306/my-app"
# .env.production (using PlanetScale connection string)
DATABASE_URL="mysql://USERNAME:PASSWORD@us-east.connect.psdb.cloud/my-app?sslaccept=strict"
```
## Prisma Integration
```prisma
// prisma/schema.prisma
datasource db {
provider = "mysql"
url = env("DATABASE_URL")
relationMode = "prisma" // required — PlanetScale does not support foreign keys
}
model User {
id Int @id @default(autoincrement())
email String @unique
name String
orders Order[]
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}
model Order {
id Int @id @default(autoincrement())
userId Int
total Decimal @db.Decimal(10, 2)
status String @default("pending")
user User @relation(fields: [userId], references: [id])
createdAt DateTime @default(now())
@@index([userId])
@@index([status])
}
```
```bash
# Push schema changes to the PlanetScale branch
npx prisma db push
# Generate the Prisma client
npx prisma generate
```
## Vitess Features and Query Insights
```bash
# Open the query insights dashboard
pscale shell my-app main
# Inside the shell, check running queries
SHOW PROCESSLIST;
# Examine query statistics (PlanetScale Insights tab in the web UI)
# Or use the API:
pscale api organizations/my-org/databases/my-app/branches/main/query-statistics
```
### Useful Vitess-Aware Queries
```sql
-- Check table sizes
SELECT table_name,
ROUND(data_length / 1024 / 1024, 2) AS data_mb,
ROUND(index_length / 1024 / 1024, 2) AS index_mb,
table_rows
FROM information_schema.tables
WHERE table_schema = 'my-app'
ORDER BY data_length DESC;
-- Show index usage
SHOW INDEX FROM users;
-- Explain a query plan
EXPLAIN SELECT * FROM orders WHERE user_id = 42 AND status = 'paid';
```
## Docker Setup for Local Development
Use a plain MySQL 8 container to mirror PlanetScale locally when you are offline or want fast iteration without the CLI proxy.
```yaml
# docker-compose.yml
version: "3.9"
services:
mysql:
image: mysql:8.0
restart: unless-stopped
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: rootpass
MYSQL_DATABASE: my-app
MYSQL_USER: myapp
MYSQL_PASSWORD: secret
volumes:
- mysql_data:/var/lib/mysql
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
command: >
--default-authentication-plugin=mysql_native_password
--character-set-server=utf8mb4
--collation-server=utf8mb4_unicode_ci
volumes:
mysql_data:
```
```bash
docker compose up -d
mysql -h 127.0.0.1 -u myapp -psecret my-app
```
## Production Best Practices
- Keep every schema change backward compatible; deploy the schema first, then the application code.
- Use deploy request reviews as a gate; require at least one approval before merging.
- Enable connection pooling (`@planetscale/database` driver or Prisma Data Proxy) for serverless workloads.
- Monitor query insights weekly and add indexes for queries exceeding 100 ms.
- Set branch promotion rules so only specific team members can deploy to `main`.
- Use read-only regions to reduce latency for geographically distributed reads.
## Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| `Access denied` on `pscale connect` | CLI not authenticated | Run `pscale auth login` |
| Deploy request shows "schema conflict" | Concurrent branch changes to the same table | Rebase: delete branch, recreate from current `main`, reapply changes |
| `foreign key constraint` error | PlanetScale does not support foreign keys | Use `relationMode = "prisma"` or remove FK definitions |
| High latency on reads | No index on queried column | Add index via a new branch and deploy request |
| `max connections` exceeded | Connection pooling not enabled | Use `@planetscale/database` serverless driver or PgBouncer-style proxy |
| `pscale connect` hangs | Firewall blocking outbound TLS | Allow outbound 443 to `*.psdb.cloud` |
## Related Skills
- [mysql](../mysql/) - MySQL tuning fundamentals
- [database-backups](../database-backups/) - Recovery planning
- [postgresql](../postgresql/) - Alternative relational database
infrastructure/databases/postgresql/SKILL.md
+327 -34
View File
@@ -9,60 +9,353 @@ metadata:
# PostgreSQL
Administer and optimize PostgreSQL databases.
Administer, optimize, and secure PostgreSQL databases in development and production environments.
## Installation & Setup
## When to Use
- You need a reliable, ACID-compliant relational database.
- Your application requires advanced features such as JSONB, full-text search, or CTEs.
- You are setting up streaming replication or point-in-time recovery.
- You need to tune an existing PostgreSQL deployment for better throughput.
## Prerequisites
- Linux server (Debian/Ubuntu or RHEL-based) or Docker.
- Root or sudo access for package installation.
- Familiarity with SQL fundamentals.
## Installation and Setup
```bash
# Install
apt install postgresql postgresql-contrib
# Debian / Ubuntu
sudo apt update
sudo apt install -y postgresql postgresql-contrib
# Access
# RHEL / Amazon Linux
sudo dnf install -y postgresql15-server postgresql15-contrib
sudo postgresql-setup --initdb
sudo systemctl enable --now postgresql
# Verify
psql --version
sudo systemctl status postgresql
```
## Initial User and Database Setup
```bash
# Switch to the postgres system user
sudo -u postgres psql
```
# Create database and user
CREATE USER myapp WITH PASSWORD 'secret';
```sql
-- Create an application user
CREATE USER myapp WITH PASSWORD 'strong_password_here';
-- Create the database owned by that user
CREATE DATABASE mydb OWNER myapp;
-- Grant connection privileges
GRANT ALL PRIVILEGES ON DATABASE mydb TO myapp;
-- Connect to the database and set default privileges
\c mydb
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO myapp;
```
## Configuration
## psql Commands Reference
```bash
# /etc/postgresql/15/main/postgresql.conf
```
\l -- list databases
\dt -- list tables in current database
\d+ tablename -- describe table with storage info
\du -- list roles
\x -- toggle expanded output
\timing on -- show query execution time
\i file.sql -- execute SQL from file
\copy -- fast client-side COPY
```
## Configuration Tuning
Edit `/etc/postgresql/15/main/postgresql.conf` (path varies by OS and version).
```ini
# Connection settings
listen_addresses = '*'
max_connections = 200
shared_buffers = 256MB
effective_cache_size = 768MB
work_mem = 4MB
maintenance_work_mem = 64MB
```
## Backup & Restore
# Memory — adjust to ~25% of total RAM for shared_buffers
shared_buffers = 4GB
effective_cache_size = 12GB
work_mem = 16MB
maintenance_work_mem = 512MB
# WAL / write performance
wal_buffers = 64MB
checkpoint_completion_target = 0.9
min_wal_size = 1GB
max_wal_size = 4GB
# Planner
random_page_cost = 1.1 # lower for SSD
effective_io_concurrency = 200 # for SSD
# Logging
log_min_duration_statement = 250 # log queries slower than 250 ms
log_checkpoints = on
log_connections = on
log_disconnections = on
log_lock_waits = on
```
```bash
# Backup
pg_dump mydb > backup.sql
pg_dump -Fc mydb > backup.dump # Custom format
# Reload configuration without restart
sudo -u postgres psql -c "SELECT pg_reload_conf();"
# Restore
psql mydb < backup.sql
pg_restore -d mydb backup.dump
# Some settings (shared_buffers, max_connections) require a full restart
sudo systemctl restart postgresql
```
## Replication
## pg_hba.conf — Client Authentication
```
# /etc/postgresql/15/main/pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
local all postgres peer
host mydb myapp 10.0.0.0/8 scram-sha-256
host all all 0.0.0.0/0 reject
```
```bash
# Primary
ALTER SYSTEM SET wal_level = replica;
CREATE USER replicator REPLICATION LOGIN PASSWORD 'secret';
# Replica
pg_basebackup -h primary -U replicator -D /var/lib/postgresql/15/main -P
sudo systemctl reload postgresql
```
## Best Practices
## Backup and Restore
- Regular VACUUM and ANALYZE
- Monitor slow queries
- Implement connection pooling (PgBouncer)
- Regular backups with pg_dump or pg_basebackup
### Logical Backups with pg_dump
```bash
# Plain SQL backup
pg_dump -U myapp -h localhost mydb > /backups/mydb_$(date +%F).sql
# Custom compressed format (recommended)
pg_dump -U myapp -h localhost -Fc mydb > /backups/mydb_$(date +%F).dump
# Backup a single table
pg_dump -U myapp -h localhost -t orders -Fc mydb > /backups/orders.dump
# Restore from custom format
pg_restore -U myapp -h localhost -d mydb --clean --if-exists /backups/mydb_2025-01-15.dump
# Restore plain SQL
psql -U myapp -h localhost -d mydb < /backups/mydb_2025-01-15.sql
```
### Physical Backups with pg_basebackup
```bash
# Full base backup (used for PITR and replica seeding)
pg_basebackup -h localhost -U replicator -D /backups/base_$(date +%F) \
--wal-method=stream --checkpoint=fast --progress --verbose
# Verify the backup
pg_verifybackup /backups/base_2025-01-15
```
## Streaming Replication
### Primary Server
```sql
-- Create replication user
CREATE USER replicator WITH REPLICATION LOGIN PASSWORD 'repl_secret';
```
```ini
# postgresql.conf on primary
wal_level = replica
max_wal_senders = 5
wal_keep_size = 1GB
```
```
# pg_hba.conf on primary
host replication replicator 10.0.0.0/8 scram-sha-256
```
### Replica Server
```bash
# Stop PostgreSQL on the replica
sudo systemctl stop postgresql
# Remove existing data directory
sudo rm -rf /var/lib/postgresql/15/main/*
# Base backup from primary
sudo -u postgres pg_basebackup \
-h 10.0.0.1 -U replicator \
-D /var/lib/postgresql/15/main \
--wal-method=stream --checkpoint=fast --progress
# Create standby signal file
sudo -u postgres touch /var/lib/postgresql/15/main/standby.signal
```
```ini
# postgresql.conf on replica
primary_conninfo = 'host=10.0.0.1 port=5432 user=replicator password=repl_secret'
hot_standby = on
```
```bash
sudo systemctl start postgresql
```
### Verify Replication
```sql
-- On primary
SELECT client_addr, state, sent_lsn, replay_lsn
FROM pg_stat_replication;
-- On replica
SELECT pg_is_in_recovery(); -- should return true
SELECT pg_last_wal_receive_lsn();
SELECT pg_last_wal_replay_lsn();
```
## Monitoring Queries
```sql
-- Active connections by state
SELECT state, COUNT(*)
FROM pg_stat_activity
GROUP BY state;
-- Long-running queries (> 30 seconds)
SELECT pid, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active'
AND now() - query_start > interval '30 seconds'
ORDER BY duration DESC;
-- Table bloat and dead tuples
SELECT relname,
n_live_tup,
n_dead_tup,
ROUND(n_dead_tup::numeric / GREATEST(n_live_tup, 1) * 100, 2) AS dead_pct
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC
LIMIT 10;
-- Index usage statistics
SELECT relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan ASC
LIMIT 10;
-- Cache hit ratio (should be > 99%)
SELECT ROUND(
100.0 * sum(blks_hit) / NULLIF(sum(blks_hit) + sum(blks_read), 0), 2
) AS cache_hit_pct
FROM pg_stat_database;
-- Database size
SELECT pg_database.datname,
pg_size_pretty(pg_database_size(pg_database.datname)) AS size
FROM pg_database
ORDER BY pg_database_size(pg_database.datname) DESC;
```
## Docker Compose Setup
```yaml
# docker-compose.yml
version: "3.9"
services:
postgres:
image: postgres:16-alpine
restart: unless-stopped
ports:
- "5432:5432"
environment:
POSTGRES_USER: myapp
POSTGRES_PASSWORD: secret
POSTGRES_DB: mydb
volumes:
- pg_data:/var/lib/postgresql/data
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
command: >
postgres
-c shared_buffers=256MB
-c work_mem=8MB
-c maintenance_work_mem=128MB
-c effective_cache_size=768MB
-c log_min_duration_statement=250
healthcheck:
test: ["CMD-SHELL", "pg_isready -U myapp -d mydb"]
interval: 10s
timeout: 5s
retries: 5
pgbouncer:
image: edoburu/pgbouncer:latest
restart: unless-stopped
ports:
- "6432:6432"
environment:
DATABASE_URL: postgres://myapp:secret@postgres:5432/mydb
POOL_MODE: transaction
MAX_CLIENT_CONN: 500
DEFAULT_POOL_SIZE: 40
depends_on:
postgres:
condition: service_healthy
volumes:
pg_data:
```
```bash
docker compose up -d
psql -h 127.0.0.1 -p 6432 -U myapp mydb
```
## Maintenance Tasks
```bash
# Manual VACUUM and ANALYZE
sudo -u postgres psql -d mydb -c "VACUUM ANALYZE;"
# Reindex a bloated index
sudo -u postgres psql -d mydb -c "REINDEX INDEX CONCURRENTLY idx_orders_user_id;"
# Check for unused indexes
sudo -u postgres psql -d mydb -c "
SELECT indexrelname, idx_scan
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;"
```
## Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| `FATAL: too many connections` | Connection limit reached | Increase `max_connections` or add PgBouncer |
| Slow SELECT on large table | Missing index or stale statistics | Run `EXPLAIN ANALYZE`; add index; run `ANALYZE` |
| High CPU from autovacuum | Large number of dead tuples | Tune `autovacuum_vacuum_cost_delay`; run manual `VACUUM` |
| Replication lag increasing | Replica under-provisioned or network bottleneck | Check `pg_stat_replication`; increase `wal_keep_size` |
| `could not access file "base/..."` | Disk full or corrupt data directory | Free disk space; restore from `pg_basebackup` |
| `FATAL: password authentication failed` | Wrong credentials or pg_hba.conf mismatch | Verify pg_hba.conf entries and reload |
## Related Skills
- [mysql](../mysql/) - Alternative relational database
- [database-backups](../database-backups/) - Automated backup strategies
- [redis](../redis/) - Caching layer to reduce database load
- [planetscale](../planetscale/) - Managed MySQL-compatible alternative
infrastructure/databases/redis/SKILL.md
+367 -36
View File
@@ -9,66 +9,397 @@ metadata:
# Redis
Configure Redis for caching and data storage.
Configure, operate, and optimize Redis for caching, queues, rate limiting, and real-time data storage.
## Installation & Setup
## When to Use
- You need a low-latency in-memory cache to reduce database load.
- Your application requires rate limiting, session storage, or leaderboards.
- You need pub/sub messaging between services.
- You want a distributed lock or job queue backed by an in-memory store.
## Prerequisites
- Linux server or Docker.
- Root or sudo access for package installation.
- Redis 7.x recommended for production.
## Installation and Setup
```bash
# Install
apt install redis-server
# Debian / Ubuntu
sudo apt update
sudo apt install -y redis-server
# Configuration
# /etc/redis/redis.conf
bind 0.0.0.0
protected-mode yes
requirepass yourpassword
maxmemory 256mb
maxmemory-policy allkeys-lru
# RHEL / Amazon Linux
sudo dnf install -y redis
# Start and enable
sudo systemctl enable --now redis-server
# Verify
redis-cli ping
# Expected output: PONG
```
## Basic Operations
## Core Configuration
Edit `/etc/redis/redis.conf`:
```ini
# Network
bind 0.0.0.0
port 6379
protected-mode yes
requirepass strong_redis_password
# Memory
maxmemory 2gb
maxmemory-policy allkeys-lru
# Connections
maxclients 10000
timeout 300
tcp-keepalive 60
# Logging
loglevel notice
logfile /var/log/redis/redis-server.log
# Security — disable dangerous commands in production
rename-command FLUSHALL ""
rename-command FLUSHDB ""
rename-command CONFIG ""
rename-command DEBUG ""
```
```bash
redis-cli -a yourpassword
sudo systemctl restart redis-server
```
# String operations
SET key "value"
GET key
SETEX key 3600 "value" # With TTL
## redis-cli Commands Reference
# Hash
HSET user:1 name "John" email "john@example.com"
```bash
# Connect with authentication
redis-cli -a strong_redis_password
# Connect to a remote host
redis-cli -h 10.0.0.5 -p 6379 -a strong_redis_password
```
### String Operations
```
SET user:1:name "Alice"
GET user:1:name
# Set with TTL (seconds)
SETEX session:abc123 3600 '{"userId":1}'
# Set only if key does not exist (distributed lock pattern)
SET lock:order:42 "worker-1" NX EX 30
# Increment counters
INCR page:views:/home
INCRBY api:quota:user:1 -1
```
### Hash Operations
```
HSET user:1 name "Alice" email "alice@example.com" plan "pro"
HGET user:1 email
HGETALL user:1
HINCRBY user:1 login_count 1
```
# List
LPUSH queue "task1"
RPOP queue
### List Operations (Queues)
```
LPUSH queue:emails '{"to":"alice@example.com","subject":"Welcome"}'
RPOP queue:emails
LLEN queue:emails
# Blocking pop (worker pattern)
BRPOP queue:emails 30
```
### Set and Sorted Set Operations
```
# Sets — unique tags
SADD article:1:tags "redis" "database" "caching"
SMEMBERS article:1:tags
SISMEMBER article:1:tags "redis"
# Sorted sets — leaderboards
ZADD leaderboard 1500 "player:1" 2300 "player:2" 1800 "player:3"
ZREVRANGE leaderboard 0 9 WITHSCORES
ZINCRBY leaderboard 100 "player:1"
ZRANK leaderboard "player:2"
```
### Key Management
```
KEYS user:* # avoid in production — use SCAN instead
SCAN 0 MATCH user:* COUNT 100
TTL session:abc123
PERSIST session:abc123
DEL user:old
EXPIRE user:1 86400
TYPE user:1
```
## Persistence
```bash
# RDB (snapshot)
### RDB Snapshots
```ini
# redis.conf — save snapshots at intervals
save 900 1 # snapshot if >= 1 key changed in 900 seconds
save 300 10 # snapshot if >= 10 keys changed in 300 seconds
save 60 10000 # snapshot if >= 10000 keys changed in 60 seconds
dbfilename dump.rdb
dir /var/lib/redis
rdbcompression yes
```
### AOF (Append-Only File)
```ini
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec # good balance of safety and performance
# Options: always (safest, slowest), everysec (recommended), no (OS decides)
# AOF rewrite thresholds
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
```
### Recommended Production Strategy
Use both RDB and AOF together. RDB provides fast restarts and compact backups. AOF provides durability down to 1-second granularity.
```ini
save 900 1
save 300 10
# AOF (append-only file)
appendonly yes
appendfsync everysec
```
## Sentinel (HA)
## Redis Sentinel (High Availability)
```bash
# sentinel.conf
Sentinel monitors Redis instances and performs automatic failover.
### Sentinel Configuration
```ini
# /etc/redis/sentinel.conf
port 26379
sentinel monitor mymaster 10.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel failover-timeout mymaster 180000
sentinel auth-pass mymaster strong_redis_password
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
```
## Best Practices
Run at least three Sentinel instances for quorum.
- Set maxmemory and eviction policy
- Use persistence for critical data
- Implement Sentinel for HA
- Monitor memory usage
```bash
# Start Sentinel
redis-sentinel /etc/redis/sentinel.conf
# Query Sentinel
redis-cli -p 26379 SENTINEL masters
redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster
redis-cli -p 26379 SENTINEL replicas mymaster
```
## Redis Cluster Mode
Cluster mode distributes data across multiple shards automatically.
```bash
# Create a 6-node cluster (3 masters + 3 replicas)
redis-cli --cluster create \
10.0.0.1:6379 10.0.0.2:6379 10.0.0.3:6379 \
10.0.0.4:6379 10.0.0.5:6379 10.0.0.6:6379 \
--cluster-replicas 1 -a strong_redis_password
# Check cluster status
redis-cli -c -a strong_redis_password CLUSTER INFO
redis-cli -c -a strong_redis_password CLUSTER NODES
# Add a new node
redis-cli --cluster add-node 10.0.0.7:6379 10.0.0.1:6379
# Rebalance slots
redis-cli --cluster rebalance 10.0.0.1:6379
```
```ini
# redis.conf for cluster nodes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
```
## Common Patterns
### Caching with TTL
```bash
# Cache a database query result for 5 minutes
SET cache:user:42:profile '{"name":"Alice","plan":"pro"}' EX 300
# Cache-aside pattern (pseudocode):
# 1. GET cache:key -> if hit, return
# 2. Query database
# 3. SET cache:key result EX 300
# 4. Return result
```
### Rate Limiting (Sliding Window)
```bash
# Allow 100 requests per minute per user
# Using a sorted set with timestamps as scores
ZADD ratelimit:user:42 1700000000.123 "req-uuid-1"
ZREMRANGEBYSCORE ratelimit:user:42 0 1699999940.000
ZCARD ratelimit:user:42
EXPIRE ratelimit:user:42 60
# If ZCARD >= 100, reject the request
```
### Pub/Sub Messaging
```bash
# Terminal 1 — subscriber
redis-cli -a strong_redis_password
SUBSCRIBE notifications:order-updates
# Terminal 2 — publisher
redis-cli -a strong_redis_password
PUBLISH notifications:order-updates '{"orderId":42,"status":"shipped"}'
# Pattern subscription
PSUBSCRIBE notifications:*
```
### Distributed Locking (Redlock Pattern)
```bash
# Acquire lock
SET lock:resource:42 "worker-abc" NX EX 30
# Returns OK if acquired, nil if already held
# Release lock (use Lua script to ensure atomicity)
redis-cli -a strong_redis_password EVAL "
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('del', KEYS[1])
else
return 0
end
" 1 lock:resource:42 "worker-abc"
```
## Docker Compose Setup
```yaml
# docker-compose.yml
version: "3.9"
services:
redis:
image: redis:7-alpine
restart: unless-stopped
ports:
- "6379:6379"
volumes:
- redis_data:/data
- ./redis.conf:/usr/local/etc/redis/redis.conf:ro
command: redis-server /usr/local/etc/redis/redis.conf
healthcheck:
test: ["CMD", "redis-cli", "-a", "strong_redis_password", "ping"]
interval: 10s
timeout: 5s
retries: 5
redis-sentinel:
image: redis:7-alpine
restart: unless-stopped
ports:
- "26379:26379"
volumes:
- ./sentinel.conf:/usr/local/etc/redis/sentinel.conf
command: redis-sentinel /usr/local/etc/redis/sentinel.conf
depends_on:
redis:
condition: service_healthy
redis-commander:
image: rediscommander/redis-commander:latest
restart: unless-stopped
ports:
- "8081:8081"
environment:
REDIS_HOSTS: "local:redis:6379:0:strong_redis_password"
depends_on:
redis:
condition: service_healthy
volumes:
redis_data:
```
```bash
docker compose up -d
redis-cli -h 127.0.0.1 -a strong_redis_password ping
```
## Monitoring
```bash
# Real-time stats
redis-cli -a strong_redis_password INFO stats
redis-cli -a strong_redis_password INFO memory
redis-cli -a strong_redis_password INFO replication
# Key metrics to watch
redis-cli -a strong_redis_password INFO stats | grep -E "keyspace_hits|keyspace_misses"
# Hit ratio = hits / (hits + misses) — aim for > 95%
# Memory usage breakdown
redis-cli -a strong_redis_password MEMORY STATS
# Slow log (queries > 10ms by default)
redis-cli -a strong_redis_password SLOWLOG GET 10
redis-cli -a strong_redis_password SLOWLOG LEN
# Monitor all commands in real time (debugging only — impacts performance)
redis-cli -a strong_redis_password MONITOR
# Connected clients
redis-cli -a strong_redis_password CLIENT LIST
```
## Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| `OOM command not allowed` | `maxmemory` limit reached | Increase `maxmemory` or set a stricter eviction policy |
| High latency spikes | RDB save or AOF rewrite forking | Use `save ""` to disable RDB if AOF is enabled; tune `auto-aof-rewrite-min-size` |
| `LOADING Redis is loading the dataset in memory` | Large dataset being restored on startup | Wait for load to complete; consider smaller dataset or faster disk |
| Cache hit ratio < 90% | TTLs too short or working set exceeds memory | Increase `maxmemory`; review TTL strategy |
| Sentinel not failing over | Fewer than quorum Sentinels reachable | Ensure >= 3 Sentinels are running and network-connected |
| `CROSSSLOT` error in cluster | Multi-key command spans slots | Use hash tags `{user:42}:profile` to colocate related keys |
## Related Skills
- [postgresql](../postgresql/) - Primary database that Redis caches
- [mysql](../mysql/) - Primary database that Redis caches
- [mongodb](../mongodb/) - Document database that Redis can front
- [database-backups](../database-backups/) - Include RDB files in backup strategy