mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
V2
This commit is contained in:
Toby
@@ -9,55 +9,364 @@ metadata:
|
||||
|
||||
# Backup and Recovery
|
||||
|
||||
Implement comprehensive backup strategies.
|
||||
Implement comprehensive backup and recovery strategies using rsync, Restic, and cloud storage backends. Covers the 3-2-1 rule, automated scheduling, S3/B2 backends, encryption, restore procedures, and verification testing.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Designing a backup strategy for servers, databases, or application data
|
||||
- Setting up Restic for encrypted, deduplicated backups to local or cloud storage
|
||||
- Automating backups with systemd timers or cron
|
||||
- Restoring data after accidental deletion, corruption, or disaster
|
||||
- Migrating data between environments using backup/restore workflows
|
||||
- Verifying backup integrity and testing recovery procedures
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `rsync` installed (included in most Linux distributions)
|
||||
- `restic` installed (v0.16+ recommended)
|
||||
- Cloud CLI configured for the backend: AWS CLI for S3, `b2` CLI for Backblaze B2
|
||||
- Sufficient storage at the backup destination (2-3x source size for retention)
|
||||
- SSH access for remote rsync targets
|
||||
- `systemd` or `cron` for scheduling
|
||||
|
||||
## The 3-2-1 Backup Rule
|
||||
|
||||
- **3** copies of your data (1 primary + 2 backups)
|
||||
- **2** different storage media or types (e.g., local disk + cloud)
|
||||
- **1** copy offsite (cloud storage, remote datacenter)
|
||||
|
||||
## rsync Backups
|
||||
|
||||
### Basic Operations
|
||||
|
||||
```bash
|
||||
# Basic sync
|
||||
rsync -avz --delete /source/ /backup/
|
||||
# Sync a local directory to a backup location
|
||||
rsync -avz --delete /data/ /backup/data/
|
||||
|
||||
# Remote backup
|
||||
rsync -avz -e ssh /data/ user@backup:/backups/
|
||||
# Flags explained:
|
||||
# -a archive mode (preserves permissions, ownership, timestamps, symlinks)
|
||||
# -v verbose output
|
||||
# -z compress data during transfer
|
||||
# --delete remove files at destination that no longer exist at source
|
||||
|
||||
# Incremental with hard links
|
||||
rsync -avz --delete --link-dest=/backup/latest /source/ /backup/$(date +%Y%m%d)/
|
||||
# Sync to a remote server over SSH
|
||||
rsync -avz -e "ssh -i ~/.ssh/backup_key" /data/ backup@remote:/backups/server01/
|
||||
|
||||
# Exclude patterns
|
||||
rsync -avz --delete \
|
||||
--exclude='*.tmp' \
|
||||
--exclude='*.log' \
|
||||
--exclude='.cache/' \
|
||||
--exclude='node_modules/' \
|
||||
/data/ /backup/data/
|
||||
|
||||
# Use an exclude file for complex patterns
|
||||
rsync -avz --delete --exclude-from=/etc/backup-excludes.txt /data/ /backup/data/
|
||||
|
||||
# Dry run (preview what would change)
|
||||
rsync -avzn --delete /data/ /backup/data/
|
||||
|
||||
# Limit bandwidth to 50 MB/s and show progress
|
||||
rsync -avz --bwlimit=50000 --progress /data/ backup@remote:/backups/
|
||||
```
|
||||
|
||||
### Incremental Backups with Hard Links
|
||||
|
||||
```bash
|
||||
# Incremental: unchanged files hard-linked to previous backup (saves space)
|
||||
rsync -avz --delete \
|
||||
--link-dest=/backup/daily/latest \
|
||||
/data/ /backup/daily/$(date +%Y-%m-%d)/
|
||||
|
||||
# Update the 'latest' symlink
|
||||
ln -snf /backup/daily/$(date +%Y-%m-%d) /backup/daily/latest
|
||||
|
||||
# Remove backups older than 30 days
|
||||
find /backup/daily -maxdepth 1 -type d -name "20*" -mtime +30 -exec rm -rf {} \;
|
||||
```
|
||||
|
||||
## Restic Backup
|
||||
|
||||
```bash
|
||||
# Initialize repository
|
||||
restic init --repo /backups
|
||||
|
||||
# Backup
|
||||
restic backup /data --repo /backups
|
||||
|
||||
# List snapshots
|
||||
restic snapshots --repo /backups
|
||||
|
||||
# Restore
|
||||
restic restore latest --target /restore --repo /backups
|
||||
|
||||
# Prune old backups
|
||||
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune
|
||||
```
|
||||
|
||||
## Cloud Backup
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# AWS S3 with restic
|
||||
restic init --repo s3:s3.amazonaws.com/bucket-name
|
||||
restic backup /data --repo s3:s3.amazonaws.com/bucket-name
|
||||
# Debian / Ubuntu
|
||||
apt install -y restic
|
||||
|
||||
# GCS
|
||||
restic init --repo gs:bucket-name:/
|
||||
# RHEL / CentOS
|
||||
dnf install -y restic
|
||||
|
||||
# Or download the latest binary
|
||||
curl -L https://github.com/restic/restic/releases/latest/download/restic_0.17.3_linux_amd64.bz2 \
|
||||
| bunzip2 > /usr/local/bin/restic
|
||||
chmod +x /usr/local/bin/restic
|
||||
|
||||
# Verify installation
|
||||
restic version
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
### Initialize a Repository
|
||||
|
||||
- Follow 3-2-1 rule
|
||||
- Test recovery regularly
|
||||
- Encrypt backups
|
||||
- Document procedures
|
||||
- Monitor backup success
|
||||
```bash
|
||||
# Local repository
|
||||
restic init --repo /backup/restic-repo
|
||||
|
||||
# AWS S3 backend
|
||||
export AWS_ACCESS_KEY_ID="AKIAEXAMPLE"
|
||||
export AWS_SECRET_ACCESS_KEY="secretkey"
|
||||
restic init --repo s3:s3.amazonaws.com/my-backup-bucket
|
||||
|
||||
# S3-compatible (MinIO)
|
||||
export AWS_ACCESS_KEY_ID="minioadmin"
|
||||
export AWS_SECRET_ACCESS_KEY="miniosecret"
|
||||
restic init --repo s3:http://minio.example.com:9000/backup-bucket
|
||||
|
||||
# Backblaze B2 backend
|
||||
export B2_ACCOUNT_ID="accountid"
|
||||
export B2_ACCOUNT_KEY="accountkey"
|
||||
restic init --repo b2:my-backup-bucket:server01
|
||||
|
||||
# SFTP backend
|
||||
restic init --repo sftp:backup@remote:/backups/server01
|
||||
|
||||
# Restic will prompt for a repository password -- store it securely
|
||||
# Use a password file for automation
|
||||
echo "my-secure-repo-password" > /etc/restic/password.txt
|
||||
chmod 600 /etc/restic/password.txt
|
||||
```
|
||||
|
||||
### Backup Operations
|
||||
|
||||
```bash
|
||||
# Basic backup
|
||||
restic backup /data --repo /backup/restic-repo --password-file /etc/restic/password.txt
|
||||
|
||||
# Backup multiple directories
|
||||
restic backup /data /etc /var/lib/postgresql \
|
||||
--repo s3:s3.amazonaws.com/my-backup-bucket \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# Backup with exclusions
|
||||
restic backup /data \
|
||||
--exclude='*.tmp' \
|
||||
--exclude='*.log' \
|
||||
--exclude-file=/etc/restic/excludes.txt \
|
||||
--repo /backup/restic-repo \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# Backup with tags (useful for filtering snapshots later)
|
||||
restic backup /data \
|
||||
--tag server01 --tag production --tag daily \
|
||||
--repo /backup/restic-repo \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# Backup stdin (e.g., database dump)
|
||||
pg_dump -U postgres mydb | restic backup --stdin --stdin-filename mydb.sql \
|
||||
--repo s3:s3.amazonaws.com/my-backup-bucket \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# Verbose output showing files processed
|
||||
restic backup /data -v \
|
||||
--repo /backup/restic-repo \
|
||||
--password-file /etc/restic/password.txt
|
||||
```
|
||||
|
||||
### Snapshot Management
|
||||
|
||||
```bash
|
||||
# List all snapshots (add --tag <tag> to filter)
|
||||
restic snapshots --repo /backup/restic-repo --password-file /etc/restic/password.txt
|
||||
|
||||
# Browse files in the latest snapshot
|
||||
restic ls latest --repo /backup/restic-repo --password-file /etc/restic/password.txt
|
||||
|
||||
# Compare two snapshots
|
||||
restic diff abc123 def456 --repo /backup/restic-repo --password-file /etc/restic/password.txt
|
||||
```
|
||||
|
||||
### Retention Policy (forget + prune)
|
||||
|
||||
```bash
|
||||
# Apply retention policy and reclaim space
|
||||
restic forget \
|
||||
--keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 3 \
|
||||
--prune \
|
||||
--repo /backup/restic-repo \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# Dry run to preview what would be removed
|
||||
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 \
|
||||
--dry-run --repo /backup/restic-repo --password-file /etc/restic/password.txt
|
||||
```
|
||||
|
||||
### Restore Procedures
|
||||
|
||||
```bash
|
||||
# Restore the latest snapshot to a target directory
|
||||
restic restore latest --target /restore \
|
||||
--repo /backup/restic-repo \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# Restore a specific snapshot by ID
|
||||
restic restore abc123 --target /restore \
|
||||
--repo /backup/restic-repo \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# Restore only specific files or directories
|
||||
restic restore latest --target /restore --include "/data/config" \
|
||||
--repo /backup/restic-repo \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# Mount a snapshot as a FUSE filesystem (browse and copy individual files)
|
||||
mkdir -p /mnt/restic
|
||||
restic mount /mnt/restic --repo /backup/restic-repo --password-file /etc/restic/password.txt &
|
||||
# Browse: ls /mnt/restic/snapshots/latest/
|
||||
# Unmount when done: umount /mnt/restic
|
||||
```
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
# Verify repository integrity (checks all data and metadata)
|
||||
restic check --repo /backup/restic-repo --password-file /etc/restic/password.txt
|
||||
|
||||
# Full data verification (reads all pack files -- slow but thorough)
|
||||
restic check --read-data --repo /backup/restic-repo --password-file /etc/restic/password.txt
|
||||
|
||||
# Verify a random subset of data (faster than full read)
|
||||
restic check --read-data-subset=5% --repo /backup/restic-repo --password-file /etc/restic/password.txt
|
||||
```
|
||||
|
||||
## Automated Backup with Environment File
|
||||
|
||||
### /etc/restic/env
|
||||
|
||||
```bash
|
||||
# Repository configuration
|
||||
export RESTIC_REPOSITORY="s3:s3.amazonaws.com/my-backup-bucket"
|
||||
export RESTIC_PASSWORD_FILE="/etc/restic/password.txt"
|
||||
export AWS_ACCESS_KEY_ID="AKIAEXAMPLE"
|
||||
export AWS_SECRET_ACCESS_KEY="secretkey"
|
||||
|
||||
# Optional: set cache directory
|
||||
export RESTIC_CACHE_DIR="/var/cache/restic"
|
||||
```
|
||||
|
||||
### Backup Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# /usr/local/bin/restic-backup.sh
|
||||
set -euo pipefail
|
||||
source /etc/restic/env
|
||||
LOG="/var/log/restic-backup.log"
|
||||
|
||||
echo "$(date): Starting backup" >> "$LOG"
|
||||
|
||||
restic backup /data /etc /var/lib/postgresql \
|
||||
--exclude-file=/etc/restic/excludes.txt \
|
||||
--tag "$(hostname)" --tag daily \
|
||||
--verbose >> "$LOG" 2>&1
|
||||
|
||||
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 \
|
||||
--prune >> "$LOG" 2>&1
|
||||
|
||||
# Weekly integrity check (Sundays)
|
||||
[ "$(date +%u)" -eq 7 ] && restic check --read-data-subset=10% >> "$LOG" 2>&1
|
||||
|
||||
echo "$(date): Backup completed" >> "$LOG"
|
||||
```
|
||||
|
||||
```bash
|
||||
chmod +x /usr/local/bin/restic-backup.sh
|
||||
```
|
||||
|
||||
## Scheduled Backups
|
||||
|
||||
### Systemd Timer (Recommended)
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/restic-backup.service
|
||||
[Unit]
|
||||
Description=Restic backup
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=root
|
||||
ExecStart=/usr/local/bin/restic-backup.sh
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
```
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/restic-backup.timer
|
||||
[Unit]
|
||||
Description=Run Restic backup daily at 2 AM
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 02:00:00
|
||||
Persistent=true
|
||||
RandomizedDelaySec=900
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
```bash
|
||||
# Enable and start the timer
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now restic-backup.timer
|
||||
|
||||
# Check timer status
|
||||
systemctl list-timers restic-backup.timer
|
||||
|
||||
# Run manually for testing
|
||||
systemctl start restic-backup.service
|
||||
journalctl -u restic-backup.service -f
|
||||
```
|
||||
|
||||
## Database Backups with Restic
|
||||
|
||||
```bash
|
||||
# PostgreSQL: stream dump directly into restic (no temp file)
|
||||
pg_dump -U postgres -Fc mydb | restic backup --stdin --stdin-filename mydb.dump \
|
||||
--tag postgres --tag mydb \
|
||||
--repo s3:s3.amazonaws.com/my-backup-bucket \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# MySQL / MariaDB: stream dump into restic
|
||||
mysqldump --all-databases --single-transaction | \
|
||||
restic backup --stdin --stdin-filename all-databases.sql \
|
||||
--tag mysql \
|
||||
--repo s3:s3.amazonaws.com/my-backup-bucket \
|
||||
--password-file /etc/restic/password.txt
|
||||
|
||||
# Restore PostgreSQL from restic
|
||||
restic dump latest mydb.dump \
|
||||
--repo s3:s3.amazonaws.com/my-backup-bucket \
|
||||
--password-file /etc/restic/password.txt \
|
||||
| pg_restore -U postgres -d mydb --clean --if-exists
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Diagnostic Command | Common Fix |
|
||||
|---|---|---|
|
||||
| "repository not initialized" | `restic cat config --repo <repo>` | Run `restic init --repo <repo>` first |
|
||||
| "wrong password" | Check env vars / password file | Verify RESTIC_PASSWORD_FILE contents and permissions |
|
||||
| Backup is slow | `restic backup -v` for progress | Check network bandwidth; exclude large unneeded dirs |
|
||||
| S3 permission denied | `aws s3 ls s3://bucket/` | Check IAM policy includes s3:GetObject, s3:PutObject |
|
||||
| "unable to create lock" | `restic unlock --repo <repo>` | A previous backup crashed; unlock the repository |
|
||||
| Restore shows empty dirs | `restic ls <snapshot-id>` | Verify correct snapshot ID; check --include path syntax |
|
||||
| Repository growing too large | `restic stats --repo <repo>` | Run `restic forget --prune` with stricter retention |
|
||||
| Check fails with pack errors | `restic check --read-data` | Rebuild index: `restic rebuild-index`; restore from another copy |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `linux-administration` -- Server maintenance and log management
|
||||
- `systemd-services` -- Scheduling backups with systemd timers
|
||||
- `object-storage` -- S3 and MinIO as backup destinations
|
||||
- `block-storage` -- LVM snapshots for consistent backups
|
||||
- `nfs-storage` -- Backing up NFS-shared data
|
||||
|
||||
@@ -9,48 +9,347 @@ metadata:
|
||||
|
||||
# Block Storage
|
||||
|
||||
Manage block storage volumes and LVM.
|
||||
Manage block storage volumes including LVM, cloud-based EBS, filesystem creation, snapshots, and RAID configurations. Covers the full lifecycle from provisioning raw disks to extending volumes in production.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Adding, partitioning, or formatting new disks on Linux servers
|
||||
- Managing LVM logical volumes for flexible storage allocation
|
||||
- Provisioning and attaching cloud block storage (AWS EBS)
|
||||
- Creating and restoring snapshots for backup or migration
|
||||
- Configuring software RAID for redundancy or performance
|
||||
- Extending existing volumes without downtime
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Root or sudo access on the target system
|
||||
- `lvm2` package installed for LVM operations
|
||||
- `mdadm` package installed for software RAID
|
||||
- AWS CLI configured for EBS operations
|
||||
- Understanding of the workload's I/O characteristics (IOPS, throughput)
|
||||
|
||||
## Disk Discovery and Partitioning
|
||||
|
||||
```bash
|
||||
# List all block devices
|
||||
lsblk
|
||||
lsblk -f # Show filesystem types and mount points
|
||||
|
||||
# Show detailed disk information
|
||||
fdisk -l /dev/sdb
|
||||
|
||||
# Identify disk model and health (requires smartmontools)
|
||||
smartctl -a /dev/sda
|
||||
smartctl -H /dev/sda # Quick health check
|
||||
|
||||
# Create a GPT partition table and a single partition
|
||||
parted /dev/sdb mklabel gpt
|
||||
parted /dev/sdb mkpart primary ext4 0% 100%
|
||||
|
||||
# Alternative: use fdisk for MBR partitioning
|
||||
fdisk /dev/sdb
|
||||
# n -> new partition, p -> primary, Enter defaults, w -> write
|
||||
|
||||
# Inform the kernel of partition table changes
|
||||
partprobe /dev/sdb
|
||||
|
||||
# Wipe filesystem signatures (prepare for LVM or RAID)
|
||||
wipefs -a /dev/sdb1
|
||||
```
|
||||
|
||||
## Filesystem Creation and Management
|
||||
|
||||
```bash
|
||||
# Create an ext4 filesystem
|
||||
mkfs.ext4 /dev/sdb1
|
||||
|
||||
# Create an ext4 filesystem with label and reserved block tuning
|
||||
mkfs.ext4 -L appdata -m 1 /dev/sdb1 # 1% reserved blocks (default is 5%)
|
||||
|
||||
# Create an XFS filesystem (recommended for large volumes)
|
||||
mkfs.xfs /dev/sdb1
|
||||
|
||||
# Create an XFS filesystem with label
|
||||
mkfs.xfs -L appdata /dev/sdb1
|
||||
|
||||
# Mount the filesystem
|
||||
mkdir -p /data
|
||||
mount /dev/sdb1 /data
|
||||
|
||||
# Add persistent mount to fstab (use UUID for reliability)
|
||||
blkid /dev/sdb1 # Get the UUID
|
||||
echo 'UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /data ext4 defaults,noatime 0 2' >> /etc/fstab
|
||||
|
||||
# Mount all entries in fstab
|
||||
mount -a
|
||||
|
||||
# Check and repair a filesystem (unmount first)
|
||||
umount /data
|
||||
fsck.ext4 -y /dev/sdb1
|
||||
xfs_repair /dev/sdb1 # For XFS
|
||||
|
||||
# Resize ext4 (can grow online while mounted)
|
||||
resize2fs /dev/sdb1
|
||||
|
||||
# Resize XFS (must be mounted to grow)
|
||||
xfs_growfs /data
|
||||
|
||||
# Check filesystem usage
|
||||
df -hT
|
||||
```
|
||||
|
||||
## LVM Management
|
||||
|
||||
### Creating an LVM Stack
|
||||
|
||||
```bash
|
||||
# Create physical volume
|
||||
pvcreate /dev/sdb
|
||||
# Step 1: Create physical volumes
|
||||
pvcreate /dev/sdb /dev/sdc
|
||||
|
||||
# Create volume group
|
||||
vgcreate data_vg /dev/sdb
|
||||
# View physical volumes
|
||||
pvs
|
||||
pvdisplay /dev/sdb
|
||||
|
||||
# Create logical volume
|
||||
lvcreate -L 50G -n app_lv data_vg
|
||||
# Step 2: Create a volume group from physical volumes
|
||||
vgcreate data_vg /dev/sdb /dev/sdc
|
||||
|
||||
# Format and mount
|
||||
# View volume groups
|
||||
vgs
|
||||
vgdisplay data_vg
|
||||
|
||||
# Step 3: Create logical volumes
|
||||
# Fixed size
|
||||
lvcreate -L 100G -n app_lv data_vg
|
||||
|
||||
# Use percentage of free space
|
||||
lvcreate -l 50%FREE -n logs_lv data_vg
|
||||
|
||||
# Use all remaining space
|
||||
lvcreate -l 100%FREE -n backup_lv data_vg
|
||||
|
||||
# View logical volumes
|
||||
lvs
|
||||
lvdisplay /dev/data_vg/app_lv
|
||||
|
||||
# Step 4: Create filesystem and mount
|
||||
mkfs.ext4 /dev/data_vg/app_lv
|
||||
mount /dev/data_vg/app_lv /data
|
||||
mkdir -p /data/app
|
||||
mount /dev/data_vg/app_lv /data/app
|
||||
|
||||
# Extend volume
|
||||
lvextend -L +10G /dev/data_vg/app_lv
|
||||
resize2fs /dev/data_vg/app_lv
|
||||
# Add to fstab
|
||||
echo '/dev/data_vg/app_lv /data/app ext4 defaults,noatime 0 2' >> /etc/fstab
|
||||
```
|
||||
|
||||
## AWS EBS
|
||||
### Extending Volumes (Online)
|
||||
|
||||
```bash
|
||||
# Create volume
|
||||
# Extend a logical volume by 20 GB
|
||||
lvextend -L +20G /dev/data_vg/app_lv
|
||||
|
||||
# Extend to fill all free space in the VG
|
||||
lvextend -l +100%FREE /dev/data_vg/app_lv
|
||||
|
||||
# Grow the ext4 filesystem (online, no unmount needed)
|
||||
resize2fs /dev/data_vg/app_lv
|
||||
|
||||
# Grow XFS filesystem (online)
|
||||
xfs_growfs /data/app
|
||||
|
||||
# Combined: extend LV and resize filesystem in one command
|
||||
lvextend -L +20G --resizefs /dev/data_vg/app_lv
|
||||
```
|
||||
|
||||
### Adding a New Disk to an Existing VG
|
||||
|
||||
```bash
|
||||
# Add a new physical volume
|
||||
pvcreate /dev/sdd
|
||||
|
||||
# Extend the volume group
|
||||
vgextend data_vg /dev/sdd
|
||||
|
||||
# Now extend any logical volume using the new space
|
||||
lvextend -l +100%FREE --resizefs /dev/data_vg/app_lv
|
||||
```
|
||||
|
||||
### LVM Snapshots
|
||||
|
||||
```bash
|
||||
# Create a snapshot (requires free space in VG)
|
||||
lvcreate -L 10G -s -n app_snap /dev/data_vg/app_lv
|
||||
|
||||
# Mount the snapshot read-only for backup
|
||||
mkdir -p /mnt/snapshot
|
||||
mount -o ro /dev/data_vg/app_snap /mnt/snapshot
|
||||
|
||||
# Perform backup from the snapshot
|
||||
tar czf /backup/app-$(date +%Y%m%d).tar.gz -C /mnt/snapshot .
|
||||
|
||||
# Unmount and remove the snapshot when done
|
||||
umount /mnt/snapshot
|
||||
lvremove -f /dev/data_vg/app_snap
|
||||
|
||||
# Restore from snapshot (reverts LV to snapshot point -- destructive)
|
||||
lvconvert --merge /dev/data_vg/app_snap
|
||||
# Note: if the LV is mounted, merge happens at next activation (reboot)
|
||||
```
|
||||
|
||||
### Reducing and Removing LVM Components
|
||||
|
||||
```bash
|
||||
# Shrink a logical volume (ext4 only -- XFS cannot shrink)
|
||||
# MUST unmount first
|
||||
umount /data/app
|
||||
e2fsck -f /dev/data_vg/app_lv
|
||||
resize2fs /dev/data_vg/app_lv 80G
|
||||
lvreduce -L 80G /dev/data_vg/app_lv
|
||||
mount /data/app
|
||||
|
||||
# Remove a logical volume
|
||||
umount /data/app
|
||||
lvremove /dev/data_vg/app_lv
|
||||
|
||||
# Remove a disk from a volume group (migrate data off first)
|
||||
pvmove /dev/sdc # Migrate extents to other PVs
|
||||
vgreduce data_vg /dev/sdc # Remove PV from VG
|
||||
pvremove /dev/sdc # Clean PV metadata
|
||||
```
|
||||
|
||||
## AWS EBS Management
|
||||
|
||||
```bash
|
||||
# Create a gp3 volume (general purpose SSD)
|
||||
aws ec2 create-volume \
|
||||
--availability-zone us-east-1a \
|
||||
--size 100 \
|
||||
--volume-type gp3 \
|
||||
--iops 3000 \
|
||||
--throughput 125 \
|
||||
--tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=app-data},{Key=Environment,Value=production}]'
|
||||
|
||||
# Create an io2 volume (provisioned IOPS SSD for databases)
|
||||
aws ec2 create-volume \
|
||||
--availability-zone us-east-1a \
|
||||
--size 500 \
|
||||
--volume-type io2 \
|
||||
--iops 10000
|
||||
|
||||
# List volumes with filters
|
||||
aws ec2 describe-volumes \
|
||||
--filters "Name=tag:Environment,Values=production" \
|
||||
--query 'Volumes[*].{ID:VolumeId,Size:Size,Type:VolumeType,State:State,AZ:AvailabilityZone}' \
|
||||
--output table
|
||||
|
||||
# Attach a volume to an instance
|
||||
aws ec2 attach-volume \
|
||||
--volume-id vol-0abc123def456789 \
|
||||
--instance-id i-0abc123def456789 \
|
||||
--device /dev/xvdf
|
||||
|
||||
# After attaching, format and mount on the instance
|
||||
lsblk # Identify the new device (e.g., /dev/nvme1n1)
|
||||
mkfs.ext4 /dev/nvme1n1
|
||||
mkdir -p /data
|
||||
mount /dev/nvme1n1 /data
|
||||
|
||||
# Modify a volume (resize without detaching -- gp3/io2)
|
||||
aws ec2 modify-volume \
|
||||
--volume-id vol-0abc123def456789 \
|
||||
--size 200
|
||||
|
||||
# After resize, grow the filesystem on the instance
|
||||
growpart /dev/nvme1n1 1 # If partitioned
|
||||
resize2fs /dev/nvme1n1 # ext4
|
||||
# xfs_growfs /data # XFS
|
||||
|
||||
# Create a snapshot
|
||||
aws ec2 create-snapshot \
|
||||
--volume-id vol-0abc123def456789 \
|
||||
--description "Pre-upgrade snapshot $(date +%Y-%m-%d)" \
|
||||
--tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=pre-upgrade}]'
|
||||
|
||||
# List snapshots
|
||||
aws ec2 describe-snapshots \
|
||||
--owner-ids self \
|
||||
--query 'Snapshots[*].{ID:SnapshotId,Vol:VolumeId,Size:VolumeSize,Date:StartTime,Desc:Description}' \
|
||||
--output table
|
||||
|
||||
# Create a volume from a snapshot (for restore or migration)
|
||||
aws ec2 create-volume \
|
||||
--snapshot-id snap-0abc123def456789 \
|
||||
--availability-zone us-east-1a \
|
||||
--volume-type gp3
|
||||
|
||||
# Attach to instance
|
||||
aws ec2 attach-volume \
|
||||
--volume-id vol-xxx \
|
||||
--instance-id i-xxx \
|
||||
--device /dev/xvdf
|
||||
# Detach a volume
|
||||
aws ec2 detach-volume --volume-id vol-0abc123def456789
|
||||
|
||||
# Delete a volume (ensure it is detached first)
|
||||
aws ec2 delete-volume --volume-id vol-0abc123def456789
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
## Software RAID (mdadm)
|
||||
|
||||
- Use LVM for flexibility
|
||||
- Implement RAID for redundancy
|
||||
- Monitor disk I/O
|
||||
- Regular disk health checks
|
||||
```bash
|
||||
# Install mdadm
|
||||
apt install -y mdadm # Debian/Ubuntu
|
||||
dnf install -y mdadm # RHEL/CentOS
|
||||
|
||||
# Create RAID 1 (mirror) with 2 disks
|
||||
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
|
||||
|
||||
# Create RAID 5 (striped with parity) with 3 disks + 1 spare
|
||||
mdadm --create /dev/md0 --level=5 --raid-devices=3 --spare-devices=1 /dev/sdb /dev/sdc /dev/sdd /dev/sde
|
||||
|
||||
# Create RAID 10 (striped mirrors) with 4 disks
|
||||
mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde
|
||||
|
||||
# Check RAID status
|
||||
cat /proc/mdstat
|
||||
mdadm --detail /dev/md0
|
||||
|
||||
# Save RAID configuration (persists across reboot)
|
||||
mdadm --detail --scan >> /etc/mdadm/mdadm.conf # Debian
|
||||
mdadm --detail --scan >> /etc/mdadm.conf # RHEL
|
||||
update-initramfs -u # Debian
|
||||
|
||||
# Create filesystem on RAID device
|
||||
mkfs.ext4 /dev/md0
|
||||
mkdir -p /data
|
||||
mount /dev/md0 /data
|
||||
|
||||
# Replace a failed disk
|
||||
mdadm --manage /dev/md0 --fail /dev/sdc
|
||||
mdadm --manage /dev/md0 --remove /dev/sdc
|
||||
# Insert new disk, then:
|
||||
mdadm --manage /dev/md0 --add /dev/sdf
|
||||
|
||||
# Monitor rebuild progress
|
||||
watch cat /proc/mdstat
|
||||
|
||||
# RAID level recommendations:
|
||||
# RAID 1: 2+ disks, mirroring, good for OS / boot drives
|
||||
# RAID 5: 3+ disks, single parity, good read performance
|
||||
# RAID 6: 4+ disks, double parity, survives 2 disk failures
|
||||
# RAID 10: 4+ disks, mirrored stripes, best I/O performance
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Diagnostic Command | Common Fix |
|
||||
|---|---|---|
|
||||
| Disk not showing up | `lsblk`, `dmesg \| tail` | Check physical connection; rescan SCSI bus |
|
||||
| Filesystem read-only | `dmesg \| grep error`, `mount` | Filesystem errors detected; run `fsck` after unmount |
|
||||
| LVM: no free space in VG | `vgs`, `pvs` | Add a new PV with `vgextend` |
|
||||
| EBS volume not visible | `lsblk` on instance | Check attach status in AWS console; NVMe naming differs |
|
||||
| RAID degraded | `cat /proc/mdstat` | Replace failed disk with `mdadm --manage --add` |
|
||||
| Cannot resize filesystem | `lvs`, `df -h` | Extend LV first, then resize FS; XFS needs to be mounted |
|
||||
| Slow I/O on EBS | `iostat -x 2`, check volume type | Upgrade to gp3/io2, increase IOPS/throughput |
|
||||
| Snapshot taking too long | AWS Console: snapshot progress | Snapshots are incremental; first one takes longest |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `linux-administration` -- Disk and filesystem basics
|
||||
- `performance-tuning` -- I/O scheduler and benchmarking with fio
|
||||
- `nfs-storage` -- Network filesystems built on top of block storage
|
||||
- `backup-recovery` -- Snapshot-based and file-level backup strategies
|
||||
- `object-storage` -- Alternative storage model for unstructured data
|
||||
|
||||
@@ -9,42 +9,279 @@ metadata:
|
||||
|
||||
# NFS Storage
|
||||
|
||||
Configure NFS for network file sharing.
|
||||
Configure NFS servers and clients for network file sharing across Linux systems. Covers NFSv4 server setup, export options, client mounting, autofs for on-demand mounts, Kerberos security, performance tuning, and Kubernetes integration.
|
||||
|
||||
## Server Configuration
|
||||
## When to Use
|
||||
|
||||
- Sharing directories between multiple Linux servers (web farms, build clusters)
|
||||
- Providing shared storage for containerized workloads (Kubernetes ReadWriteMany)
|
||||
- Centralizing home directories or application data across a fleet
|
||||
- Setting up a development environment with shared project files
|
||||
- Migrating from local storage to network-attached storage
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- NFS server: `nfs-kernel-server` (Debian/Ubuntu) or `nfs-utils` (RHEL/CentOS)
|
||||
- NFS client: `nfs-common` (Debian/Ubuntu) or `nfs-utils` (RHEL/CentOS)
|
||||
- Network connectivity between server and clients (TCP/UDP 2049 for NFSv4)
|
||||
- Firewall rules allowing NFS traffic
|
||||
- For NFSv4 Kerberos: `krb5-user` and a functioning KDC
|
||||
|
||||
## NFS Server Setup
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# Install
|
||||
apt install nfs-kernel-server
|
||||
# Debian / Ubuntu
|
||||
apt update && apt install -y nfs-kernel-server
|
||||
|
||||
# Configure exports
|
||||
# RHEL / CentOS
|
||||
dnf install -y nfs-utils
|
||||
|
||||
# Enable and start the NFS server
|
||||
systemctl enable --now nfs-server
|
||||
|
||||
# Verify NFS is running
|
||||
systemctl status nfs-server
|
||||
rpcinfo -p | grep nfs
|
||||
```
|
||||
|
||||
### Export Configuration (/etc/exports)
|
||||
|
||||
```bash
|
||||
# /etc/exports
|
||||
# Syntax: <directory> <client-spec>(options)
|
||||
|
||||
# Share /data to a specific subnet with read-write access
|
||||
/data 10.0.0.0/24(rw,sync,no_subtree_check,no_root_squash)
|
||||
|
||||
# Share /shared read-only to everyone
|
||||
/shared *(ro,sync,no_subtree_check)
|
||||
|
||||
# Apply changes
|
||||
exportfs -ra
|
||||
# Share /home to specific hosts
|
||||
/home server01.example.com(rw,sync,no_subtree_check)
|
||||
/home server02.example.com(rw,sync,no_subtree_check)
|
||||
|
||||
# Start service
|
||||
systemctl enable --now nfs-kernel-server
|
||||
# Share /var/nfs/projects with root squash (default, map root to nobody)
|
||||
/var/nfs/projects 10.0.0.0/24(rw,sync,no_subtree_check,root_squash)
|
||||
|
||||
# NFSv4 pseudo-root export (recommended for NFSv4)
|
||||
/srv/nfs 10.0.0.0/24(rw,sync,fsid=0,crossmnt,no_subtree_check)
|
||||
/srv/nfs/data 10.0.0.0/24(rw,sync,no_subtree_check,no_root_squash)
|
||||
/srv/nfs/shared 10.0.0.0/24(ro,sync,no_subtree_check)
|
||||
```
|
||||
|
||||
## Client Configuration
|
||||
### Export Options Explained
|
||||
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `rw` | Read-write access |
|
||||
| `ro` | Read-only access |
|
||||
| `sync` | Write data to disk before replying (safe, slower) |
|
||||
| `async` | Reply before data is written to disk (fast, risk of corruption) |
|
||||
| `no_subtree_check` | Disable subtree checking (improves reliability) |
|
||||
| `root_squash` | Map remote root (UID 0) to `nobody` (default, more secure) |
|
||||
| `no_root_squash` | Allow remote root to act as root on the server (use cautiously) |
|
||||
| `all_squash` | Map all remote UIDs/GIDs to `nobody` |
|
||||
| `anonuid=1000` | Map anonymous users to a specific UID |
|
||||
| `anongid=1000` | Map anonymous groups to a specific GID |
|
||||
| `crossmnt` | Allow clients to traverse into sub-mounts |
|
||||
| `fsid=0` | Mark as the NFSv4 pseudo-root |
|
||||
|
||||
### Applying Export Changes
|
||||
|
||||
```bash
|
||||
# Install
|
||||
apt install nfs-common
|
||||
# Apply changes to exports (no server restart needed)
|
||||
exportfs -ra
|
||||
|
||||
# Mount
|
||||
mount -t nfs server:/data /mnt/data
|
||||
# Show current exports
|
||||
exportfs -v
|
||||
|
||||
# /etc/fstab
|
||||
server:/data /mnt/data nfs defaults,_netdev 0 0
|
||||
# Export a new directory on the fly (temporary, not persistent)
|
||||
exportfs -o rw,sync,no_subtree_check 10.0.0.0/24:/tmp/share
|
||||
|
||||
# Unexport a directory
|
||||
exportfs -u 10.0.0.0/24:/tmp/share
|
||||
```
|
||||
|
||||
## Kubernetes NFS
|
||||
### Server Firewall Configuration
|
||||
|
||||
```bash
|
||||
# UFW (Ubuntu)
|
||||
ufw allow from 10.0.0.0/24 to any port nfs
|
||||
ufw allow from 10.0.0.0/24 to any port 111 # rpcbind (NFSv3)
|
||||
|
||||
# firewalld (RHEL/CentOS)
|
||||
firewall-cmd --permanent --add-service=nfs
|
||||
firewall-cmd --permanent --add-service=rpc-bind
|
||||
firewall-cmd --permanent --add-service=mountd
|
||||
firewall-cmd --reload
|
||||
|
||||
# NFSv4 only needs TCP 2049 (no rpcbind or mountd)
|
||||
firewall-cmd --permanent --add-port=2049/tcp
|
||||
firewall-cmd --reload
|
||||
```
|
||||
|
||||
## NFS Client Configuration
|
||||
|
||||
### Manual Mounting
|
||||
|
||||
```bash
|
||||
# Install NFS client
|
||||
apt install -y nfs-common # Debian/Ubuntu
|
||||
dnf install -y nfs-utils # RHEL/CentOS
|
||||
|
||||
# Discover exports from the server
|
||||
showmount -e nfs-server.example.com
|
||||
|
||||
# Mount an NFS share manually
|
||||
mkdir -p /mnt/data
|
||||
mount -t nfs nfs-server.example.com:/data /mnt/data
|
||||
|
||||
# Mount with specific NFS version and options
|
||||
mount -t nfs -o vers=4.2,tcp,hard,intr nfs-server.example.com:/data /mnt/data
|
||||
|
||||
# Verify the mount
|
||||
mount | grep nfs
|
||||
df -hT /mnt/data
|
||||
|
||||
# Unmount
|
||||
umount /mnt/data
|
||||
```
|
||||
|
||||
### Persistent Mounts via /etc/fstab
|
||||
|
||||
```bash
|
||||
# /etc/fstab entries for NFS
|
||||
|
||||
# Basic NFSv4 mount
|
||||
nfs-server.example.com:/data /mnt/data nfs4 defaults,_netdev 0 0
|
||||
|
||||
# Mount with performance and reliability options
|
||||
nfs-server.example.com:/data /mnt/data nfs4 hard,intr,rsize=1048576,wsize=1048576,timeo=600,retrans=3,_netdev 0 0
|
||||
|
||||
# Read-only mount
|
||||
nfs-server.example.com:/shared /mnt/shared nfs4 ro,_netdev 0 0
|
||||
|
||||
# Mount with specific UID/GID mapping (useful for containers)
|
||||
nfs-server.example.com:/data /mnt/data nfs4 defaults,_netdev,uid=1000,gid=1000 0 0
|
||||
```
|
||||
|
||||
```bash
|
||||
# Mount all fstab entries
|
||||
mount -a
|
||||
|
||||
# Test fstab entry without actually mounting
|
||||
mount --fake -a -v
|
||||
```
|
||||
|
||||
### Mount Options Explained
|
||||
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `hard` | Retry NFS requests indefinitely (recommended for data integrity) |
|
||||
| `soft` | Return error after `retrans` retries (risk of data corruption) |
|
||||
| `intr` | Allow interruption of hard-mounted NFS requests |
|
||||
| `rsize=1048576` | Read buffer size in bytes (1 MB, max for NFSv4) |
|
||||
| `wsize=1048576` | Write buffer size in bytes (1 MB) |
|
||||
| `timeo=600` | Timeout in tenths of a second (60 seconds) |
|
||||
| `retrans=3` | Number of retries before error (soft) or message (hard) |
|
||||
| `_netdev` | Wait for network before mounting (critical for boot) |
|
||||
| `noatime` | Do not update access time (improves performance) |
|
||||
| `nconnect=8` | Use multiple TCP connections (kernel 5.3+, improves throughput) |
|
||||
|
||||
## Autofs (On-Demand Mounting)
|
||||
|
||||
```bash
|
||||
# Install autofs
|
||||
apt install -y autofs # Debian/Ubuntu
|
||||
dnf install -y autofs # RHEL/CentOS
|
||||
|
||||
# Configure the master map
|
||||
# /etc/auto.master or /etc/auto.master.d/nfs.autofs
|
||||
/mnt/nfs /etc/auto.nfs --timeout=300
|
||||
```
|
||||
|
||||
### /etc/auto.nfs
|
||||
|
||||
```text
|
||||
# Format: mount-point options location
|
||||
# Mounts will appear under /mnt/nfs/<mount-point>
|
||||
|
||||
data -rw,hard,intr,rsize=1048576,wsize=1048576 nfs-server.example.com:/data
|
||||
shared -ro,hard,intr nfs-server.example.com:/shared
|
||||
home -rw,hard,intr nfs-server.example.com:/home/&
|
||||
|
||||
# Wildcard: mount any subdirectory from the server automatically
|
||||
# /etc/auto.master entry: /mnt/nfs /etc/auto.nfs
|
||||
* -rw,hard,intr nfs-server.example.com:/srv/nfs/&
|
||||
```
|
||||
|
||||
```bash
|
||||
# Enable and start autofs
|
||||
systemctl enable --now autofs
|
||||
|
||||
# Test: simply cd into the mount point and it appears
|
||||
ls /mnt/nfs/data # Triggers auto-mount
|
||||
# The share unmounts automatically after the timeout (300 seconds idle)
|
||||
|
||||
# Check autofs status
|
||||
systemctl status autofs
|
||||
automount -v # Verbose debugging mode (foreground)
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Server-Side Tuning
|
||||
|
||||
```bash
|
||||
# Increase the number of NFS daemon threads (default is 8)
|
||||
# /etc/default/nfs-kernel-server (Debian) or /etc/sysconfig/nfs (RHEL)
|
||||
RPCNFSDCOUNT=32
|
||||
|
||||
# Or set at runtime
|
||||
echo 32 > /proc/fs/nfsd/threads
|
||||
|
||||
# Restart NFS to apply
|
||||
systemctl restart nfs-server
|
||||
|
||||
# Tune NFS server read/write sizes in sysctl
|
||||
# These are auto-negotiated but can be adjusted
|
||||
echo 1048576 > /proc/fs/nfsd/max_block_size
|
||||
|
||||
# Kernel network buffer tuning (see performance-tuning skill)
|
||||
sysctl -w net.core.rmem_max=134217728
|
||||
sysctl -w net.core.wmem_max=134217728
|
||||
```
|
||||
|
||||
### Client-Side Tuning
|
||||
|
||||
```bash
|
||||
# Use large read/write buffer sizes in mount options
|
||||
mount -t nfs4 -o rsize=1048576,wsize=1048576,noatime nfs-server:/data /mnt/data
|
||||
|
||||
# Use multiple TCP connections (Linux kernel 5.3+)
|
||||
mount -t nfs4 -o nconnect=8 nfs-server:/data /mnt/data
|
||||
|
||||
# Check current mount options and NFS statistics
|
||||
nfsstat -c # Client NFS statistics
|
||||
nfsstat -s # Server NFS statistics
|
||||
mountstats /mnt/data # Detailed per-mount stats
|
||||
|
||||
# Test NFS throughput with dd
|
||||
dd if=/dev/zero of=/mnt/data/testfile bs=1M count=1024 oflag=direct
|
||||
dd if=/mnt/data/testfile of=/dev/null bs=1M iflag=direct
|
||||
rm /mnt/data/testfile
|
||||
|
||||
# Test with fio for more realistic workloads
|
||||
fio --name=nfs-test --directory=/mnt/data --ioengine=libaio --direct=1 \
|
||||
--rw=randrw --bs=4k --numjobs=4 --size=1G --runtime=60 --group_reporting
|
||||
```
|
||||
|
||||
## Kubernetes NFS Integration
|
||||
|
||||
```yaml
|
||||
# nfs-pv.yaml -- Static PersistentVolume
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
@@ -54,14 +291,50 @@ spec:
|
||||
storage: 100Gi
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: nfs
|
||||
nfs:
|
||||
server: nfs-server.example.com
|
||||
path: /data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: nfs-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: nfs
|
||||
resources:
|
||||
requests:
|
||||
storage: 100Gi
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
```bash
|
||||
# For dynamic provisioning, install the NFS CSI driver via Helm
|
||||
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
|
||||
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system
|
||||
# Then create a StorageClass pointing to your NFS server and share path.
|
||||
```
|
||||
|
||||
- Use proper export options
|
||||
- Implement firewall rules
|
||||
- Monitor NFS performance
|
||||
- Use NFSv4 for security
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Diagnostic Command | Common Fix |
|
||||
|---|---|---|
|
||||
| mount: access denied | `showmount -e server`, `exportfs -v` | Check /etc/exports, run `exportfs -ra`, verify subnet |
|
||||
| mount hangs | `mount -v`, check network | Verify firewall allows TCP 2049; use `bg` mount option |
|
||||
| Stale file handle | `ls /mnt/data` returns stale error | Unmount and remount: `umount -f /mnt/data && mount -a` |
|
||||
| Permission denied on files | `ls -la`, check UID mapping | Match UIDs or use `all_squash,anonuid=1000,anongid=1000` |
|
||||
| Slow NFS performance | `nfsstat -c`, `mountstats /mnt/data` | Increase rsize/wsize, add nconnect=8, tune NFS threads |
|
||||
| Autofs not mounting | `systemctl status autofs`, `automount -v` | Check /etc/auto.master syntax, verify server is reachable |
|
||||
| NFSv4 ID mapping wrong | `id username` on both sides | Ensure matching domain in `/etc/idmapd.conf` |
|
||||
| Boot hangs waiting for NFS | Check fstab options | Add `_netdev` and `bg` options to fstab entry |
|
||||
| Docker volume mount fails | `docker volume inspect`, `dmesg` | Verify NFS client packages installed on Docker host |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `linux-administration` -- Server setup and network configuration
|
||||
- `block-storage` -- Underlying storage for NFS server data directories
|
||||
- `performance-tuning` -- Kernel and network tuning for NFS throughput
|
||||
- `backup-recovery` -- Backing up NFS-hosted data
|
||||
- `object-storage` -- Alternative storage model for cloud-native workloads
|
||||
|
||||
@@ -9,44 +9,354 @@ metadata:
|
||||
|
||||
# Object Storage
|
||||
|
||||
Configure and manage object storage solutions.
|
||||
Configure and manage object storage solutions including AWS S3, MinIO (self-hosted), and compatible providers. Covers CLI operations, bucket policies, lifecycle rules, versioning, encryption, and the MinIO client (mc).
|
||||
|
||||
## AWS S3
|
||||
## When to Use
|
||||
|
||||
- Storing application assets, backups, logs, or media files
|
||||
- Setting up an S3-compatible object store on-premises with MinIO
|
||||
- Configuring lifecycle rules to transition or expire objects automatically
|
||||
- Implementing access control with bucket policies and IAM
|
||||
- Syncing data between local filesystems and object storage
|
||||
- Serving static content from S3 or MinIO
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- AWS CLI v2 installed and configured (`aws configure`) for S3 operations
|
||||
- Docker installed for MinIO self-hosted setup
|
||||
- MinIO client (`mc`) installed for MinIO management
|
||||
- IAM credentials with appropriate S3 permissions
|
||||
- Network access to the object storage endpoint
|
||||
|
||||
## AWS S3 CLI Operations
|
||||
|
||||
### Bucket Management
|
||||
|
||||
```bash
|
||||
# Create bucket
|
||||
aws s3 mb s3://my-bucket
|
||||
# Create a new bucket
|
||||
aws s3 mb s3://my-app-assets-prod
|
||||
|
||||
# Upload/Download
|
||||
aws s3 cp file.txt s3://my-bucket/
|
||||
aws s3 sync ./local s3://my-bucket/remote
|
||||
# Create a bucket in a specific region
|
||||
aws s3 mb s3://my-app-assets-eu --region eu-west-1
|
||||
|
||||
# Configure lifecycle
|
||||
# List all buckets
|
||||
aws s3 ls
|
||||
|
||||
# List objects in a bucket (with sizes)
|
||||
aws s3 ls s3://my-app-assets-prod --recursive --human-readable --summarize
|
||||
|
||||
# Delete an empty bucket
|
||||
aws s3 rb s3://my-old-bucket
|
||||
|
||||
# Delete a bucket and ALL its contents (destructive)
|
||||
aws s3 rb s3://my-old-bucket --force
|
||||
```
|
||||
|
||||
### Upload and Download
|
||||
|
||||
```bash
|
||||
# Upload a single file
|
||||
aws s3 cp ./report.pdf s3://my-app-assets-prod/reports/
|
||||
|
||||
# Upload with a specific storage class
|
||||
aws s3 cp ./archive.tar.gz s3://my-app-assets-prod/archives/ --storage-class GLACIER
|
||||
|
||||
# Upload with server-side encryption (AES-256)
|
||||
aws s3 cp ./sensitive.dat s3://my-app-assets-prod/data/ --sse AES256
|
||||
|
||||
# Download a file
|
||||
aws s3 cp s3://my-app-assets-prod/reports/report.pdf ./downloads/
|
||||
|
||||
# Sync a local directory to S3 (upload only changed files)
|
||||
aws s3 sync ./build/ s3://my-app-assets-prod/static/ --delete
|
||||
|
||||
# Sync from S3 to local
|
||||
aws s3 sync s3://my-app-assets-prod/static/ ./local-copy/
|
||||
|
||||
# Sync with exclusion patterns
|
||||
aws s3 sync ./logs/ s3://my-app-logs/ --exclude "*.tmp" --exclude ".git/*"
|
||||
|
||||
# Copy between buckets
|
||||
aws s3 sync s3://source-bucket/ s3://destination-bucket/ --source-region us-east-1 --region eu-west-1
|
||||
|
||||
# Generate a pre-signed URL (temporary access, 1 hour)
|
||||
aws s3 presign s3://my-app-assets-prod/reports/report.pdf --expires-in 3600
|
||||
|
||||
# Recursive delete of a prefix
|
||||
aws s3 rm s3://my-app-assets-prod/old-data/ --recursive
|
||||
```
|
||||
|
||||
### Versioning
|
||||
|
||||
```bash
|
||||
# Enable versioning on a bucket
|
||||
aws s3api put-bucket-versioning \
|
||||
--bucket my-app-assets-prod \
|
||||
--versioning-configuration Status=Enabled
|
||||
|
||||
# Check versioning status
|
||||
aws s3api get-bucket-versioning --bucket my-app-assets-prod
|
||||
|
||||
# List object versions
|
||||
aws s3api list-object-versions --bucket my-app-assets-prod --prefix reports/
|
||||
|
||||
# Restore a previous version (copy old version to current)
|
||||
aws s3api copy-object \
|
||||
--bucket my-app-assets-prod \
|
||||
--copy-source "my-app-assets-prod/reports/report.pdf?versionId=abc123" \
|
||||
--key reports/report.pdf
|
||||
|
||||
# Delete a specific version permanently
|
||||
aws s3api delete-object \
|
||||
--bucket my-app-assets-prod \
|
||||
--key reports/old-report.pdf \
|
||||
--version-id abc123
|
||||
```
|
||||
|
||||
### Bucket Policies
|
||||
|
||||
```bash
|
||||
# Apply a bucket policy from a JSON file
|
||||
aws s3api put-bucket-policy --bucket my-app-assets-prod --policy file://policy.json
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "PublicReadForStaticSite",
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": "s3:GetObject",
|
||||
"Resource": "arn:aws:s3:::my-app-assets-prod/static/*"
|
||||
},
|
||||
{
|
||||
"Sid": "DenyUnencryptedUploads",
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "s3:PutObject",
|
||||
"Resource": "arn:aws:s3:::my-app-assets-prod/*",
|
||||
"Condition": {
|
||||
"StringNotEquals": {
|
||||
"s3:x-amz-server-side-encryption": "AES256"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"Sid": "RestrictToVPC",
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "s3:*",
|
||||
"Resource": [
|
||||
"arn:aws:s3:::my-app-assets-prod",
|
||||
"arn:aws:s3:::my-app-assets-prod/*"
|
||||
],
|
||||
"Condition": {
|
||||
"StringNotEquals": {
|
||||
"aws:sourceVpce": "vpce-abc123"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Lifecycle Rules
|
||||
|
||||
```bash
|
||||
# Apply lifecycle configuration
|
||||
aws s3api put-bucket-lifecycle-configuration \
|
||||
--bucket my-bucket \
|
||||
--bucket my-app-assets-prod \
|
||||
--lifecycle-configuration file://lifecycle.json
|
||||
```
|
||||
|
||||
## MinIO (Self-Hosted)
|
||||
|
||||
```bash
|
||||
# Deploy
|
||||
docker run -d \
|
||||
-p 9000:9000 -p 9001:9001 \
|
||||
-e MINIO_ROOT_USER=admin \
|
||||
-e MINIO_ROOT_PASSWORD=password \
|
||||
-v /data:/data \
|
||||
minio/minio server /data --console-address ":9001"
|
||||
|
||||
# Configure mc client
|
||||
mc alias set myminio http://localhost:9000 admin password
|
||||
mc mb myminio/mybucket
|
||||
```json
|
||||
{
|
||||
"Rules": [
|
||||
{
|
||||
"ID": "TransitionLogsToIA",
|
||||
"Filter": { "Prefix": "logs/" },
|
||||
"Status": "Enabled",
|
||||
"Transitions": [
|
||||
{
|
||||
"Days": 30,
|
||||
"StorageClass": "STANDARD_IA"
|
||||
},
|
||||
{
|
||||
"Days": 90,
|
||||
"StorageClass": "GLACIER"
|
||||
}
|
||||
],
|
||||
"Expiration": {
|
||||
"Days": 365
|
||||
}
|
||||
},
|
||||
{
|
||||
"ID": "CleanupIncompleteUploads",
|
||||
"Filter": { "Prefix": "" },
|
||||
"Status": "Enabled",
|
||||
"AbortIncompleteMultipartUpload": {
|
||||
"DaysAfterInitiation": 7
|
||||
}
|
||||
},
|
||||
{
|
||||
"ID": "ExpireOldVersions",
|
||||
"Filter": { "Prefix": "" },
|
||||
"Status": "Enabled",
|
||||
"NoncurrentVersionExpiration": {
|
||||
"NoncurrentDays": 30
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
```bash
|
||||
# View current lifecycle rules
|
||||
aws s3api get-bucket-lifecycle-configuration --bucket my-app-assets-prod
|
||||
|
||||
- Enable versioning
|
||||
- Implement lifecycle policies
|
||||
- Use server-side encryption
|
||||
- Configure access logging
|
||||
- Implement bucket policies
|
||||
# Enable S3 access logging
|
||||
aws s3api put-bucket-logging --bucket my-app-assets-prod --bucket-logging-status '{
|
||||
"LoggingEnabled": {
|
||||
"TargetBucket": "my-app-logs",
|
||||
"TargetPrefix": "s3-access-logs/"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
## MinIO Self-Hosted Setup
|
||||
|
||||
### Docker Deployment
|
||||
|
||||
```bash
|
||||
# Single-node MinIO with persistent storage
|
||||
docker run -d \
|
||||
--name minio \
|
||||
--restart unless-stopped \
|
||||
-p 9000:9000 \
|
||||
-p 9001:9001 \
|
||||
-e MINIO_ROOT_USER=minioadmin \
|
||||
-e MINIO_ROOT_PASSWORD=minio-secret-key-change-me \
|
||||
-v /data/minio:/data \
|
||||
minio/minio server /data --console-address ":9001"
|
||||
```
|
||||
|
||||
### Docker Compose (Multi-Drive)
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
version: "3.8"
|
||||
services:
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
command: server /data{1...4} --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: minioadmin
|
||||
MINIO_ROOT_PASSWORD: minio-secret-key-change-me
|
||||
MINIO_BROWSER_REDIRECT_URL: https://minio-console.example.com
|
||||
ports:
|
||||
- "9000:9000"
|
||||
- "9001:9001"
|
||||
volumes:
|
||||
- minio-data1:/data1
|
||||
- minio-data2:/data2
|
||||
- minio-data3:/data3
|
||||
- minio-data4:/data4
|
||||
healthcheck:
|
||||
test: ["CMD", "mc", "ready", "local"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
minio-data1:
|
||||
minio-data2:
|
||||
minio-data3:
|
||||
minio-data4:
|
||||
```
|
||||
|
||||
```bash
|
||||
# Start the stack
|
||||
docker compose up -d
|
||||
|
||||
# Check health
|
||||
docker compose ps
|
||||
curl -s http://localhost:9000/minio/health/live
|
||||
```
|
||||
|
||||
### MinIO Client (mc) Commands
|
||||
|
||||
```bash
|
||||
# Install mc
|
||||
curl -O https://dl.min.io/client/mc/release/linux-amd64/mc
|
||||
chmod +x mc && mv mc /usr/local/bin/
|
||||
|
||||
# Configure an alias for the MinIO server
|
||||
mc alias set myminio http://localhost:9000 minioadmin minio-secret-key-change-me
|
||||
|
||||
# Configure an alias for AWS S3
|
||||
mc alias set aws https://s3.amazonaws.com AKIAEXAMPLE SECRETKEYEXAMPLE
|
||||
|
||||
# Bucket operations
|
||||
mc mb myminio/app-data
|
||||
mc mb myminio/backups
|
||||
mc ls myminio/
|
||||
|
||||
# Upload and download
|
||||
mc cp ./backup.tar.gz myminio/backups/
|
||||
mc cp myminio/backups/backup.tar.gz ./restore/
|
||||
|
||||
# Sync a directory (mirror)
|
||||
mc mirror ./static/ myminio/app-data/static/
|
||||
mc mirror --watch ./static/ myminio/app-data/static/ # Continuous sync
|
||||
|
||||
# Set bucket policy (download = public read)
|
||||
mc anonymous set download myminio/app-data/static
|
||||
|
||||
# Set a specific policy from JSON
|
||||
mc anonymous set-json policy.json myminio/app-data
|
||||
|
||||
# Enable versioning
|
||||
mc version enable myminio/app-data
|
||||
|
||||
# Set lifecycle rule: expire objects in tmp/ after 7 days
|
||||
mc ilm rule add --expiry-days 7 --prefix "tmp/" myminio/app-data
|
||||
|
||||
# List lifecycle rules
|
||||
mc ilm rule ls myminio/app-data
|
||||
|
||||
# Create a service account (for applications)
|
||||
mc admin user svcacct add myminio minioadmin --access-key myapp-key --secret-key myapp-secret
|
||||
|
||||
# View server info and disk usage
|
||||
mc admin info myminio
|
||||
|
||||
# Check bucket disk usage
|
||||
mc du myminio/app-data
|
||||
|
||||
# Set a notification target (webhook on object creation)
|
||||
mc event add myminio/app-data arn:minio:sqs::myqueue:webhook --event put
|
||||
mc event ls myminio/app-data
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Diagnostic Command | Common Fix |
|
||||
|---|---|---|
|
||||
| Access Denied on S3 | `aws s3api get-bucket-policy --bucket name` | Check IAM policy, bucket policy, and block public access settings |
|
||||
| Slow uploads | `aws s3 cp --debug` | Use multipart: `aws configure set s3.multipart_threshold 64MB` |
|
||||
| 403 on pre-signed URL | Check clock skew, URL expiry | Sync system clock with NTP; regenerate URL |
|
||||
| MinIO unhealthy | `mc admin info myminio` | Check disk space, container logs, port availability |
|
||||
| Lifecycle rules not applying | `aws s3api get-bucket-lifecycle-configuration` | Rules run once per day; check Filter prefix matches |
|
||||
| Objects not versioned | `aws s3api get-bucket-versioning` | Enable versioning; it does not apply retroactively |
|
||||
| mc: connection refused | `mc alias ls` | Verify endpoint URL, port, and credentials |
|
||||
| Large sync is slow | Monitor with `mc mirror --watch` | Use `--multi-thread` flag, increase bandwidth |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `block-storage` -- Underlying disk storage for MinIO data volumes
|
||||
- `backup-recovery` -- Using S3/MinIO as a backup destination with restic
|
||||
- `nfs-storage` -- Alternative shared storage for file-level access
|
||||
- `linux-administration` -- Server setup and maintenance for MinIO hosts
|
||||
|
||||
Reference in New Issue
Block a user