feat: add incremental backups, S3 cleanup, and cron scheduling
Incremental backups using BC API's lastModifiedDateTime filter to only
export records changed since the last successful run. Runs every 15
minutes via cron, with a daily full backup for complete snapshots.
bc-export.ps1:
- Add -SinceDateTime parameter for incremental filtering
- Append $filter=lastModifiedDateTime gt {timestamp} to all entity URLs
- Exit code 2 when no records changed (skip archive/upload)
- Record mode and sinceDateTime in export-metadata.json
bc-backup.sh:
- Accept --mode full|incremental flag (default: incremental)
- State file (last-run-state.json) tracks last successful run timestamp
- Auto-fallback to full when no state file exists
- Skip archive/encrypt/upload when incremental finds 0 changes
- Lock file (.backup.lock) prevents overlapping cron runs
- S3 keys organized by mode: backups/full/ vs backups/incremental/
bc-cleanup.sh (new):
- Lists all S3 objects under backups/ prefix
- Deletes objects older than RETENTION_DAYS (default 30)
- Handles pagination for large buckets
- Gracefully handles COMPLIANCE-locked objects
bc-backup.conf.template:
- Add BACKUP_MODE_DEFAULT option
cron-examples.txt:
- Recommended setup: 15-min incremental + daily full + daily cleanup
- Alternative schedules (30-min, hourly)
- Systemd timer examples
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -2,92 +2,91 @@
|
||||
# Add these to your crontab with: crontab -e
|
||||
|
||||
# ===================================
|
||||
# Hourly Backup (Every hour at minute 0)
|
||||
# RECOMMENDED: Incremental + Full + Cleanup
|
||||
# ===================================
|
||||
0 * * * * /home/malin/c0ding/bcbak/bc-backup.sh >> /home/malin/c0ding/bcbak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Every 2 hours
|
||||
# ===================================
|
||||
0 */2 * * * /home/malin/c0ding/bcbak/bc-backup.sh >> /home/malin/c0ding/bcbak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Every 4 hours
|
||||
# ===================================
|
||||
0 */4 * * * /home/malin/c0ding/bcbak/bc-backup.sh >> /home/malin/c0ding/bcbak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Every 6 hours (at 00:00, 06:00, 12:00, 18:00)
|
||||
# ===================================
|
||||
0 0,6,12,18 * * * /home/malin/c0ding/bcbak/bc-backup.sh >> /home/malin/c0ding/bcbak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Daily at 2:00 AM
|
||||
# ===================================
|
||||
0 2 * * * /home/malin/c0ding/bcbak/bc-backup.sh >> /home/malin/c0ding/bcbak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Multiple times per day (8 AM, 12 PM, 4 PM, 8 PM)
|
||||
# ===================================
|
||||
0 8,12,16,20 * * * /home/malin/c0ding/bcbak/bc-backup.sh >> /home/malin/c0ding/bcbak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Business hours only (9 AM - 5 PM, hourly)
|
||||
# ===================================
|
||||
0 9-17 * * 1-5 /home/malin/c0ding/bcbak/bc-backup.sh >> /home/malin/c0ding/bcbak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# With email notifications (requires mail/sendmail)
|
||||
# ===================================
|
||||
MAILTO=your-email@example.com
|
||||
0 * * * * /home/malin/c0ding/bcbak/bc-backup.sh >> /home/malin/c0ding/bcbak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# With environment variables
|
||||
# ===================================
|
||||
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
0 * * * * /home/malin/c0ding/bcbak/bc-backup.sh >> /home/malin/c0ding/bcbak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Systemd Timer Alternative (More Reliable)
|
||||
# ===================================
|
||||
# Instead of cron, you can use systemd timers.
|
||||
# Create files in /etc/systemd/system/:
|
||||
# This is the recommended setup for production use:
|
||||
# - Incremental every 15 minutes (only changed records)
|
||||
# - Full backup daily at 2 AM (complete snapshot)
|
||||
# - S3 cleanup daily at 3 AM (delete expired backups)
|
||||
#
|
||||
# bc-backup.service:
|
||||
# The lock file prevents overlapping runs, so if a full backup
|
||||
# is still running at :15, the incremental will skip gracefully.
|
||||
|
||||
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
|
||||
|
||||
# Incremental backup every 15 minutes
|
||||
*/15 * * * * /root/BC-bak/bc-backup.sh --mode incremental >> /root/BC-bak/logs/cron.log 2>&1
|
||||
|
||||
# Full backup daily at 2:00 AM
|
||||
0 2 * * * /root/BC-bak/bc-backup.sh --mode full >> /root/BC-bak/logs/cron.log 2>&1
|
||||
|
||||
# S3 cleanup daily at 3:00 AM (delete backups older than RETENTION_DAYS)
|
||||
0 3 * * * /root/BC-bak/bc-cleanup.sh >> /root/BC-bak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Alternative: Incremental every 30 minutes
|
||||
# ===================================
|
||||
# */30 * * * * /root/BC-bak/bc-backup.sh --mode incremental >> /root/BC-bak/logs/cron.log 2>&1
|
||||
# 0 2 * * * /root/BC-bak/bc-backup.sh --mode full >> /root/BC-bak/logs/cron.log 2>&1
|
||||
# 0 3 * * * /root/BC-bak/bc-cleanup.sh >> /root/BC-bak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Alternative: Incremental hourly
|
||||
# ===================================
|
||||
# 0 * * * * /root/BC-bak/bc-backup.sh --mode incremental >> /root/BC-bak/logs/cron.log 2>&1
|
||||
# 0 2 * * * /root/BC-bak/bc-backup.sh --mode full >> /root/BC-bak/logs/cron.log 2>&1
|
||||
# 0 3 * * * /root/BC-bak/bc-cleanup.sh >> /root/BC-bak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Full backup only (no incremental)
|
||||
# ===================================
|
||||
# 0 2 * * * /root/BC-bak/bc-backup.sh --mode full >> /root/BC-bak/logs/cron.log 2>&1
|
||||
# 0 3 * * * /root/BC-bak/bc-cleanup.sh >> /root/BC-bak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# With email notifications on failure
|
||||
# ===================================
|
||||
# MAILTO=admin@example.com
|
||||
# */15 * * * * /root/BC-bak/bc-backup.sh --mode incremental >> /root/BC-bak/logs/cron.log 2>&1
|
||||
|
||||
# ===================================
|
||||
# Systemd Timer Alternative
|
||||
# ===================================
|
||||
# Create /etc/systemd/system/bc-backup-incremental.service:
|
||||
# [Unit]
|
||||
# Description=Business Central Database Backup
|
||||
# Description=BC Incremental Backup
|
||||
#
|
||||
# [Service]
|
||||
# Type=oneshot
|
||||
# User=malin
|
||||
# WorkingDirectory=/home/malin/c0ding/bcbak
|
||||
# ExecStart=/home/malin/c0ding/bcbak/bc-backup.sh
|
||||
# StandardOutput=append:/home/malin/c0ding/bcbak/logs/backup.log
|
||||
# StandardError=append:/home/malin/c0ding/bcbak/logs/backup.log
|
||||
# ExecStart=/root/BC-bak/bc-backup.sh --mode incremental
|
||||
# StandardOutput=append:/root/BC-bak/logs/backup.log
|
||||
# StandardError=append:/root/BC-bak/logs/backup.log
|
||||
#
|
||||
# bc-backup.timer:
|
||||
# Create /etc/systemd/system/bc-backup-incremental.timer:
|
||||
# [Unit]
|
||||
# Description=Run BC Backup Every Hour
|
||||
# Description=Run BC Incremental Backup Every 15 Minutes
|
||||
#
|
||||
# [Timer]
|
||||
# OnCalendar=hourly
|
||||
# OnCalendar=*:0/15
|
||||
# Persistent=true
|
||||
#
|
||||
# [Install]
|
||||
# WantedBy=timers.target
|
||||
#
|
||||
# Create similar for full backup (OnCalendar=*-*-* 02:00:00)
|
||||
# and cleanup (OnCalendar=*-*-* 03:00:00)
|
||||
#
|
||||
# Enable with:
|
||||
# sudo systemctl daemon-reload
|
||||
# sudo systemctl enable bc-backup.timer
|
||||
# sudo systemctl start bc-backup.timer
|
||||
# sudo systemctl status bc-backup.timer
|
||||
# sudo systemctl enable --now bc-backup-incremental.timer
|
||||
|
||||
# ===================================
|
||||
# Useful Cron Management Commands
|
||||
# Useful Commands
|
||||
# ===================================
|
||||
# Edit crontab: crontab -e
|
||||
# List crontab: crontab -l
|
||||
# Remove all cron jobs: crontab -r
|
||||
# View cron logs: grep CRON /var/log/syslog
|
||||
# Test cron environment: * * * * * env > /tmp/cron-env.txt
|
||||
# Manual full backup: /root/BC-bak/bc-backup.sh --mode full
|
||||
# Manual incremental: /root/BC-bak/bc-backup.sh --mode incremental
|
||||
# Manual cleanup: /root/BC-bak/bc-cleanup.sh
|
||||
# Check backup state: cat /root/BC-bak/last-run-state.json
|
||||
|
||||
Reference in New Issue
Block a user