diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a7c63a..894cc63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Removed emojis from UI elements - Improved navigation flow (edit page returns to detail view) - Enhanced cron job logging with threshold display +- Streamlined installation process + - Encryption key auto-generation during migration + - No separate script needed for encryption key setup ### Fixed - Notification channel type display error in domain view diff --git a/README.md b/README.md index dc2003f..8d7ca72 100644 --- a/README.md +++ b/README.md @@ -85,15 +85,12 @@ DB_PORT=3306 DB_DATABASE=domain_monitor DB_USERNAME=root DB_PASSWORD=your_password - -# Email (if using email notifications) -MAIL_HOST=smtp.mailtrap.io -MAIL_PORT=2525 -MAIL_USERNAME=your_username -MAIL_PASSWORD=your_password -MAIL_FROM_ADDRESS=noreply@domainmonitor.com ``` +**Note:** +- The encryption key (APP_ENCRYPTION_KEY) will be automatically generated during migration +- Application name, URL, timezone, email settings, and monitoring schedules are configured through the web interface in **Settings** (not .env) + ### 4. Create Database Create a MySQL database: @@ -108,9 +105,19 @@ CREATE DATABASE domain_monitor CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; php database/migrate.php ``` -**⚠️ IMPORTANT:** The migration will generate a random admin password and display it **only once**: +**⚠️ IMPORTANT:** The migration will: +1. **Generate an encryption key** (if not already set) and save it to `.env` +2. **Generate a random admin password** and display it **only once** +Example output: ``` +🔑 Generating encryption key... +✓ Encryption key generated and saved to .env + Key: base64_encoded_key_here + ⚠️ Keep this key secret and backup securely! + +... + 🔑 Admin credentials (SAVE THESE!): ═══════════════════════════════════════ Username: admin @@ -120,7 +127,9 @@ php database/migrate.php 💾 Save it to a secure password manager. ``` -**Save this password immediately** - you'll need it to access the dashboard! +**Save these immediately:** +- The encryption key is needed to decrypt sensitive data (backup securely!) +- The admin password is needed to access the dashboard ### 6. Import TLD Registry Data (Optional but Recommended) @@ -162,20 +171,27 @@ Then visit: `http://localhost:8000` ## 🔧 Configuration +### Application & Email Settings + +All application and email settings are now managed through the **Settings** page in the web interface: + +1. Navigate to **Settings** → **Application** tab + - Set application name, URL, and timezone + +2. Navigate to **Settings** → **Email** tab + - Configure SMTP server (host, port, encryption) + - Set authentication credentials + - Configure from address and name + +**Example Email Settings:** +- SMTP Host: `smtp.gmail.com` +- SMTP Port: `587` +- Encryption: `TLS` +- Username: `your-email@gmail.com` +- Password: `your-app-password` + ### Notification Channels -#### 📧 Email - -Configure SMTP settings in `.env`: - -```ini -MAIL_HOST=smtp.gmail.com -MAIL_PORT=587 -MAIL_USERNAME=your-email@gmail.com -MAIL_PASSWORD=your-app-password -MAIL_ENCRYPTION=tls -``` - #### ✈️ Telegram 1. Create a bot using [@BotFather](https://t.me/BotFather) @@ -274,23 +290,36 @@ By default, notifications are sent at these intervals before expiration: - 1 day (tomorrow!) - When expired (immediate alert) -Configure this in `.env`: +### Configure Settings -```ini -NOTIFICATION_DAYS_BEFORE=60,30,21,14,7,5,3,2,1 -``` +All system settings are managed through the **Settings** page (`/settings`) in your browser: -**Customization Examples:** -```ini -# Minimal alerts -NOTIFICATION_DAYS_BEFORE=30,7,1 +#### Application Settings +- **Application Name**: Customize the display name +- **Application URL**: Base URL for links in emails +- **Timezone**: Set your preferred timezone -# More frequent alerts -NOTIFICATION_DAYS_BEFORE=90,60,45,30,21,14,10,7,5,3,2,1 +#### Email Settings +- **SMTP Configuration**: Host, port, encryption +- **Authentication**: Username and password +- **From Address**: Email sender details -# Business-focused (weekday planning) -NOTIFICATION_DAYS_BEFORE=60,30,14,7,3,1 -``` +#### Monitoring Settings +- **Notification Schedule**: Choose from presets or create custom + - **Minimal**: 30, 7, 1 days + - **Standard**: 60, 30, 21, 14, 7, 5, 3, 2, 1 days + - **Frequent**: 90, 60, 45, 30, 21, 14, 10, 7, 5, 3, 2, 1 days + - **Business Focused**: 60, 30, 14, 7, 3, 1 days + - **Custom**: Enter your own comma-separated days + +- **Check Interval**: How often to check domains + - Every 6 hours + - Every 12 hours + - Daily (24 hours) + - Every 2 days + - Weekly + +All settings are stored in the database and can be updated at any time through the web interface. ## 📁 Project Structure diff --git a/database/migrate.php b/database/migrate.php index 8fee30c..41a8bf2 100644 --- a/database/migrate.php +++ b/database/migrate.php @@ -10,14 +10,50 @@ $dotenv = Dotenv::createImmutable(__DIR__ . '/..'); $dotenv->load(); try { - // Check if encryption key is set + // Check if encryption key is set, if not generate and save it if (empty($_ENV['APP_ENCRYPTION_KEY'])) { - echo "⚠️ WARNING: APP_ENCRYPTION_KEY is not set in .env\n"; - echo " This key is required to encrypt sensitive data (like SMTP passwords).\n\n"; - echo " Generate one using:\n"; - echo " php scripts/generate-encryption-key.php\n\n"; - echo " Then add it to your .env file and run migrations again.\n\n"; - exit(1); + echo "🔑 Generating encryption key...\n"; + + // Generate a secure 32-byte (256-bit) key + $encryptionKey = base64_encode(random_bytes(32)); + + // Path to .env file + $envFile = __DIR__ . '/../.env'; + + if (!file_exists($envFile)) { + echo "✗ Error: .env file not found. Please create it first.\n"; + exit(1); + } + + // Read current .env content + $envContent = file_get_contents($envFile); + + // Check if APP_ENCRYPTION_KEY line exists + if (strpos($envContent, 'APP_ENCRYPTION_KEY=') !== false) { + // Replace empty value with generated key + $envContent = preg_replace( + '/APP_ENCRYPTION_KEY=.*$/m', + "APP_ENCRYPTION_KEY=$encryptionKey", + $envContent + ); + } else { + // Append the key to the file + $envContent .= "\nAPP_ENCRYPTION_KEY=$encryptionKey\n"; + } + + // Write updated content back to .env + if (!file_put_contents($envFile, $envContent)) { + echo "✗ Error: Could not write to .env file.\n"; + exit(1); + } + + // Reload environment variables + $dotenv = Dotenv::createImmutable(__DIR__ . '/..'); + $dotenv->load(); + + echo "✓ Encryption key generated and saved to .env\n"; + echo " Key: $encryptionKey\n"; + echo " ⚠️ Keep this key secret and backup securely!\n\n"; } $host = $_ENV['DB_HOST']; @@ -47,6 +83,7 @@ try { __DIR__ . '/migrations/005_update_tld_import_logs.sql', __DIR__ . '/migrations/006_add_complete_workflow_import_type.sql', __DIR__ . '/migrations/007_add_app_and_email_settings.sql', + __DIR__ . '/migrations/008_remove_mail_driver.sql', ]; foreach ($migrationFiles as $migrationFile) { diff --git a/env.example.txt b/env.example.txt index 77c9b25..eec0fca 100644 --- a/env.example.txt +++ b/env.example.txt @@ -1,7 +1,7 @@ # Application APP_ENV=development -# Security - Generate encryption key using: php -r "echo base64_encode(random_bytes(32));" +# Security - Encryption key (auto-generated during migration if not set) # This key is used to encrypt sensitive data in the database (like SMTP passwords) # IMPORTANT: Never share this key or commit it to version control! APP_ENCRYPTION_KEY= diff --git a/logs/QUICK_START.md b/logs/QUICK_START.md deleted file mode 100644 index deb28f4..0000000 --- a/logs/QUICK_START.md +++ /dev/null @@ -1,60 +0,0 @@ -# Quick Start: Using the TLD Import Logs - -## 📁 Log File Location - -Your TLD import logs are here: -``` -logs/tld_import_2025-10-08.log (today's log) -``` - -## 🔍 Quick Commands - -### Watch import in real-time (Windows PowerShell): -```powershell -Get-Content logs\tld_import_2025-10-08.log -Wait -Tail 50 -``` - -### Check for errors: -```powershell -Select-String -Path logs\tld_import_*.log -Pattern "ERROR|CRITICAL" -``` - -### Find last processed TLD: -```powershell -Select-String -Path logs\tld_import_2025-10-08.log -Pattern "Processing TLD" | Select-Object -Last 1 -``` - -### Check completion status: -```powershell -Select-String -Path logs\tld_import_2025-10-08.log -Pattern "Batch statistics|complete" -``` - -## 🚨 If Import Fails - -1. **Check the log file** for the date of your import -2. **Look for the last line** - it should show which TLD was being processed -3. **Search for "ERROR" or "FAILED"** to find problems -4. **Check "last_processed_id"** to see where it stopped - -## 📊 What the Logs Show - -- ✅ Each TLD being processed with ID -- ✅ Time taken per TLD (in milliseconds) -- ✅ Success/failure status -- ✅ Data found (WHOIS server, registry URL, dates) -- ✅ Progress tracking (processed/remaining) -- ✅ Batch statistics (total time, average per TLD) -- ✅ Detailed error messages with context - -## 🔧 Next Steps After Your Import - -1. **Run your import again** - the logging is now active -2. **Watch the logs** while it runs -3. **If it times out** - check the log to see where -4. **Share the log file** if you need help debugging - -## 📖 Full Documentation - -See `logs/README.md` for complete documentation and troubleshooting guide. -See `LOGGING_SYSTEM.md` for technical details about the implementation. - diff --git a/logs/README.md b/logs/README.md deleted file mode 100644 index 10c9c77..0000000 --- a/logs/README.md +++ /dev/null @@ -1,193 +0,0 @@ -# Logs Directory - -This directory contains detailed application logs for debugging and monitoring purposes. - -## Log Files - -### TLD Import Logs (`tld_import_YYYY-MM-DD.log`) - -Comprehensive logs for TLD registry import operations including: - -- **Start/End Operations**: Marked with `=== START:` and `=== END:` separators -- **Batch Processing**: Each batch's start time, TLDs processed, and duration -- **Individual TLD Processing**: - - TLD name and ID - - Fetch time (in milliseconds) - - Database update time - - Data found (WHOIS server, registry URL, dates) - - Success/failure status - - Error messages with exception details -- **Progress Tracking**: - - Current step in workflow - - Last processed ID - - Completion percentage - - Remaining TLDs -- **Statistics**: - - Batch processing time - - Average time per TLD - - Success/failure counts - - Overall progress - -### Cron Job Logs (`cron.log`) - -Logs from automated domain checking cron jobs (created by `cron/check_domains.php`). - -## Log Levels - -- **DEBUG**: Detailed diagnostic information -- **INFO**: General informational messages (normal operations) -- **WARNING**: Warning messages (non-critical issues) -- **ERROR**: Error messages (failures that don't stop execution) -- **CRITICAL**: Critical errors (failures that stop execution) - -## Log Format - -``` -[YYYY-MM-DD HH:MM:SS] [LEVEL] Message | Context: {"key":"value"} -``` - -Example: -``` -[2025-10-08 10:04:23] [INFO] Processing TLD [1/50]: .aaa (ID: 1) -[2025-10-08 10:04:23] [INFO] TLD .aaa: SUCCESS - Found WHOIS server, registry URL | Context: {"fetch_time_ms":245.67,"update_time_ms":12.34,"data_fields":2} -``` - -## Troubleshooting TLD Import Issues - -### Issue: Import Stuck at Same Progress - -**Check the logs for:** -1. Look for "last_processed_id" - is it advancing? -2. Check for FAILED messages - which TLDs are failing? -3. Look at "batch_time_seconds" - is it taking too long? - -**Example log analysis:** -```bash -# Find failed TLDs -grep "FAILED" logs/tld_import_2025-10-08.log - -# Check last processed IDs -grep "Updating last processed ID" logs/tld_import_2025-10-08.log - -# Find slow TLDs (over 5 seconds) -grep "fetch_time_ms" logs/tld_import_2025-10-08.log | grep -E '"fetch_time_ms":[5-9][0-9]{3}|[0-9]{5,}' -``` - -### Issue: FastCGI Timeout - -**Symptoms in logs:** -- Last log entry shows a TLD being processed -- No "Batch statistics" or "END:" marker -- Apache/Nginx error log shows "Connection reset by peer" - -**Solutions:** -1. Reduce batch size in `TldRegistryService.php`: - ```php - $tldsNeedingWhois = $this->getTldsNeedingWhoisData(25, $lastProcessedId); // Reduced from 50 - ``` - -2. Increase PHP/FastCGI timeout: - ```ini - ; php.ini - max_execution_time = 300 - ``` - -### Issue: Repeated Failures on Same TLDs - -**Check logs for:** -```bash -# Find TLDs that consistently fail -grep "FAILED" logs/tld_import_*.log | sort | uniq -c | sort -rn -``` - -**Common causes:** -- TLD doesn't have IANA RDAP/WHOIS data -- Network timeout to IANA servers -- Invalid TLD format - -## Analyzing Performance - -### Average Processing Time -```bash -# Get average fetch time per TLD -grep "fetch_time_ms" logs/tld_import_2025-10-08.log | grep -oP '"fetch_time_ms":\K[0-9.]+' | awk '{sum+=$1; n++} END {print "Average: " sum/n " ms"}' -``` - -### Slowest TLDs -```bash -# Find slowest 10 TLDs -grep "fetch_time_ms" logs/tld_import_2025-10-08.log | grep -oP 'TLD \.\w+.*fetch_time_ms":\K[0-9.]+' | sort -rn | head -10 -``` - -### Batch Performance -```bash -# Show all batch statistics -grep "Batch statistics" logs/tld_import_2025-10-08.log -``` - -## Log Retention - -Logs are organized by date and are not automatically deleted. You may want to: - -1. **Manual cleanup**: Delete old logs periodically - ```bash - find logs/ -name "*.log" -mtime +30 -delete # Delete logs older than 30 days - ``` - -2. **Set up logrotate** (Linux): - ``` - /path/to/Domain Monitor/logs/*.log { - daily - rotate 30 - compress - missingok - notifempty - } - ``` - -## Accessing Logs - -### Via Command Line -```bash -# View latest TLD import log -tail -f logs/tld_import_$(date +%Y-%m-%d).log - -# View last 100 lines -tail -100 logs/tld_import_2025-10-08.log - -# Search for errors -grep ERROR logs/tld_import_2025-10-08.log - -# Watch for specific TLD -grep ".example" logs/tld_import_2025-10-08.log -``` - -### Via PHP -The `Logger` class provides a `tail()` method: -```php -$logger = new Logger('tld_import'); -$lastLines = $logger->tail(100); // Get last 100 lines -``` - -## Important Notes - -- **Log files grow large**: A complete TLD import (~1500 TLDs) generates ~2-5 MB of logs -- **Context data is JSON**: Can be parsed programmatically for analysis -- **Timestamps are in server timezone**: Check your PHP timezone setting -- **Logs are NOT web-accessible**: The `.htaccess` in the project root blocks access to this directory - -## Related Files - -- `app/Services/Logger.php` - Logger implementation -- `app/Services/TldRegistryService.php` - Uses logger for TLD import operations -- `cron/check_domains.php` - Uses file_put_contents for cron.log - -## Support - -If you're experiencing import issues: -1. Check the relevant log file for error messages -2. Look for patterns in failed TLDs -3. Check Apache/Nginx error logs for timeout issues -4. Verify network connectivity to IANA servers -5. Consider reducing batch size for slower servers - diff --git a/scripts/generate-encryption-key.php b/scripts/generate-encryption-key.php deleted file mode 100644 index cffa8e0..0000000 --- a/scripts/generate-encryption-key.php +++ /dev/null @@ -1,36 +0,0 @@ -#!/usr/bin/env php -