Fixed Encryption key generation and updated Readme

2025-10-08 19:08:27 +03:00
parent 146df224bd
commit 8e5ff9ab39
7 changed files with 111 additions and 331 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -31,6 +31,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
  - Removed emojis from UI elements
 - Improved navigation flow (edit page returns to detail view)
 - Enhanced cron job logging with threshold display
+- Streamlined installation process
+  - Encryption key auto-generation during migration
+  - No separate script needed for encryption key setup

 ### Fixed
 - Notification channel type display error in domain view
--- a/README.md
+++ b/README.md
@@ -85,15 +85,12 @@ DB_PORT=3306
 DB_DATABASE=domain_monitor
 DB_USERNAME=root
 DB_PASSWORD=your_password
-
-# Email (if using email notifications)
-MAIL_HOST=smtp.mailtrap.io
-MAIL_PORT=2525
-MAIL_USERNAME=your_username
-MAIL_PASSWORD=your_password
-MAIL_FROM_ADDRESS=noreply@domainmonitor.com
 ```

+**Note:** 
+- The encryption key (APP_ENCRYPTION_KEY) will be automatically generated during migration
+- Application name, URL, timezone, email settings, and monitoring schedules are configured through the web interface in **Settings** (not .env)
+
 ### 4. Create Database

 Create a MySQL database:
@@ -108,9 +105,19 @@ CREATE DATABASE domain_monitor CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 php database/migrate.php
 ```

-**⚠️ IMPORTANT:** The migration will generate a random admin password and display it **only once**:
+**⚠️ IMPORTANT:** The migration will:
+1. **Generate an encryption key** (if not already set) and save it to `.env`
+2. **Generate a random admin password** and display it **only once**

+Example output:
 ```
+🔑 Generating encryption key...
+✓ Encryption key generated and saved to .env
+   Key: base64_encoded_key_here
+   ⚠️  Keep this key secret and backup securely!
+
+...
+
 🔑 Admin credentials (SAVE THESE!):
   ═══════════════════════════════════════
   Username: admin
@@ -120,7 +127,9 @@ php database/migrate.php
   💾 Save it to a secure password manager.
 ```

-**Save this password immediately** - you'll need it to access the dashboard!
+**Save these immediately:**
+- The encryption key is needed to decrypt sensitive data (backup securely!)
+- The admin password is needed to access the dashboard

 ### 6. Import TLD Registry Data (Optional but Recommended)

@@ -162,20 +171,27 @@ Then visit: `http://localhost:8000`

 ## 🔧 Configuration

+### Application & Email Settings
+
+All application and email settings are now managed through the **Settings** page in the web interface:
+
+1. Navigate to **Settings** → **Application** tab
+   - Set application name, URL, and timezone
+
+2. Navigate to **Settings** → **Email** tab
+   - Configure SMTP server (host, port, encryption)
+   - Set authentication credentials
+   - Configure from address and name
+
+**Example Email Settings:**
+- SMTP Host: `smtp.gmail.com`
+- SMTP Port: `587`
+- Encryption: `TLS`
+- Username: `your-email@gmail.com`
+- Password: `your-app-password`
+
 ### Notification Channels

-#### 📧 Email
-
-Configure SMTP settings in `.env`:
-
-```ini
-MAIL_HOST=smtp.gmail.com
-MAIL_PORT=587
-MAIL_USERNAME=your-email@gmail.com
-MAIL_PASSWORD=your-app-password
-MAIL_ENCRYPTION=tls
-```
-
 #### ✈️ Telegram

 1. Create a bot using [@BotFather](https://t.me/BotFather)
@@ -274,23 +290,36 @@ By default, notifications are sent at these intervals before expiration:
 - 1 day (tomorrow!)
 - When expired (immediate alert)

-Configure this in `.env`:
+### Configure Settings

-```ini
-NOTIFICATION_DAYS_BEFORE=60,30,21,14,7,5,3,2,1
-```
+All system settings are managed through the **Settings** page (`/settings`) in your browser:

-**Customization Examples:**
-```ini
-# Minimal alerts
-NOTIFICATION_DAYS_BEFORE=30,7,1
+#### Application Settings
+- **Application Name**: Customize the display name
+- **Application URL**: Base URL for links in emails
+- **Timezone**: Set your preferred timezone

-# More frequent alerts
-NOTIFICATION_DAYS_BEFORE=90,60,45,30,21,14,10,7,5,3,2,1
+#### Email Settings
+- **SMTP Configuration**: Host, port, encryption
+- **Authentication**: Username and password
+- **From Address**: Email sender details

-# Business-focused (weekday planning)
-NOTIFICATION_DAYS_BEFORE=60,30,14,7,3,1
-```
+#### Monitoring Settings
+- **Notification Schedule**: Choose from presets or create custom
+  - **Minimal**: 30, 7, 1 days
+  - **Standard**: 60, 30, 21, 14, 7, 5, 3, 2, 1 days
+  - **Frequent**: 90, 60, 45, 30, 21, 14, 10, 7, 5, 3, 2, 1 days
+  - **Business Focused**: 60, 30, 14, 7, 3, 1 days
+  - **Custom**: Enter your own comma-separated days
+
+- **Check Interval**: How often to check domains
+  - Every 6 hours
+  - Every 12 hours  
+  - Daily (24 hours)
+  - Every 2 days
+  - Weekly
+
+All settings are stored in the database and can be updated at any time through the web interface.

 ## 📁 Project Structure

--- a/database/migrate.php
+++ b/database/migrate.php
@@ -10,16 +10,52 @@ $dotenv = Dotenv::createImmutable(__DIR__ . '/..');
 $dotenv->load();

 try {
-    // Check if encryption key is set
+    // Check if encryption key is set, if not generate and save it
    if (empty($_ENV['APP_ENCRYPTION_KEY'])) {
-        echo "⚠️  WARNING: APP_ENCRYPTION_KEY is not set in .env\n";
-        echo "   This key is required to encrypt sensitive data (like SMTP passwords).\n\n";
-        echo "   Generate one using:\n";
-        echo "   php scripts/generate-encryption-key.php\n\n";
-        echo "   Then add it to your .env file and run migrations again.\n\n";
+        echo "🔑 Generating encryption key...\n";
+        
+        // Generate a secure 32-byte (256-bit) key
+        $encryptionKey = base64_encode(random_bytes(32));
+        
+        // Path to .env file
+        $envFile = __DIR__ . '/../.env';
+        
+        if (!file_exists($envFile)) {
+            echo "✗ Error: .env file not found. Please create it first.\n";
            exit(1);
        }
        
+        // Read current .env content
+        $envContent = file_get_contents($envFile);
+        
+        // Check if APP_ENCRYPTION_KEY line exists
+        if (strpos($envContent, 'APP_ENCRYPTION_KEY=') !== false) {
+            // Replace empty value with generated key
+            $envContent = preg_replace(
+                '/APP_ENCRYPTION_KEY=.*$/m',
+                "APP_ENCRYPTION_KEY=$encryptionKey",
+                $envContent
+            );
+        } else {
+            // Append the key to the file
+            $envContent .= "\nAPP_ENCRYPTION_KEY=$encryptionKey\n";
+        }
+        
+        // Write updated content back to .env
+        if (!file_put_contents($envFile, $envContent)) {
+            echo "✗ Error: Could not write to .env file.\n";
+            exit(1);
+        }
+        
+        // Reload environment variables
+        $dotenv = Dotenv::createImmutable(__DIR__ . '/..');
+        $dotenv->load();
+        
+        echo "✓ Encryption key generated and saved to .env\n";
+        echo "   Key: $encryptionKey\n";
+        echo "   ⚠️  Keep this key secret and backup securely!\n\n";
+    }
+
    $host = $_ENV['DB_HOST'];
    $port = $_ENV['DB_PORT'];
    $database = $_ENV['DB_DATABASE'];
@@ -47,6 +83,7 @@ try {
        __DIR__ . '/migrations/005_update_tld_import_logs.sql',
        __DIR__ . '/migrations/006_add_complete_workflow_import_type.sql',
        __DIR__ . '/migrations/007_add_app_and_email_settings.sql',
+        __DIR__ . '/migrations/008_remove_mail_driver.sql',
    ];

    foreach ($migrationFiles as $migrationFile) {
--- a/env.example.txt
+++ b/env.example.txt
@@ -1,7 +1,7 @@
 # Application
 APP_ENV=development

-# Security - Generate encryption key using: php -r "echo base64_encode(random_bytes(32));"
+# Security - Encryption key (auto-generated during migration if not set)
 # This key is used to encrypt sensitive data in the database (like SMTP passwords)
 # IMPORTANT: Never share this key or commit it to version control!
 APP_ENCRYPTION_KEY=
--- a/logs/QUICK_START.md
+++ b/logs/QUICK_START.md
@@ -1,60 +0,0 @@
-# Quick Start: Using the TLD Import Logs
-
-## 📁 Log File Location
-
-Your TLD import logs are here:
-```
-logs/tld_import_2025-10-08.log  (today's log)
-```
-
-## 🔍 Quick Commands
-
-### Watch import in real-time (Windows PowerShell):
-```powershell
-Get-Content logs\tld_import_2025-10-08.log -Wait -Tail 50
-```
-
-### Check for errors:
-```powershell
-Select-String -Path logs\tld_import_*.log -Pattern "ERROR|CRITICAL"
-```
-
-### Find last processed TLD:
-```powershell
-Select-String -Path logs\tld_import_2025-10-08.log -Pattern "Processing TLD" | Select-Object -Last 1
-```
-
-### Check completion status:
-```powershell
-Select-String -Path logs\tld_import_2025-10-08.log -Pattern "Batch statistics|complete"
-```
-
-## 🚨 If Import Fails
-
-1. **Check the log file** for the date of your import
-2. **Look for the last line** - it should show which TLD was being processed
-3. **Search for "ERROR" or "FAILED"** to find problems
-4. **Check "last_processed_id"** to see where it stopped
-
-## 📊 What the Logs Show
-
- ✅ Each TLD being processed with ID
- ✅ Time taken per TLD (in milliseconds)
- ✅ Success/failure status
- ✅ Data found (WHOIS server, registry URL, dates)
- ✅ Progress tracking (processed/remaining)
- ✅ Batch statistics (total time, average per TLD)
- ✅ Detailed error messages with context
-
-## 🔧 Next Steps After Your Import
-
-1. **Run your import again** - the logging is now active
-2. **Watch the logs** while it runs
-3. **If it times out** - check the log to see where
-4. **Share the log file** if you need help debugging
-
-## 📖 Full Documentation
-
-See `logs/README.md` for complete documentation and troubleshooting guide.
-See `LOGGING_SYSTEM.md` for technical details about the implementation.
-
--- a/logs/README.md
+++ b/logs/README.md
@@ -1,193 +0,0 @@
-# Logs Directory
-
-This directory contains detailed application logs for debugging and monitoring purposes.
-
-## Log Files
-
-### TLD Import Logs (`tld_import_YYYY-MM-DD.log`)
-
-Comprehensive logs for TLD registry import operations including:
-
- **Start/End Operations**: Marked with `=== START:` and `=== END:` separators
- **Batch Processing**: Each batch's start time, TLDs processed, and duration
- **Individual TLD Processing**: 
-  - TLD name and ID
-  - Fetch time (in milliseconds)
-  - Database update time
-  - Data found (WHOIS server, registry URL, dates)
-  - Success/failure status
-  - Error messages with exception details
- **Progress Tracking**:
-  - Current step in workflow
-  - Last processed ID
-  - Completion percentage
-  - Remaining TLDs
- **Statistics**:
-  - Batch processing time
-  - Average time per TLD
-  - Success/failure counts
-  - Overall progress
-
-### Cron Job Logs (`cron.log`)
-
-Logs from automated domain checking cron jobs (created by `cron/check_domains.php`).
-
-## Log Levels
-
- **DEBUG**: Detailed diagnostic information
- **INFO**: General informational messages (normal operations)
- **WARNING**: Warning messages (non-critical issues)
- **ERROR**: Error messages (failures that don't stop execution)
- **CRITICAL**: Critical errors (failures that stop execution)
-
-## Log Format
-
-```
-[YYYY-MM-DD HH:MM:SS] [LEVEL] Message | Context: {"key":"value"}
-```
-
-Example:
-```
-[2025-10-08 10:04:23] [INFO] Processing TLD [1/50]: .aaa (ID: 1)
-[2025-10-08 10:04:23] [INFO] TLD .aaa: SUCCESS - Found WHOIS server, registry URL | Context: {"fetch_time_ms":245.67,"update_time_ms":12.34,"data_fields":2}
-```
-
-## Troubleshooting TLD Import Issues
-
-### Issue: Import Stuck at Same Progress
-
-**Check the logs for:**
-1. Look for "last_processed_id" - is it advancing?
-2. Check for FAILED messages - which TLDs are failing?
-3. Look at "batch_time_seconds" - is it taking too long?
-
-**Example log analysis:**
-```bash
-# Find failed TLDs
-grep "FAILED" logs/tld_import_2025-10-08.log
-
-# Check last processed IDs
-grep "Updating last processed ID" logs/tld_import_2025-10-08.log
-
-# Find slow TLDs (over 5 seconds)
-grep "fetch_time_ms" logs/tld_import_2025-10-08.log | grep -E '"fetch_time_ms":[5-9][0-9]{3}|[0-9]{5,}'
-```
-
-### Issue: FastCGI Timeout
-
-**Symptoms in logs:**
- Last log entry shows a TLD being processed
- No "Batch statistics" or "END:" marker
- Apache/Nginx error log shows "Connection reset by peer"
-
-**Solutions:**
-1. Reduce batch size in `TldRegistryService.php`:
-   ```php
-   $tldsNeedingWhois = $this->getTldsNeedingWhoisData(25, $lastProcessedId); // Reduced from 50
-   ```
-
-2. Increase PHP/FastCGI timeout:
-   ```ini
-   ; php.ini
-   max_execution_time = 300
-   ```
-
-### Issue: Repeated Failures on Same TLDs
-
-**Check logs for:**
-```bash
-# Find TLDs that consistently fail
-grep "FAILED" logs/tld_import_*.log | sort | uniq -c | sort -rn
-```
-
-**Common causes:**
- TLD doesn't have IANA RDAP/WHOIS data
- Network timeout to IANA servers
- Invalid TLD format
-
-## Analyzing Performance
-
-### Average Processing Time
-```bash
-# Get average fetch time per TLD
-grep "fetch_time_ms" logs/tld_import_2025-10-08.log | grep -oP '"fetch_time_ms":\K[0-9.]+' | awk '{sum+=$1; n++} END {print "Average: " sum/n " ms"}'
-```
-
-### Slowest TLDs
-```bash
-# Find slowest 10 TLDs
-grep "fetch_time_ms" logs/tld_import_2025-10-08.log | grep -oP 'TLD \.\w+.*fetch_time_ms":\K[0-9.]+' | sort -rn | head -10
-```
-
-### Batch Performance
-```bash
-# Show all batch statistics
-grep "Batch statistics" logs/tld_import_2025-10-08.log
-```
-
-## Log Retention
-
-Logs are organized by date and are not automatically deleted. You may want to:
-
-1. **Manual cleanup**: Delete old logs periodically
-   ```bash
-   find logs/ -name "*.log" -mtime +30 -delete  # Delete logs older than 30 days
-   ```
-
-2. **Set up logrotate** (Linux):
-   ```
-   /path/to/Domain Monitor/logs/*.log {
-       daily
-       rotate 30
-       compress
-       missingok
-       notifempty
-   }
-   ```
-
-## Accessing Logs
-
-### Via Command Line
-```bash
-# View latest TLD import log
-tail -f logs/tld_import_$(date +%Y-%m-%d).log
-
-# View last 100 lines
-tail -100 logs/tld_import_2025-10-08.log
-
-# Search for errors
-grep ERROR logs/tld_import_2025-10-08.log
-
-# Watch for specific TLD
-grep ".example" logs/tld_import_2025-10-08.log
-```
-
-### Via PHP
-The `Logger` class provides a `tail()` method:
-```php
-$logger = new Logger('tld_import');
-$lastLines = $logger->tail(100); // Get last 100 lines
-```
-
-## Important Notes
-
- **Log files grow large**: A complete TLD import (~1500 TLDs) generates ~2-5 MB of logs
- **Context data is JSON**: Can be parsed programmatically for analysis
- **Timestamps are in server timezone**: Check your PHP timezone setting
- **Logs are NOT web-accessible**: The `.htaccess` in the project root blocks access to this directory
-
-## Related Files
-
- `app/Services/Logger.php` - Logger implementation
- `app/Services/TldRegistryService.php` - Uses logger for TLD import operations
- `cron/check_domains.php` - Uses file_put_contents for cron.log
-
-## Support
-
-If you're experiencing import issues:
-1. Check the relevant log file for error messages
-2. Look for patterns in failed TLDs
-3. Check Apache/Nginx error logs for timeout issues
-4. Verify network connectivity to IANA servers
-5. Consider reducing batch size for slower servers
-
--- a/scripts/generate-encryption-key.php
+++ b/scripts/generate-encryption-key.php
@@ -1,36 +0,0 @@
-#!/usr/bin/env php
-<?php
-
-/**
- * Generate Encryption Key
- * 
- * This script generates a secure encryption key for the application.
- * The key is used to encrypt sensitive data in the database (like SMTP passwords).
- * 
- * Usage: php scripts/generate-encryption-key.php
- */
-
-echo "===========================================\n";
-echo "  Domain Monitor - Encryption Key Generator\n";
-echo "===========================================\n\n";
-
-// Generate a secure 32-byte (256-bit) key
-$key = random_bytes(32);
-$encodedKey = base64_encode($key);
-
-echo "Your encryption key has been generated:\n\n";
-echo "\033[1;32m$encodedKey\033[0m\n\n";
-
-echo "Add this to your .env file:\n\n";
-echo "APP_ENCRYPTION_KEY=$encodedKey\n\n";
-
-echo "⚠️  IMPORTANT SECURITY NOTES:\n";
-echo "-------------------------------------------\n";
-echo "1. Keep this key SECRET - never share it\n";
-echo "2. Never commit this key to version control\n";
-echo "3. If you lose this key, encrypted data cannot be recovered\n";
-echo "4. Don't change this key after encrypting data\n";
-echo "5. Store a backup of this key in a secure location\n\n";
-
-echo "✅ Done! Copy the key to your .env file.\n";
-