Malin/domnitor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fixed Encryption key generation and updated Readme

Browse Source
This commit is contained in:
Hosteroid
2025-10-08 19:08:27 +03:00
parent 146df224bd
commit 8e5ff9ab39
7 changed files with 111 additions and 331 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@@ -31,6 +31,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Removed emojis from UI elements - Removed emojis from UI elements
- Improved navigation flow (edit page returns to detail view) - Improved navigation flow (edit page returns to detail view)
- Enhanced cron job logging with threshold display - Enhanced cron job logging with threshold display
- Streamlined installation process
- Encryption key auto-generation during migration
- No separate script needed for encryption key setup
### Fixed ### Fixed
- Notification channel type display error in domain view - Notification channel type display error in domain view

97
README.md
View File

@@ -85,15 +85,12 @@ DB_PORT=3306
DB_DATABASE=domain_monitor DB_DATABASE=domain_monitor
DB_USERNAME=root DB_USERNAME=root
DB_PASSWORD=your_password DB_PASSWORD=your_password
# Email (if using email notifications)
MAIL_HOST=smtp.mailtrap.io
MAIL_PORT=2525
MAIL_USERNAME=your_username
MAIL_PASSWORD=your_password
MAIL_FROM_ADDRESS=noreply@domainmonitor.com
``` ```
**Note:**
- The encryption key (APP_ENCRYPTION_KEY) will be automatically generated during migration
- Application name, URL, timezone, email settings, and monitoring schedules are configured through the web interface in **Settings** (not .env)
### 4. Create Database ### 4. Create Database
Create a MySQL database: Create a MySQL database:
@@ -108,9 +105,19 @@ CREATE DATABASE domain_monitor CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
php database/migrate.php php database/migrate.php
``` ```
**⚠️ IMPORTANT:** The migration will generate a random admin password and display it **only once**: **⚠️ IMPORTANT:** The migration will:
1. **Generate an encryption key** (if not already set) and save it to `.env`
2. **Generate a random admin password** and display it **only once**
Example output:
``` ```
🔑 Generating encryption key...
✓ Encryption key generated and saved to .env
Key: base64_encoded_key_here
⚠️ Keep this key secret and backup securely!
...
🔑 Admin credentials (SAVE THESE!): 🔑 Admin credentials (SAVE THESE!):
═══════════════════════════════════════ ═══════════════════════════════════════
Username: admin Username: admin
@@ -120,7 +127,9 @@ php database/migrate.php
💾 Save it to a secure password manager. 💾 Save it to a secure password manager.
``` ```
**Save this password immediately** - you'll need it to access the dashboard! **Save these immediately:**
- The encryption key is needed to decrypt sensitive data (backup securely!)
- The admin password is needed to access the dashboard
### 6. Import TLD Registry Data (Optional but Recommended) ### 6. Import TLD Registry Data (Optional but Recommended)
@@ -162,20 +171,27 @@ Then visit: `http://localhost:8000`
## 🔧 Configuration ## 🔧 Configuration
### Application & Email Settings
All application and email settings are now managed through the **Settings** page in the web interface:
1. Navigate to **Settings** → **Application** tab
- Set application name, URL, and timezone
2. Navigate to **Settings** → **Email** tab
- Configure SMTP server (host, port, encryption)
- Set authentication credentials
- Configure from address and name
**Example Email Settings:**
- SMTP Host: `smtp.gmail.com`
- SMTP Port: `587`
- Encryption: `TLS`
- Username: `your-email@gmail.com`
- Password: `your-app-password`
### Notification Channels ### Notification Channels
#### 📧 Email
Configure SMTP settings in `.env`:
```ini
MAIL_HOST=smtp.gmail.com
MAIL_PORT=587
MAIL_USERNAME=your-email@gmail.com
MAIL_PASSWORD=your-app-password
MAIL_ENCRYPTION=tls
```
#### ✈️ Telegram #### ✈️ Telegram
1. Create a bot using [@BotFather](https://t.me/BotFather) 1. Create a bot using [@BotFather](https://t.me/BotFather)
@@ -274,23 +290,36 @@ By default, notifications are sent at these intervals before expiration:
- 1 day (tomorrow!) - 1 day (tomorrow!)
- When expired (immediate alert) - When expired (immediate alert)
Configure this in `.env`: ### Configure Settings
```ini All system settings are managed through the **Settings** page (`/settings`) in your browser:
NOTIFICATION_DAYS_BEFORE=60,30,21,14,7,5,3,2,1
```
**Customization Examples:** #### Application Settings
```ini - **Application Name**: Customize the display name
# Minimal alerts - **Application URL**: Base URL for links in emails
NOTIFICATION_DAYS_BEFORE=30,7,1 - **Timezone**: Set your preferred timezone
# More frequent alerts #### Email Settings
NOTIFICATION_DAYS_BEFORE=90,60,45,30,21,14,10,7,5,3,2,1 - **SMTP Configuration**: Host, port, encryption
- **Authentication**: Username and password
- **From Address**: Email sender details
# Business-focused (weekday planning) #### Monitoring Settings
NOTIFICATION_DAYS_BEFORE=60,30,14,7,3,1 - **Notification Schedule**: Choose from presets or create custom
``` - **Minimal**: 30, 7, 1 days
- **Standard**: 60, 30, 21, 14, 7, 5, 3, 2, 1 days
- **Frequent**: 90, 60, 45, 30, 21, 14, 10, 7, 5, 3, 2, 1 days
- **Business Focused**: 60, 30, 14, 7, 3, 1 days
- **Custom**: Enter your own comma-separated days
- **Check Interval**: How often to check domains
- Every 6 hours
- Every 12 hours
- Daily (24 hours)
- Every 2 days
- Weekly
All settings are stored in the database and can be updated at any time through the web interface.
## 📁 Project Structure ## 📁 Project Structure

51
database/migrate.php
View File

@@ -10,14 +10,50 @@ $dotenv = Dotenv::createImmutable(__DIR__ . '/..');
$dotenv->load(); $dotenv->load();
try { try {
// Check if encryption key is set // Check if encryption key is set, if not generate and save it
if (empty($_ENV['APP_ENCRYPTION_KEY'])) { if (empty($_ENV['APP_ENCRYPTION_KEY'])) {
echo "⚠️ WARNING: APP_ENCRYPTION_KEY is not set in .env\n"; echo "🔑 Generating encryption key...\n";
echo " This key is required to encrypt sensitive data (like SMTP passwords).\n\n";
echo " Generate one using:\n"; // Generate a secure 32-byte (256-bit) key
echo " php scripts/generate-encryption-key.php\n\n"; $encryptionKey = base64_encode(random_bytes(32));
echo " Then add it to your .env file and run migrations again.\n\n";
exit(1); // Path to .env file
$envFile = __DIR__ . '/../.env';
if (!file_exists($envFile)) {
echo "✗ Error: .env file not found. Please create it first.\n";
exit(1);
}
// Read current .env content
$envContent = file_get_contents($envFile);
// Check if APP_ENCRYPTION_KEY line exists
if (strpos($envContent, 'APP_ENCRYPTION_KEY=') !== false) {
// Replace empty value with generated key
$envContent = preg_replace(
'/APP_ENCRYPTION_KEY=.*$/m',
"APP_ENCRYPTION_KEY=$encryptionKey",
$envContent
);
} else {
// Append the key to the file
$envContent .= "\nAPP_ENCRYPTION_KEY=$encryptionKey\n";
}
// Write updated content back to .env
if (!file_put_contents($envFile, $envContent)) {
echo "✗ Error: Could not write to .env file.\n";
exit(1);
}
// Reload environment variables
$dotenv = Dotenv::createImmutable(__DIR__ . '/..');
$dotenv->load();
echo "✓ Encryption key generated and saved to .env\n";
echo " Key: $encryptionKey\n";
echo " ⚠️ Keep this key secret and backup securely!\n\n";
} }
$host = $_ENV['DB_HOST']; $host = $_ENV['DB_HOST'];
@@ -47,6 +83,7 @@ try {
__DIR__ . '/migrations/005_update_tld_import_logs.sql', __DIR__ . '/migrations/005_update_tld_import_logs.sql',
__DIR__ . '/migrations/006_add_complete_workflow_import_type.sql', __DIR__ . '/migrations/006_add_complete_workflow_import_type.sql',
__DIR__ . '/migrations/007_add_app_and_email_settings.sql', __DIR__ . '/migrations/007_add_app_and_email_settings.sql',
__DIR__ . '/migrations/008_remove_mail_driver.sql',
]; ];
foreach ($migrationFiles as $migrationFile) { foreach ($migrationFiles as $migrationFile) {

2
env.example.txt
View File

@@ -1,7 +1,7 @@
# Application # Application
APP_ENV=development APP_ENV=development
# Security - Generate encryption key using: php -r "echo base64_encode(random_bytes(32));" # Security - Encryption key (auto-generated during migration if not set)
# This key is used to encrypt sensitive data in the database (like SMTP passwords) # This key is used to encrypt sensitive data in the database (like SMTP passwords)
# IMPORTANT: Never share this key or commit it to version control! # IMPORTANT: Never share this key or commit it to version control!
APP_ENCRYPTION_KEY= APP_ENCRYPTION_KEY=

60
logs/QUICK_START.md
View File

@@ -1,60 +0,0 @@
# Quick Start: Using the TLD Import Logs
## 📁 Log File Location
Your TLD import logs are here:
```
logs/tld_import_2025-10-08.log (today's log)
```
## 🔍 Quick Commands
### Watch import in real-time (Windows PowerShell):
```powershell
Get-Content logs\tld_import_2025-10-08.log -Wait -Tail 50
```
### Check for errors:
```powershell
Select-String -Path logs\tld_import_*.log -Pattern "ERROR|CRITICAL"
```
### Find last processed TLD:
```powershell
Select-String -Path logs\tld_import_2025-10-08.log -Pattern "Processing TLD" | Select-Object -Last 1
```
### Check completion status:
```powershell
Select-String -Path logs\tld_import_2025-10-08.log -Pattern "Batch statistics|complete"
```
## 🚨 If Import Fails
1. **Check the log file** for the date of your import
2. **Look for the last line** - it should show which TLD was being processed
3. **Search for "ERROR" or "FAILED"** to find problems
4. **Check "last_processed_id"** to see where it stopped
## 📊 What the Logs Show
- ✅ Each TLD being processed with ID
- ✅ Time taken per TLD (in milliseconds)
- ✅ Success/failure status
- ✅ Data found (WHOIS server, registry URL, dates)
- ✅ Progress tracking (processed/remaining)
- ✅ Batch statistics (total time, average per TLD)
- ✅ Detailed error messages with context
## 🔧 Next Steps After Your Import
1. **Run your import again** - the logging is now active
2. **Watch the logs** while it runs
3. **If it times out** - check the log to see where
4. **Share the log file** if you need help debugging
## 📖 Full Documentation
See `logs/README.md` for complete documentation and troubleshooting guide.
See `LOGGING_SYSTEM.md` for technical details about the implementation.

193
logs/README.md
View File

@@ -1,193 +0,0 @@
# Logs Directory
This directory contains detailed application logs for debugging and monitoring purposes.
## Log Files
### TLD Import Logs (`tld_import_YYYY-MM-DD.log`)
Comprehensive logs for TLD registry import operations including:
- **Start/End Operations**: Marked with `=== START:` and `=== END:` separators
- **Batch Processing**: Each batch's start time, TLDs processed, and duration
- **Individual TLD Processing**:
- TLD name and ID
- Fetch time (in milliseconds)
- Database update time
- Data found (WHOIS server, registry URL, dates)
- Success/failure status
- Error messages with exception details
- **Progress Tracking**:
- Current step in workflow
- Last processed ID
- Completion percentage
- Remaining TLDs
- **Statistics**:
- Batch processing time
- Average time per TLD
- Success/failure counts
- Overall progress
### Cron Job Logs (`cron.log`)
Logs from automated domain checking cron jobs (created by `cron/check_domains.php`).
## Log Levels
- **DEBUG**: Detailed diagnostic information
- **INFO**: General informational messages (normal operations)
- **WARNING**: Warning messages (non-critical issues)
- **ERROR**: Error messages (failures that don't stop execution)
- **CRITICAL**: Critical errors (failures that stop execution)
## Log Format
```
[YYYY-MM-DD HH:MM:SS] [LEVEL] Message | Context: {"key":"value"}
```
Example:
```
[2025-10-08 10:04:23] [INFO] Processing TLD [1/50]: .aaa (ID: 1)
[2025-10-08 10:04:23] [INFO] TLD .aaa: SUCCESS - Found WHOIS server, registry URL | Context: {"fetch_time_ms":245.67,"update_time_ms":12.34,"data_fields":2}
```
## Troubleshooting TLD Import Issues
### Issue: Import Stuck at Same Progress
**Check the logs for:**
1. Look for "last_processed_id" - is it advancing?
2. Check for FAILED messages - which TLDs are failing?
3. Look at "batch_time_seconds" - is it taking too long?
**Example log analysis:**
```bash
# Find failed TLDs
grep "FAILED" logs/tld_import_2025-10-08.log
# Check last processed IDs
grep "Updating last processed ID" logs/tld_import_2025-10-08.log
# Find slow TLDs (over 5 seconds)
grep "fetch_time_ms" logs/tld_import_2025-10-08.log | grep -E '"fetch_time_ms":[5-9][0-9]{3}|[0-9]{5,}'
```
### Issue: FastCGI Timeout
**Symptoms in logs:**
- Last log entry shows a TLD being processed
- No "Batch statistics" or "END:" marker
- Apache/Nginx error log shows "Connection reset by peer"
**Solutions:**
1. Reduce batch size in `TldRegistryService.php`:
```php
$tldsNeedingWhois = $this->getTldsNeedingWhoisData(25, $lastProcessedId); // Reduced from 50
```
2. Increase PHP/FastCGI timeout:
```ini
; php.ini
max_execution_time = 300
```
### Issue: Repeated Failures on Same TLDs
**Check logs for:**
```bash
# Find TLDs that consistently fail
grep "FAILED" logs/tld_import_*.log | sort | uniq -c | sort -rn
```
**Common causes:**
- TLD doesn't have IANA RDAP/WHOIS data
- Network timeout to IANA servers
- Invalid TLD format
## Analyzing Performance
### Average Processing Time
```bash
# Get average fetch time per TLD
grep "fetch_time_ms" logs/tld_import_2025-10-08.log | grep -oP '"fetch_time_ms":\K[0-9.]+' | awk '{sum+=$1; n++} END {print "Average: " sum/n " ms"}'
```
### Slowest TLDs
```bash
# Find slowest 10 TLDs
grep "fetch_time_ms" logs/tld_import_2025-10-08.log | grep -oP 'TLD \.\w+.*fetch_time_ms":\K[0-9.]+' | sort -rn | head -10
```
### Batch Performance
```bash
# Show all batch statistics
grep "Batch statistics" logs/tld_import_2025-10-08.log
```
## Log Retention
Logs are organized by date and are not automatically deleted. You may want to:
1. **Manual cleanup**: Delete old logs periodically
```bash
find logs/ -name "*.log" -mtime +30 -delete # Delete logs older than 30 days
```
2. **Set up logrotate** (Linux):
```
/path/to/Domain Monitor/logs/*.log {
daily
rotate 30
compress
missingok
notifempty
}
```
## Accessing Logs
### Via Command Line
```bash
# View latest TLD import log
tail -f logs/tld_import_$(date +%Y-%m-%d).log
# View last 100 lines
tail -100 logs/tld_import_2025-10-08.log
# Search for errors
grep ERROR logs/tld_import_2025-10-08.log
# Watch for specific TLD
grep ".example" logs/tld_import_2025-10-08.log
```
### Via PHP
The `Logger` class provides a `tail()` method:
```php
$logger = new Logger('tld_import');
$lastLines = $logger->tail(100); // Get last 100 lines
```
## Important Notes
- **Log files grow large**: A complete TLD import (~1500 TLDs) generates ~2-5 MB of logs
- **Context data is JSON**: Can be parsed programmatically for analysis
- **Timestamps are in server timezone**: Check your PHP timezone setting
- **Logs are NOT web-accessible**: The `.htaccess` in the project root blocks access to this directory
## Related Files
- `app/Services/Logger.php` - Logger implementation
- `app/Services/TldRegistryService.php` - Uses logger for TLD import operations
- `cron/check_domains.php` - Uses file_put_contents for cron.log
## Support
If you're experiencing import issues:
1. Check the relevant log file for error messages
2. Look for patterns in failed TLDs
3. Check Apache/Nginx error logs for timeout issues
4. Verify network connectivity to IANA servers
5. Consider reducing batch size for slower servers

36
scripts/generate-encryption-key.php
View File

@@ -1,36 +0,0 @@
#!/usr/bin/env php
<?php
/**
* Generate Encryption Key
*
* This script generates a secure encryption key for the application.
* The key is used to encrypt sensitive data in the database (like SMTP passwords).
*
* Usage: php scripts/generate-encryption-key.php
*/
echo "===========================================\n";
echo " Domain Monitor - Encryption Key Generator\n";
echo "===========================================\n\n";
// Generate a secure 32-byte (256-bit) key
$key = random_bytes(32);
$encodedKey = base64_encode($key);
echo "Your encryption key has been generated:\n\n";
echo "\033[1;32m$encodedKey\033[0m\n\n";
echo "Add this to your .env file:\n\n";
echo "APP_ENCRYPTION_KEY=$encodedKey\n\n";
echo "⚠️ IMPORTANT SECURITY NOTES:\n";
echo "-------------------------------------------\n";
echo "1. Keep this key SECRET - never share it\n";
echo "2. Never commit this key to version control\n";
echo "3. If you lose this key, encrypted data cannot be recovered\n";
echo "4. Don't change this key after encrypting data\n";
echo "5. Store a backup of this key in a secure location\n\n";
echo "✅ Done! Copy the key to your .env file.\n";