mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
V2
This commit is contained in:
Toby
@@ -9,74 +9,445 @@ metadata:
|
||||
|
||||
# Audit Logging
|
||||
|
||||
Implement comprehensive audit logging for compliance.
|
||||
Implement comprehensive audit logging for compliance, security monitoring, and forensic analysis across infrastructure and applications.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Setting up centralized logging for compliance frameworks (SOC 2, HIPAA, PCI DSS)
|
||||
- Implementing security event monitoring and alerting
|
||||
- Building audit trails for regulatory requirements
|
||||
- Configuring log retention and tamper-proof storage
|
||||
- Integrating application logs with SIEM platforms
|
||||
|
||||
## Log Categories
|
||||
|
||||
```yaml
|
||||
audit_events:
|
||||
authentication:
|
||||
- Login attempts
|
||||
- MFA events
|
||||
- Session management
|
||||
|
||||
- Login attempts (success and failure)
|
||||
- MFA enrollment and verification events
|
||||
- Session creation, renewal, and termination
|
||||
- Password changes and resets
|
||||
- API key and token generation
|
||||
|
||||
authorization:
|
||||
- Access grants
|
||||
- Permission changes
|
||||
- Role assignments
|
||||
|
||||
- Access grants and denials
|
||||
- Permission changes and role assignments
|
||||
- Privilege escalation events
|
||||
- Resource sharing modifications
|
||||
- Policy evaluation results
|
||||
|
||||
data_access:
|
||||
- Read operations
|
||||
- Write operations
|
||||
- Delete operations
|
||||
|
||||
- Read operations on sensitive data
|
||||
- Write and update operations
|
||||
- Delete and purge operations
|
||||
- Bulk export and download events
|
||||
- Data classification changes
|
||||
|
||||
administrative:
|
||||
- Configuration changes
|
||||
- User management
|
||||
- System changes
|
||||
- User and group management
|
||||
- System startup and shutdown
|
||||
- Backup and restore operations
|
||||
- Network and firewall rule changes
|
||||
|
||||
system:
|
||||
- Service health state changes
|
||||
- Resource provisioning and deprovisioning
|
||||
- Certificate and key rotation events
|
||||
- Scheduled job execution results
|
||||
- Integration and webhook events
|
||||
```
|
||||
|
||||
## Application Logging
|
||||
## Rsyslog Configuration for Centralized Logging
|
||||
|
||||
```bash
|
||||
# /etc/rsyslog.d/50-audit.conf
|
||||
|
||||
# Load imfile module to read application logs
|
||||
module(load="imfile")
|
||||
|
||||
# Forward auth logs
|
||||
input(type="imfile"
|
||||
File="/var/log/auth.log"
|
||||
Tag="auth"
|
||||
Severity="info"
|
||||
Facility="auth"
|
||||
)
|
||||
|
||||
# Forward application audit logs
|
||||
input(type="imfile"
|
||||
File="/var/log/app/audit.log"
|
||||
Tag="app-audit"
|
||||
Severity="info"
|
||||
Facility="local0"
|
||||
)
|
||||
|
||||
# Structured JSON template
|
||||
template(name="json-audit" type="list") {
|
||||
constant(value="{")
|
||||
constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
|
||||
constant(value="\",\"host\":\"") property(name="hostname")
|
||||
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
|
||||
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
|
||||
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
|
||||
constant(value="\",\"message\":\"") property(name="msg" format="json")
|
||||
constant(value="\"}\n")
|
||||
}
|
||||
|
||||
# Forward to central syslog server over TLS
|
||||
action(
|
||||
type="omfwd"
|
||||
target="syslog.internal.example.com"
|
||||
port="6514"
|
||||
protocol="tcp"
|
||||
StreamDriver="gtls"
|
||||
StreamDriverMode="1"
|
||||
StreamDriverAuthMode="x509/name"
|
||||
template="json-audit"
|
||||
queue.type="LinkedList"
|
||||
queue.size="50000"
|
||||
queue.filename="fwd_audit"
|
||||
queue.saveonshutdown="on"
|
||||
action.resumeRetryCount="-1"
|
||||
)
|
||||
```
|
||||
|
||||
## Journald Configuration for Persistent Logging
|
||||
|
||||
```ini
|
||||
# /etc/systemd/journald.conf
|
||||
[Journal]
|
||||
Storage=persistent
|
||||
Compress=yes
|
||||
Seal=yes
|
||||
SplitMode=uid
|
||||
MaxRetentionSec=365d
|
||||
MaxFileSec=30d
|
||||
SystemMaxUse=10G
|
||||
SystemKeepFree=2G
|
||||
ForwardToSyslog=yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# Query journald for audit events
|
||||
journalctl _TRANSPORT=audit --since "24 hours ago" --output json-pretty
|
||||
|
||||
# Filter by specific audit types
|
||||
journalctl _AUDIT_TYPE=1112 --since today # user login events
|
||||
journalctl _AUDIT_TYPE=1100 --since today # user auth events
|
||||
|
||||
# Export for offline analysis
|
||||
journalctl --since "7 days ago" --output export > /backup/journal-export.bin
|
||||
```
|
||||
|
||||
## Application Logging with Structured JSON
|
||||
|
||||
```python
|
||||
import logging
|
||||
import json
|
||||
import hashlib
|
||||
from datetime import datetime, timezone
|
||||
from functools import wraps
|
||||
|
||||
class AuditLogger:
|
||||
def log_event(self, event_type, user, resource, action, result):
|
||||
def __init__(self, service_name, logger_name="audit"):
|
||||
self.service = service_name
|
||||
self.logger = logging.getLogger(logger_name)
|
||||
handler = logging.FileHandler("/var/log/app/audit.log")
|
||||
handler.setFormatter(logging.Formatter("%(message)s"))
|
||||
self.logger.addHandler(handler)
|
||||
self.logger.setLevel(logging.INFO)
|
||||
self._prev_hash = None
|
||||
|
||||
def log_event(self, event_type, user, resource, action, result,
|
||||
metadata=None, source_ip=None):
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat(),
|
||||
'event_type': event_type,
|
||||
'user': user,
|
||||
'resource': resource,
|
||||
'action': action,
|
||||
'result': result,
|
||||
'source_ip': request.remote_addr
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"service": self.service,
|
||||
"event_type": event_type,
|
||||
"user": user,
|
||||
"resource": resource,
|
||||
"action": action,
|
||||
"result": result,
|
||||
"source_ip": source_ip,
|
||||
"metadata": metadata or {},
|
||||
}
|
||||
logger.info(json.dumps(log_entry))
|
||||
# Chain hash for tamper detection
|
||||
raw = json.dumps(log_entry, sort_keys=True)
|
||||
log_entry["prev_hash"] = self._prev_hash
|
||||
log_entry["hash"] = hashlib.sha256(
|
||||
f"{self._prev_hash}:{raw}".encode()
|
||||
).hexdigest()
|
||||
self._prev_hash = log_entry["hash"]
|
||||
self.logger.info(json.dumps(log_entry))
|
||||
|
||||
def log_auth(self, user, action, success, source_ip=None, mfa=False):
|
||||
self.log_event(
|
||||
event_type="authentication",
|
||||
user=user,
|
||||
resource="auth-service",
|
||||
action=action,
|
||||
result="success" if success else "failure",
|
||||
metadata={"mfa_used": mfa},
|
||||
source_ip=source_ip,
|
||||
)
|
||||
|
||||
def log_data_access(self, user, resource, operation, record_count=0,
|
||||
source_ip=None):
|
||||
self.log_event(
|
||||
event_type="data_access",
|
||||
user=user,
|
||||
resource=resource,
|
||||
action=operation,
|
||||
result="success",
|
||||
metadata={"record_count": record_count},
|
||||
source_ip=source_ip,
|
||||
)
|
||||
|
||||
|
||||
def audit_trail(audit_logger, resource_name):
|
||||
"""Decorator to automatically audit function calls."""
|
||||
def decorator(func):
|
||||
@wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
user = kwargs.get("current_user", "system")
|
||||
try:
|
||||
result = func(*args, **kwargs)
|
||||
audit_logger.log_event(
|
||||
event_type="operation",
|
||||
user=user,
|
||||
resource=resource_name,
|
||||
action=func.__name__,
|
||||
result="success",
|
||||
)
|
||||
return result
|
||||
except Exception as e:
|
||||
audit_logger.log_event(
|
||||
event_type="operation",
|
||||
user=user,
|
||||
resource=resource_name,
|
||||
action=func.__name__,
|
||||
result="failure",
|
||||
metadata={"error": str(e)},
|
||||
)
|
||||
raise
|
||||
return wrapper
|
||||
return decorator
|
||||
```
|
||||
|
||||
## Centralized Logging
|
||||
## Fluentd / Fluent Bit Log Aggregation
|
||||
|
||||
```yaml
|
||||
# Fluentd configuration
|
||||
<source>
|
||||
@type tail
|
||||
path /var/log/audit/*.log
|
||||
tag audit.*
|
||||
</source>
|
||||
# fluent-bit.conf - lightweight agent on each node
|
||||
[SERVICE]
|
||||
Flush 5
|
||||
Daemon Off
|
||||
Log_Level info
|
||||
Parsers_File parsers.conf
|
||||
|
||||
<match audit.**>
|
||||
@type elasticsearch
|
||||
host elasticsearch.example.com
|
||||
index_name audit-logs
|
||||
</match>
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/app/audit.log
|
||||
Parser json
|
||||
Tag audit.app
|
||||
Refresh_Interval 5
|
||||
Rotate_Wait 30
|
||||
|
||||
[INPUT]
|
||||
Name systemd
|
||||
Tag audit.system
|
||||
Systemd_Filter _TRANSPORT=audit
|
||||
|
||||
[FILTER]
|
||||
Name modify
|
||||
Match audit.*
|
||||
Add cluster ${CLUSTER_NAME}
|
||||
Add node ${NODE_NAME}
|
||||
|
||||
[OUTPUT]
|
||||
Name es
|
||||
Match audit.*
|
||||
Host elasticsearch.internal.example.com
|
||||
Port 9200
|
||||
Index audit-logs
|
||||
Type _doc
|
||||
tls On
|
||||
tls.verify On
|
||||
Retry_Limit 5
|
||||
|
||||
[OUTPUT]
|
||||
Name s3
|
||||
Match audit.*
|
||||
region us-east-1
|
||||
bucket audit-logs-archive
|
||||
total_file_size 50M
|
||||
upload_timeout 10m
|
||||
s3_key_format /logs/%Y/%m/%d/$TAG/%H_%M_%S.gz
|
||||
compression gzip
|
||||
```
|
||||
|
||||
## Elasticsearch Index Lifecycle for Retention
|
||||
|
||||
```json
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"hot": {
|
||||
"min_age": "0ms",
|
||||
"actions": {
|
||||
"rollover": {
|
||||
"max_size": "50gb",
|
||||
"max_age": "1d"
|
||||
},
|
||||
"set_priority": { "priority": 100 }
|
||||
}
|
||||
},
|
||||
"warm": {
|
||||
"min_age": "7d",
|
||||
"actions": {
|
||||
"shrink": { "number_of_shards": 1 },
|
||||
"forcemerge": { "max_num_segments": 1 },
|
||||
"set_priority": { "priority": 50 }
|
||||
}
|
||||
},
|
||||
"cold": {
|
||||
"min_age": "30d",
|
||||
"actions": {
|
||||
"freeze": {},
|
||||
"set_priority": { "priority": 0 }
|
||||
}
|
||||
},
|
||||
"delete": {
|
||||
"min_age": "365d",
|
||||
"actions": { "delete": {} }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Retention Policy by Compliance Framework
|
||||
|
||||
```yaml
|
||||
retention_requirements:
|
||||
soc2:
|
||||
minimum: 1 year
|
||||
recommended: 3 years
|
||||
notes: "Based on audit period and report requirements"
|
||||
|
||||
hipaa:
|
||||
minimum: 6 years
|
||||
notes: "From date of creation or last effective date"
|
||||
|
||||
pci_dss:
|
||||
minimum: 1 year
|
||||
immediately_available: 3 months
|
||||
notes: "Req 10.7 - retain for at least one year, 3 months immediately available"
|
||||
|
||||
gdpr:
|
||||
minimum: "As long as necessary for processing purpose"
|
||||
notes: "Apply data minimization; delete when no longer needed"
|
||||
|
||||
fedramp:
|
||||
minimum: 3 years
|
||||
notes: "AU-11 control requirement"
|
||||
|
||||
iso27001:
|
||||
minimum: "Defined by organization policy"
|
||||
recommended: 3 years
|
||||
notes: "A.12.4.1 - retention period must be defined"
|
||||
```
|
||||
|
||||
## Log Integrity Verification Script
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# verify-log-integrity.sh - Verify log file checksums against stored hashes
|
||||
|
||||
LOG_DIR="/var/log/app"
|
||||
HASH_FILE="/var/log/app/.checksums"
|
||||
ALERT_WEBHOOK="${ALERT_WEBHOOK_URL}"
|
||||
|
||||
verify_logs() {
|
||||
local failures=0
|
||||
while IFS=' ' read -r stored_hash filename; do
|
||||
if [ -f "$filename" ]; then
|
||||
current_hash=$(sha256sum "$filename" | awk '{print $1}')
|
||||
if [ "$stored_hash" != "$current_hash" ]; then
|
||||
echo "TAMPER DETECTED: $filename"
|
||||
failures=$((failures + 1))
|
||||
curl -s -X POST "$ALERT_WEBHOOK" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"text\":\"ALERT: Audit log tamper detected on $(hostname): $filename\"}"
|
||||
fi
|
||||
else
|
||||
echo "MISSING: $filename"
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
done < "$HASH_FILE"
|
||||
|
||||
return $failures
|
||||
}
|
||||
|
||||
update_checksums() {
|
||||
find "$LOG_DIR" -name "*.log" -type f -exec sha256sum {} \; > "$HASH_FILE"
|
||||
chmod 440 "$HASH_FILE"
|
||||
}
|
||||
|
||||
case "${1:-verify}" in
|
||||
verify) verify_logs ;;
|
||||
update) update_checksums ;;
|
||||
*) echo "Usage: $0 {verify|update}" ;;
|
||||
esac
|
||||
```
|
||||
|
||||
## SIEM Integration Checklist
|
||||
|
||||
```yaml
|
||||
siem_integration:
|
||||
log_sources:
|
||||
- [ ] Operating system auth logs (syslog, journald)
|
||||
- [ ] Application audit logs (structured JSON)
|
||||
- [ ] Cloud provider audit trails (CloudTrail, Activity Log, Audit Logs)
|
||||
- [ ] Database query and access logs
|
||||
- [ ] Network flow logs and firewall logs
|
||||
- [ ] Container and orchestrator logs (Kubernetes audit)
|
||||
- [ ] WAF and CDN access logs
|
||||
- [ ] VPN and remote access logs
|
||||
|
||||
normalization:
|
||||
- [ ] Common event format (CEF) or OCSF schema
|
||||
- [ ] Consistent timestamp format (ISO 8601 / UTC)
|
||||
- [ ] Unified user identity fields
|
||||
- [ ] Standardized severity levels
|
||||
|
||||
alerting_rules:
|
||||
- [ ] Multiple failed login attempts (brute force)
|
||||
- [ ] Login from unusual location or device
|
||||
- [ ] Privilege escalation events
|
||||
- [ ] Sensitive data bulk export
|
||||
- [ ] Administrative action outside change window
|
||||
- [ ] Service account anomalous activity
|
||||
- [ ] Log forwarding gap or interruption
|
||||
|
||||
operational:
|
||||
- [ ] Log pipeline health monitoring
|
||||
- [ ] Storage capacity alerting
|
||||
- [ ] Retention policy enforcement verified
|
||||
- [ ] Backup of log archives confirmed
|
||||
- [ ] Access to log systems restricted and audited
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Structured logging (JSON)
|
||||
- Centralized collection
|
||||
- Tamper-proof storage
|
||||
- Retention policies
|
||||
- Alerting on anomalies
|
||||
- Use structured logging (JSON) with consistent field names across all services
|
||||
- Ship logs to a centralized platform with write-once storage for tamper protection
|
||||
- Implement hash chaining or digital signatures for log integrity verification
|
||||
- Define and enforce retention policies per compliance framework requirements
|
||||
- Set up real-time alerting for high-severity security events
|
||||
- Separate audit logs from application debug logs to reduce noise
|
||||
- Never log sensitive data (passwords, tokens, PII) in audit entries
|
||||
- Monitor the logging pipeline itself to detect gaps in coverage
|
||||
- Regularly test log restoration from archives to verify recoverability
|
||||
- Rotate and compress logs to manage storage while meeting retention windows
|
||||
|
||||
Reference in New Issue
Block a user