mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
V2
This commit is contained in:
Toby
@@ -9,74 +9,445 @@ metadata:
|
||||
|
||||
# Audit Logging
|
||||
|
||||
Implement comprehensive audit logging for compliance.
|
||||
Implement comprehensive audit logging for compliance, security monitoring, and forensic analysis across infrastructure and applications.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Setting up centralized logging for compliance frameworks (SOC 2, HIPAA, PCI DSS)
|
||||
- Implementing security event monitoring and alerting
|
||||
- Building audit trails for regulatory requirements
|
||||
- Configuring log retention and tamper-proof storage
|
||||
- Integrating application logs with SIEM platforms
|
||||
|
||||
## Log Categories
|
||||
|
||||
```yaml
|
||||
audit_events:
|
||||
authentication:
|
||||
- Login attempts
|
||||
- MFA events
|
||||
- Session management
|
||||
|
||||
- Login attempts (success and failure)
|
||||
- MFA enrollment and verification events
|
||||
- Session creation, renewal, and termination
|
||||
- Password changes and resets
|
||||
- API key and token generation
|
||||
|
||||
authorization:
|
||||
- Access grants
|
||||
- Permission changes
|
||||
- Role assignments
|
||||
|
||||
- Access grants and denials
|
||||
- Permission changes and role assignments
|
||||
- Privilege escalation events
|
||||
- Resource sharing modifications
|
||||
- Policy evaluation results
|
||||
|
||||
data_access:
|
||||
- Read operations
|
||||
- Write operations
|
||||
- Delete operations
|
||||
|
||||
- Read operations on sensitive data
|
||||
- Write and update operations
|
||||
- Delete and purge operations
|
||||
- Bulk export and download events
|
||||
- Data classification changes
|
||||
|
||||
administrative:
|
||||
- Configuration changes
|
||||
- User management
|
||||
- System changes
|
||||
- User and group management
|
||||
- System startup and shutdown
|
||||
- Backup and restore operations
|
||||
- Network and firewall rule changes
|
||||
|
||||
system:
|
||||
- Service health state changes
|
||||
- Resource provisioning and deprovisioning
|
||||
- Certificate and key rotation events
|
||||
- Scheduled job execution results
|
||||
- Integration and webhook events
|
||||
```
|
||||
|
||||
## Application Logging
|
||||
## Rsyslog Configuration for Centralized Logging
|
||||
|
||||
```bash
|
||||
# /etc/rsyslog.d/50-audit.conf
|
||||
|
||||
# Load imfile module to read application logs
|
||||
module(load="imfile")
|
||||
|
||||
# Forward auth logs
|
||||
input(type="imfile"
|
||||
File="/var/log/auth.log"
|
||||
Tag="auth"
|
||||
Severity="info"
|
||||
Facility="auth"
|
||||
)
|
||||
|
||||
# Forward application audit logs
|
||||
input(type="imfile"
|
||||
File="/var/log/app/audit.log"
|
||||
Tag="app-audit"
|
||||
Severity="info"
|
||||
Facility="local0"
|
||||
)
|
||||
|
||||
# Structured JSON template
|
||||
template(name="json-audit" type="list") {
|
||||
constant(value="{")
|
||||
constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
|
||||
constant(value="\",\"host\":\"") property(name="hostname")
|
||||
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
|
||||
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
|
||||
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
|
||||
constant(value="\",\"message\":\"") property(name="msg" format="json")
|
||||
constant(value="\"}\n")
|
||||
}
|
||||
|
||||
# Forward to central syslog server over TLS
|
||||
action(
|
||||
type="omfwd"
|
||||
target="syslog.internal.example.com"
|
||||
port="6514"
|
||||
protocol="tcp"
|
||||
StreamDriver="gtls"
|
||||
StreamDriverMode="1"
|
||||
StreamDriverAuthMode="x509/name"
|
||||
template="json-audit"
|
||||
queue.type="LinkedList"
|
||||
queue.size="50000"
|
||||
queue.filename="fwd_audit"
|
||||
queue.saveonshutdown="on"
|
||||
action.resumeRetryCount="-1"
|
||||
)
|
||||
```
|
||||
|
||||
## Journald Configuration for Persistent Logging
|
||||
|
||||
```ini
|
||||
# /etc/systemd/journald.conf
|
||||
[Journal]
|
||||
Storage=persistent
|
||||
Compress=yes
|
||||
Seal=yes
|
||||
SplitMode=uid
|
||||
MaxRetentionSec=365d
|
||||
MaxFileSec=30d
|
||||
SystemMaxUse=10G
|
||||
SystemKeepFree=2G
|
||||
ForwardToSyslog=yes
|
||||
```
|
||||
|
||||
```bash
|
||||
# Query journald for audit events
|
||||
journalctl _TRANSPORT=audit --since "24 hours ago" --output json-pretty
|
||||
|
||||
# Filter by specific audit types
|
||||
journalctl _AUDIT_TYPE=1112 --since today # user login events
|
||||
journalctl _AUDIT_TYPE=1100 --since today # user auth events
|
||||
|
||||
# Export for offline analysis
|
||||
journalctl --since "7 days ago" --output export > /backup/journal-export.bin
|
||||
```
|
||||
|
||||
## Application Logging with Structured JSON
|
||||
|
||||
```python
|
||||
import logging
|
||||
import json
|
||||
import hashlib
|
||||
from datetime import datetime, timezone
|
||||
from functools import wraps
|
||||
|
||||
class AuditLogger:
|
||||
def log_event(self, event_type, user, resource, action, result):
|
||||
def __init__(self, service_name, logger_name="audit"):
|
||||
self.service = service_name
|
||||
self.logger = logging.getLogger(logger_name)
|
||||
handler = logging.FileHandler("/var/log/app/audit.log")
|
||||
handler.setFormatter(logging.Formatter("%(message)s"))
|
||||
self.logger.addHandler(handler)
|
||||
self.logger.setLevel(logging.INFO)
|
||||
self._prev_hash = None
|
||||
|
||||
def log_event(self, event_type, user, resource, action, result,
|
||||
metadata=None, source_ip=None):
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat(),
|
||||
'event_type': event_type,
|
||||
'user': user,
|
||||
'resource': resource,
|
||||
'action': action,
|
||||
'result': result,
|
||||
'source_ip': request.remote_addr
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"service": self.service,
|
||||
"event_type": event_type,
|
||||
"user": user,
|
||||
"resource": resource,
|
||||
"action": action,
|
||||
"result": result,
|
||||
"source_ip": source_ip,
|
||||
"metadata": metadata or {},
|
||||
}
|
||||
logger.info(json.dumps(log_entry))
|
||||
# Chain hash for tamper detection
|
||||
raw = json.dumps(log_entry, sort_keys=True)
|
||||
log_entry["prev_hash"] = self._prev_hash
|
||||
log_entry["hash"] = hashlib.sha256(
|
||||
f"{self._prev_hash}:{raw}".encode()
|
||||
).hexdigest()
|
||||
self._prev_hash = log_entry["hash"]
|
||||
self.logger.info(json.dumps(log_entry))
|
||||
|
||||
def log_auth(self, user, action, success, source_ip=None, mfa=False):
|
||||
self.log_event(
|
||||
event_type="authentication",
|
||||
user=user,
|
||||
resource="auth-service",
|
||||
action=action,
|
||||
result="success" if success else "failure",
|
||||
metadata={"mfa_used": mfa},
|
||||
source_ip=source_ip,
|
||||
)
|
||||
|
||||
def log_data_access(self, user, resource, operation, record_count=0,
|
||||
source_ip=None):
|
||||
self.log_event(
|
||||
event_type="data_access",
|
||||
user=user,
|
||||
resource=resource,
|
||||
action=operation,
|
||||
result="success",
|
||||
metadata={"record_count": record_count},
|
||||
source_ip=source_ip,
|
||||
)
|
||||
|
||||
|
||||
def audit_trail(audit_logger, resource_name):
|
||||
"""Decorator to automatically audit function calls."""
|
||||
def decorator(func):
|
||||
@wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
user = kwargs.get("current_user", "system")
|
||||
try:
|
||||
result = func(*args, **kwargs)
|
||||
audit_logger.log_event(
|
||||
event_type="operation",
|
||||
user=user,
|
||||
resource=resource_name,
|
||||
action=func.__name__,
|
||||
result="success",
|
||||
)
|
||||
return result
|
||||
except Exception as e:
|
||||
audit_logger.log_event(
|
||||
event_type="operation",
|
||||
user=user,
|
||||
resource=resource_name,
|
||||
action=func.__name__,
|
||||
result="failure",
|
||||
metadata={"error": str(e)},
|
||||
)
|
||||
raise
|
||||
return wrapper
|
||||
return decorator
|
||||
```
|
||||
|
||||
## Centralized Logging
|
||||
## Fluentd / Fluent Bit Log Aggregation
|
||||
|
||||
```yaml
|
||||
# Fluentd configuration
|
||||
<source>
|
||||
@type tail
|
||||
path /var/log/audit/*.log
|
||||
tag audit.*
|
||||
</source>
|
||||
# fluent-bit.conf - lightweight agent on each node
|
||||
[SERVICE]
|
||||
Flush 5
|
||||
Daemon Off
|
||||
Log_Level info
|
||||
Parsers_File parsers.conf
|
||||
|
||||
<match audit.**>
|
||||
@type elasticsearch
|
||||
host elasticsearch.example.com
|
||||
index_name audit-logs
|
||||
</match>
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/app/audit.log
|
||||
Parser json
|
||||
Tag audit.app
|
||||
Refresh_Interval 5
|
||||
Rotate_Wait 30
|
||||
|
||||
[INPUT]
|
||||
Name systemd
|
||||
Tag audit.system
|
||||
Systemd_Filter _TRANSPORT=audit
|
||||
|
||||
[FILTER]
|
||||
Name modify
|
||||
Match audit.*
|
||||
Add cluster ${CLUSTER_NAME}
|
||||
Add node ${NODE_NAME}
|
||||
|
||||
[OUTPUT]
|
||||
Name es
|
||||
Match audit.*
|
||||
Host elasticsearch.internal.example.com
|
||||
Port 9200
|
||||
Index audit-logs
|
||||
Type _doc
|
||||
tls On
|
||||
tls.verify On
|
||||
Retry_Limit 5
|
||||
|
||||
[OUTPUT]
|
||||
Name s3
|
||||
Match audit.*
|
||||
region us-east-1
|
||||
bucket audit-logs-archive
|
||||
total_file_size 50M
|
||||
upload_timeout 10m
|
||||
s3_key_format /logs/%Y/%m/%d/$TAG/%H_%M_%S.gz
|
||||
compression gzip
|
||||
```
|
||||
|
||||
## Elasticsearch Index Lifecycle for Retention
|
||||
|
||||
```json
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"hot": {
|
||||
"min_age": "0ms",
|
||||
"actions": {
|
||||
"rollover": {
|
||||
"max_size": "50gb",
|
||||
"max_age": "1d"
|
||||
},
|
||||
"set_priority": { "priority": 100 }
|
||||
}
|
||||
},
|
||||
"warm": {
|
||||
"min_age": "7d",
|
||||
"actions": {
|
||||
"shrink": { "number_of_shards": 1 },
|
||||
"forcemerge": { "max_num_segments": 1 },
|
||||
"set_priority": { "priority": 50 }
|
||||
}
|
||||
},
|
||||
"cold": {
|
||||
"min_age": "30d",
|
||||
"actions": {
|
||||
"freeze": {},
|
||||
"set_priority": { "priority": 0 }
|
||||
}
|
||||
},
|
||||
"delete": {
|
||||
"min_age": "365d",
|
||||
"actions": { "delete": {} }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Retention Policy by Compliance Framework
|
||||
|
||||
```yaml
|
||||
retention_requirements:
|
||||
soc2:
|
||||
minimum: 1 year
|
||||
recommended: 3 years
|
||||
notes: "Based on audit period and report requirements"
|
||||
|
||||
hipaa:
|
||||
minimum: 6 years
|
||||
notes: "From date of creation or last effective date"
|
||||
|
||||
pci_dss:
|
||||
minimum: 1 year
|
||||
immediately_available: 3 months
|
||||
notes: "Req 10.7 - retain for at least one year, 3 months immediately available"
|
||||
|
||||
gdpr:
|
||||
minimum: "As long as necessary for processing purpose"
|
||||
notes: "Apply data minimization; delete when no longer needed"
|
||||
|
||||
fedramp:
|
||||
minimum: 3 years
|
||||
notes: "AU-11 control requirement"
|
||||
|
||||
iso27001:
|
||||
minimum: "Defined by organization policy"
|
||||
recommended: 3 years
|
||||
notes: "A.12.4.1 - retention period must be defined"
|
||||
```
|
||||
|
||||
## Log Integrity Verification Script
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# verify-log-integrity.sh - Verify log file checksums against stored hashes
|
||||
|
||||
LOG_DIR="/var/log/app"
|
||||
HASH_FILE="/var/log/app/.checksums"
|
||||
ALERT_WEBHOOK="${ALERT_WEBHOOK_URL}"
|
||||
|
||||
verify_logs() {
|
||||
local failures=0
|
||||
while IFS=' ' read -r stored_hash filename; do
|
||||
if [ -f "$filename" ]; then
|
||||
current_hash=$(sha256sum "$filename" | awk '{print $1}')
|
||||
if [ "$stored_hash" != "$current_hash" ]; then
|
||||
echo "TAMPER DETECTED: $filename"
|
||||
failures=$((failures + 1))
|
||||
curl -s -X POST "$ALERT_WEBHOOK" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"text\":\"ALERT: Audit log tamper detected on $(hostname): $filename\"}"
|
||||
fi
|
||||
else
|
||||
echo "MISSING: $filename"
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
done < "$HASH_FILE"
|
||||
|
||||
return $failures
|
||||
}
|
||||
|
||||
update_checksums() {
|
||||
find "$LOG_DIR" -name "*.log" -type f -exec sha256sum {} \; > "$HASH_FILE"
|
||||
chmod 440 "$HASH_FILE"
|
||||
}
|
||||
|
||||
case "${1:-verify}" in
|
||||
verify) verify_logs ;;
|
||||
update) update_checksums ;;
|
||||
*) echo "Usage: $0 {verify|update}" ;;
|
||||
esac
|
||||
```
|
||||
|
||||
## SIEM Integration Checklist
|
||||
|
||||
```yaml
|
||||
siem_integration:
|
||||
log_sources:
|
||||
- [ ] Operating system auth logs (syslog, journald)
|
||||
- [ ] Application audit logs (structured JSON)
|
||||
- [ ] Cloud provider audit trails (CloudTrail, Activity Log, Audit Logs)
|
||||
- [ ] Database query and access logs
|
||||
- [ ] Network flow logs and firewall logs
|
||||
- [ ] Container and orchestrator logs (Kubernetes audit)
|
||||
- [ ] WAF and CDN access logs
|
||||
- [ ] VPN and remote access logs
|
||||
|
||||
normalization:
|
||||
- [ ] Common event format (CEF) or OCSF schema
|
||||
- [ ] Consistent timestamp format (ISO 8601 / UTC)
|
||||
- [ ] Unified user identity fields
|
||||
- [ ] Standardized severity levels
|
||||
|
||||
alerting_rules:
|
||||
- [ ] Multiple failed login attempts (brute force)
|
||||
- [ ] Login from unusual location or device
|
||||
- [ ] Privilege escalation events
|
||||
- [ ] Sensitive data bulk export
|
||||
- [ ] Administrative action outside change window
|
||||
- [ ] Service account anomalous activity
|
||||
- [ ] Log forwarding gap or interruption
|
||||
|
||||
operational:
|
||||
- [ ] Log pipeline health monitoring
|
||||
- [ ] Storage capacity alerting
|
||||
- [ ] Retention policy enforcement verified
|
||||
- [ ] Backup of log archives confirmed
|
||||
- [ ] Access to log systems restricted and audited
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Structured logging (JSON)
|
||||
- Centralized collection
|
||||
- Tamper-proof storage
|
||||
- Retention policies
|
||||
- Alerting on anomalies
|
||||
- Use structured logging (JSON) with consistent field names across all services
|
||||
- Ship logs to a centralized platform with write-once storage for tamper protection
|
||||
- Implement hash chaining or digital signatures for log integrity verification
|
||||
- Define and enforce retention policies per compliance framework requirements
|
||||
- Set up real-time alerting for high-severity security events
|
||||
- Separate audit logs from application debug logs to reduce noise
|
||||
- Never log sensitive data (passwords, tokens, PII) in audit entries
|
||||
- Monitor the logging pipeline itself to detect gaps in coverage
|
||||
- Regularly test log restoration from archives to verify recoverability
|
||||
- Rotate and compress logs to manage storage while meeting retention windows
|
||||
|
||||
@@ -9,56 +9,455 @@ metadata:
|
||||
|
||||
# AWS CloudTrail
|
||||
|
||||
Audit AWS account activity with CloudTrail.
|
||||
Audit AWS account activity with CloudTrail for compliance, security investigation, and operational troubleshooting.
|
||||
|
||||
## Create Trail
|
||||
## When to Use
|
||||
|
||||
- Enabling organization-wide audit logging across all AWS accounts
|
||||
- Investigating security incidents or unauthorized API activity
|
||||
- Meeting compliance requirements for SOC 2, HIPAA, PCI DSS, or FedRAMP
|
||||
- Setting up automated alerting on sensitive AWS API calls
|
||||
- Querying historical AWS activity for forensic analysis
|
||||
|
||||
## Create an Organization Trail
|
||||
|
||||
```bash
|
||||
# Create organization trail
|
||||
# Create the S3 bucket for log storage
|
||||
aws s3api create-bucket \
|
||||
--bucket org-cloudtrail-audit-logs \
|
||||
--region us-east-1
|
||||
|
||||
# Apply bucket policy allowing CloudTrail to write
|
||||
aws s3api put-bucket-policy \
|
||||
--bucket org-cloudtrail-audit-logs \
|
||||
--policy '{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "AWSCloudTrailAclCheck",
|
||||
"Effect": "Allow",
|
||||
"Principal": {"Service": "cloudtrail.amazonaws.com"},
|
||||
"Action": "s3:GetBucketAcl",
|
||||
"Resource": "arn:aws:s3:::org-cloudtrail-audit-logs"
|
||||
},
|
||||
{
|
||||
"Sid": "AWSCloudTrailWrite",
|
||||
"Effect": "Allow",
|
||||
"Principal": {"Service": "cloudtrail.amazonaws.com"},
|
||||
"Action": "s3:PutObject",
|
||||
"Resource": "arn:aws:s3:::org-cloudtrail-audit-logs/AWSLogs/*",
|
||||
"Condition": {
|
||||
"StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}
|
||||
}
|
||||
}
|
||||
]
|
||||
}'
|
||||
|
||||
# Block public access on the audit bucket
|
||||
aws s3api put-public-access-block \
|
||||
--bucket org-cloudtrail-audit-logs \
|
||||
--public-access-block-configuration \
|
||||
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
|
||||
|
||||
# Enable versioning for tamper protection
|
||||
aws s3api put-bucket-versioning \
|
||||
--bucket org-cloudtrail-audit-logs \
|
||||
--versioning-configuration Status=Enabled
|
||||
|
||||
# Enable server-side encryption
|
||||
aws s3api put-bucket-encryption \
|
||||
--bucket org-cloudtrail-audit-logs \
|
||||
--server-side-encryption-configuration '{
|
||||
"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "aws:kms", "KMSMasterKeyID": "alias/cloudtrail-key"}}]
|
||||
}'
|
||||
|
||||
# Set lifecycle policy for log retention
|
||||
aws s3api put-bucket-lifecycle-configuration \
|
||||
--bucket org-cloudtrail-audit-logs \
|
||||
--lifecycle-configuration '{
|
||||
"Rules": [
|
||||
{
|
||||
"ID": "TransitionToGlacier",
|
||||
"Status": "Enabled",
|
||||
"Filter": {"Prefix": "AWSLogs/"},
|
||||
"Transitions": [
|
||||
{"Days": 90, "StorageClass": "GLACIER"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"ID": "ExpireOldLogs",
|
||||
"Status": "Enabled",
|
||||
"Filter": {"Prefix": "AWSLogs/"},
|
||||
"Expiration": {"Days": 2555}
|
||||
}
|
||||
]
|
||||
}'
|
||||
|
||||
# Create the organization trail
|
||||
aws cloudtrail create-trail \
|
||||
--name org-audit-trail \
|
||||
--s3-bucket-name audit-logs-bucket \
|
||||
--s3-bucket-name org-cloudtrail-audit-logs \
|
||||
--is-organization-trail \
|
||||
--is-multi-region-trail \
|
||||
--enable-log-file-validation \
|
||||
--kms-key-id arn:aws:kms:...
|
||||
--kms-key-id arn:aws:kms:us-east-1:123456789012:alias/cloudtrail-key \
|
||||
--cloud-watch-logs-log-group-arn arn:aws:logs:us-east-1:123456789012:log-group:CloudTrail:* \
|
||||
--cloud-watch-logs-role-arn arn:aws:iam::123456789012:role/CloudTrail-CWLogs-Role
|
||||
|
||||
# Start logging
|
||||
aws cloudtrail start-logging --name org-audit-trail
|
||||
```
|
||||
|
||||
## Event Selectors
|
||||
## Event Selectors for Management and Data Events
|
||||
|
||||
```bash
|
||||
# Log all management and data events
|
||||
# Configure advanced event selectors for granular control
|
||||
aws cloudtrail put-event-selectors \
|
||||
--trail-name org-audit-trail \
|
||||
--event-selectors '[{
|
||||
"ReadWriteType": "All",
|
||||
"IncludeManagementEvents": true,
|
||||
"DataResources": [{
|
||||
"Type": "AWS::S3::Object",
|
||||
"Values": ["arn:aws:s3:::sensitive-bucket/"]
|
||||
}]
|
||||
}]'
|
||||
--advanced-event-selectors '[
|
||||
{
|
||||
"Name": "AllManagementEvents",
|
||||
"FieldSelectors": [
|
||||
{"Field": "eventCategory", "Equals": ["Management"]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "S3DataEventsForSensitiveBuckets",
|
||||
"FieldSelectors": [
|
||||
{"Field": "eventCategory", "Equals": ["Data"]},
|
||||
{"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
|
||||
{"Field": "resources.ARN", "StartsWith": [
|
||||
"arn:aws:s3:::sensitive-data-bucket/",
|
||||
"arn:aws:s3:::pii-bucket/",
|
||||
"arn:aws:s3:::financial-data/"
|
||||
]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "LambdaInvocations",
|
||||
"FieldSelectors": [
|
||||
{"Field": "eventCategory", "Equals": ["Data"]},
|
||||
{"Field": "resources.type", "Equals": ["AWS::Lambda::Function"]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "DynamoDBDataEvents",
|
||||
"FieldSelectors": [
|
||||
{"Field": "eventCategory", "Equals": ["Data"]},
|
||||
{"Field": "resources.type", "Equals": ["AWS::DynamoDB::Table"]}
|
||||
]
|
||||
}
|
||||
]'
|
||||
```
|
||||
|
||||
## CloudTrail Lake
|
||||
## CloudWatch Alerts for Sensitive Activity
|
||||
|
||||
```bash
|
||||
# Create metric filter for unauthorized API calls
|
||||
aws logs put-metric-filter \
|
||||
--log-group-name CloudTrail \
|
||||
--filter-name UnauthorizedAPICalls \
|
||||
--filter-pattern '{ ($.errorCode = "*UnauthorizedAccess*") || ($.errorCode = "AccessDenied*") }' \
|
||||
--metric-transformations \
|
||||
metricName=UnauthorizedAPICalls,metricNamespace=CloudTrailMetrics,metricValue=1
|
||||
|
||||
# Create alarm for unauthorized calls
|
||||
aws cloudwatch put-metric-alarm \
|
||||
--alarm-name UnauthorizedAPICallsAlarm \
|
||||
--metric-name UnauthorizedAPICalls \
|
||||
--namespace CloudTrailMetrics \
|
||||
--statistic Sum \
|
||||
--period 300 \
|
||||
--threshold 5 \
|
||||
--comparison-operator GreaterThanOrEqualToThreshold \
|
||||
--evaluation-periods 1 \
|
||||
--alarm-actions arn:aws:sns:us-east-1:123456789012:security-alerts
|
||||
|
||||
# Root account usage alarm
|
||||
aws logs put-metric-filter \
|
||||
--log-group-name CloudTrail \
|
||||
--filter-name RootAccountUsage \
|
||||
--filter-pattern '{ ($.userIdentity.type = "Root") && ($.userIdentity.invokedBy NOT EXISTS) && ($.eventType != "AwsServiceEvent") }' \
|
||||
--metric-transformations \
|
||||
metricName=RootAccountUsage,metricNamespace=CloudTrailMetrics,metricValue=1
|
||||
|
||||
aws cloudwatch put-metric-alarm \
|
||||
--alarm-name RootAccountUsageAlarm \
|
||||
--metric-name RootAccountUsage \
|
||||
--namespace CloudTrailMetrics \
|
||||
--statistic Sum \
|
||||
--period 300 \
|
||||
--threshold 1 \
|
||||
--comparison-operator GreaterThanOrEqualToThreshold \
|
||||
--evaluation-periods 1 \
|
||||
--alarm-actions arn:aws:sns:us-east-1:123456789012:security-alerts
|
||||
|
||||
# Console login without MFA
|
||||
aws logs put-metric-filter \
|
||||
--log-group-name CloudTrail \
|
||||
--filter-name ConsoleLoginWithoutMFA \
|
||||
--filter-pattern '{ ($.eventName = "ConsoleLogin") && ($.additionalEventData.MFAUsed != "Yes") && ($.userIdentity.type = "IAMUser") }' \
|
||||
--metric-transformations \
|
||||
metricName=ConsoleLoginWithoutMFA,metricNamespace=CloudTrailMetrics,metricValue=1
|
||||
|
||||
# IAM policy changes
|
||||
aws logs put-metric-filter \
|
||||
--log-group-name CloudTrail \
|
||||
--filter-name IAMPolicyChanges \
|
||||
--filter-pattern '{ ($.eventName=CreatePolicy) || ($.eventName=DeletePolicy) || ($.eventName=AttachRolePolicy) || ($.eventName=DetachRolePolicy) || ($.eventName=AttachUserPolicy) || ($.eventName=PutUserPolicy) }' \
|
||||
--metric-transformations \
|
||||
metricName=IAMPolicyChanges,metricNamespace=CloudTrailMetrics,metricValue=1
|
||||
|
||||
# Security group changes
|
||||
aws logs put-metric-filter \
|
||||
--log-group-name CloudTrail \
|
||||
--filter-name SecurityGroupChanges \
|
||||
--filter-pattern '{ ($.eventName=AuthorizeSecurityGroupIngress) || ($.eventName=RevokeSecurityGroupIngress) || ($.eventName=CreateSecurityGroup) || ($.eventName=DeleteSecurityGroup) }' \
|
||||
--metric-transformations \
|
||||
metricName=SecurityGroupChanges,metricNamespace=CloudTrailMetrics,metricValue=1
|
||||
```
|
||||
|
||||
## Athena Queries for CloudTrail Analysis
|
||||
|
||||
```sql
|
||||
-- Query events
|
||||
SELECT eventTime, userIdentity.userName, eventName, sourceIPAddress
|
||||
-- Create Athena table for CloudTrail logs
|
||||
CREATE EXTERNAL TABLE IF NOT EXISTS cloudtrail_logs (
|
||||
eventVersion STRING,
|
||||
userIdentity STRUCT<
|
||||
type: STRING,
|
||||
principalId: STRING,
|
||||
arn: STRING,
|
||||
accountId: STRING,
|
||||
invokedBy: STRING,
|
||||
accessKeyId: STRING,
|
||||
userName: STRING,
|
||||
sessionContext: STRUCT<
|
||||
attributes: STRUCT<mfaAuthenticated: STRING, creationDate: STRING>,
|
||||
sessionIssuer: STRUCT<type: STRING, principalId: STRING, arn: STRING, accountId: STRING, userName: STRING>
|
||||
>
|
||||
>,
|
||||
eventTime STRING,
|
||||
eventSource STRING,
|
||||
eventName STRING,
|
||||
awsRegion STRING,
|
||||
sourceIPAddress STRING,
|
||||
userAgent STRING,
|
||||
errorCode STRING,
|
||||
errorMessage STRING,
|
||||
requestParameters STRING,
|
||||
responseElements STRING,
|
||||
additionalEventData STRING,
|
||||
requestId STRING,
|
||||
eventId STRING,
|
||||
readOnly STRING,
|
||||
resources ARRAY<STRUCT<arn: STRING, accountId: STRING, type: STRING>>,
|
||||
eventType STRING,
|
||||
recipientAccountId STRING
|
||||
)
|
||||
PARTITIONED BY (region STRING, year STRING, month STRING, day STRING)
|
||||
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
|
||||
LOCATION 's3://org-cloudtrail-audit-logs/AWSLogs/123456789012/CloudTrail/';
|
||||
|
||||
-- Find all delete operations in the last 7 days
|
||||
SELECT eventTime, userIdentity.arn, eventName, sourceIPAddress,
|
||||
requestParameters
|
||||
FROM cloudtrail_logs
|
||||
WHERE eventTime > '2024-01-01'
|
||||
AND eventName LIKE '%Delete%'
|
||||
WHERE eventName LIKE '%Delete%'
|
||||
AND eventTime > date_format(date_add('day', -7, current_date), '%Y-%m-%dT%H:%i:%sZ')
|
||||
ORDER BY eventTime DESC
|
||||
LIMIT 100
|
||||
LIMIT 100;
|
||||
|
||||
-- Identify console logins from unusual IP addresses
|
||||
SELECT eventTime, userIdentity.userName, sourceIPAddress,
|
||||
additionalEventData
|
||||
FROM cloudtrail_logs
|
||||
WHERE eventName = 'ConsoleLogin'
|
||||
AND sourceIPAddress NOT IN ('198.51.100.0/24', '203.0.113.0/24')
|
||||
AND eventTime > date_format(date_add('day', -30, current_date), '%Y-%m-%dT%H:%i:%sZ')
|
||||
ORDER BY eventTime DESC;
|
||||
|
||||
-- Access key usage patterns per principal
|
||||
SELECT userIdentity.arn,
|
||||
count(*) AS api_call_count,
|
||||
count(DISTINCT eventName) AS unique_actions,
|
||||
count(DISTINCT sourceIPAddress) AS unique_ips,
|
||||
min(eventTime) AS first_seen,
|
||||
max(eventTime) AS last_seen
|
||||
FROM cloudtrail_logs
|
||||
WHERE eventTime > date_format(date_add('day', -30, current_date), '%Y-%m-%dT%H:%i:%sZ')
|
||||
GROUP BY userIdentity.arn
|
||||
ORDER BY api_call_count DESC
|
||||
LIMIT 50;
|
||||
|
||||
-- Failed API calls indicating permission issues or reconnaissance
|
||||
SELECT eventTime, userIdentity.arn, eventName, errorCode, errorMessage,
|
||||
sourceIPAddress
|
||||
FROM cloudtrail_logs
|
||||
WHERE errorCode IN ('AccessDenied', 'UnauthorizedAccess', 'Client.UnauthorizedAccess')
|
||||
AND eventTime > date_format(date_add('day', -7, current_date), '%Y-%m-%dT%H:%i:%sZ')
|
||||
ORDER BY eventTime DESC
|
||||
LIMIT 200;
|
||||
|
||||
-- Track KMS key usage
|
||||
SELECT eventTime, userIdentity.arn, eventName, requestParameters,
|
||||
resources[1].arn AS key_arn
|
||||
FROM cloudtrail_logs
|
||||
WHERE eventSource = 'kms.amazonaws.com'
|
||||
AND eventName IN ('Decrypt', 'Encrypt', 'GenerateDataKey', 'DisableKey', 'ScheduleKeyDeletion')
|
||||
AND eventTime > date_format(date_add('day', -7, current_date), '%Y-%m-%dT%H:%i:%sZ')
|
||||
ORDER BY eventTime DESC;
|
||||
```
|
||||
|
||||
## CloudTrail Lake (Event Data Store)
|
||||
|
||||
```bash
|
||||
# Create an event data store for long-term queryable storage
|
||||
aws cloudtrail create-event-data-store \
|
||||
--name org-audit-event-store \
|
||||
--multi-region-enabled \
|
||||
--organization-enabled \
|
||||
--retention-period 2555 \
|
||||
--advanced-event-selectors '[
|
||||
{
|
||||
"Name": "AllManagementEvents",
|
||||
"FieldSelectors": [
|
||||
{"Field": "eventCategory", "Equals": ["Management"]}
|
||||
]
|
||||
}
|
||||
]'
|
||||
```
|
||||
|
||||
```sql
|
||||
-- CloudTrail Lake SQL queries (run in console or via StartQuery API)
|
||||
-- Investigate a specific user's activity
|
||||
SELECT eventTime, eventName, eventSource, sourceIPAddress,
|
||||
errorCode, requestParameters
|
||||
FROM EVENT_DATA_STORE_ID
|
||||
WHERE userIdentity.arn = 'arn:aws:iam::123456789012:user/suspicious-user'
|
||||
AND eventTime > '2024-01-01 00:00:00'
|
||||
ORDER BY eventTime DESC;
|
||||
|
||||
-- Cross-account activity summary
|
||||
SELECT recipientAccountId, userIdentity.arn,
|
||||
count(*) AS event_count
|
||||
FROM EVENT_DATA_STORE_ID
|
||||
WHERE eventTime > '2024-01-01 00:00:00'
|
||||
GROUP BY recipientAccountId, userIdentity.arn
|
||||
ORDER BY event_count DESC;
|
||||
```
|
||||
|
||||
## Validate Trail Integrity
|
||||
|
||||
```bash
|
||||
# Validate log file integrity for a date range
|
||||
aws cloudtrail validate-logs \
|
||||
--trail-arn arn:aws:cloudtrail:us-east-1:123456789012:trail/org-audit-trail \
|
||||
--start-time "2024-01-01T00:00:00Z" \
|
||||
--end-time "2024-01-31T23:59:59Z"
|
||||
|
||||
# Check trail status
|
||||
aws cloudtrail get-trail-status --name org-audit-trail
|
||||
|
||||
# Describe the trail configuration
|
||||
aws cloudtrail describe-trails --trail-name-list org-audit-trail
|
||||
```
|
||||
|
||||
## Terraform Configuration
|
||||
|
||||
```hcl
|
||||
resource "aws_cloudtrail" "org_trail" {
|
||||
name = "org-audit-trail"
|
||||
s3_bucket_name = aws_s3_bucket.cloudtrail.id
|
||||
is_organization_trail = true
|
||||
is_multi_region_trail = true
|
||||
enable_log_file_validation = true
|
||||
kms_key_id = aws_kms_key.cloudtrail.arn
|
||||
cloud_watch_logs_group_arn = "${aws_cloudwatch_log_group.cloudtrail.arn}:*"
|
||||
cloud_watch_logs_role_arn = aws_iam_role.cloudtrail_cw.arn
|
||||
include_global_service_events = true
|
||||
|
||||
advanced_event_selector {
|
||||
name = "AllManagementEvents"
|
||||
field_selector {
|
||||
field = "eventCategory"
|
||||
equals = ["Management"]
|
||||
}
|
||||
}
|
||||
|
||||
advanced_event_selector {
|
||||
name = "SensitiveS3DataEvents"
|
||||
field_selector {
|
||||
field = "eventCategory"
|
||||
equals = ["Data"]
|
||||
}
|
||||
field_selector {
|
||||
field = "resources.type"
|
||||
equals = ["AWS::S3::Object"]
|
||||
}
|
||||
field_selector {
|
||||
field = "resources.ARN"
|
||||
starts_with = ["arn:aws:s3:::sensitive-data-bucket/"]
|
||||
}
|
||||
}
|
||||
|
||||
tags = {
|
||||
Environment = "production"
|
||||
Compliance = "soc2,hipaa"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Setup Checklist
|
||||
|
||||
```yaml
|
||||
cloudtrail_checklist:
|
||||
trail_configuration:
|
||||
- [ ] Organization trail enabled across all accounts
|
||||
- [ ] Multi-region trail enabled
|
||||
- [ ] Log file validation enabled
|
||||
- [ ] KMS encryption configured with dedicated key
|
||||
- [ ] CloudWatch Logs integration active
|
||||
- [ ] S3 bucket policy restricts access to CloudTrail service only
|
||||
|
||||
s3_bucket_hardening:
|
||||
- [ ] Public access blocked
|
||||
- [ ] Versioning enabled
|
||||
- [ ] Server-side encryption enabled
|
||||
- [ ] Lifecycle policy set for retention and archival
|
||||
- [ ] Access logging enabled on the bucket itself
|
||||
- [ ] Object Lock enabled for WORM compliance (if required)
|
||||
|
||||
monitoring_and_alerting:
|
||||
- [ ] Metric filters for unauthorized API calls
|
||||
- [ ] Alarm on root account usage
|
||||
- [ ] Alarm on console login without MFA
|
||||
- [ ] Alarm on IAM policy changes
|
||||
- [ ] Alarm on security group and NACL changes
|
||||
- [ ] Alarm on CloudTrail configuration changes
|
||||
- [ ] Alarm on S3 bucket policy changes
|
||||
|
||||
analysis:
|
||||
- [ ] Athena table created for ad-hoc queries
|
||||
- [ ] CloudTrail Lake event data store for long-term queries
|
||||
- [ ] Regular review of high-risk API patterns
|
||||
- [ ] Automated reports for compliance evidence
|
||||
|
||||
operational:
|
||||
- [ ] Trail status health check automated
|
||||
- [ ] Log delivery latency monitored
|
||||
- [ ] Log file validation run periodically
|
||||
- [ ] SNS notification for trail configuration changes
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Organization-wide trails
|
||||
- Enable log file validation
|
||||
- Encrypt with KMS
|
||||
- CloudWatch Logs integration
|
||||
- Event alerting
|
||||
- Enable organization-wide trails from the management account for full coverage
|
||||
- Always enable log file validation to detect tampering
|
||||
- Encrypt logs with a customer-managed KMS key and restrict key usage
|
||||
- Use advanced event selectors to capture data events on sensitive resources without logging everything
|
||||
- Integrate with CloudWatch Logs for real-time metric filters and alarms
|
||||
- Set up Athena or CloudTrail Lake for efficient querying during investigations
|
||||
- Apply S3 lifecycle policies to transition old logs to Glacier and enforce retention
|
||||
- Monitor the trail itself (delivery errors, configuration changes) as a meta-control
|
||||
- Validate log integrity periodically as part of compliance evidence collection
|
||||
- Restrict access to the CloudTrail S3 bucket and KMS key with least-privilege IAM policies
|
||||
|
||||
@@ -9,49 +9,348 @@ metadata:
|
||||
|
||||
# Azure Monitor Audit
|
||||
|
||||
Audit Azure activity with Monitor and Activity Logs.
|
||||
Audit Azure activity with Monitor, Activity Logs, and Log Analytics for compliance, security, and operational visibility.
|
||||
|
||||
## Diagnostic Settings
|
||||
## When to Use
|
||||
|
||||
- Enabling centralized audit logging across Azure subscriptions
|
||||
- Meeting compliance requirements for SOC 2, HIPAA, PCI DSS, or ISO 27001
|
||||
- Investigating security incidents or unauthorized activity in Azure
|
||||
- Setting up alerting on administrative and security events
|
||||
- Building compliance dashboards and automated evidence collection
|
||||
|
||||
## Create Log Analytics Workspace
|
||||
|
||||
```bash
|
||||
# Enable diagnostic settings
|
||||
az monitor diagnostic-settings create \
|
||||
--name audit-logs \
|
||||
--resource /subscriptions/{sub}/resourceGroups/{rg}/providers/... \
|
||||
--logs '[{"category":"AuditEvent","enabled":true}]' \
|
||||
--workspace /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{workspace}
|
||||
# Create resource group for audit resources
|
||||
az group create \
|
||||
--name rg-audit \
|
||||
--location eastus
|
||||
|
||||
# Create Log Analytics workspace
|
||||
az monitor log-analytics workspace create \
|
||||
--resource-group rg-audit \
|
||||
--workspace-name audit-workspace \
|
||||
--location eastus \
|
||||
--retention-time 365 \
|
||||
--sku PerGB2018
|
||||
|
||||
# Get workspace ID for later use
|
||||
WORKSPACE_ID=$(az monitor log-analytics workspace show \
|
||||
--resource-group rg-audit \
|
||||
--workspace-name audit-workspace \
|
||||
--query id -o tsv)
|
||||
|
||||
# Enable audit solutions
|
||||
az monitor log-analytics solution create \
|
||||
--resource-group rg-audit \
|
||||
--solution-type SecurityCenterFree \
|
||||
--workspace audit-workspace
|
||||
```
|
||||
|
||||
## Activity Log Export
|
||||
## Configure Diagnostic Settings for Subscription Activity Log
|
||||
|
||||
```bash
|
||||
# Export activity log to Log Analytics
|
||||
# Export subscription activity log to Log Analytics
|
||||
az monitor diagnostic-settings subscription create \
|
||||
--name activity-log-export \
|
||||
--name activity-log-to-workspace \
|
||||
--location global \
|
||||
--logs '[{"category":"Administrative","enabled":true},{"category":"Security","enabled":true}]' \
|
||||
--workspace /subscriptions/.../workspaces/audit-workspace
|
||||
--workspace "$WORKSPACE_ID" \
|
||||
--logs '[
|
||||
{"category": "Administrative", "enabled": true},
|
||||
{"category": "Security", "enabled": true},
|
||||
{"category": "ServiceHealth", "enabled": true},
|
||||
{"category": "Alert", "enabled": true},
|
||||
{"category": "Recommendation", "enabled": true},
|
||||
{"category": "Policy", "enabled": true},
|
||||
{"category": "Autoscale", "enabled": true},
|
||||
{"category": "ResourceHealth", "enabled": true}
|
||||
]'
|
||||
|
||||
# Also archive to storage account for long-term retention
|
||||
az storage account create \
|
||||
--name auditlogsarchive \
|
||||
--resource-group rg-audit \
|
||||
--location eastus \
|
||||
--sku Standard_GRS \
|
||||
--kind StorageV2 \
|
||||
--min-tls-version TLS1_2 \
|
||||
--allow-blob-public-access false
|
||||
|
||||
az monitor diagnostic-settings subscription create \
|
||||
--name activity-log-to-storage \
|
||||
--location global \
|
||||
--storage-account /subscriptions/{sub}/resourceGroups/rg-audit/providers/Microsoft.Storage/storageAccounts/auditlogsarchive \
|
||||
--logs '[
|
||||
{"category": "Administrative", "enabled": true, "retentionPolicy": {"enabled": true, "days": 2555}},
|
||||
{"category": "Security", "enabled": true, "retentionPolicy": {"enabled": true, "days": 2555}}
|
||||
]'
|
||||
```
|
||||
|
||||
## Log Analytics Queries
|
||||
## Resource-Level Diagnostic Settings
|
||||
|
||||
```bash
|
||||
# Enable diagnostics for Azure Key Vault
|
||||
az monitor diagnostic-settings create \
|
||||
--name keyvault-audit \
|
||||
--resource /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.KeyVault/vaults/{vault} \
|
||||
--workspace "$WORKSPACE_ID" \
|
||||
--logs '[
|
||||
{"category": "AuditEvent", "enabled": true, "retentionPolicy": {"enabled": true, "days": 365}},
|
||||
{"category": "AzurePolicyEvaluationDetails", "enabled": true}
|
||||
]' \
|
||||
--metrics '[
|
||||
{"category": "AllMetrics", "enabled": true}
|
||||
]'
|
||||
|
||||
# Enable diagnostics for Azure SQL Database
|
||||
az monitor diagnostic-settings create \
|
||||
--name sql-audit \
|
||||
--resource /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Sql/servers/{server}/databases/{db} \
|
||||
--workspace "$WORKSPACE_ID" \
|
||||
--logs '[
|
||||
{"category": "SQLSecurityAuditEvents", "enabled": true},
|
||||
{"category": "SQLInsights", "enabled": true},
|
||||
{"category": "AutomaticTuning", "enabled": true}
|
||||
]'
|
||||
|
||||
# Enable diagnostics for Azure App Service
|
||||
az monitor diagnostic-settings create \
|
||||
--name appservice-audit \
|
||||
--resource /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Web/sites/{app} \
|
||||
--workspace "$WORKSPACE_ID" \
|
||||
--logs '[
|
||||
{"category": "AppServiceHTTPLogs", "enabled": true},
|
||||
{"category": "AppServiceAuditLogs", "enabled": true},
|
||||
{"category": "AppServiceIPSecAuditLogs", "enabled": true},
|
||||
{"category": "AppServicePlatformLogs", "enabled": true}
|
||||
]'
|
||||
|
||||
# Enable diagnostics for Network Security Groups
|
||||
az monitor diagnostic-settings create \
|
||||
--name nsg-flow-logs \
|
||||
--resource /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Network/networkSecurityGroups/{nsg} \
|
||||
--workspace "$WORKSPACE_ID" \
|
||||
--logs '[
|
||||
{"category": "NetworkSecurityGroupEvent", "enabled": true},
|
||||
{"category": "NetworkSecurityGroupRuleCounter", "enabled": true}
|
||||
]'
|
||||
```
|
||||
|
||||
## Azure Policy for Diagnostic Settings Enforcement
|
||||
|
||||
```bash
|
||||
# Assign built-in policy to require diagnostic settings on Key Vaults
|
||||
az policy assignment create \
|
||||
--name require-kv-diagnostics \
|
||||
--policy "951af2fa-529b-416e-ab6e-066fd85ac459" \
|
||||
--scope /subscriptions/{sub} \
|
||||
--params '{
|
||||
"logAnalytics": {"value": "'$WORKSPACE_ID'"},
|
||||
"effect": {"value": "DeployIfNotExists"}
|
||||
}'
|
||||
|
||||
# Assign policy to require diagnostic settings on SQL databases
|
||||
az policy assignment create \
|
||||
--name require-sql-diagnostics \
|
||||
--policy "b79fa14e-238a-4c2d-b376-442ce508fc84" \
|
||||
--scope /subscriptions/{sub} \
|
||||
--params '{
|
||||
"logAnalyticsWorkspaceId": {"value": "'$WORKSPACE_ID'"}
|
||||
}'
|
||||
```
|
||||
|
||||
## KQL Queries for Security Investigation
|
||||
|
||||
```kusto
|
||||
// Failed login attempts
|
||||
AuditLogs
|
||||
// Failed sign-in attempts with location and device details
|
||||
SigninLogs
|
||||
| where TimeGenerated > ago(24h)
|
||||
| where ResultType != "0"
|
||||
| project TimeGenerated, Identity, ResultDescription, IPAddress
|
||||
| summarize FailureCount = count(),
|
||||
DistinctIPs = dcount(IPAddress),
|
||||
Locations = make_set(LocationDetails.city)
|
||||
by UserPrincipalName, ResultDescription, AppDisplayName
|
||||
| where FailureCount > 5
|
||||
| order by FailureCount desc
|
||||
|
||||
// Administrative changes
|
||||
// Successful sign-ins from unusual locations
|
||||
SigninLogs
|
||||
| where TimeGenerated > ago(7d)
|
||||
| where ResultType == "0"
|
||||
| extend City = tostring(LocationDetails.city),
|
||||
Country = tostring(LocationDetails.countryOrRegion)
|
||||
| summarize LoginCount = count(),
|
||||
Cities = make_set(City),
|
||||
Countries = make_set(Country)
|
||||
by UserPrincipalName
|
||||
| where array_length(Countries) > 2
|
||||
|
||||
// Risky sign-ins requiring investigation
|
||||
SigninLogs
|
||||
| where TimeGenerated > ago(7d)
|
||||
| where RiskLevelDuringSignIn in ("medium", "high")
|
||||
| project TimeGenerated, UserPrincipalName, IPAddress,
|
||||
LocationDetails.city, RiskLevelDuringSignIn,
|
||||
RiskEventTypes_V2, AppDisplayName
|
||||
| order by TimeGenerated desc
|
||||
|
||||
// Administrative operations across subscriptions
|
||||
AzureActivity
|
||||
| where TimeGenerated > ago(24h)
|
||||
| where CategoryValue == "Administrative"
|
||||
| where OperationNameValue contains "write" or OperationNameValue contains "delete"
|
||||
| project TimeGenerated, Caller, OperationNameValue, ResourceGroup
|
||||
| where ActivityStatusValue == "Success"
|
||||
| project TimeGenerated, Caller, OperationNameValue,
|
||||
ResourceGroup, Resource, SubscriptionId
|
||||
| order by TimeGenerated desc
|
||||
|
||||
// Key Vault access patterns
|
||||
AzureDiagnostics
|
||||
| where ResourceType == "VAULTS"
|
||||
| where TimeGenerated > ago(24h)
|
||||
| where OperationName in ("SecretGet", "SecretSet", "SecretDelete",
|
||||
"KeySign", "KeyDecrypt", "CertificateGet")
|
||||
| project TimeGenerated, CallerIPAddress, identity_claim_upn_s,
|
||||
OperationName, id_s, ResultType
|
||||
| order by TimeGenerated desc
|
||||
|
||||
// Detect changes to Network Security Groups
|
||||
AzureActivity
|
||||
| where TimeGenerated > ago(7d)
|
||||
| where OperationNameValue has_any ("securityRules/write", "securityRules/delete",
|
||||
"networkSecurityGroups/write")
|
||||
| where ActivityStatusValue == "Success"
|
||||
| project TimeGenerated, Caller, OperationNameValue,
|
||||
ResourceGroup, Properties
|
||||
| order by TimeGenerated desc
|
||||
|
||||
// Azure Policy compliance drift
|
||||
PolicyInsights
|
||||
| where TimeGenerated > ago(7d)
|
||||
| where ComplianceState == "NonCompliant"
|
||||
| summarize NonCompliantCount = count() by PolicyDefinitionName, ResourceType
|
||||
| order by NonCompliantCount desc
|
||||
|
||||
// Privileged role assignments (PIM)
|
||||
AuditLogs
|
||||
| where TimeGenerated > ago(30d)
|
||||
| where OperationName has_any ("Add member to role", "Add eligible member to role")
|
||||
| extend RoleName = tostring(TargetResources[0].displayName),
|
||||
AssignedUser = tostring(TargetResources[2].displayName),
|
||||
AssignedBy = InitiatedBy.user.userPrincipalName
|
||||
| project TimeGenerated, AssignedBy, AssignedUser, RoleName, OperationName
|
||||
| order by TimeGenerated desc
|
||||
```
|
||||
|
||||
## Alert Rules
|
||||
|
||||
```bash
|
||||
# Create action group for security notifications
|
||||
az monitor action-group create \
|
||||
--resource-group rg-audit \
|
||||
--name security-team \
|
||||
--short-name SecTeam \
|
||||
--email-receivers name=SecurityLead email=security@example.com \
|
||||
--webhook-receivers name=PagerDuty uri=https://events.pagerduty.com/integration/{key}/enqueue
|
||||
|
||||
# Alert on multiple failed sign-ins (brute force detection)
|
||||
az monitor scheduled-query create \
|
||||
--resource-group rg-audit \
|
||||
--name brute-force-detection \
|
||||
--scopes "$WORKSPACE_ID" \
|
||||
--condition "count > 10" \
|
||||
--condition-query "SigninLogs | where ResultType != '0' | summarize count() by UserPrincipalName, bin(TimeGenerated, 5m) | where count_ > 10" \
|
||||
--evaluation-frequency 5m \
|
||||
--window-size 5m \
|
||||
--severity 2 \
|
||||
--action-groups /subscriptions/{sub}/resourceGroups/rg-audit/providers/Microsoft.Insights/actionGroups/security-team
|
||||
|
||||
# Alert on Key Vault secret access outside business hours
|
||||
az monitor scheduled-query create \
|
||||
--resource-group rg-audit \
|
||||
--name keyvault-offhours-access \
|
||||
--scopes "$WORKSPACE_ID" \
|
||||
--condition "count > 0" \
|
||||
--condition-query "AzureDiagnostics | where ResourceType == 'VAULTS' | where OperationName in ('SecretGet','SecretList') | where hourofday(TimeGenerated) < 6 or hourofday(TimeGenerated) > 22" \
|
||||
--evaluation-frequency 15m \
|
||||
--window-size 15m \
|
||||
--severity 3 \
|
||||
--action-groups /subscriptions/{sub}/resourceGroups/rg-audit/providers/Microsoft.Insights/actionGroups/security-team
|
||||
|
||||
# Alert on subscription-level administrative changes
|
||||
az monitor activity-log alert create \
|
||||
--resource-group rg-audit \
|
||||
--name critical-admin-changes \
|
||||
--condition category=Administrative and operationName="Microsoft.Authorization/roleAssignments/write" \
|
||||
--action-group /subscriptions/{sub}/resourceGroups/rg-audit/providers/Microsoft.Insights/actionGroups/security-team \
|
||||
--description "Alert on new role assignments"
|
||||
```
|
||||
|
||||
## Workbook for Compliance Dashboard (ARM Template Snippet)
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "Microsoft.Insights/workbooks",
|
||||
"apiVersion": "2022-04-01",
|
||||
"name": "[guid('compliance-dashboard')]",
|
||||
"location": "[resourceGroup().location]",
|
||||
"kind": "shared",
|
||||
"properties": {
|
||||
"displayName": "Compliance Audit Dashboard",
|
||||
"serializedData": "{\"version\":\"Notebook/1.0\",\"items\":[{\"type\":1,\"content\":{\"json\":\"## Compliance Audit Dashboard\"},\"name\":\"title\"},{\"type\":3,\"content\":{\"version\":\"KqlItem/1.0\",\"query\":\"SigninLogs | where TimeGenerated > ago(24h) | where ResultType != '0' | summarize count() by bin(TimeGenerated, 1h)\",\"size\":0,\"title\":\"Failed Sign-ins (24h)\",\"timeContext\":{\"durationMs\":86400000},\"queryType\":0},\"name\":\"failed-signins\"}]}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Setup Checklist
|
||||
|
||||
```yaml
|
||||
azure_monitor_checklist:
|
||||
workspace_setup:
|
||||
- [ ] Log Analytics workspace created in appropriate region
|
||||
- [ ] Retention period configured (minimum per compliance framework)
|
||||
- [ ] Daily cap configured to prevent cost overruns
|
||||
- [ ] RBAC permissions set (Log Analytics Reader for auditors)
|
||||
|
||||
diagnostic_settings:
|
||||
- [ ] Subscription activity log exported to Log Analytics
|
||||
- [ ] Subscription activity log archived to storage account
|
||||
- [ ] Key Vault audit events enabled
|
||||
- [ ] Azure SQL audit logging enabled
|
||||
- [ ] NSG flow logs enabled
|
||||
- [ ] App Service audit logs enabled
|
||||
- [ ] Azure AD sign-in and audit logs connected
|
||||
|
||||
policy_enforcement:
|
||||
- [ ] Azure Policy assigned to enforce diagnostic settings
|
||||
- [ ] DeployIfNotExists policies for critical resource types
|
||||
- [ ] Compliance state monitored via Policy Insights
|
||||
|
||||
alerting:
|
||||
- [ ] Action groups configured for security and operations teams
|
||||
- [ ] Alert on brute force sign-in attempts
|
||||
- [ ] Alert on privileged role assignments
|
||||
- [ ] Alert on Key Vault sensitive operations
|
||||
- [ ] Alert on NSG rule changes
|
||||
- [ ] Alert on resource deletions in production
|
||||
|
||||
reporting:
|
||||
- [ ] Compliance workbook deployed
|
||||
- [ ] Weekly automated query reports exported
|
||||
- [ ] Quarterly access review queries prepared
|
||||
- [ ] Evidence collection queries documented for auditors
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Centralize to Log Analytics
|
||||
- Long-term archive to Storage
|
||||
- Configure alerts
|
||||
- Regular query reviews
|
||||
- Centralize all audit data into a single Log Analytics workspace per tenant
|
||||
- Archive logs to immutable storage for long-term retention and compliance
|
||||
- Use Azure Policy with DeployIfNotExists to enforce diagnostic settings on new resources
|
||||
- Create saved KQL queries for common investigation and compliance scenarios
|
||||
- Set up scheduled query alerts for security-critical events
|
||||
- Assign Log Analytics Reader role to auditors without granting broader access
|
||||
- Monitor the diagnostic settings pipeline itself for delivery failures
|
||||
- Use workbooks for visual compliance dashboards shared with stakeholders
|
||||
- Export query results on a schedule for compliance evidence packages
|
||||
- Separate operational and security alerting to avoid alert fatigue
|
||||
|
||||
@@ -9,63 +9,466 @@ metadata:
|
||||
|
||||
# GCP Audit Logs
|
||||
|
||||
Audit GCP activity with Cloud Audit Logs.
|
||||
Audit GCP activity with Cloud Audit Logs for compliance, security investigation, and operational monitoring.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Enabling organization-wide audit logging across GCP projects
|
||||
- Meeting compliance requirements for SOC 2, HIPAA, PCI DSS, or FedRAMP
|
||||
- Investigating unauthorized access or suspicious API activity
|
||||
- Setting up alerting on administrative and data access events
|
||||
- Exporting logs to BigQuery for long-term analysis and reporting
|
||||
|
||||
## Audit Log Types
|
||||
|
||||
```yaml
|
||||
log_types:
|
||||
admin_activity:
|
||||
- Always enabled
|
||||
- API calls that modify resources
|
||||
- No charge
|
||||
|
||||
description: API calls that modify resource configuration or metadata
|
||||
enabled: Always (cannot be disabled)
|
||||
retention: 400 days (default)
|
||||
cost: No charge
|
||||
examples:
|
||||
- Creating or deleting VM instances
|
||||
- Changing IAM policies
|
||||
- Modifying firewall rules
|
||||
|
||||
data_access:
|
||||
- Must be enabled
|
||||
- Read/write data operations
|
||||
- Can be high volume
|
||||
|
||||
description: API calls that read resource configuration, metadata, or user data
|
||||
enabled: Must be explicitly enabled (except BigQuery)
|
||||
retention: 30 days (default)
|
||||
cost: Can be significant at high volume
|
||||
subtypes:
|
||||
ADMIN_READ: Read resource configuration/metadata
|
||||
DATA_READ: Read user-provided data
|
||||
DATA_WRITE: Write user-provided data
|
||||
|
||||
system_event:
|
||||
- Always enabled
|
||||
- GCP system actions
|
||||
|
||||
description: Actions performed by GCP systems on behalf of resources
|
||||
enabled: Always (cannot be disabled)
|
||||
retention: 400 days (default)
|
||||
cost: No charge
|
||||
examples:
|
||||
- Live migration of VM instances
|
||||
- Automatic scaling events
|
||||
|
||||
policy_denied:
|
||||
- Always enabled
|
||||
- Access denials
|
||||
description: Actions denied by VPC Service Controls or organization policies
|
||||
enabled: Always (cannot be disabled)
|
||||
retention: 400 days (default)
|
||||
cost: No charge
|
||||
```
|
||||
|
||||
## Enable Data Access Logs
|
||||
## Enable Data Access Logs for an Organization
|
||||
|
||||
```bash
|
||||
# Enable for all services
|
||||
gcloud logging sinks create audit-sink \
|
||||
storage.googleapis.com/audit-logs-bucket \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com"'
|
||||
# Get current org IAM policy
|
||||
gcloud organizations get-iam-policy ORG_ID --format=json > org-policy.json
|
||||
|
||||
# IAM policy for data access logs
|
||||
gcloud projects get-iam-policy PROJECT_ID > policy.yaml
|
||||
# Add auditConfigs section
|
||||
gcloud projects set-iam-policy PROJECT_ID policy.yaml
|
||||
# Add audit config to org-policy.json:
|
||||
# {
|
||||
# "auditConfigs": [
|
||||
# {
|
||||
# "service": "allServices",
|
||||
# "auditLogConfigs": [
|
||||
# {"logType": "ADMIN_READ"},
|
||||
# {"logType": "DATA_READ"},
|
||||
# {"logType": "DATA_WRITE"}
|
||||
# ]
|
||||
# }
|
||||
# ],
|
||||
# ...existing bindings...
|
||||
# }
|
||||
|
||||
# Apply the updated policy
|
||||
gcloud organizations set-iam-policy ORG_ID org-policy.json
|
||||
|
||||
# Enable data access logs for specific services at project level
|
||||
gcloud projects get-iam-policy PROJECT_ID --format=json > project-policy.json
|
||||
|
||||
# Example: enable only for Cloud Storage and BigQuery
|
||||
# {
|
||||
# "auditConfigs": [
|
||||
# {
|
||||
# "service": "storage.googleapis.com",
|
||||
# "auditLogConfigs": [
|
||||
# {"logType": "DATA_READ"},
|
||||
# {"logType": "DATA_WRITE"}
|
||||
# ]
|
||||
# },
|
||||
# {
|
||||
# "service": "bigquery.googleapis.com",
|
||||
# "auditLogConfigs": [
|
||||
# {"logType": "DATA_READ"},
|
||||
# {"logType": "DATA_WRITE"}
|
||||
# ]
|
||||
# }
|
||||
# ]
|
||||
# }
|
||||
|
||||
gcloud projects set-iam-policy PROJECT_ID project-policy.json
|
||||
```
|
||||
|
||||
## BigQuery Analysis
|
||||
## Configure Log Sinks for Export
|
||||
|
||||
```bash
|
||||
# Create BigQuery dataset for audit log export
|
||||
bq mk --dataset \
|
||||
--description "Audit log export" \
|
||||
--default_table_expiration 0 \
|
||||
--location US \
|
||||
PROJECT_ID:audit_logs
|
||||
|
||||
# Create organization-level log sink to BigQuery
|
||||
gcloud logging sinks create org-audit-bigquery \
|
||||
bigquery.googleapis.com/projects/PROJECT_ID/datasets/audit_logs \
|
||||
--organization=ORG_ID \
|
||||
--include-children \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com"'
|
||||
|
||||
# Get the sink writer identity and grant BigQuery access
|
||||
SINK_SA=$(gcloud logging sinks describe org-audit-bigquery \
|
||||
--organization=ORG_ID --format='value(writerIdentity)')
|
||||
|
||||
bq add-iam-policy-binding \
|
||||
--member="$SINK_SA" \
|
||||
--role="roles/bigquery.dataEditor" \
|
||||
PROJECT_ID:audit_logs
|
||||
|
||||
# Create Cloud Storage sink for long-term archive
|
||||
gsutil mb -l US -b on gs://org-audit-logs-archive
|
||||
gsutil retention set 7y gs://org-audit-logs-archive
|
||||
|
||||
gcloud logging sinks create org-audit-storage \
|
||||
storage.googleapis.com/org-audit-logs-archive \
|
||||
--organization=ORG_ID \
|
||||
--include-children \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com"'
|
||||
|
||||
STORAGE_SA=$(gcloud logging sinks describe org-audit-storage \
|
||||
--organization=ORG_ID --format='value(writerIdentity)')
|
||||
|
||||
gsutil iam ch "$STORAGE_SA:objectCreator" gs://org-audit-logs-archive
|
||||
|
||||
# Create Pub/Sub sink for real-time streaming to SIEM
|
||||
gcloud pubsub topics create audit-log-stream
|
||||
|
||||
gcloud logging sinks create org-audit-pubsub \
|
||||
pubsub.googleapis.com/projects/PROJECT_ID/topics/audit-log-stream \
|
||||
--organization=ORG_ID \
|
||||
--include-children \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com" AND (protoPayload.methodName:"delete" OR protoPayload.methodName:"setIamPolicy" OR severity>=WARNING)'
|
||||
|
||||
PUBSUB_SA=$(gcloud logging sinks describe org-audit-pubsub \
|
||||
--organization=ORG_ID --format='value(writerIdentity)')
|
||||
|
||||
gcloud pubsub topics add-iam-policy-binding audit-log-stream \
|
||||
--member="$PUBSUB_SA" \
|
||||
--role="roles/pubsub.publisher"
|
||||
```
|
||||
|
||||
## Logging Queries (Cloud Logging Explorer)
|
||||
|
||||
```bash
|
||||
# View admin activity logs for the last 24 hours
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/activity"
|
||||
AND timestamp>="2024-01-01T00:00:00Z"' \
|
||||
--project=PROJECT_ID \
|
||||
--format=json \
|
||||
--limit=100
|
||||
|
||||
# Find IAM policy changes
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/activity"
|
||||
AND protoPayload.methodName="SetIamPolicy"' \
|
||||
--project=PROJECT_ID \
|
||||
--freshness=7d
|
||||
|
||||
# Find resource deletions
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/activity"
|
||||
AND protoPayload.methodName=~"delete"
|
||||
AND severity>=NOTICE' \
|
||||
--project=PROJECT_ID \
|
||||
--freshness=7d
|
||||
|
||||
# Data access audit log entries
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/data_access"
|
||||
AND protoPayload.serviceName="storage.googleapis.com"
|
||||
AND protoPayload.methodName="storage.objects.get"' \
|
||||
--project=PROJECT_ID \
|
||||
--freshness=24h
|
||||
|
||||
# Failed authorization attempts
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/policy"' \
|
||||
--project=PROJECT_ID \
|
||||
--freshness=7d
|
||||
```
|
||||
|
||||
## BigQuery Analysis Queries
|
||||
|
||||
```sql
|
||||
-- Query audit logs from BigQuery export
|
||||
-- All destructive operations in the last 30 days
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail,
|
||||
protopayload_auditlog.methodName,
|
||||
resource.labels.project_id
|
||||
FROM `project.dataset.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS principal,
|
||||
protopayload_auditlog.methodName AS method,
|
||||
protopayload_auditlog.resourceName AS resource,
|
||||
resource.labels.project_id AS project,
|
||||
protopayload_auditlog.status.code AS status_code,
|
||||
protopayload_auditlog.status.message AS status_message
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
|
||||
AND protopayload_auditlog.methodName LIKE '%delete%'
|
||||
ORDER BY timestamp DESC
|
||||
LIMIT 500;
|
||||
|
||||
-- IAM policy changes across the organization
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS changed_by,
|
||||
resource.labels.project_id AS project,
|
||||
protopayload_auditlog.resourceName AS resource,
|
||||
protopayload_auditlog.servicedata_v1_iam.policyDelta.bindingDeltas
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
|
||||
AND protopayload_auditlog.methodName = 'SetIamPolicy'
|
||||
ORDER BY timestamp DESC;
|
||||
|
||||
-- Activity per principal (detect anomalous usage)
|
||||
SELECT
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS principal,
|
||||
COUNT(*) AS action_count,
|
||||
COUNT(DISTINCT protopayload_auditlog.methodName) AS unique_methods,
|
||||
COUNT(DISTINCT protopayload_auditlog.requestMetadata.callerIp) AS unique_ips,
|
||||
MIN(timestamp) AS first_activity,
|
||||
MAX(timestamp) AS last_activity
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY))
|
||||
GROUP BY principal
|
||||
ORDER BY action_count DESC
|
||||
LIMIT 50;
|
||||
|
||||
-- Service account key creation events (security risk indicator)
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS created_by,
|
||||
protopayload_auditlog.resourceName AS service_account,
|
||||
protopayload_auditlog.requestMetadata.callerIp AS source_ip
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY))
|
||||
AND protopayload_auditlog.methodName = 'google.iam.admin.v1.CreateServiceAccountKey'
|
||||
ORDER BY timestamp DESC;
|
||||
|
||||
-- Data access patterns for sensitive Cloud Storage buckets
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS accessor,
|
||||
protopayload_auditlog.resourceName AS object_path,
|
||||
protopayload_auditlog.methodName AS access_type,
|
||||
protopayload_auditlog.requestMetadata.callerIp AS source_ip
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_data_access_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY))
|
||||
AND protopayload_auditlog.resourceName LIKE '%sensitive-bucket%'
|
||||
ORDER BY timestamp DESC
|
||||
LIMIT 1000;
|
||||
|
||||
-- Failed operations indicating permission issues
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS principal,
|
||||
protopayload_auditlog.methodName AS method,
|
||||
protopayload_auditlog.status.code AS error_code,
|
||||
protopayload_auditlog.status.message AS error_message,
|
||||
protopayload_auditlog.requestMetadata.callerIp AS source_ip
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY))
|
||||
AND protopayload_auditlog.status.code != 0
|
||||
ORDER BY timestamp DESC
|
||||
LIMIT 500;
|
||||
```
|
||||
|
||||
## Alerting Policies
|
||||
|
||||
```bash
|
||||
# Alert on service account key creation
|
||||
gcloud alpha monitoring policies create \
|
||||
--display-name="SA Key Created" \
|
||||
--condition-display-name="Service Account Key Creation" \
|
||||
--condition-filter='resource.type="audited_resource" AND protoPayload.methodName="google.iam.admin.v1.CreateServiceAccountKey"' \
|
||||
--condition-threshold-value=0 \
|
||||
--condition-threshold-comparison=COMPARISON_GT \
|
||||
--condition-threshold-duration=0s \
|
||||
--notification-channels=projects/PROJECT_ID/notificationChannels/CHANNEL_ID \
|
||||
--combiner=OR
|
||||
|
||||
# Create a log-based metric for IAM changes
|
||||
gcloud logging metrics create iam-policy-changes \
|
||||
--description="Count of IAM policy changes" \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com/activity" AND protoPayload.methodName="SetIamPolicy"'
|
||||
|
||||
# Create alerting policy using the log-based metric
|
||||
gcloud alpha monitoring policies create \
|
||||
--display-name="IAM Policy Changes" \
|
||||
--condition-display-name="IAM Changes Detected" \
|
||||
--condition-filter='metric.type="logging.googleapis.com/user/iam-policy-changes"' \
|
||||
--condition-threshold-value=0 \
|
||||
--condition-threshold-comparison=COMPARISON_GT \
|
||||
--condition-threshold-duration=0s \
|
||||
--notification-channels=projects/PROJECT_ID/notificationChannels/CHANNEL_ID
|
||||
|
||||
# Create log-based metric for firewall changes
|
||||
gcloud logging metrics create firewall-rule-changes \
|
||||
--description="Count of firewall rule changes" \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com/activity"
|
||||
AND (protoPayload.methodName="v1.compute.firewalls.insert"
|
||||
OR protoPayload.methodName="v1.compute.firewalls.delete"
|
||||
OR protoPayload.methodName="v1.compute.firewalls.patch")'
|
||||
|
||||
# Create log-based metric for VPC network changes
|
||||
gcloud logging metrics create vpc-network-changes \
|
||||
--description="Count of VPC network changes" \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com/activity"
|
||||
AND resource.type="gce_network"
|
||||
AND (protoPayload.methodName=~"insert$" OR protoPayload.methodName=~"delete$")'
|
||||
```
|
||||
|
||||
## Terraform Configuration
|
||||
|
||||
```hcl
|
||||
# Organization-level audit log sink to BigQuery
|
||||
resource "google_logging_organization_sink" "audit_bigquery" {
|
||||
name = "org-audit-bigquery"
|
||||
org_id = var.org_id
|
||||
destination = "bigquery.googleapis.com/projects/${var.project_id}/datasets/${google_bigquery_dataset.audit_logs.dataset_id}"
|
||||
filter = "logName:\"cloudaudit.googleapis.com\""
|
||||
include_children = true
|
||||
|
||||
bigquery_options {
|
||||
use_partitioned_tables = true
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_bigquery_dataset" "audit_logs" {
|
||||
dataset_id = "audit_logs"
|
||||
project = var.project_id
|
||||
location = "US"
|
||||
description = "Organization audit log export"
|
||||
|
||||
default_table_expiration_ms = null # No auto-expiry
|
||||
|
||||
access {
|
||||
role = "WRITER"
|
||||
user_by_email = google_logging_organization_sink.audit_bigquery.writer_identity
|
||||
}
|
||||
|
||||
access {
|
||||
role = "READER"
|
||||
group_by_email = "security-auditors@example.com"
|
||||
}
|
||||
}
|
||||
|
||||
# Retention bucket with bucket lock
|
||||
resource "google_storage_bucket" "audit_archive" {
|
||||
name = "org-audit-logs-archive"
|
||||
location = "US"
|
||||
force_destroy = false
|
||||
project = var.project_id
|
||||
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
retention_policy {
|
||||
is_locked = true
|
||||
retention_period = 220752000 # 7 years in seconds
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 90
|
||||
}
|
||||
action {
|
||||
type = "SetStorageClass"
|
||||
storage_class = "COLDLINE"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Log-based alerting
|
||||
resource "google_logging_metric" "iam_changes" {
|
||||
name = "iam-policy-changes"
|
||||
project = var.project_id
|
||||
filter = "logName:\"cloudaudit.googleapis.com/activity\" AND protoPayload.methodName=\"SetIamPolicy\""
|
||||
|
||||
metric_descriptor {
|
||||
metric_kind = "DELTA"
|
||||
value_type = "INT64"
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_monitoring_alert_policy" "iam_changes" {
|
||||
display_name = "IAM Policy Changes Detected"
|
||||
project = var.project_id
|
||||
combiner = "OR"
|
||||
|
||||
conditions {
|
||||
display_name = "IAM policy change count"
|
||||
condition_threshold {
|
||||
filter = "metric.type=\"logging.googleapis.com/user/iam-policy-changes\" AND resource.type=\"global\""
|
||||
comparison = "COMPARISON_GT"
|
||||
threshold_value = 0
|
||||
duration = "0s"
|
||||
}
|
||||
}
|
||||
|
||||
notification_channels = [var.notification_channel_id]
|
||||
}
|
||||
```
|
||||
|
||||
## Setup Checklist
|
||||
|
||||
```yaml
|
||||
gcp_audit_logs_checklist:
|
||||
log_enablement:
|
||||
- [ ] Admin activity logs verified active (always on)
|
||||
- [ ] Data access logs enabled for sensitive services
|
||||
- [ ] Data access exemptions configured to exclude high-volume, low-risk operations
|
||||
- [ ] System event logs verified active (always on)
|
||||
|
||||
log_routing:
|
||||
- [ ] Organization-level sink to BigQuery for analysis
|
||||
- [ ] Organization-level sink to Cloud Storage for long-term archive
|
||||
- [ ] Pub/Sub sink for real-time SIEM streaming (high severity events)
|
||||
- [ ] Sink writer identities granted appropriate destination permissions
|
||||
- [ ] Inclusion filters verified to capture all audit log types
|
||||
|
||||
storage_and_retention:
|
||||
- [ ] BigQuery dataset created with appropriate access controls
|
||||
- [ ] Cloud Storage bucket with retention policy and bucket lock
|
||||
- [ ] Storage class lifecycle rules configured (Standard to Coldline)
|
||||
- [ ] Default log retention in Cloud Logging extended if needed
|
||||
|
||||
alerting:
|
||||
- [ ] Notification channels configured (email, PagerDuty, Slack)
|
||||
- [ ] Log-based metric for IAM policy changes
|
||||
- [ ] Log-based metric for firewall rule changes
|
||||
- [ ] Log-based metric for service account key creation
|
||||
- [ ] Alert policy for each critical metric
|
||||
- [ ] Alert notification tested end-to-end
|
||||
|
||||
access_control:
|
||||
- [ ] Logging Admin role restricted to security team
|
||||
- [ ] BigQuery dataset read access granted to auditors only
|
||||
- [ ] Storage bucket access restricted with IAM
|
||||
- [ ] Sink configuration changes monitored via admin activity logs
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Export to BigQuery for analysis
|
||||
- Configure log retention
|
||||
- Enable data access logs for sensitive resources
|
||||
- Set up alerting policies
|
||||
- Enable data access logs selectively on sensitive services to control cost and volume
|
||||
- Use organization-level sinks with include-children to capture all projects automatically
|
||||
- Export to BigQuery with partitioned tables for efficient querying over large time ranges
|
||||
- Archive to Cloud Storage with bucket lock and retention policies for immutable long-term storage
|
||||
- Create log-based metrics and alerting policies for high-severity events
|
||||
- Stream critical audit events via Pub/Sub to SIEM for real-time correlation
|
||||
- Apply exemptions to exclude high-volume read-only service accounts from data access logs
|
||||
- Restrict access to audit log sinks and destinations with least-privilege IAM bindings
|
||||
- Regularly run BigQuery analysis queries to detect anomalous patterns and generate compliance reports
|
||||
- Monitor log sink health and delivery latency to ensure continuous audit coverage
|
||||
|
||||
Reference in New Issue
Block a user