mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
V2
This commit is contained in:
Toby
@@ -9,63 +9,466 @@ metadata:
|
||||
|
||||
# GCP Audit Logs
|
||||
|
||||
Audit GCP activity with Cloud Audit Logs.
|
||||
Audit GCP activity with Cloud Audit Logs for compliance, security investigation, and operational monitoring.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Enabling organization-wide audit logging across GCP projects
|
||||
- Meeting compliance requirements for SOC 2, HIPAA, PCI DSS, or FedRAMP
|
||||
- Investigating unauthorized access or suspicious API activity
|
||||
- Setting up alerting on administrative and data access events
|
||||
- Exporting logs to BigQuery for long-term analysis and reporting
|
||||
|
||||
## Audit Log Types
|
||||
|
||||
```yaml
|
||||
log_types:
|
||||
admin_activity:
|
||||
- Always enabled
|
||||
- API calls that modify resources
|
||||
- No charge
|
||||
|
||||
description: API calls that modify resource configuration or metadata
|
||||
enabled: Always (cannot be disabled)
|
||||
retention: 400 days (default)
|
||||
cost: No charge
|
||||
examples:
|
||||
- Creating or deleting VM instances
|
||||
- Changing IAM policies
|
||||
- Modifying firewall rules
|
||||
|
||||
data_access:
|
||||
- Must be enabled
|
||||
- Read/write data operations
|
||||
- Can be high volume
|
||||
|
||||
description: API calls that read resource configuration, metadata, or user data
|
||||
enabled: Must be explicitly enabled (except BigQuery)
|
||||
retention: 30 days (default)
|
||||
cost: Can be significant at high volume
|
||||
subtypes:
|
||||
ADMIN_READ: Read resource configuration/metadata
|
||||
DATA_READ: Read user-provided data
|
||||
DATA_WRITE: Write user-provided data
|
||||
|
||||
system_event:
|
||||
- Always enabled
|
||||
- GCP system actions
|
||||
|
||||
description: Actions performed by GCP systems on behalf of resources
|
||||
enabled: Always (cannot be disabled)
|
||||
retention: 400 days (default)
|
||||
cost: No charge
|
||||
examples:
|
||||
- Live migration of VM instances
|
||||
- Automatic scaling events
|
||||
|
||||
policy_denied:
|
||||
- Always enabled
|
||||
- Access denials
|
||||
description: Actions denied by VPC Service Controls or organization policies
|
||||
enabled: Always (cannot be disabled)
|
||||
retention: 400 days (default)
|
||||
cost: No charge
|
||||
```
|
||||
|
||||
## Enable Data Access Logs
|
||||
## Enable Data Access Logs for an Organization
|
||||
|
||||
```bash
|
||||
# Enable for all services
|
||||
gcloud logging sinks create audit-sink \
|
||||
storage.googleapis.com/audit-logs-bucket \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com"'
|
||||
# Get current org IAM policy
|
||||
gcloud organizations get-iam-policy ORG_ID --format=json > org-policy.json
|
||||
|
||||
# IAM policy for data access logs
|
||||
gcloud projects get-iam-policy PROJECT_ID > policy.yaml
|
||||
# Add auditConfigs section
|
||||
gcloud projects set-iam-policy PROJECT_ID policy.yaml
|
||||
# Add audit config to org-policy.json:
|
||||
# {
|
||||
# "auditConfigs": [
|
||||
# {
|
||||
# "service": "allServices",
|
||||
# "auditLogConfigs": [
|
||||
# {"logType": "ADMIN_READ"},
|
||||
# {"logType": "DATA_READ"},
|
||||
# {"logType": "DATA_WRITE"}
|
||||
# ]
|
||||
# }
|
||||
# ],
|
||||
# ...existing bindings...
|
||||
# }
|
||||
|
||||
# Apply the updated policy
|
||||
gcloud organizations set-iam-policy ORG_ID org-policy.json
|
||||
|
||||
# Enable data access logs for specific services at project level
|
||||
gcloud projects get-iam-policy PROJECT_ID --format=json > project-policy.json
|
||||
|
||||
# Example: enable only for Cloud Storage and BigQuery
|
||||
# {
|
||||
# "auditConfigs": [
|
||||
# {
|
||||
# "service": "storage.googleapis.com",
|
||||
# "auditLogConfigs": [
|
||||
# {"logType": "DATA_READ"},
|
||||
# {"logType": "DATA_WRITE"}
|
||||
# ]
|
||||
# },
|
||||
# {
|
||||
# "service": "bigquery.googleapis.com",
|
||||
# "auditLogConfigs": [
|
||||
# {"logType": "DATA_READ"},
|
||||
# {"logType": "DATA_WRITE"}
|
||||
# ]
|
||||
# }
|
||||
# ]
|
||||
# }
|
||||
|
||||
gcloud projects set-iam-policy PROJECT_ID project-policy.json
|
||||
```
|
||||
|
||||
## BigQuery Analysis
|
||||
## Configure Log Sinks for Export
|
||||
|
||||
```bash
|
||||
# Create BigQuery dataset for audit log export
|
||||
bq mk --dataset \
|
||||
--description "Audit log export" \
|
||||
--default_table_expiration 0 \
|
||||
--location US \
|
||||
PROJECT_ID:audit_logs
|
||||
|
||||
# Create organization-level log sink to BigQuery
|
||||
gcloud logging sinks create org-audit-bigquery \
|
||||
bigquery.googleapis.com/projects/PROJECT_ID/datasets/audit_logs \
|
||||
--organization=ORG_ID \
|
||||
--include-children \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com"'
|
||||
|
||||
# Get the sink writer identity and grant BigQuery access
|
||||
SINK_SA=$(gcloud logging sinks describe org-audit-bigquery \
|
||||
--organization=ORG_ID --format='value(writerIdentity)')
|
||||
|
||||
bq add-iam-policy-binding \
|
||||
--member="$SINK_SA" \
|
||||
--role="roles/bigquery.dataEditor" \
|
||||
PROJECT_ID:audit_logs
|
||||
|
||||
# Create Cloud Storage sink for long-term archive
|
||||
gsutil mb -l US -b on gs://org-audit-logs-archive
|
||||
gsutil retention set 7y gs://org-audit-logs-archive
|
||||
|
||||
gcloud logging sinks create org-audit-storage \
|
||||
storage.googleapis.com/org-audit-logs-archive \
|
||||
--organization=ORG_ID \
|
||||
--include-children \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com"'
|
||||
|
||||
STORAGE_SA=$(gcloud logging sinks describe org-audit-storage \
|
||||
--organization=ORG_ID --format='value(writerIdentity)')
|
||||
|
||||
gsutil iam ch "$STORAGE_SA:objectCreator" gs://org-audit-logs-archive
|
||||
|
||||
# Create Pub/Sub sink for real-time streaming to SIEM
|
||||
gcloud pubsub topics create audit-log-stream
|
||||
|
||||
gcloud logging sinks create org-audit-pubsub \
|
||||
pubsub.googleapis.com/projects/PROJECT_ID/topics/audit-log-stream \
|
||||
--organization=ORG_ID \
|
||||
--include-children \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com" AND (protoPayload.methodName:"delete" OR protoPayload.methodName:"setIamPolicy" OR severity>=WARNING)'
|
||||
|
||||
PUBSUB_SA=$(gcloud logging sinks describe org-audit-pubsub \
|
||||
--organization=ORG_ID --format='value(writerIdentity)')
|
||||
|
||||
gcloud pubsub topics add-iam-policy-binding audit-log-stream \
|
||||
--member="$PUBSUB_SA" \
|
||||
--role="roles/pubsub.publisher"
|
||||
```
|
||||
|
||||
## Logging Queries (Cloud Logging Explorer)
|
||||
|
||||
```bash
|
||||
# View admin activity logs for the last 24 hours
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/activity"
|
||||
AND timestamp>="2024-01-01T00:00:00Z"' \
|
||||
--project=PROJECT_ID \
|
||||
--format=json \
|
||||
--limit=100
|
||||
|
||||
# Find IAM policy changes
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/activity"
|
||||
AND protoPayload.methodName="SetIamPolicy"' \
|
||||
--project=PROJECT_ID \
|
||||
--freshness=7d
|
||||
|
||||
# Find resource deletions
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/activity"
|
||||
AND protoPayload.methodName=~"delete"
|
||||
AND severity>=NOTICE' \
|
||||
--project=PROJECT_ID \
|
||||
--freshness=7d
|
||||
|
||||
# Data access audit log entries
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/data_access"
|
||||
AND protoPayload.serviceName="storage.googleapis.com"
|
||||
AND protoPayload.methodName="storage.objects.get"' \
|
||||
--project=PROJECT_ID \
|
||||
--freshness=24h
|
||||
|
||||
# Failed authorization attempts
|
||||
gcloud logging read 'logName:"cloudaudit.googleapis.com/policy"' \
|
||||
--project=PROJECT_ID \
|
||||
--freshness=7d
|
||||
```
|
||||
|
||||
## BigQuery Analysis Queries
|
||||
|
||||
```sql
|
||||
-- Query audit logs from BigQuery export
|
||||
-- All destructive operations in the last 30 days
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail,
|
||||
protopayload_auditlog.methodName,
|
||||
resource.labels.project_id
|
||||
FROM `project.dataset.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS principal,
|
||||
protopayload_auditlog.methodName AS method,
|
||||
protopayload_auditlog.resourceName AS resource,
|
||||
resource.labels.project_id AS project,
|
||||
protopayload_auditlog.status.code AS status_code,
|
||||
protopayload_auditlog.status.message AS status_message
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
|
||||
AND protopayload_auditlog.methodName LIKE '%delete%'
|
||||
ORDER BY timestamp DESC
|
||||
LIMIT 500;
|
||||
|
||||
-- IAM policy changes across the organization
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS changed_by,
|
||||
resource.labels.project_id AS project,
|
||||
protopayload_auditlog.resourceName AS resource,
|
||||
protopayload_auditlog.servicedata_v1_iam.policyDelta.bindingDeltas
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
|
||||
AND protopayload_auditlog.methodName = 'SetIamPolicy'
|
||||
ORDER BY timestamp DESC;
|
||||
|
||||
-- Activity per principal (detect anomalous usage)
|
||||
SELECT
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS principal,
|
||||
COUNT(*) AS action_count,
|
||||
COUNT(DISTINCT protopayload_auditlog.methodName) AS unique_methods,
|
||||
COUNT(DISTINCT protopayload_auditlog.requestMetadata.callerIp) AS unique_ips,
|
||||
MIN(timestamp) AS first_activity,
|
||||
MAX(timestamp) AS last_activity
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY))
|
||||
GROUP BY principal
|
||||
ORDER BY action_count DESC
|
||||
LIMIT 50;
|
||||
|
||||
-- Service account key creation events (security risk indicator)
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS created_by,
|
||||
protopayload_auditlog.resourceName AS service_account,
|
||||
protopayload_auditlog.requestMetadata.callerIp AS source_ip
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY))
|
||||
AND protopayload_auditlog.methodName = 'google.iam.admin.v1.CreateServiceAccountKey'
|
||||
ORDER BY timestamp DESC;
|
||||
|
||||
-- Data access patterns for sensitive Cloud Storage buckets
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS accessor,
|
||||
protopayload_auditlog.resourceName AS object_path,
|
||||
protopayload_auditlog.methodName AS access_type,
|
||||
protopayload_auditlog.requestMetadata.callerIp AS source_ip
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_data_access_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY))
|
||||
AND protopayload_auditlog.resourceName LIKE '%sensitive-bucket%'
|
||||
ORDER BY timestamp DESC
|
||||
LIMIT 1000;
|
||||
|
||||
-- Failed operations indicating permission issues
|
||||
SELECT
|
||||
timestamp,
|
||||
protopayload_auditlog.authenticationInfo.principalEmail AS principal,
|
||||
protopayload_auditlog.methodName AS method,
|
||||
protopayload_auditlog.status.code AS error_code,
|
||||
protopayload_auditlog.status.message AS error_message,
|
||||
protopayload_auditlog.requestMetadata.callerIp AS source_ip
|
||||
FROM `project.audit_logs.cloudaudit_googleapis_com_activity_*`
|
||||
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY))
|
||||
AND protopayload_auditlog.status.code != 0
|
||||
ORDER BY timestamp DESC
|
||||
LIMIT 500;
|
||||
```
|
||||
|
||||
## Alerting Policies
|
||||
|
||||
```bash
|
||||
# Alert on service account key creation
|
||||
gcloud alpha monitoring policies create \
|
||||
--display-name="SA Key Created" \
|
||||
--condition-display-name="Service Account Key Creation" \
|
||||
--condition-filter='resource.type="audited_resource" AND protoPayload.methodName="google.iam.admin.v1.CreateServiceAccountKey"' \
|
||||
--condition-threshold-value=0 \
|
||||
--condition-threshold-comparison=COMPARISON_GT \
|
||||
--condition-threshold-duration=0s \
|
||||
--notification-channels=projects/PROJECT_ID/notificationChannels/CHANNEL_ID \
|
||||
--combiner=OR
|
||||
|
||||
# Create a log-based metric for IAM changes
|
||||
gcloud logging metrics create iam-policy-changes \
|
||||
--description="Count of IAM policy changes" \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com/activity" AND protoPayload.methodName="SetIamPolicy"'
|
||||
|
||||
# Create alerting policy using the log-based metric
|
||||
gcloud alpha monitoring policies create \
|
||||
--display-name="IAM Policy Changes" \
|
||||
--condition-display-name="IAM Changes Detected" \
|
||||
--condition-filter='metric.type="logging.googleapis.com/user/iam-policy-changes"' \
|
||||
--condition-threshold-value=0 \
|
||||
--condition-threshold-comparison=COMPARISON_GT \
|
||||
--condition-threshold-duration=0s \
|
||||
--notification-channels=projects/PROJECT_ID/notificationChannels/CHANNEL_ID
|
||||
|
||||
# Create log-based metric for firewall changes
|
||||
gcloud logging metrics create firewall-rule-changes \
|
||||
--description="Count of firewall rule changes" \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com/activity"
|
||||
AND (protoPayload.methodName="v1.compute.firewalls.insert"
|
||||
OR protoPayload.methodName="v1.compute.firewalls.delete"
|
||||
OR protoPayload.methodName="v1.compute.firewalls.patch")'
|
||||
|
||||
# Create log-based metric for VPC network changes
|
||||
gcloud logging metrics create vpc-network-changes \
|
||||
--description="Count of VPC network changes" \
|
||||
--log-filter='logName:"cloudaudit.googleapis.com/activity"
|
||||
AND resource.type="gce_network"
|
||||
AND (protoPayload.methodName=~"insert$" OR protoPayload.methodName=~"delete$")'
|
||||
```
|
||||
|
||||
## Terraform Configuration
|
||||
|
||||
```hcl
|
||||
# Organization-level audit log sink to BigQuery
|
||||
resource "google_logging_organization_sink" "audit_bigquery" {
|
||||
name = "org-audit-bigquery"
|
||||
org_id = var.org_id
|
||||
destination = "bigquery.googleapis.com/projects/${var.project_id}/datasets/${google_bigquery_dataset.audit_logs.dataset_id}"
|
||||
filter = "logName:\"cloudaudit.googleapis.com\""
|
||||
include_children = true
|
||||
|
||||
bigquery_options {
|
||||
use_partitioned_tables = true
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_bigquery_dataset" "audit_logs" {
|
||||
dataset_id = "audit_logs"
|
||||
project = var.project_id
|
||||
location = "US"
|
||||
description = "Organization audit log export"
|
||||
|
||||
default_table_expiration_ms = null # No auto-expiry
|
||||
|
||||
access {
|
||||
role = "WRITER"
|
||||
user_by_email = google_logging_organization_sink.audit_bigquery.writer_identity
|
||||
}
|
||||
|
||||
access {
|
||||
role = "READER"
|
||||
group_by_email = "security-auditors@example.com"
|
||||
}
|
||||
}
|
||||
|
||||
# Retention bucket with bucket lock
|
||||
resource "google_storage_bucket" "audit_archive" {
|
||||
name = "org-audit-logs-archive"
|
||||
location = "US"
|
||||
force_destroy = false
|
||||
project = var.project_id
|
||||
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
retention_policy {
|
||||
is_locked = true
|
||||
retention_period = 220752000 # 7 years in seconds
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 90
|
||||
}
|
||||
action {
|
||||
type = "SetStorageClass"
|
||||
storage_class = "COLDLINE"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Log-based alerting
|
||||
resource "google_logging_metric" "iam_changes" {
|
||||
name = "iam-policy-changes"
|
||||
project = var.project_id
|
||||
filter = "logName:\"cloudaudit.googleapis.com/activity\" AND protoPayload.methodName=\"SetIamPolicy\""
|
||||
|
||||
metric_descriptor {
|
||||
metric_kind = "DELTA"
|
||||
value_type = "INT64"
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_monitoring_alert_policy" "iam_changes" {
|
||||
display_name = "IAM Policy Changes Detected"
|
||||
project = var.project_id
|
||||
combiner = "OR"
|
||||
|
||||
conditions {
|
||||
display_name = "IAM policy change count"
|
||||
condition_threshold {
|
||||
filter = "metric.type=\"logging.googleapis.com/user/iam-policy-changes\" AND resource.type=\"global\""
|
||||
comparison = "COMPARISON_GT"
|
||||
threshold_value = 0
|
||||
duration = "0s"
|
||||
}
|
||||
}
|
||||
|
||||
notification_channels = [var.notification_channel_id]
|
||||
}
|
||||
```
|
||||
|
||||
## Setup Checklist
|
||||
|
||||
```yaml
|
||||
gcp_audit_logs_checklist:
|
||||
log_enablement:
|
||||
- [ ] Admin activity logs verified active (always on)
|
||||
- [ ] Data access logs enabled for sensitive services
|
||||
- [ ] Data access exemptions configured to exclude high-volume, low-risk operations
|
||||
- [ ] System event logs verified active (always on)
|
||||
|
||||
log_routing:
|
||||
- [ ] Organization-level sink to BigQuery for analysis
|
||||
- [ ] Organization-level sink to Cloud Storage for long-term archive
|
||||
- [ ] Pub/Sub sink for real-time SIEM streaming (high severity events)
|
||||
- [ ] Sink writer identities granted appropriate destination permissions
|
||||
- [ ] Inclusion filters verified to capture all audit log types
|
||||
|
||||
storage_and_retention:
|
||||
- [ ] BigQuery dataset created with appropriate access controls
|
||||
- [ ] Cloud Storage bucket with retention policy and bucket lock
|
||||
- [ ] Storage class lifecycle rules configured (Standard to Coldline)
|
||||
- [ ] Default log retention in Cloud Logging extended if needed
|
||||
|
||||
alerting:
|
||||
- [ ] Notification channels configured (email, PagerDuty, Slack)
|
||||
- [ ] Log-based metric for IAM policy changes
|
||||
- [ ] Log-based metric for firewall rule changes
|
||||
- [ ] Log-based metric for service account key creation
|
||||
- [ ] Alert policy for each critical metric
|
||||
- [ ] Alert notification tested end-to-end
|
||||
|
||||
access_control:
|
||||
- [ ] Logging Admin role restricted to security team
|
||||
- [ ] BigQuery dataset read access granted to auditors only
|
||||
- [ ] Storage bucket access restricted with IAM
|
||||
- [ ] Sink configuration changes monitored via admin activity logs
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Export to BigQuery for analysis
|
||||
- Configure log retention
|
||||
- Enable data access logs for sensitive resources
|
||||
- Set up alerting policies
|
||||
- Enable data access logs selectively on sensitive services to control cost and volume
|
||||
- Use organization-level sinks with include-children to capture all projects automatically
|
||||
- Export to BigQuery with partitioned tables for efficient querying over large time ranges
|
||||
- Archive to Cloud Storage with bucket lock and retention policies for immutable long-term storage
|
||||
- Create log-based metrics and alerting policies for high-severity events
|
||||
- Stream critical audit events via Pub/Sub to SIEM for real-time correlation
|
||||
- Apply exemptions to exclude high-volume read-only service accounts from data access logs
|
||||
- Restrict access to audit log sinks and destinations with least-privilege IAM bindings
|
||||
- Regularly run BigQuery analysis queries to detect anomalous patterns and generate compliance reports
|
||||
- Monitor log sink health and delivery latency to ensure continuous audit coverage
|
||||
|
||||
Reference in New Issue
Block a user