mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
V2
This commit is contained in:
Toby
@@ -9,80 +9,410 @@ metadata:
|
||||
|
||||
# AWS S3
|
||||
|
||||
Manage object storage with Amazon S3.
|
||||
Manage Amazon S3 object storage with production-grade security, lifecycle policies, replication, and access controls.
|
||||
|
||||
## Create Bucket
|
||||
## When to Use This Skill
|
||||
|
||||
- Creating S3 buckets with security hardening (encryption, public access block, versioning)
|
||||
- Writing bucket policies to enforce HTTPS, restrict IP ranges, or grant cross-account access
|
||||
- Setting up lifecycle rules to transition objects between storage classes
|
||||
- Configuring cross-region replication for disaster recovery
|
||||
- Generating presigned URLs for temporary access to private objects
|
||||
- Setting up static website hosting or CloudFront origins
|
||||
- Troubleshooting access denied errors or policy conflicts
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- AWS CLI v2 installed and configured
|
||||
- IAM permissions: `s3:*`, `s3-object-lambda:*`, `kms:*` (for SSE-KMS)
|
||||
- For replication: IAM role with replication permissions and destination bucket in target region
|
||||
- For logging: a separate logging bucket with appropriate ACL
|
||||
|
||||
## Create and Secure a Bucket
|
||||
|
||||
```bash
|
||||
# Create a bucket (us-east-1 does not need LocationConstraint)
|
||||
aws s3api create-bucket \
|
||||
--bucket my-bucket \
|
||||
--bucket my-app-data-prod \
|
||||
--region us-east-1
|
||||
|
||||
# Enable versioning
|
||||
aws s3api put-bucket-versioning \
|
||||
--bucket my-bucket \
|
||||
--versioning-configuration Status=Enabled
|
||||
# Create a bucket in another region
|
||||
aws s3api create-bucket \
|
||||
--bucket my-app-data-dr \
|
||||
--region us-west-2 \
|
||||
--create-bucket-configuration LocationConstraint=us-west-2
|
||||
|
||||
# Block public access
|
||||
# Block ALL public access (always do this first)
|
||||
aws s3api put-public-access-block \
|
||||
--bucket my-bucket \
|
||||
--bucket my-app-data-prod \
|
||||
--public-access-block-configuration '{
|
||||
"BlockPublicAcls": true,
|
||||
"IgnorePublicAcls": true,
|
||||
"BlockPublicPolicy": true,
|
||||
"RestrictPublicBuckets": true
|
||||
}'
|
||||
|
||||
# Enable versioning
|
||||
aws s3api put-bucket-versioning \
|
||||
--bucket my-app-data-prod \
|
||||
--versioning-configuration Status=Enabled
|
||||
|
||||
# Enable server-side encryption with SSE-KMS
|
||||
aws s3api put-bucket-encryption \
|
||||
--bucket my-app-data-prod \
|
||||
--server-side-encryption-configuration '{
|
||||
"Rules": [{
|
||||
"ApplyServerSideEncryptionByDefault": {
|
||||
"SSEAlgorithm": "aws:kms",
|
||||
"KMSMasterKeyID": "alias/s3-key"
|
||||
},
|
||||
"BucketKeyEnabled": true
|
||||
}]
|
||||
}'
|
||||
|
||||
# Enable access logging
|
||||
aws s3api put-bucket-logging \
|
||||
--bucket my-app-data-prod \
|
||||
--bucket-logging-status '{
|
||||
"LoggingEnabled": {
|
||||
"TargetBucket": "my-access-logs-bucket",
|
||||
"TargetPrefix": "s3-logs/my-app-data-prod/"
|
||||
}
|
||||
}'
|
||||
|
||||
# Add tags
|
||||
aws s3api put-bucket-tagging \
|
||||
--bucket my-app-data-prod \
|
||||
--tagging '{
|
||||
"TagSet": [
|
||||
{"Key": "Environment", "Value": "production"},
|
||||
{"Key": "Team", "Value": "platform"},
|
||||
{"Key": "DataClassification", "Value": "confidential"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## Bucket Policy
|
||||
## Bucket Policies
|
||||
|
||||
```bash
|
||||
# Apply a bucket policy (enforce HTTPS and restrict to VPC endpoint)
|
||||
aws s3api put-bucket-policy \
|
||||
--bucket my-app-data-prod \
|
||||
--policy '{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "DenyInsecureTransport",
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "s3:*",
|
||||
"Resource": [
|
||||
"arn:aws:s3:::my-app-data-prod",
|
||||
"arn:aws:s3:::my-app-data-prod/*"
|
||||
],
|
||||
"Condition": {
|
||||
"Bool": {"aws:SecureTransport": "false"}
|
||||
}
|
||||
},
|
||||
{
|
||||
"Sid": "RestrictToVPCEndpoint",
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "s3:*",
|
||||
"Resource": [
|
||||
"arn:aws:s3:::my-app-data-prod",
|
||||
"arn:aws:s3:::my-app-data-prod/*"
|
||||
],
|
||||
"Condition": {
|
||||
"StringNotEquals": {
|
||||
"aws:sourceVpce": "vpce-abc123"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
Cross-account access policy:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [{
|
||||
"Sid": "EnforceHTTPS",
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "s3:*",
|
||||
"Resource": [
|
||||
"arn:aws:s3:::my-bucket",
|
||||
"arn:aws:s3:::my-bucket/*"
|
||||
],
|
||||
"Condition": {
|
||||
"Bool": {"aws:SecureTransport": "false"}
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "CrossAccountRead",
|
||||
"Effect": "Allow",
|
||||
"Principal": {
|
||||
"AWS": "arn:aws:iam::987654321098:role/DataAnalystRole"
|
||||
},
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:ListBucket"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::my-app-data-prod",
|
||||
"arn:aws:s3:::my-app-data-prod/shared/*"
|
||||
]
|
||||
}
|
||||
}]
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Lifecycle Rules
|
||||
|
||||
```bash
|
||||
# Apply a comprehensive lifecycle configuration
|
||||
aws s3api put-bucket-lifecycle-configuration \
|
||||
--bucket my-bucket \
|
||||
--bucket my-app-data-prod \
|
||||
--lifecycle-configuration '{
|
||||
"Rules": [{
|
||||
"ID": "Archive old objects",
|
||||
"Status": "Enabled",
|
||||
"Filter": {"Prefix": "logs/"},
|
||||
"Transitions": [{
|
||||
"Days": 30,
|
||||
"StorageClass": "GLACIER"
|
||||
}],
|
||||
"Expiration": {"Days": 365}
|
||||
}]
|
||||
"Rules": [
|
||||
{
|
||||
"ID": "TierDownOldData",
|
||||
"Status": "Enabled",
|
||||
"Filter": {"Prefix": "data/"},
|
||||
"Transitions": [
|
||||
{"Days": 30, "StorageClass": "STANDARD_IA"},
|
||||
{"Days": 90, "StorageClass": "GLACIER_IR"},
|
||||
{"Days": 180, "StorageClass": "GLACIER"},
|
||||
{"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"ID": "ExpireLogs",
|
||||
"Status": "Enabled",
|
||||
"Filter": {"Prefix": "logs/"},
|
||||
"Expiration": {"Days": 90},
|
||||
"Transitions": [
|
||||
{"Days": 7, "StorageClass": "STANDARD_IA"},
|
||||
{"Days": 30, "StorageClass": "GLACIER"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"ID": "CleanupOldVersions",
|
||||
"Status": "Enabled",
|
||||
"Filter": {"Prefix": ""},
|
||||
"NoncurrentVersionTransitions": [
|
||||
{"NoncurrentDays": 30, "StorageClass": "STANDARD_IA"},
|
||||
{"NoncurrentDays": 90, "StorageClass": "GLACIER"}
|
||||
],
|
||||
"NoncurrentVersionExpiration": {"NoncurrentDays": 180}
|
||||
},
|
||||
{
|
||||
"ID": "AbortIncompleteUploads",
|
||||
"Status": "Enabled",
|
||||
"Filter": {"Prefix": ""},
|
||||
"AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
|
||||
},
|
||||
{
|
||||
"ID": "ExpireDeleteMarkers",
|
||||
"Status": "Enabled",
|
||||
"Filter": {"Prefix": ""},
|
||||
"Expiration": {"ExpiredObjectDeleteMarker": true}
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
## Cross-Region Replication
|
||||
|
||||
- Enable versioning
|
||||
- Block public access
|
||||
- Use encryption (SSE-S3 or SSE-KMS)
|
||||
- Implement lifecycle policies
|
||||
- Enable access logging
|
||||
```bash
|
||||
# Enable replication (requires versioning on both buckets)
|
||||
aws s3api put-bucket-replication \
|
||||
--bucket my-app-data-prod \
|
||||
--replication-configuration '{
|
||||
"Role": "arn:aws:iam::123456789012:role/S3ReplicationRole",
|
||||
"Rules": [
|
||||
{
|
||||
"ID": "ReplicateAll",
|
||||
"Status": "Enabled",
|
||||
"Priority": 1,
|
||||
"Filter": {"Prefix": ""},
|
||||
"Destination": {
|
||||
"Bucket": "arn:aws:s3:::my-app-data-dr",
|
||||
"StorageClass": "STANDARD_IA",
|
||||
"EncryptionConfiguration": {
|
||||
"ReplicaKmsKeyID": "arn:aws:kms:us-west-2:123456789012:key/dr-key-id"
|
||||
},
|
||||
"Metrics": {"Status": "Enabled", "EventThreshold": {"Minutes": 15}},
|
||||
"ReplicationTime": {"Status": "Enabled", "Time": {"Minutes": 15}}
|
||||
},
|
||||
"DeleteMarkerReplication": {"Status": "Enabled"},
|
||||
"SourceSelectionCriteria": {
|
||||
"SseKmsEncryptedObjects": {"Status": "Enabled"}
|
||||
}
|
||||
}
|
||||
]
|
||||
}'
|
||||
|
||||
# Check replication status
|
||||
aws s3api head-object \
|
||||
--bucket my-app-data-prod \
|
||||
--key data/important-file.json \
|
||||
--query "ReplicationStatus"
|
||||
```
|
||||
|
||||
## Presigned URLs
|
||||
|
||||
```bash
|
||||
# Generate a presigned URL for downloading (valid 1 hour)
|
||||
aws s3 presign s3://my-app-data-prod/reports/quarterly.pdf \
|
||||
--expires-in 3600
|
||||
|
||||
# Generate a presigned URL for uploading
|
||||
aws s3 presign s3://my-app-data-prod/uploads/user-file.zip \
|
||||
--expires-in 3600
|
||||
|
||||
# Presigned URL with specific content type (using the API directly)
|
||||
aws s3api generate-presigned-url \
|
||||
--client-method put_object \
|
||||
--params '{"Bucket":"my-app-data-prod","Key":"uploads/photo.jpg","ContentType":"image/jpeg"}' \
|
||||
--expires-in 3600
|
||||
```
|
||||
|
||||
## Common S3 Operations
|
||||
|
||||
```bash
|
||||
# Sync a local directory to S3
|
||||
aws s3 sync ./build s3://my-app-data-prod/static/ \
|
||||
--delete \
|
||||
--exclude "*.tmp" \
|
||||
--cache-control "max-age=31536000" \
|
||||
--content-encoding "gzip"
|
||||
|
||||
# Copy with storage class
|
||||
aws s3 cp large-archive.tar.gz s3://my-app-data-prod/archives/ \
|
||||
--storage-class GLACIER_IR
|
||||
|
||||
# List objects with size summary
|
||||
aws s3 ls s3://my-app-data-prod/ --recursive --summarize --human-readable
|
||||
|
||||
# Remove all objects with a prefix
|
||||
aws s3 rm s3://my-app-data-prod/temp/ --recursive
|
||||
|
||||
# Get bucket size via CloudWatch (most efficient for large buckets)
|
||||
aws cloudwatch get-metric-statistics \
|
||||
--namespace AWS/S3 \
|
||||
--metric-name BucketSizeBytes \
|
||||
--dimensions Name=BucketName,Value=my-app-data-prod Name=StorageType,Value=StandardStorage \
|
||||
--start-time "$(date -u -d '2 days ago' +%Y-%m-%dT%H:%M:%SZ)" \
|
||||
--end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
|
||||
--period 86400 \
|
||||
--statistics Average \
|
||||
--output table
|
||||
```
|
||||
|
||||
## Terraform S3 Bucket
|
||||
|
||||
```hcl
|
||||
resource "aws_s3_bucket" "main" {
|
||||
bucket = "my-app-data-prod"
|
||||
|
||||
tags = {
|
||||
Environment = "production"
|
||||
DataClassification = "confidential"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_s3_bucket_versioning" "main" {
|
||||
bucket = aws_s3_bucket.main.id
|
||||
versioning_configuration {
|
||||
status = "Enabled"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_s3_bucket_public_access_block" "main" {
|
||||
bucket = aws_s3_bucket.main.id
|
||||
|
||||
block_public_acls = true
|
||||
block_public_policy = true
|
||||
ignore_public_acls = true
|
||||
restrict_public_buckets = true
|
||||
}
|
||||
|
||||
resource "aws_s3_bucket_server_side_encryption_configuration" "main" {
|
||||
bucket = aws_s3_bucket.main.id
|
||||
|
||||
rule {
|
||||
apply_server_side_encryption_by_default {
|
||||
sse_algorithm = "aws:kms"
|
||||
kms_master_key_id = aws_kms_key.s3.arn
|
||||
}
|
||||
bucket_key_enabled = true
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_s3_bucket_lifecycle_configuration" "main" {
|
||||
bucket = aws_s3_bucket.main.id
|
||||
|
||||
rule {
|
||||
id = "tier-down"
|
||||
status = "Enabled"
|
||||
|
||||
transition {
|
||||
days = 30
|
||||
storage_class = "STANDARD_IA"
|
||||
}
|
||||
|
||||
transition {
|
||||
days = 90
|
||||
storage_class = "GLACIER"
|
||||
}
|
||||
|
||||
noncurrent_version_transition {
|
||||
noncurrent_days = 30
|
||||
storage_class = "GLACIER"
|
||||
}
|
||||
|
||||
noncurrent_version_expiration {
|
||||
noncurrent_days = 180
|
||||
}
|
||||
|
||||
abort_incomplete_multipart_upload {
|
||||
days_after_initiation = 7
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_s3_bucket_policy" "enforce_https" {
|
||||
bucket = aws_s3_bucket.main.id
|
||||
|
||||
policy = jsonencode({
|
||||
Version = "2012-10-17"
|
||||
Statement = [{
|
||||
Sid = "DenyInsecureTransport"
|
||||
Effect = "Deny"
|
||||
Principal = "*"
|
||||
Action = "s3:*"
|
||||
Resource = [
|
||||
aws_s3_bucket.main.arn,
|
||||
"${aws_s3_bucket.main.arn}/*"
|
||||
]
|
||||
Condition = {
|
||||
Bool = { "aws:SecureTransport" = "false" }
|
||||
}
|
||||
}]
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Cause | Fix |
|
||||
|---|---|---|
|
||||
| Access Denied on GetObject | Bucket policy or IAM denies access | Check bucket policy, IAM policy, and public access block |
|
||||
| Access Denied on PutObject | Missing encryption header when required | Add SSE header; check bucket policy encryption conditions |
|
||||
| 403 on presigned URL | URL expired or wrong region | Regenerate; ensure region matches bucket region |
|
||||
| Replication not working | Versioning disabled on source or dest | Enable versioning on both buckets |
|
||||
| Lifecycle not transitioning | Rule filter does not match objects | Verify prefix and tag filters; check rule status |
|
||||
| Bucket delete fails | Bucket not empty or has versioned objects | Delete all objects and versions first; disable versioning |
|
||||
| Slow uploads for large files | Single-part upload | Use `aws s3 cp` (auto multipart) or set multipart threshold |
|
||||
| Cross-account access denied | Both bucket policy AND IAM policy needed | Grant in bucket policy and in caller's IAM policy |
|
||||
| Object Lock prevents deletion | Governance or compliance mode active | Use governance bypass (with permission) or wait for retention |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [terraform-aws](../terraform-aws/) - IaC deployment
|
||||
- [aws-iam](../aws-iam/) - Access policies
|
||||
- [aws-iam](../aws-iam/) - Bucket and object access policies
|
||||
- [aws-vpc](../aws-vpc/) - VPC endpoints for private S3 access
|
||||
- [aws-cost-optimization](../aws-cost-optimization/) - Storage class optimization
|
||||
- [terraform-aws](../terraform-aws/) - IaC deployment for S3
|
||||
- [cloudformation](../cloudformation/) - AWS-native S3 templates
|
||||
|
||||
Reference in New Issue
Block a user