mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
V2
This commit is contained in:
Toby
@@ -14,72 +14,449 @@ Securely store, manage, and rotate secrets in AWS.
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Storing database credentials
|
||||
- Managing API keys in AWS
|
||||
- Implementing automatic secret rotation
|
||||
- Integrating secrets with AWS services
|
||||
- Storing database credentials, API keys, or tokens in AWS
|
||||
- Implementing automatic credential rotation for RDS or other services
|
||||
- Replacing hardcoded secrets in application code or config files
|
||||
- Integrating secrets into ECS, EKS, or Lambda workloads
|
||||
- Meeting compliance requirements for secret management and rotation
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- AWS account
|
||||
- AWS CLI configured
|
||||
- IAM permissions for Secrets Manager
|
||||
- AWS account with appropriate IAM permissions
|
||||
- AWS CLI v2 installed and configured
|
||||
- IAM policy allowing `secretsmanager:*` actions (or scoped permissions)
|
||||
- For rotation: Lambda execution role and VPC access to target services
|
||||
- Python 3.9+ with `boto3` for SDK examples
|
||||
|
||||
## Basic Operations
|
||||
## Secret Creation and Management
|
||||
|
||||
```bash
|
||||
# Create secret
|
||||
# Create a secret with JSON structure
|
||||
aws secretsmanager create-secret \
|
||||
--name myapp/database \
|
||||
--secret-string '{"username":"admin","password":"secret123"}'
|
||||
--name myapp/production/database \
|
||||
--description "Production database credentials" \
|
||||
--secret-string '{"username":"dbadmin","password":"S3cur3P@ssw0rd!","engine":"postgres","host":"db.internal.example.com","port":5432,"dbname":"myapp"}' \
|
||||
--tags '[{"Key":"Environment","Value":"production"},{"Key":"Team","Value":"platform"}]'
|
||||
|
||||
# Get secret
|
||||
aws secretsmanager get-secret-value --secret-id myapp/database
|
||||
# Create a secret with KMS encryption (custom key)
|
||||
aws secretsmanager create-secret \
|
||||
--name myapp/production/api-key \
|
||||
--description "Third-party API key" \
|
||||
--secret-string "ak_live_xxxxxxxxxxxx" \
|
||||
--kms-key-id alias/secrets-key
|
||||
|
||||
# Update secret
|
||||
# Create a binary secret (certificates, keys)
|
||||
aws secretsmanager create-secret \
|
||||
--name myapp/production/tls-cert \
|
||||
--secret-binary fileb://server.pfx
|
||||
|
||||
# Get secret value
|
||||
aws secretsmanager get-secret-value \
|
||||
--secret-id myapp/production/database \
|
||||
--query 'SecretString' --output text | jq .
|
||||
|
||||
# Get a specific version
|
||||
aws secretsmanager get-secret-value \
|
||||
--secret-id myapp/production/database \
|
||||
--version-stage AWSPREVIOUS
|
||||
|
||||
# Update secret value
|
||||
aws secretsmanager put-secret-value \
|
||||
--secret-id myapp/database \
|
||||
--secret-string '{"username":"admin","password":"newpassword"}'
|
||||
--secret-id myapp/production/database \
|
||||
--secret-string '{"username":"dbadmin","password":"N3wS3cur3P@ss!","engine":"postgres","host":"db.internal.example.com","port":5432,"dbname":"myapp"}'
|
||||
|
||||
# Delete secret
|
||||
aws secretsmanager delete-secret --secret-id myapp/database --recovery-window-in-days 7
|
||||
# List all secrets
|
||||
aws secretsmanager list-secrets \
|
||||
--filters Key=name,Values=myapp/production
|
||||
|
||||
# Delete secret (with recovery window)
|
||||
aws secretsmanager delete-secret \
|
||||
--secret-id myapp/production/old-key \
|
||||
--recovery-window-in-days 7
|
||||
|
||||
# Restore a deleted secret
|
||||
aws secretsmanager restore-secret \
|
||||
--secret-id myapp/production/old-key
|
||||
|
||||
# Tag a secret
|
||||
aws secretsmanager tag-resource \
|
||||
--secret-id myapp/production/database \
|
||||
--tags '[{"Key":"RotationEnabled","Value":"true"}]'
|
||||
```
|
||||
|
||||
## Automatic Rotation
|
||||
|
||||
### Enable Rotation
|
||||
|
||||
```bash
|
||||
# Enable rotation with Lambda
|
||||
# Enable rotation with an existing Lambda function
|
||||
aws secretsmanager rotate-secret \
|
||||
--secret-id myapp/database \
|
||||
--rotation-lambda-arn arn:aws:lambda:region:account:function:rotation-function \
|
||||
--rotation-rules AutomaticallyAfterDays=30
|
||||
--secret-id myapp/production/database \
|
||||
--rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRDSPostgreSQLRotation \
|
||||
--rotation-rules '{"AutomaticallyAfterDays":30,"ScheduleExpression":"rate(30 days)"}'
|
||||
|
||||
# Trigger immediate rotation
|
||||
aws secretsmanager rotate-secret \
|
||||
--secret-id myapp/production/database
|
||||
|
||||
# Check rotation status
|
||||
aws secretsmanager describe-secret \
|
||||
--secret-id myapp/production/database \
|
||||
--query '{RotationEnabled:RotationEnabled,RotationLambdaARN:RotationLambdaARN,RotationRules:RotationRules,LastRotatedDate:LastRotatedDate}'
|
||||
```
|
||||
|
||||
### Lambda Rotation Function
|
||||
|
||||
```python
|
||||
"""rotation_function.py - Custom rotation Lambda for database credentials."""
|
||||
|
||||
import boto3
|
||||
import json
|
||||
import logging
|
||||
import psycopg2
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""Secrets Manager rotation handler.
|
||||
|
||||
The rotation process has four steps:
|
||||
1. createSecret - Generate new secret value
|
||||
2. setSecret - Apply the new secret to the target service
|
||||
3. testSecret - Verify the new secret works
|
||||
4. finishSecret - Mark rotation complete
|
||||
"""
|
||||
secret_arn = event['SecretId']
|
||||
token = event['ClientRequestToken']
|
||||
step = event['Step']
|
||||
|
||||
client = boto3.client('secretsmanager')
|
||||
|
||||
metadata = client.describe_secret(SecretId=secret_arn)
|
||||
if not metadata.get('RotationEnabled'):
|
||||
raise ValueError(f"Secret {secret_arn} does not have rotation enabled")
|
||||
|
||||
versions = metadata.get('VersionIdsToStages', {})
|
||||
if token not in versions:
|
||||
raise ValueError(f"Secret version {token} has no stage for rotation")
|
||||
|
||||
if step == "createSecret":
|
||||
create_secret(client, secret_arn, token)
|
||||
elif step == "setSecret":
|
||||
set_secret(client, secret_arn, token)
|
||||
elif step == "testSecret":
|
||||
test_secret(client, secret_arn, token)
|
||||
elif step == "finishSecret":
|
||||
finish_secret(client, secret_arn, token)
|
||||
else:
|
||||
raise ValueError(f"Invalid step: {step}")
|
||||
|
||||
|
||||
def create_secret(client, secret_arn, token):
|
||||
"""Generate a new secret value."""
|
||||
current = client.get_secret_value(
|
||||
SecretId=secret_arn, VersionStage="AWSCURRENT"
|
||||
)
|
||||
current_dict = json.loads(current['SecretString'])
|
||||
|
||||
new_password = client.get_random_password(
|
||||
PasswordLength=32,
|
||||
ExcludeCharacters='/@"\\',
|
||||
RequireEachIncludedType=True,
|
||||
)['RandomPassword']
|
||||
|
||||
current_dict['password'] = new_password
|
||||
client.put_secret_value(
|
||||
SecretId=secret_arn,
|
||||
ClientRequestToken=token,
|
||||
SecretString=json.dumps(current_dict),
|
||||
VersionStages=['AWSPENDING'],
|
||||
)
|
||||
logger.info(f"createSecret: New secret version created for {secret_arn}")
|
||||
|
||||
|
||||
def set_secret(client, secret_arn, token):
|
||||
"""Apply the new secret to the target database."""
|
||||
pending = client.get_secret_value(
|
||||
SecretId=secret_arn, VersionId=token, VersionStage="AWSPENDING"
|
||||
)
|
||||
pending_dict = json.loads(pending['SecretString'])
|
||||
|
||||
current = client.get_secret_value(
|
||||
SecretId=secret_arn, VersionStage="AWSCURRENT"
|
||||
)
|
||||
current_dict = json.loads(current['SecretString'])
|
||||
|
||||
conn = psycopg2.connect(
|
||||
host=current_dict['host'],
|
||||
port=current_dict.get('port', 5432),
|
||||
user=current_dict['username'],
|
||||
password=current_dict['password'],
|
||||
dbname=current_dict.get('dbname', 'postgres'),
|
||||
)
|
||||
conn.autocommit = True
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"ALTER USER %s WITH PASSWORD %s",
|
||||
(pending_dict['username'], pending_dict['password']),
|
||||
)
|
||||
conn.close()
|
||||
logger.info(f"setSecret: Password updated in database for {secret_arn}")
|
||||
|
||||
|
||||
def test_secret(client, secret_arn, token):
|
||||
"""Verify the new secret works."""
|
||||
pending = client.get_secret_value(
|
||||
SecretId=secret_arn, VersionId=token, VersionStage="AWSPENDING"
|
||||
)
|
||||
pending_dict = json.loads(pending['SecretString'])
|
||||
|
||||
conn = psycopg2.connect(
|
||||
host=pending_dict['host'],
|
||||
port=pending_dict.get('port', 5432),
|
||||
user=pending_dict['username'],
|
||||
password=pending_dict['password'],
|
||||
dbname=pending_dict.get('dbname', 'postgres'),
|
||||
)
|
||||
conn.close()
|
||||
logger.info(f"testSecret: New credentials verified for {secret_arn}")
|
||||
|
||||
|
||||
def finish_secret(client, secret_arn, token):
|
||||
"""Finalize the rotation by updating version stages."""
|
||||
metadata = client.describe_secret(SecretId=secret_arn)
|
||||
versions = metadata.get('VersionIdsToStages', {})
|
||||
|
||||
current_version = None
|
||||
for version_id, stages in versions.items():
|
||||
if "AWSCURRENT" in stages:
|
||||
if version_id == token:
|
||||
logger.info("finishSecret: Version already marked AWSCURRENT")
|
||||
return
|
||||
current_version = version_id
|
||||
break
|
||||
|
||||
client.update_secret_version_stage(
|
||||
SecretId=secret_arn,
|
||||
VersionStage="AWSCURRENT",
|
||||
MoveToVersionId=token,
|
||||
RemoveFromVersionId=current_version,
|
||||
)
|
||||
logger.info(f"finishSecret: Rotation complete for {secret_arn}")
|
||||
```
|
||||
|
||||
### Rotation Lambda Terraform
|
||||
|
||||
```hcl
|
||||
resource "aws_lambda_function" "rotation" {
|
||||
filename = "rotation_function.zip"
|
||||
function_name = "secrets-rotation-postgresql"
|
||||
role = aws_iam_role.rotation.arn
|
||||
handler = "rotation_function.lambda_handler"
|
||||
runtime = "python3.11"
|
||||
timeout = 60
|
||||
|
||||
vpc_config {
|
||||
subnet_ids = var.private_subnet_ids
|
||||
security_group_ids = [aws_security_group.rotation.id]
|
||||
}
|
||||
|
||||
environment {
|
||||
variables = {
|
||||
SECRETS_MANAGER_ENDPOINT = "https://secretsmanager.${var.region}.amazonaws.com"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_lambda_permission" "secrets_manager" {
|
||||
action = "lambda:InvokeFunction"
|
||||
function_name = aws_lambda_function.rotation.function_name
|
||||
principal = "secretsmanager.amazonaws.com"
|
||||
statement_id = "AllowSecretsManager"
|
||||
}
|
||||
|
||||
resource "aws_secretsmanager_secret_rotation" "db" {
|
||||
secret_id = aws_secretsmanager_secret.db.id
|
||||
rotation_lambda_arn = aws_lambda_function.rotation.arn
|
||||
rotation_rules {
|
||||
automatically_after_days = 30
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Application Integration
|
||||
|
||||
### Python SDK
|
||||
|
||||
```python
|
||||
import boto3
|
||||
import json
|
||||
from functools import lru_cache
|
||||
|
||||
def get_secret(secret_name):
|
||||
client = boto3.client('secretsmanager')
|
||||
def get_secret(secret_name: str, region: str = "us-east-1") -> dict:
|
||||
"""Retrieve and parse a secret from AWS Secrets Manager."""
|
||||
client = boto3.client("secretsmanager", region_name=region)
|
||||
response = client.get_secret_value(SecretId=secret_name)
|
||||
return json.loads(response['SecretString'])
|
||||
if "SecretString" in response:
|
||||
return json.loads(response["SecretString"])
|
||||
else:
|
||||
import base64
|
||||
return base64.b64decode(response["SecretBinary"])
|
||||
|
||||
@lru_cache(maxsize=32)
|
||||
def get_cached_secret(secret_name: str) -> dict:
|
||||
"""Cached secret retrieval. Clear cache on rotation events."""
|
||||
return get_secret(secret_name)
|
||||
|
||||
# Usage
|
||||
creds = get_secret('myapp/database')
|
||||
db_connect(creds['username'], creds['password'])
|
||||
creds = get_secret("myapp/production/database")
|
||||
connection_string = (
|
||||
f"postgresql://{creds['username']}:{creds['password']}"
|
||||
f"@{creds['host']}:{creds['port']}/{creds['dbname']}"
|
||||
)
|
||||
```
|
||||
|
||||
### ECS Task Definition
|
||||
|
||||
```json
|
||||
{
|
||||
"containerDefinitions": [
|
||||
{
|
||||
"name": "myapp",
|
||||
"image": "ghcr.io/acme/myapp:v1.0.0",
|
||||
"secrets": [
|
||||
{
|
||||
"name": "DB_USERNAME",
|
||||
"valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:myapp/production/database:username::"
|
||||
},
|
||||
{
|
||||
"name": "DB_PASSWORD",
|
||||
"valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:myapp/production/database:password::"
|
||||
},
|
||||
{
|
||||
"name": "API_KEY",
|
||||
"valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:myapp/production/api-key"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"executionRoleArn": "arn:aws:iam::123456789:role/ecsTaskExecutionRole"
|
||||
}
|
||||
```
|
||||
|
||||
### EKS with External Secrets Operator
|
||||
|
||||
```yaml
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: SecretStore
|
||||
metadata:
|
||||
name: aws-secrets-manager
|
||||
namespace: production
|
||||
spec:
|
||||
provider:
|
||||
aws:
|
||||
service: SecretsManager
|
||||
region: us-east-1
|
||||
auth:
|
||||
jwt:
|
||||
serviceAccountRef:
|
||||
name: external-secrets-sa
|
||||
---
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: db-credentials
|
||||
namespace: production
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: aws-secrets-manager
|
||||
kind: SecretStore
|
||||
target:
|
||||
name: db-credentials
|
||||
creationPolicy: Owner
|
||||
data:
|
||||
- secretKey: username
|
||||
remoteRef:
|
||||
key: myapp/production/database
|
||||
property: username
|
||||
- secretKey: password
|
||||
remoteRef:
|
||||
key: myapp/production/database
|
||||
property: password
|
||||
```
|
||||
|
||||
## Resource-Based Policy
|
||||
|
||||
```bash
|
||||
# Restrict secret access to specific roles
|
||||
aws secretsmanager put-resource-policy \
|
||||
--secret-id myapp/production/database \
|
||||
--resource-policy '{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": {
|
||||
"AWS": [
|
||||
"arn:aws:iam::123456789:role/myapp-ecs-task-role",
|
||||
"arn:aws:iam::123456789:role/myapp-lambda-role"
|
||||
]
|
||||
},
|
||||
"Action": [
|
||||
"secretsmanager:GetSecretValue"
|
||||
],
|
||||
"Resource": "*",
|
||||
"Condition": {
|
||||
"StringEquals": {
|
||||
"aws:RequestedRegion": "us-east-1"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "secretsmanager:GetSecretValue",
|
||||
"Resource": "*",
|
||||
"Condition": {
|
||||
"StringNotEquals": {
|
||||
"aws:PrincipalAccount": "123456789012"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Cause | Solution |
|
||||
|---------|-------|----------|
|
||||
| `AccessDeniedException` on GetSecretValue | IAM policy missing permission | Add `secretsmanager:GetSecretValue` to the role; check resource-based policy |
|
||||
| Rotation fails with Lambda timeout | Lambda cannot reach database | Ensure Lambda is in same VPC with route to DB; check security groups |
|
||||
| Secret value is empty after rotation | createSecret step failed | Check Lambda CloudWatch logs; verify random password generation works |
|
||||
| ECS container fails to start | Secret ARN format incorrect | Use full ARN with `::` for JSON key extraction; verify secret exists |
|
||||
| Application uses old credentials after rotation | Client caching stale values | Implement cache invalidation on rotation; reduce cache TTL |
|
||||
| Rotation Lambda permission error | Missing `lambda:InvokeFunction` permission | Add `aws_lambda_permission` for secretsmanager.amazonaws.com principal |
|
||||
| KMS decrypt fails | Secret KMS key policy missing role | Add the accessing role to the KMS key policy's `kms:Decrypt` principals |
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Enable automatic rotation
|
||||
- Use resource-based policies
|
||||
- Enable encryption with KMS
|
||||
- Implement least-privilege access
|
||||
- Use versioning for rollback
|
||||
- Enable automatic rotation with 30-day intervals minimum
|
||||
- Use resource-based policies in addition to IAM policies (defense in depth)
|
||||
- Encrypt secrets with customer-managed KMS keys (not default)
|
||||
- Implement least-privilege access (only the roles that need each secret)
|
||||
- Use secret versioning for safe rollback during rotation issues
|
||||
- Monitor secret access with CloudTrail and alert on unusual patterns
|
||||
- Structure secret names hierarchically: `{app}/{env}/{secret-type}`
|
||||
- Never log secret values; log only secret ARNs and access metadata
|
||||
- Test rotation in staging before enabling in production
|
||||
- Set up CloudWatch alarms for rotation failures
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [hashicorp-vault](../hashicorp-vault/) - Multi-cloud secrets
|
||||
- [aws-iam](../../../infrastructure/cloud-aws/aws-iam/) - IAM policies
|
||||
- [azure-keyvault](../azure-keyvault/) - Azure secret management
|
||||
- [gcp-secret-manager](../gcp-secret-manager/) - GCP secret management
|
||||
|
||||
Reference in New Issue
Block a user