mirror of
https://github.com/BagelHole/DevOps-Security-Agent-Skills.git
synced 2026-08-22 12:49:53 +02:00
V2
This commit is contained in:
@@ -9,41 +9,260 @@ metadata:
|
||||
|
||||
# GCP Cloud Functions
|
||||
|
||||
Build serverless applications with Cloud Functions.
|
||||
Build and deploy event-driven serverless applications with Google Cloud Functions (Gen1 and Gen2).
|
||||
|
||||
## Deploy Function
|
||||
## When to Use
|
||||
|
||||
- Processing webhooks, API endpoints, or lightweight HTTP backends
|
||||
- Reacting to events from Pub/Sub, Cloud Storage, Firestore, or Eventarc
|
||||
- Running scheduled tasks (cron) without maintaining a server
|
||||
- Building data-processing pipelines triggered by file uploads
|
||||
- Prototyping microservices before committing to Cloud Run or GKE
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Google Cloud SDK (`gcloud`) installed and authenticated
|
||||
- APIs enabled: Cloud Functions, Cloud Build, Artifact Registry, Cloud Run (Gen2)
|
||||
- IAM role `roles/cloudfunctions.developer` (or `roles/run.developer` for Gen2)
|
||||
|
||||
```bash
|
||||
# Deploy HTTP function
|
||||
gcloud functions deploy hello \
|
||||
--runtime=python311 \
|
||||
--trigger-http \
|
||||
--allow-unauthenticated \
|
||||
--entry-point=hello_http
|
||||
|
||||
# Deploy Pub/Sub triggered function
|
||||
gcloud functions deploy process-message \
|
||||
--runtime=python311 \
|
||||
--trigger-topic=my-topic \
|
||||
--entry-point=process
|
||||
gcloud services enable cloudfunctions.googleapis.com cloudbuild.googleapis.com \
|
||||
artifactregistry.googleapis.com run.googleapis.com eventarc.googleapis.com
|
||||
```
|
||||
|
||||
## Function Code
|
||||
## Gen1 vs Gen2 Comparison
|
||||
|
||||
| Feature | Gen1 | Gen2 (recommended) |
|
||||
|---------|------|---------------------|
|
||||
| Runtime | Cloud Functions infra | Built on Cloud Run |
|
||||
| Max timeout | 9 minutes | 60 minutes |
|
||||
| Max memory | 8 GB | 32 GB |
|
||||
| Concurrency | 1 request/instance | Up to 1000/instance |
|
||||
| Traffic splitting | No | Yes |
|
||||
| Eventarc triggers | No | Yes |
|
||||
|
||||
## Deploy an HTTP Function (Gen2)
|
||||
|
||||
```bash
|
||||
# Python HTTP function
|
||||
gcloud functions deploy hello-http \
|
||||
--gen2 --region=us-central1 --runtime=python312 \
|
||||
--trigger-http --allow-unauthenticated \
|
||||
--entry-point=hello_http \
|
||||
--memory=256Mi --timeout=60s \
|
||||
--min-instances=0 --max-instances=100 \
|
||||
--set-env-vars=APP_ENV=production --source=.
|
||||
|
||||
# Node.js HTTP function
|
||||
gcloud functions deploy hello-node \
|
||||
--gen2 --region=us-central1 --runtime=nodejs20 \
|
||||
--trigger-http --allow-unauthenticated \
|
||||
--entry-point=helloNode --memory=256Mi --source=.
|
||||
```
|
||||
|
||||
## Deploy a Pub/Sub Triggered Function
|
||||
|
||||
```bash
|
||||
gcloud pubsub topics create order-events
|
||||
|
||||
gcloud functions deploy process-order \
|
||||
--gen2 --region=us-central1 --runtime=python312 \
|
||||
--trigger-topic=order-events \
|
||||
--entry-point=process_order \
|
||||
--memory=512Mi --timeout=120s --retry \
|
||||
--service-account=order-processor@${PROJECT_ID}.iam.gserviceaccount.com \
|
||||
--source=.
|
||||
```
|
||||
|
||||
## Deploy a Cloud Storage Triggered Function
|
||||
|
||||
```bash
|
||||
gcloud functions deploy process-upload \
|
||||
--gen2 --region=us-central1 --runtime=python312 \
|
||||
--trigger-event-filters="type=google.cloud.storage.object.v1.finalized" \
|
||||
--trigger-event-filters="bucket=my-upload-bucket" \
|
||||
--entry-point=process_upload \
|
||||
--memory=1Gi --timeout=300s --source=.
|
||||
```
|
||||
|
||||
## Deploy a Scheduled Function
|
||||
|
||||
```bash
|
||||
gcloud functions deploy daily-cleanup \
|
||||
--gen2 --region=us-central1 --runtime=python312 \
|
||||
--trigger-http --no-allow-unauthenticated \
|
||||
--entry-point=daily_cleanup --source=.
|
||||
|
||||
gcloud scheduler jobs create http daily-cleanup-job \
|
||||
--schedule="0 2 * * *" \
|
||||
--uri="https://us-central1-${PROJECT_ID}.cloudfunctions.net/daily-cleanup" \
|
||||
--http-method=POST \
|
||||
--oidc-service-account-email=scheduler-sa@${PROJECT_ID}.iam.gserviceaccount.com \
|
||||
--location=us-central1
|
||||
```
|
||||
|
||||
## Python Function Examples
|
||||
|
||||
```python
|
||||
# main.py
|
||||
def hello_http(request):
|
||||
return 'Hello, World!'
|
||||
import functions_framework
|
||||
import base64, json
|
||||
from flask import jsonify
|
||||
from google.cloud import firestore
|
||||
|
||||
def process(event, context):
|
||||
import base64
|
||||
data = base64.b64decode(event['data']).decode('utf-8')
|
||||
print(f"Received: {data}")
|
||||
@functions_framework.http
|
||||
def hello_http(request):
|
||||
"""HTTP Cloud Function."""
|
||||
name = request.args.get("name", "World")
|
||||
return jsonify({"message": f"Hello, {name}!", "status": "ok"}), 200
|
||||
|
||||
@functions_framework.cloud_event
|
||||
def process_order(cloud_event):
|
||||
"""Triggered by a Pub/Sub message."""
|
||||
data = base64.b64decode(cloud_event.data["message"]["data"]).decode("utf-8")
|
||||
order = json.loads(data)
|
||||
db = firestore.Client()
|
||||
db.collection("orders").document(order["id"]).set({
|
||||
"status": "processing", "items": order["items"], "total": order["total"],
|
||||
})
|
||||
|
||||
@functions_framework.cloud_event
|
||||
def process_upload(cloud_event):
|
||||
"""Triggered when a file is uploaded to Cloud Storage."""
|
||||
data = cloud_event.data
|
||||
bucket_name, file_name = data["bucket"], data["name"]
|
||||
if not file_name.lower().endswith((".png", ".jpg", ".jpeg")):
|
||||
return
|
||||
from google.cloud import vision
|
||||
client = vision.ImageAnnotatorClient()
|
||||
image = vision.Image(source=vision.ImageSource(
|
||||
gcs_image_uri=f"gs://{bucket_name}/{file_name}"))
|
||||
labels = [l.description for l in client.label_detection(image=image).label_annotations]
|
||||
print(f"Labels for {file_name}: {labels}")
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
```
|
||||
# requirements.txt
|
||||
functions-framework==3.*
|
||||
google-cloud-firestore==2.*
|
||||
google-cloud-storage==2.*
|
||||
google-cloud-vision==3.*
|
||||
flask>=2.0
|
||||
```
|
||||
|
||||
- Use 2nd gen functions for better performance
|
||||
- Implement proper error handling
|
||||
- Use environment variables for configuration
|
||||
- Monitor with Cloud Logging
|
||||
## Node.js Function Examples
|
||||
|
||||
```javascript
|
||||
// index.js
|
||||
const functions = require("@google-cloud/functions-framework");
|
||||
|
||||
functions.http("helloNode", (req, res) => {
|
||||
const name = req.query.name || "World";
|
||||
res.json({ message: `Hello, ${name}!`, status: "ok" });
|
||||
});
|
||||
|
||||
functions.cloudEvent("processMessage", (cloudEvent) => {
|
||||
const data = Buffer.from(cloudEvent.data.message.data, "base64").toString();
|
||||
console.log(`Processing: ${JSON.parse(data)}`);
|
||||
});
|
||||
```
|
||||
|
||||
## Managing Deployed Functions
|
||||
|
||||
```bash
|
||||
gcloud functions list --gen2 --region=us-central1
|
||||
gcloud functions describe hello-http --gen2 --region=us-central1
|
||||
gcloud functions logs read hello-http --gen2 --region=us-central1 --limit=50
|
||||
gcloud functions delete hello-http --gen2 --region=us-central1 --quiet
|
||||
|
||||
# Update env vars without redeploying code
|
||||
gcloud functions deploy hello-http --gen2 --region=us-central1 \
|
||||
--update-env-vars=APP_ENV=staging
|
||||
|
||||
# Test locally before deploying
|
||||
functions-framework --target=hello_http --port=8080
|
||||
```
|
||||
|
||||
## Terraform Configuration
|
||||
|
||||
```hcl
|
||||
resource "google_cloudfunctions2_function" "api" {
|
||||
name = "hello-http"
|
||||
location = "us-central1"
|
||||
|
||||
build_config {
|
||||
runtime = "python312"
|
||||
entry_point = "hello_http"
|
||||
source {
|
||||
storage_source {
|
||||
bucket = google_storage_bucket.source.name
|
||||
object = google_storage_bucket_object.source.name
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
service_config {
|
||||
min_instance_count = 0
|
||||
max_instance_count = 100
|
||||
available_memory = "256Mi"
|
||||
timeout_seconds = 60
|
||||
service_account_email = google_service_account.fn.email
|
||||
environment_variables = { APP_ENV = "production" }
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_cloud_run_service_iam_member" "invoker" {
|
||||
location = google_cloudfunctions2_function.api.location
|
||||
service = google_cloudfunctions2_function.api.name
|
||||
role = "roles/run.invoker"
|
||||
member = "allUsers"
|
||||
}
|
||||
|
||||
resource "google_cloudfunctions2_function" "processor" {
|
||||
name = "process-order"
|
||||
location = "us-central1"
|
||||
|
||||
build_config {
|
||||
runtime = "python312"
|
||||
entry_point = "process_order"
|
||||
source {
|
||||
storage_source {
|
||||
bucket = google_storage_bucket.source.name
|
||||
object = google_storage_bucket_object.source.name
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
service_config {
|
||||
max_instance_count = 50
|
||||
available_memory = "512Mi"
|
||||
timeout_seconds = 120
|
||||
service_account_email = google_service_account.fn.email
|
||||
}
|
||||
|
||||
event_trigger {
|
||||
trigger_region = "us-central1"
|
||||
event_type = "google.cloud.pubsub.topic.v1.messagePublished"
|
||||
pubsub_topic = google_pubsub_topic.orders.id
|
||||
retry_policy = "RETRY_POLICY_RETRY"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| `PERMISSION_DENIED` on deploy | Missing Cloud Build or Artifact Registry perms | Grant `roles/cloudbuild.builds.builder` to Cloud Build SA |
|
||||
| Function deploys but returns 403 | Missing `roles/run.invoker` for Gen2 | Add `--allow-unauthenticated` or grant invoker role |
|
||||
| Cold start latency > 5s | Large dependencies or no min instances | Set `--min-instances=1`; reduce deps; use lazy imports |
|
||||
| Pub/Sub messages redelivered | Function errors or times out | Increase `--timeout`; fix error handling; add dead-letter topic |
|
||||
| `Build failed` during deploy | Syntax error or missing dependency | Check `gcloud builds log`; verify requirements.txt |
|
||||
| Cannot connect to VPC resource | Function not on VPC connector | Add `--vpc-connector=my-connector` to deploy |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- **gcp-networking** - VPC connectors for accessing private resources from functions
|
||||
- **gcp-cloud-sql** - Connecting Cloud Functions to managed databases
|
||||
- **terraform-gcp** - Deploy Cloud Functions with Infrastructure as Code
|
||||
- **gcp-gke** - When workloads outgrow serverless and need Kubernetes
|
||||
|
||||
@@ -9,41 +9,253 @@ metadata:
|
||||
|
||||
# GCP Cloud SQL
|
||||
|
||||
Deploy managed databases on Google Cloud.
|
||||
Deploy and manage fully managed relational databases (PostgreSQL, MySQL, SQL Server) on Google Cloud.
|
||||
|
||||
## Create Instance
|
||||
## When to Use
|
||||
|
||||
- Running production relational databases without managing replication, patching, or backups
|
||||
- Migrating on-premises PostgreSQL or MySQL workloads to a managed service
|
||||
- Applications requiring ACID transactions, relational schemas, and SQL query support
|
||||
- Workloads that need automated high availability with regional failover
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Google Cloud SDK (`gcloud`) installed and authenticated
|
||||
- Cloud SQL Admin API and Service Networking API enabled
|
||||
- IAM role `roles/cloudsql.admin` for full management
|
||||
|
||||
```bash
|
||||
gcloud sql instances create mydb \
|
||||
--database-version=POSTGRES_15 \
|
||||
--tier=db-f1-micro \
|
||||
--region=us-central1 \
|
||||
--root-password=secretpassword \
|
||||
--storage-auto-increase \
|
||||
--backup-start-time=02:00
|
||||
|
||||
# Create database
|
||||
gcloud sql databases create myapp --instance=mydb
|
||||
|
||||
# Create user
|
||||
gcloud sql users create appuser \
|
||||
--instance=mydb \
|
||||
--password=userpassword
|
||||
gcloud services enable sqladmin.googleapis.com servicenetworking.googleapis.com
|
||||
```
|
||||
|
||||
## High Availability
|
||||
## Instance Tiers Reference
|
||||
|
||||
| Tier | vCPUs | Memory | Use Case |
|
||||
|------|-------|--------|----------|
|
||||
| db-f1-micro | Shared | 0.6 GB | Dev/test only |
|
||||
| db-g1-small | Shared | 1.7 GB | Low-traffic staging |
|
||||
| db-custom-2-8192 | 2 | 8 GB | Small production |
|
||||
| db-custom-4-16384 | 4 | 16 GB | Medium production |
|
||||
| db-custom-8-32768 | 8 | 32 GB | High-traffic production |
|
||||
|
||||
## Create a PostgreSQL Instance
|
||||
|
||||
```bash
|
||||
gcloud sql instances create mydb \
|
||||
--database-version=POSTGRES_15 \
|
||||
--tier=db-custom-2-8192 \
|
||||
gcloud sql instances create prod-db \
|
||||
--database-version=POSTGRES_16 \
|
||||
--tier=db-custom-4-16384 \
|
||||
--region=us-central1 \
|
||||
--availability-type=REGIONAL
|
||||
--availability-type=REGIONAL \
|
||||
--storage-type=SSD --storage-size=100GB --storage-auto-increase \
|
||||
--backup-start-time=02:00 --enable-point-in-time-recovery \
|
||||
--retained-backups-count=14 \
|
||||
--maintenance-window-day=SUN --maintenance-window-hour=4 \
|
||||
--database-flags=max_connections=200,log_min_duration_statement=1000 \
|
||||
--root-password=$(openssl rand -base64 24) \
|
||||
--labels=env=production,team=backend
|
||||
|
||||
gcloud sql databases create myapp --instance=prod-db --charset=UTF8
|
||||
gcloud sql users create appuser --instance=prod-db \
|
||||
--password=$(openssl rand -base64 24)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
## Create a MySQL Instance
|
||||
|
||||
- Enable automated backups
|
||||
- Use Cloud SQL Proxy for connections
|
||||
- Implement private IP
|
||||
- Use read replicas for scaling
|
||||
```bash
|
||||
gcloud sql instances create mysql-prod \
|
||||
--database-version=MYSQL_8_0 \
|
||||
--tier=db-custom-4-16384 --region=us-central1 \
|
||||
--availability-type=REGIONAL \
|
||||
--storage-type=SSD --storage-size=100GB --storage-auto-increase \
|
||||
--backup-start-time=02:00 --enable-bin-log --retained-backups-count=14 \
|
||||
--database-flags=slow_query_log=on,long_query_time=2,max_connections=500 \
|
||||
--root-password=$(openssl rand -base64 24)
|
||||
```
|
||||
|
||||
## Private IP Configuration
|
||||
|
||||
```bash
|
||||
# Allocate IP range and create private connection
|
||||
gcloud compute addresses create google-managed-services \
|
||||
--global --purpose=VPC_PEERING --prefix-length=16 --network=my-vpc
|
||||
|
||||
gcloud services vpc-peerings connect \
|
||||
--service=servicenetworking.googleapis.com \
|
||||
--ranges=google-managed-services --network=my-vpc
|
||||
|
||||
# Create instance with private IP only
|
||||
gcloud sql instances create private-db \
|
||||
--database-version=POSTGRES_16 --tier=db-custom-2-8192 \
|
||||
--region=us-central1 \
|
||||
--network=projects/${PROJECT_ID}/global/networks/my-vpc \
|
||||
--no-assign-ip --availability-type=REGIONAL \
|
||||
--storage-type=SSD --storage-size=50GB --storage-auto-increase
|
||||
```
|
||||
|
||||
## Read Replicas
|
||||
|
||||
```bash
|
||||
# Same-region replica
|
||||
gcloud sql instances create prod-db-replica-1 \
|
||||
--master-instance-name=prod-db --tier=db-custom-4-16384 \
|
||||
--region=us-central1 --availability-type=ZONAL
|
||||
|
||||
# Cross-region replica for DR
|
||||
gcloud sql instances create prod-db-replica-eu \
|
||||
--master-instance-name=prod-db --tier=db-custom-4-16384 \
|
||||
--region=europe-west1 --availability-type=ZONAL
|
||||
|
||||
# Promote a replica to standalone (disaster recovery)
|
||||
gcloud sql instances promote-replica prod-db-replica-eu
|
||||
```
|
||||
|
||||
## Backups and Restore
|
||||
|
||||
```bash
|
||||
gcloud sql backups create --instance=prod-db --description="pre-migration"
|
||||
gcloud sql backups list --instance=prod-db
|
||||
|
||||
# Point-in-time recovery
|
||||
gcloud sql instances clone prod-db prod-db-pitr \
|
||||
--point-in-time="2026-03-23T10:00:00Z"
|
||||
|
||||
# Export / import
|
||||
gcloud sql export sql prod-db gs://my-bucket/export.sql.gz --database=myapp
|
||||
gcloud sql import sql prod-db gs://my-bucket/export.sql.gz --database=myapp
|
||||
```
|
||||
|
||||
## Cloud SQL Auth Proxy
|
||||
|
||||
```bash
|
||||
curl -o cloud-sql-proxy \
|
||||
https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.11.0/cloud-sql-proxy.linux.amd64
|
||||
chmod +x cloud-sql-proxy
|
||||
|
||||
./cloud-sql-proxy ${PROJECT_ID}:us-central1:prod-db --port=5432 --auto-iam-authn
|
||||
|
||||
# Unix socket (for Kubernetes sidecar pattern)
|
||||
./cloud-sql-proxy ${PROJECT_ID}:us-central1:prod-db --unix-socket=/tmp/cloudsql
|
||||
psql "host=/tmp/cloudsql/${PROJECT_ID}:us-central1:prod-db user=appuser dbname=myapp"
|
||||
```
|
||||
|
||||
## Connection Methods Summary
|
||||
|
||||
| Method | Use Case | Requirement |
|
||||
|--------|----------|-------------|
|
||||
| Public IP + SSL | Dev/test access | Authorized networks configured |
|
||||
| Cloud SQL Auth Proxy | Production on GCE/GKE | SA with `roles/cloudsql.client` |
|
||||
| Private IP | VPC-native apps | VPC peering configured |
|
||||
| Cloud SQL Connector lib | App-level integration | SA credentials |
|
||||
|
||||
## Terraform Configuration
|
||||
|
||||
```hcl
|
||||
resource "google_sql_database_instance" "main" {
|
||||
name = "prod-db"
|
||||
database_version = "POSTGRES_16"
|
||||
region = "us-central1"
|
||||
|
||||
settings {
|
||||
tier = "db-custom-4-16384"
|
||||
availability_type = "REGIONAL"
|
||||
disk_type = "PD_SSD"
|
||||
disk_size = 100
|
||||
disk_autoresize = true
|
||||
|
||||
backup_configuration {
|
||||
enabled = true
|
||||
start_time = "02:00"
|
||||
point_in_time_recovery_enabled = true
|
||||
backup_retention_settings { retained_backups = 14 }
|
||||
}
|
||||
|
||||
ip_configuration {
|
||||
ipv4_enabled = false
|
||||
private_network = google_compute_network.vpc.id
|
||||
require_ssl = true
|
||||
}
|
||||
|
||||
maintenance_window { day = 7; hour = 4 }
|
||||
database_flags { name = "max_connections"; value = "200" }
|
||||
|
||||
user_labels = { env = "production" }
|
||||
}
|
||||
|
||||
deletion_protection = true
|
||||
depends_on = [google_service_networking_connection.private_vpc]
|
||||
}
|
||||
|
||||
resource "google_sql_database" "app" {
|
||||
name = "myapp"
|
||||
instance = google_sql_database_instance.main.name
|
||||
}
|
||||
|
||||
resource "google_sql_user" "app" {
|
||||
name = "appuser"
|
||||
instance = google_sql_database_instance.main.name
|
||||
password = random_password.db_password.result
|
||||
}
|
||||
|
||||
resource "google_sql_database_instance" "replica" {
|
||||
name = "prod-db-replica-1"
|
||||
master_instance_name = google_sql_database_instance.main.name
|
||||
region = "us-central1"
|
||||
database_version = "POSTGRES_16"
|
||||
|
||||
replica_configuration { failover_target = false }
|
||||
|
||||
settings {
|
||||
tier = "db-custom-4-16384"
|
||||
disk_type = "PD_SSD"
|
||||
disk_autoresize = true
|
||||
ip_configuration {
|
||||
ipv4_enabled = false
|
||||
private_network = google_compute_network.vpc.id
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_global_address" "private_ip" {
|
||||
name = "google-managed-services"
|
||||
purpose = "VPC_PEERING"
|
||||
address_type = "INTERNAL"
|
||||
prefix_length = 16
|
||||
network = google_compute_network.vpc.id
|
||||
}
|
||||
|
||||
resource "google_service_networking_connection" "private_vpc" {
|
||||
network = google_compute_network.vpc.id
|
||||
service = "servicenetworking.googleapis.com"
|
||||
reserved_peering_ranges = [google_compute_global_address.private_ip.name]
|
||||
}
|
||||
```
|
||||
|
||||
## Common Operations
|
||||
|
||||
```bash
|
||||
gcloud sql instances list
|
||||
gcloud sql instances describe prod-db \
|
||||
--format="yaml(state,settings.tier,settings.availabilityType,ipAddresses)"
|
||||
gcloud sql instances patch prod-db --storage-size=200GB
|
||||
gcloud sql instances patch prod-db --database-flags=max_connections=300
|
||||
gcloud sql instances restart prod-db
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| `Connection refused` via public IP | IP not in authorized networks | Add IP with `gcloud sql instances patch --authorized-networks` |
|
||||
| `SSL required` error | `require_ssl=true` but client not using SSL | Use Cloud SQL Proxy or pass `sslmode=require` |
|
||||
| High replication lag | Replica tier too small or write-heavy primary | Increase replica tier; reduce write load |
|
||||
| Instance slow despite RUNNABLE | Under-provisioned CPU/memory | Scale tier with `gcloud sql instances patch --tier` |
|
||||
| Proxy returns `ECONNREFUSED` | Wrong connection name or missing IAM role | Verify `project:region:instance` format; grant `roles/cloudsql.client` |
|
||||
| Cannot create private IP instance | VPC peering not established | Run `gcloud services vpc-peerings connect` first |
|
||||
| Backup restore fails | Incompatible version | Ensure same major database version between source and target |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- **gcp-networking** - VPC and private service connect for Cloud SQL private IP
|
||||
- **terraform-gcp** - Provision Cloud SQL with Infrastructure as Code
|
||||
- **gcp-gke** - Connecting Kubernetes workloads to Cloud SQL via sidecar proxy
|
||||
- **gcp-compute** - Running applications on Compute Engine that connect to Cloud SQL
|
||||
|
||||
@@ -9,34 +9,295 @@ metadata:
|
||||
|
||||
# GCP Compute Engine
|
||||
|
||||
Deploy and manage Compute Engine instances.
|
||||
Deploy, manage, and scale Compute Engine virtual machines on Google Cloud Platform.
|
||||
|
||||
## Create Instance
|
||||
## When to Use
|
||||
|
||||
- Deploying web servers, application backends, or batch-processing workloads on GCP
|
||||
- Running workloads that need full OS-level control (unlike Cloud Run or App Engine)
|
||||
- Creating managed instance groups for auto-healing and auto-scaling behind a load balancer
|
||||
- Provisioning GPU-attached VMs for ML training or rendering pipelines
|
||||
- Cost-optimizing non-critical workloads with preemptible or spot VMs
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Google Cloud SDK (`gcloud`) installed and authenticated
|
||||
- A GCP project with the Compute Engine API enabled
|
||||
- IAM role `roles/compute.admin` or scoped roles for instance management
|
||||
|
||||
```bash
|
||||
gcloud auth list
|
||||
gcloud config set project $PROJECT_ID
|
||||
gcloud services enable compute.googleapis.com
|
||||
```
|
||||
|
||||
## Machine Types Reference
|
||||
|
||||
| Family | Example | vCPUs | Memory | Use Case |
|
||||
|--------|---------|-------|--------|----------|
|
||||
| E2 | e2-micro | 0.25 | 1 GB | Dev/test, microservices |
|
||||
| E2 | e2-medium | 1 | 4 GB | Light web servers |
|
||||
| N2 | n2-standard-4 | 4 | 16 GB | General-purpose production |
|
||||
| N2 | n2-highmem-8 | 8 | 64 GB | In-memory caches, databases |
|
||||
| C2 | c2-standard-16 | 16 | 64 GB | Compute-intensive, HPC |
|
||||
|
||||
```bash
|
||||
# List machine types available in a zone
|
||||
gcloud compute machine-types list --zones=us-central1-a --filter="name~'e2-'"
|
||||
|
||||
# Create a custom machine type (6 vCPUs, 24 GB RAM)
|
||||
gcloud compute instances create custom-vm \
|
||||
--custom-cpu=6 --custom-memory=24GB \
|
||||
--zone=us-central1-a \
|
||||
--image-family=debian-12 --image-project=debian-cloud
|
||||
```
|
||||
|
||||
## Create an Instance
|
||||
|
||||
```bash
|
||||
# Production instance with shielded VM and startup script
|
||||
gcloud compute instances create web-server \
|
||||
--machine-type=e2-medium \
|
||||
--zone=us-central1-a \
|
||||
--image-family=debian-11 \
|
||||
--image-family=debian-12 \
|
||||
--image-project=debian-cloud \
|
||||
--boot-disk-size=20GB \
|
||||
--tags=http-server
|
||||
--boot-disk-type=pd-balanced \
|
||||
--tags=http-server,https-server \
|
||||
--labels=env=production,team=backend \
|
||||
--metadata=enable-oslogin=TRUE \
|
||||
--shielded-secure-boot \
|
||||
--shielded-vtpm \
|
||||
--shielded-integrity-monitoring
|
||||
|
||||
# Create from instance template
|
||||
gcloud compute instance-templates create web-template \
|
||||
--machine-type=e2-medium \
|
||||
--image-family=debian-11 \
|
||||
--image-project=debian-cloud
|
||||
# Instance with a startup script and service account
|
||||
gcloud compute instances create app-server \
|
||||
--machine-type=e2-standard-2 \
|
||||
--zone=us-central1-a \
|
||||
--image-family=ubuntu-2204-lts \
|
||||
--image-project=ubuntu-os-cloud \
|
||||
--boot-disk-size=50GB \
|
||||
--metadata-from-file=startup-script=startup.sh \
|
||||
--service-account=app-sa@${PROJECT_ID}.iam.gserviceaccount.com \
|
||||
--scopes=cloud-platform
|
||||
|
||||
gcloud compute instance-groups managed create web-group \
|
||||
--template=web-template \
|
||||
--size=3 \
|
||||
--zone=us-central1-a
|
||||
# Instance with an additional data disk
|
||||
gcloud compute instances create db-server \
|
||||
--machine-type=n2-highmem-4 \
|
||||
--zone=us-central1-a \
|
||||
--image-family=debian-12 --image-project=debian-cloud \
|
||||
--boot-disk-size=20GB \
|
||||
--create-disk=name=data-disk,size=200GB,type=pd-ssd,auto-delete=no
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
## Startup Script Example
|
||||
|
||||
- Use managed instance groups
|
||||
- Implement preemptible VMs for cost savings
|
||||
- Use custom images for consistency
|
||||
- Enable shielded VMs
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# startup.sh - runs on first boot and every reboot
|
||||
set -euo pipefail
|
||||
apt-get update && apt-get install -y nginx
|
||||
systemctl enable nginx && systemctl start nginx
|
||||
curl -X PUT -H "Metadata-Flavor: Google" \
|
||||
"http://metadata.google.internal/computeMetadata/v1/instance/guest-attributes/startup/status" \
|
||||
-d "complete"
|
||||
```
|
||||
|
||||
## Instance Templates and Managed Instance Groups
|
||||
|
||||
```bash
|
||||
# Create an instance template
|
||||
gcloud compute instance-templates create web-template \
|
||||
--machine-type=e2-medium \
|
||||
--image-family=debian-12 --image-project=debian-cloud \
|
||||
--boot-disk-size=20GB --tags=http-server \
|
||||
--metadata-from-file=startup-script=startup.sh
|
||||
|
||||
# Create a regional managed instance group (MIG) with health check
|
||||
gcloud compute health-checks create http http-health-check \
|
||||
--port=80 --request-path=/healthz \
|
||||
--check-interval=10s --timeout=5s \
|
||||
--healthy-threshold=2 --unhealthy-threshold=3
|
||||
|
||||
gcloud compute instance-groups managed create web-mig \
|
||||
--template=web-template --size=3 \
|
||||
--region=us-central1 \
|
||||
--health-check=http-health-check --initial-delay=120
|
||||
|
||||
# Configure autoscaling
|
||||
gcloud compute instance-groups managed set-autoscaling web-mig \
|
||||
--region=us-central1 \
|
||||
--min-num-replicas=2 --max-num-replicas=10 \
|
||||
--target-cpu-utilization=0.65 --cool-down-period=90
|
||||
|
||||
# Rolling update to a new template
|
||||
gcloud compute instance-groups managed rolling-action start-update web-mig \
|
||||
--version=template=web-template-v2 \
|
||||
--region=us-central1 --max-surge=3 --max-unavailable=0
|
||||
```
|
||||
|
||||
## Preemptible and Spot VMs
|
||||
|
||||
```bash
|
||||
# Spot VM (recommended over legacy preemptible)
|
||||
gcloud compute instances create spot-worker \
|
||||
--machine-type=n2-standard-8 \
|
||||
--zone=us-central1-a \
|
||||
--image-family=debian-12 --image-project=debian-cloud \
|
||||
--provisioning-model=SPOT \
|
||||
--instance-termination-action=STOP
|
||||
|
||||
# Spot instance template for batch MIG
|
||||
gcloud compute instance-templates create batch-template \
|
||||
--machine-type=n2-standard-4 \
|
||||
--image-family=debian-12 --image-project=debian-cloud \
|
||||
--provisioning-model=SPOT \
|
||||
--instance-termination-action=DELETE
|
||||
```
|
||||
|
||||
## Snapshots and Images
|
||||
|
||||
```bash
|
||||
# Create a snapshot
|
||||
gcloud compute disks snapshot web-server \
|
||||
--zone=us-central1-a \
|
||||
--snapshot-names=web-server-snap-$(date +%Y%m%d)
|
||||
|
||||
# Scheduled snapshot policy
|
||||
gcloud compute resource-policies create snapshot-schedule daily-backup \
|
||||
--region=us-central1 --max-retention-days=14 \
|
||||
--daily-schedule --start-time=03:00
|
||||
|
||||
gcloud compute disks add-resource-policies web-server \
|
||||
--zone=us-central1-a --resource-policies=daily-backup
|
||||
|
||||
# Create a custom image from an instance
|
||||
gcloud compute instances stop web-server --zone=us-central1-a
|
||||
gcloud compute images create web-golden-image \
|
||||
--source-disk=web-server --source-disk-zone=us-central1-a \
|
||||
--family=web-server --labels=version=v1
|
||||
```
|
||||
|
||||
## Terraform Configuration
|
||||
|
||||
```hcl
|
||||
resource "google_compute_instance" "web" {
|
||||
name = "web-server"
|
||||
machine_type = "e2-medium"
|
||||
zone = "us-central1-a"
|
||||
tags = ["http-server", "https-server"]
|
||||
|
||||
boot_disk {
|
||||
initialize_params {
|
||||
image = "debian-cloud/debian-12"
|
||||
size = 20
|
||||
type = "pd-balanced"
|
||||
}
|
||||
}
|
||||
|
||||
network_interface {
|
||||
subnetwork = google_compute_subnetwork.main.id
|
||||
access_config {}
|
||||
}
|
||||
|
||||
metadata_startup_script = file("${path.module}/startup.sh")
|
||||
|
||||
service_account {
|
||||
email = google_service_account.app.email
|
||||
scopes = ["cloud-platform"]
|
||||
}
|
||||
|
||||
shielded_instance_config {
|
||||
enable_secure_boot = true
|
||||
enable_vtpm = true
|
||||
enable_integrity_monitoring = true
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_instance_template" "web" {
|
||||
name_prefix = "web-"
|
||||
machine_type = "e2-medium"
|
||||
|
||||
disk {
|
||||
source_image = "debian-cloud/debian-12"
|
||||
auto_delete = true
|
||||
boot = true
|
||||
disk_size_gb = 20
|
||||
}
|
||||
|
||||
network_interface {
|
||||
subnetwork = google_compute_subnetwork.main.id
|
||||
}
|
||||
|
||||
lifecycle { create_before_destroy = true }
|
||||
}
|
||||
|
||||
resource "google_compute_region_instance_group_manager" "web" {
|
||||
name = "web-mig"
|
||||
base_instance_name = "web"
|
||||
region = "us-central1"
|
||||
|
||||
version {
|
||||
instance_template = google_compute_instance_template.web.id
|
||||
}
|
||||
|
||||
target_size = 3
|
||||
named_port { name = "http"; port = 80 }
|
||||
|
||||
auto_healing_policies {
|
||||
health_check = google_compute_health_check.http.id
|
||||
initial_delay_sec = 120
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_region_autoscaler" "web" {
|
||||
name = "web-autoscaler"
|
||||
region = "us-central1"
|
||||
target = google_compute_region_instance_group_manager.web.id
|
||||
|
||||
autoscaling_policy {
|
||||
min_replicas = 2
|
||||
max_replicas = 10
|
||||
cooldown_period = 90
|
||||
cpu_utilization { target = 0.65 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Common Operations
|
||||
|
||||
```bash
|
||||
# SSH into an instance
|
||||
gcloud compute ssh web-server --zone=us-central1-a
|
||||
|
||||
# List all instances with status
|
||||
gcloud compute instances list \
|
||||
--format="table(name,zone,status,machineType.basename())"
|
||||
|
||||
# Stop / start / resize
|
||||
gcloud compute instances stop web-server --zone=us-central1-a
|
||||
gcloud compute instances set-machine-type web-server \
|
||||
--machine-type=e2-standard-4 --zone=us-central1-a
|
||||
gcloud compute instances start web-server --zone=us-central1-a
|
||||
|
||||
# View serial port output (debug startup scripts)
|
||||
gcloud compute instances get-serial-port-output web-server --zone=us-central1-a
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| Instance stuck in STAGING | Quota exceeded or resource unavailable | Check quota with `gcloud compute project-info describe`; try another zone |
|
||||
| Startup script not running | Syntax errors or wrong metadata key | Check serial output; ensure key is `startup-script` not `startup_script` |
|
||||
| Cannot SSH | Firewall blocks port 22 or OS Login misconfigured | Add firewall rule for `tcp:22`; verify `enable-oslogin` metadata |
|
||||
| Preempted too often | Zone resource pressure | Use Spot VM with `STOP` action; spread across zones in a MIG |
|
||||
| Disk out of space | Boot disk too small | Use `gcloud compute disks resize`; enable `--storage-auto-increase` for data disks |
|
||||
| MIG not healing | Health check misconfigured or initial delay too short | Verify health check path returns 200; increase `--initial-delay` |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- **gcp-networking** - VPC, firewall rules, and load balancers for Compute Engine
|
||||
- **terraform-gcp** - Provision Compute Engine resources with Infrastructure as Code
|
||||
- **gcp-gke** - When workloads are better suited for containers than VMs
|
||||
- **gcp-cloud-sql** - Managed databases that Compute Engine applications connect to
|
||||
|
||||
@@ -7,49 +7,285 @@ metadata:
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# Google Kubernetes Engine
|
||||
# Google Kubernetes Engine (GKE)
|
||||
|
||||
Deploy managed Kubernetes clusters on GCP.
|
||||
Deploy, operate, and scale managed Kubernetes clusters on Google Cloud Platform.
|
||||
|
||||
## Create Cluster
|
||||
## When to Use
|
||||
|
||||
- Running containerized microservices at scale with automatic scaling and healing
|
||||
- Workloads requiring fine-grained orchestration, service mesh, or custom scheduling
|
||||
- Teams already invested in Kubernetes tooling (Helm, Argo CD, Flux)
|
||||
- When Cloud Run's request-based model does not fit (long-running, stateful workloads)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Google Cloud SDK (`gcloud`) and `kubectl` installed
|
||||
- APIs enabled: Kubernetes Engine, Compute Engine
|
||||
- IAM role `roles/container.admin` for cluster management
|
||||
|
||||
```bash
|
||||
gcloud container clusters create my-cluster \
|
||||
--num-nodes=3 \
|
||||
--machine-type=e2-medium \
|
||||
--zone=us-central1-a \
|
||||
--enable-autoscaling \
|
||||
--min-nodes=1 \
|
||||
--max-nodes=5 \
|
||||
--workload-pool=${PROJECT_ID}.svc.id.goog
|
||||
gcloud services enable container.googleapis.com compute.googleapis.com
|
||||
gcloud components install kubectl
|
||||
```
|
||||
|
||||
# Get credentials
|
||||
gcloud container clusters get-credentials my-cluster --zone=us-central1-a
|
||||
## Standard vs Autopilot
|
||||
|
||||
| Feature | Standard | Autopilot |
|
||||
|---------|----------|-----------|
|
||||
| Node management | You manage node pools | Google manages nodes |
|
||||
| Pricing | Pay per node (VM) | Pay per pod resource request |
|
||||
| GPU/TPU | Full support | Supported (with limits) |
|
||||
| DaemonSets | Allowed | Restricted |
|
||||
| Best for | Full control, specialized HW | Hands-off, cost-optimized |
|
||||
|
||||
## Create a Standard Cluster
|
||||
|
||||
```bash
|
||||
gcloud container clusters create prod-cluster \
|
||||
--region=us-central1 --num-nodes=2 \
|
||||
--machine-type=e2-standard-4 --disk-size=100 \
|
||||
--enable-autoscaling --min-nodes=1 --max-nodes=5 \
|
||||
--enable-autorepair --enable-autoupgrade \
|
||||
--release-channel=regular \
|
||||
--workload-pool=${PROJECT_ID}.svc.id.goog \
|
||||
--enable-ip-alias --enable-network-policy \
|
||||
--enable-shielded-nodes \
|
||||
--logging=SYSTEM,WORKLOAD --monitoring=SYSTEM,WORKLOAD \
|
||||
--labels=env=production,team=platform
|
||||
|
||||
gcloud container clusters get-credentials prod-cluster --region=us-central1
|
||||
```
|
||||
|
||||
## Create an Autopilot Cluster
|
||||
|
||||
```bash
|
||||
gcloud container clusters create-auto autopilot-prod \
|
||||
--region=us-central1 --release-channel=regular \
|
||||
--workload-pool=${PROJECT_ID}.svc.id.goog \
|
||||
--network=my-vpc --subnetwork=gke-subnet
|
||||
```
|
||||
|
||||
## Node Pools
|
||||
|
||||
```bash
|
||||
# High-memory pool with taint
|
||||
gcloud container node-pools create highmem-pool \
|
||||
--cluster=prod-cluster --region=us-central1 \
|
||||
--machine-type=n2-highmem-8 --disk-size=200 --disk-type=pd-ssd \
|
||||
--num-nodes=1 --enable-autoscaling --min-nodes=0 --max-nodes=4 \
|
||||
--node-labels=workload=memory-intensive \
|
||||
--node-taints=dedicated=highmem:NoSchedule
|
||||
|
||||
# GPU pool
|
||||
gcloud container node-pools create gpu-pool \
|
||||
--cluster=my-cluster \
|
||||
--zone=us-central1-a \
|
||||
--machine-type=n1-standard-4 \
|
||||
--accelerator=type=nvidia-tesla-k80,count=1 \
|
||||
--num-nodes=1
|
||||
--cluster=prod-cluster --region=us-central1 \
|
||||
--machine-type=n1-standard-8 \
|
||||
--accelerator=type=nvidia-tesla-t4,count=1 \
|
||||
--num-nodes=0 --enable-autoscaling --min-nodes=0 --max-nodes=4 \
|
||||
--node-taints=nvidia.com/gpu=present:NoSchedule
|
||||
|
||||
# Spot pool for batch workloads
|
||||
gcloud container node-pools create spot-pool \
|
||||
--cluster=prod-cluster --region=us-central1 \
|
||||
--machine-type=e2-standard-4 --spot \
|
||||
--num-nodes=0 --enable-autoscaling --min-nodes=0 --max-nodes=20 \
|
||||
--node-taints=cloud.google.com/gke-spot=true:NoSchedule
|
||||
```
|
||||
|
||||
## Workload Identity
|
||||
|
||||
```bash
|
||||
# Create GSA and grant permissions
|
||||
gcloud iam service-accounts create app-gsa
|
||||
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
|
||||
--member="serviceAccount:app-gsa@${PROJECT_ID}.iam.gserviceaccount.com" \
|
||||
--role="roles/storage.objectViewer"
|
||||
|
||||
# Create KSA and bind to GSA
|
||||
kubectl create namespace myapp
|
||||
kubectl create serviceaccount app-ksa --namespace=myapp
|
||||
gcloud iam service-accounts add-iam-policy-binding \
|
||||
app-gsa@${PROJECT_ID}.iam.gserviceaccount.com \
|
||||
--role=roles/iam.workloadIdentityUser \
|
||||
--member="serviceAccount:${PROJECT_ID}.svc.id.goog[NAMESPACE/KSA_NAME]" \
|
||||
GSA_NAME@${PROJECT_ID}.iam.gserviceaccount.com
|
||||
--member="serviceAccount:${PROJECT_ID}.svc.id.goog[myapp/app-ksa]"
|
||||
kubectl annotate serviceaccount app-ksa --namespace=myapp \
|
||||
iam.gke.io/gcp-service-account=app-gsa@${PROJECT_ID}.iam.gserviceaccount.com
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
## Deploying Workloads
|
||||
|
||||
- Use Workload Identity
|
||||
- Enable VPC-native clusters
|
||||
- Implement node auto-provisioning
|
||||
- Use regional clusters for HA
|
||||
```yaml
|
||||
# deployment.yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: web-app
|
||||
namespace: myapp
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels: { app: web-app }
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: web-app }
|
||||
spec:
|
||||
serviceAccountName: app-ksa
|
||||
containers:
|
||||
- name: web
|
||||
image: us-central1-docker.pkg.dev/PROJECT_ID/repo/web-app:v1.2.0
|
||||
ports: [{ containerPort: 8080 }]
|
||||
resources:
|
||||
requests: { cpu: 250m, memory: 512Mi }
|
||||
limits: { cpu: 500m, memory: 1Gi }
|
||||
readinessProbe:
|
||||
httpGet: { path: /healthz, port: 8080 }
|
||||
initialDelaySeconds: 5
|
||||
livenessProbe:
|
||||
httpGet: { path: /healthz, port: 8080 }
|
||||
initialDelaySeconds: 15
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: topology.kubernetes.io/zone
|
||||
whenUnsatisfiable: DoNotSchedule
|
||||
labelSelector:
|
||||
matchLabels: { app: web-app }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata: { name: web-app, namespace: myapp }
|
||||
spec:
|
||||
selector: { app: web-app }
|
||||
ports: [{ port: 80, targetPort: 8080 }]
|
||||
type: ClusterIP
|
||||
```
|
||||
|
||||
## Ingress with Managed SSL
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: web-ingress
|
||||
namespace: myapp
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: "gce"
|
||||
networking.gke.io/managed-certificates: "web-cert"
|
||||
kubernetes.io/ingress.global-static-ip-name: "web-static-ip"
|
||||
spec:
|
||||
rules:
|
||||
- host: app.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service: { name: web-app, port: { number: 80 } }
|
||||
---
|
||||
apiVersion: networking.gke.io/v1
|
||||
kind: ManagedCertificate
|
||||
metadata: { name: web-cert, namespace: myapp }
|
||||
spec:
|
||||
domains: [app.example.com]
|
||||
```
|
||||
|
||||
```bash
|
||||
gcloud compute addresses create web-static-ip --global
|
||||
```
|
||||
|
||||
## Terraform Configuration
|
||||
|
||||
```hcl
|
||||
resource "google_container_cluster" "primary" {
|
||||
name = "prod-cluster"
|
||||
location = "us-central1"
|
||||
|
||||
release_channel { channel = "REGULAR" }
|
||||
workload_identity_config { workload_pool = "${var.project_id}.svc.id.goog" }
|
||||
|
||||
network = google_compute_network.vpc.name
|
||||
subnetwork = google_compute_subnetwork.gke.name
|
||||
|
||||
ip_allocation_policy {
|
||||
cluster_secondary_range_name = "pods"
|
||||
services_secondary_range_name = "services"
|
||||
}
|
||||
|
||||
private_cluster_config {
|
||||
enable_private_nodes = true
|
||||
master_ipv4_cidr_block = "172.16.0.0/28"
|
||||
}
|
||||
|
||||
network_policy { enabled = true }
|
||||
logging_config { enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"] }
|
||||
monitoring_config {
|
||||
enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
|
||||
managed_prometheus { enabled = true }
|
||||
}
|
||||
|
||||
remove_default_node_pool = true
|
||||
initial_node_count = 1
|
||||
}
|
||||
|
||||
resource "google_container_node_pool" "primary" {
|
||||
name = "primary-pool"
|
||||
cluster = google_container_cluster.primary.name
|
||||
location = "us-central1"
|
||||
|
||||
initial_node_count = 2
|
||||
autoscaling { min_node_count = 1; max_node_count = 5 }
|
||||
management { auto_repair = true; auto_upgrade = true }
|
||||
|
||||
node_config {
|
||||
machine_type = "e2-standard-4"
|
||||
disk_size_gb = 100
|
||||
disk_type = "pd-balanced"
|
||||
oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
|
||||
shielded_instance_config {
|
||||
enable_secure_boot = true
|
||||
enable_integrity_monitoring = true
|
||||
}
|
||||
metadata = { disable-legacy-endpoints = "true" }
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_subnetwork" "gke" {
|
||||
name = "gke-subnet"
|
||||
ip_cidr_range = "10.0.0.0/20"
|
||||
region = "us-central1"
|
||||
network = google_compute_network.vpc.id
|
||||
|
||||
secondary_ip_range { range_name = "pods"; ip_cidr_range = "10.4.0.0/14" }
|
||||
secondary_ip_range { range_name = "services"; ip_cidr_range = "10.8.0.0/20" }
|
||||
}
|
||||
```
|
||||
|
||||
## Common Operations
|
||||
|
||||
```bash
|
||||
gcloud container clusters list
|
||||
gcloud container clusters upgrade prod-cluster --region=us-central1 --master
|
||||
kubectl top nodes && kubectl top pods --namespace=myapp
|
||||
kubectl scale deployment web-app --replicas=5 --namespace=myapp
|
||||
kubectl autoscale deployment web-app --namespace=myapp --min=3 --max=20 --cpu-percent=70
|
||||
kubectl logs -f deployment/web-app --namespace=myapp --all-containers
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| Pods stuck in `Pending` | No nodes with enough resources | Check autoscaler; add larger node pool; verify resource requests |
|
||||
| `ImagePullBackOff` | Wrong image path or missing AR access | Verify image URL; grant `roles/artifactregistry.reader` to node SA |
|
||||
| Workload Identity wrong account | KSA annotation missing | Re-annotate KSA; restart pods to pick up new token |
|
||||
| Nodes `NotReady` | Disk/memory pressure or network issue | Run `kubectl describe node`; check taints and conditions |
|
||||
| Ingress returns 502 | Backend pods failing health check | Verify readiness probe; check NEG health in Console |
|
||||
| Cluster create quota error | Insufficient regional CPU/IP quota | Request quota increase in IAM & Admin > Quotas |
|
||||
| Network policy not working | Not enabled on cluster | Recreate with `--enable-network-policy` or use Dataplane V2 |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- **gcp-networking** - VPC, firewall rules, and load balancers for GKE clusters
|
||||
- **terraform-gcp** - Provision GKE clusters with Infrastructure as Code
|
||||
- **gcp-compute** - When workloads are better suited for VMs than containers
|
||||
- **gcp-cloud-sql** - Connecting GKE pods to Cloud SQL via sidecar proxy
|
||||
|
||||
@@ -9,51 +9,269 @@ metadata:
|
||||
|
||||
# GCP Networking
|
||||
|
||||
Design and implement GCP network infrastructure.
|
||||
Design, implement, and secure network infrastructure on Google Cloud Platform.
|
||||
|
||||
## Create VPC
|
||||
## When to Use
|
||||
|
||||
- Building VPC networks for new GCP projects or multi-project architectures
|
||||
- Configuring firewall rules to control traffic between services
|
||||
- Setting up Cloud NAT for outbound internet access from private instances
|
||||
- Deploying load balancers for HTTP(S), TCP/UDP, or internal traffic
|
||||
- Implementing Private Service Connect or Shared VPC
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Google Cloud SDK (`gcloud`) installed and authenticated
|
||||
- Compute Engine API enabled
|
||||
- IAM role `roles/compute.networkAdmin` for network management
|
||||
|
||||
```bash
|
||||
gcloud compute networks create my-vpc --subnet-mode=custom
|
||||
gcloud services enable compute.googleapis.com servicenetworking.googleapis.com
|
||||
```
|
||||
|
||||
gcloud compute networks subnets create my-subnet \
|
||||
--network=my-vpc \
|
||||
--region=us-central1 \
|
||||
--range=10.0.0.0/24
|
||||
## VPC Network Creation
|
||||
|
||||
```bash
|
||||
gcloud compute networks create prod-vpc \
|
||||
--subnet-mode=custom --bgp-routing-mode=regional --mtu=1460
|
||||
|
||||
gcloud compute networks subnets create us-subnet \
|
||||
--network=prod-vpc --region=us-central1 --range=10.0.0.0/20 \
|
||||
--enable-private-ip-google-access --enable-flow-logs \
|
||||
--logging-flow-sampling=0.5
|
||||
|
||||
gcloud compute networks subnets create eu-subnet \
|
||||
--network=prod-vpc --region=europe-west1 --range=10.1.0.0/20 \
|
||||
--enable-private-ip-google-access --enable-flow-logs
|
||||
|
||||
# Subnet with secondary ranges for GKE
|
||||
gcloud compute networks subnets create gke-subnet \
|
||||
--network=prod-vpc --region=us-central1 --range=10.2.0.0/20 \
|
||||
--secondary-range=pods=10.4.0.0/14,services=10.8.0.0/20 \
|
||||
--enable-private-ip-google-access
|
||||
|
||||
# Proxy-only subnet (required for regional L7 LBs)
|
||||
gcloud compute networks subnets create proxy-only-subnet \
|
||||
--network=prod-vpc --region=us-central1 --range=10.129.0.0/23 \
|
||||
--purpose=REGIONAL_MANAGED_PROXY --role=ACTIVE
|
||||
```
|
||||
|
||||
## Firewall Rules
|
||||
|
||||
```bash
|
||||
gcloud compute firewall-rules create allow-http \
|
||||
--network=my-vpc \
|
||||
--allow=tcp:80,tcp:443 \
|
||||
--source-ranges=0.0.0.0/0 \
|
||||
--target-tags=http-server
|
||||
gcloud compute firewall-rules create allow-http-https \
|
||||
--network=prod-vpc --allow=tcp:80,tcp:443 \
|
||||
--source-ranges=0.0.0.0/0 --target-tags=http-server --priority=1000
|
||||
|
||||
gcloud compute firewall-rules create allow-internal \
|
||||
--network=my-vpc \
|
||||
--allow=tcp,udp,icmp \
|
||||
--source-ranges=10.0.0.0/8
|
||||
--network=prod-vpc --allow=tcp,udp,icmp \
|
||||
--source-ranges=10.0.0.0/8 --priority=1000
|
||||
|
||||
gcloud compute firewall-rules create allow-iap-ssh \
|
||||
--network=prod-vpc --allow=tcp:22 \
|
||||
--source-ranges=35.235.240.0/20 --priority=1000
|
||||
|
||||
gcloud compute firewall-rules create allow-health-checks \
|
||||
--network=prod-vpc --allow=tcp:80,tcp:443,tcp:8080 \
|
||||
--source-ranges=130.211.0.0/22,35.191.0.0/16 \
|
||||
--target-tags=http-server --priority=900
|
||||
|
||||
# List firewall rules
|
||||
gcloud compute firewall-rules list --filter="network=prod-vpc" \
|
||||
--format="table(name,direction,priority,allowed[].map().firewall_rule().list():label=ALLOW)"
|
||||
```
|
||||
|
||||
## Cloud NAT
|
||||
|
||||
```bash
|
||||
gcloud compute routers create my-router \
|
||||
--network=my-vpc \
|
||||
--region=us-central1
|
||||
gcloud compute routers create prod-router \
|
||||
--network=prod-vpc --region=us-central1
|
||||
|
||||
gcloud compute routers nats create my-nat \
|
||||
--router=my-router \
|
||||
--region=us-central1 \
|
||||
--nat-all-subnet-ip-ranges \
|
||||
--auto-allocate-nat-external-ips
|
||||
gcloud compute routers nats create prod-nat \
|
||||
--router=prod-router --region=us-central1 \
|
||||
--nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips \
|
||||
--min-ports-per-vm=256 --max-ports-per-vm=4096 \
|
||||
--enable-logging --log-filter=ERRORS_ONLY
|
||||
|
||||
# Static NAT IPs (stable egress)
|
||||
gcloud compute addresses create nat-ip-1 nat-ip-2 --region=us-central1
|
||||
gcloud compute routers nats create prod-nat-static \
|
||||
--router=prod-router --region=us-central1 \
|
||||
--nat-all-subnet-ip-ranges --nat-external-ip-pool=nat-ip-1,nat-ip-2
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
## External HTTP(S) Load Balancer
|
||||
|
||||
- Use Shared VPC for multi-project
|
||||
- Implement Cloud Armor for DDoS
|
||||
- Use Private Google Access
|
||||
- Enable VPC Flow Logs
|
||||
```bash
|
||||
gcloud compute addresses create web-lb-ip --global
|
||||
|
||||
gcloud compute health-checks create http web-hc \
|
||||
--port=80 --request-path=/healthz --check-interval=10s --timeout=5s
|
||||
|
||||
gcloud compute backend-services create web-backend \
|
||||
--protocol=HTTP --port-name=http --health-checks=web-hc \
|
||||
--global --enable-cdn --enable-logging
|
||||
|
||||
gcloud compute backend-services add-backend web-backend \
|
||||
--instance-group=web-mig --instance-group-region=us-central1 \
|
||||
--balancing-mode=UTILIZATION --max-utilization=0.8 --global
|
||||
|
||||
gcloud compute url-maps create web-url-map --default-service=web-backend
|
||||
|
||||
gcloud compute ssl-certificates create web-cert \
|
||||
--domains=app.example.com --global
|
||||
|
||||
gcloud compute target-https-proxies create web-proxy \
|
||||
--url-map=web-url-map --ssl-certificates=web-cert
|
||||
|
||||
gcloud compute forwarding-rules create web-https \
|
||||
--address=web-lb-ip --target-https-proxy=web-proxy --ports=443 --global
|
||||
```
|
||||
|
||||
## Internal Load Balancer
|
||||
|
||||
```bash
|
||||
gcloud compute backend-services create internal-backend \
|
||||
--protocol=TCP --region=us-central1 \
|
||||
--health-checks=web-hc --health-checks-region=us-central1 \
|
||||
--load-balancing-scheme=INTERNAL
|
||||
|
||||
gcloud compute forwarding-rules create internal-lb \
|
||||
--region=us-central1 --load-balancing-scheme=INTERNAL \
|
||||
--network=prod-vpc --subnet=us-subnet \
|
||||
--backend-service=internal-backend --ports=8080
|
||||
```
|
||||
|
||||
## Cloud Armor (DDoS and WAF)
|
||||
|
||||
```bash
|
||||
gcloud compute security-policies create web-armor
|
||||
|
||||
gcloud compute security-policies rules create 1000 \
|
||||
--security-policy=web-armor \
|
||||
--expression="origin.region_code == 'XX'" --action=deny-403
|
||||
|
||||
gcloud compute security-policies rules create 2000 \
|
||||
--security-policy=web-armor --expression="true" \
|
||||
--action=rate-based-ban \
|
||||
--rate-limit-threshold-count=100 \
|
||||
--rate-limit-threshold-interval-sec=60 --ban-duration-sec=600
|
||||
|
||||
gcloud compute backend-services update web-backend \
|
||||
--security-policy=web-armor --global
|
||||
```
|
||||
|
||||
## Private Service Connect
|
||||
|
||||
```bash
|
||||
gcloud compute addresses create psc-google-apis \
|
||||
--global --purpose=PRIVATE_SERVICE_CONNECT \
|
||||
--addresses=10.255.255.254 --network=prod-vpc
|
||||
|
||||
gcloud compute forwarding-rules create psc-google-apis \
|
||||
--global --network=prod-vpc --address=psc-google-apis \
|
||||
--target-google-apis-bundle=all-apis
|
||||
```
|
||||
|
||||
## Shared VPC
|
||||
|
||||
```bash
|
||||
gcloud compute shared-vpc enable $HOST_PROJECT_ID
|
||||
gcloud compute shared-vpc associated-projects add $SERVICE_PROJECT_ID \
|
||||
--host-project=$HOST_PROJECT_ID
|
||||
```
|
||||
|
||||
## Terraform Configuration
|
||||
|
||||
```hcl
|
||||
resource "google_compute_network" "vpc" {
|
||||
name = "prod-vpc"
|
||||
auto_create_subnetworks = false
|
||||
routing_mode = "REGIONAL"
|
||||
}
|
||||
|
||||
resource "google_compute_subnetwork" "us" {
|
||||
name = "us-subnet"
|
||||
ip_cidr_range = "10.0.0.0/20"
|
||||
region = "us-central1"
|
||||
network = google_compute_network.vpc.id
|
||||
private_ip_google_access = true
|
||||
log_config { aggregation_interval = "INTERVAL_5_SEC"; flow_sampling = 0.5 }
|
||||
}
|
||||
|
||||
resource "google_compute_firewall" "allow_http" {
|
||||
name = "allow-http-https"
|
||||
network = google_compute_network.vpc.name
|
||||
allow { protocol = "tcp"; ports = ["80", "443"] }
|
||||
source_ranges = ["0.0.0.0/0"]
|
||||
target_tags = ["http-server"]
|
||||
}
|
||||
|
||||
resource "google_compute_firewall" "allow_iap" {
|
||||
name = "allow-iap-ssh"
|
||||
network = google_compute_network.vpc.name
|
||||
allow { protocol = "tcp"; ports = ["22"] }
|
||||
source_ranges = ["35.235.240.0/20"]
|
||||
}
|
||||
|
||||
resource "google_compute_router" "router" {
|
||||
name = "prod-router"
|
||||
region = "us-central1"
|
||||
network = google_compute_network.vpc.id
|
||||
}
|
||||
|
||||
resource "google_compute_router_nat" "nat" {
|
||||
name = "prod-nat"
|
||||
router = google_compute_router.router.name
|
||||
region = "us-central1"
|
||||
nat_ip_allocate_option = "AUTO_ONLY"
|
||||
source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
|
||||
min_ports_per_vm = 256
|
||||
log_config { enable = true; filter = "ERRORS_ONLY" }
|
||||
}
|
||||
|
||||
resource "google_compute_security_policy" "waf" {
|
||||
name = "web-armor"
|
||||
rule {
|
||||
action = "deny(403)"
|
||||
priority = 1000
|
||||
match { expr { expression = "evaluatePreconfiguredExpr('xss-v33-stable')" } }
|
||||
}
|
||||
rule {
|
||||
action = "allow"
|
||||
priority = 2147483647
|
||||
match { versioned_expr = "SRC_IPS_V1"; config { src_ip_ranges = ["*"] } }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Common Operations
|
||||
|
||||
```bash
|
||||
gcloud compute networks list
|
||||
gcloud compute networks subnets list --network=prod-vpc
|
||||
gcloud compute networks subnets describe us-subnet --region=us-central1
|
||||
gcloud network-management connectivity-tests create test-web-to-db \
|
||||
--source-instance=projects/${PROJECT_ID}/zones/us-central1-a/instances/web \
|
||||
--destination-instance=projects/${PROJECT_ID}/zones/us-central1-a/instances/db \
|
||||
--destination-port=5432 --protocol=TCP
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| Instance cannot reach internet | No external IP and no Cloud NAT | Configure Cloud NAT on the subnet's router |
|
||||
| Firewall rule not taking effect | Wrong target tags or priority | Verify tags match instance; check priority ordering |
|
||||
| Load balancer returns 502 | Backend failing health checks | Check health check path/port; allow `130.211.0.0/22`, `35.191.0.0/16` |
|
||||
| Cannot reach Google APIs from private VM | Private Google Access disabled | Enable `--enable-private-ip-google-access` on subnet |
|
||||
| Cloud NAT port exhaustion | Too many connections per VM | Increase `--min-ports-per-vm`; enable dynamic port allocation |
|
||||
| Shared VPC project cannot create VMs | Missing `compute.networkUser` role | Grant `roles/compute.networkUser` on host project |
|
||||
| SSL cert stuck PROVISIONING | DNS not pointing to LB IP | Update A record to reserved static IP; wait up to 60 min |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- **gcp-compute** - Compute Engine instances that use VPC networks and firewall rules
|
||||
- **gcp-gke** - GKE clusters deployed in VPC subnets with secondary ranges
|
||||
- **gcp-cloud-sql** - Private IP database connectivity through VPC peering
|
||||
- **terraform-gcp** - Provision networking resources with Infrastructure as Code
|
||||
|
||||
@@ -9,58 +9,345 @@ metadata:
|
||||
|
||||
# Terraform GCP
|
||||
|
||||
Provision Google Cloud infrastructure with Terraform.
|
||||
Provision and manage Google Cloud Platform infrastructure using Terraform with the `hashicorp/google` provider.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Defining GCP infrastructure as code for repeatable, auditable deployments
|
||||
- Managing multi-environment setups (dev, staging, production) from a single codebase
|
||||
- Provisioning complex resource graphs (VPC + GKE + Cloud SQL + IAM) in one plan
|
||||
- Integrating infrastructure changes into CI/CD pipelines with plan/apply stages
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Terraform >= 1.5 installed
|
||||
- Google Cloud SDK or a service account key for CI
|
||||
- A GCP project with billing enabled
|
||||
|
||||
```bash
|
||||
gcloud auth application-default login # local dev
|
||||
export GOOGLE_APPLICATION_CREDENTIALS="sa.json" # CI/CD
|
||||
terraform version
|
||||
```
|
||||
|
||||
## Provider Configuration
|
||||
|
||||
```hcl
|
||||
# versions.tf
|
||||
terraform {
|
||||
required_version = ">= 1.5"
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
backend "gcs" {
|
||||
bucket = "tf-state-bucket"
|
||||
prefix = "terraform/state"
|
||||
google = { source = "hashicorp/google"; version = "~> 5.0" }
|
||||
google-beta = { source = "hashicorp/google-beta"; version = "~> 5.0" }
|
||||
}
|
||||
backend "gcs" { bucket = "my-project-tf-state"; prefix = "terraform/state" }
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
provider "google" { project = var.project_id; region = var.region }
|
||||
provider "google-beta" { project = var.project_id; region = var.region }
|
||||
```
|
||||
|
||||
## Example Resources
|
||||
|
||||
```hcl
|
||||
resource "google_compute_network" "vpc" {
|
||||
name = "main-vpc"
|
||||
auto_create_subnetworks = false
|
||||
}
|
||||
|
||||
resource "google_compute_instance" "vm" {
|
||||
name = "web-server"
|
||||
machine_type = "e2-micro"
|
||||
zone = "us-central1-a"
|
||||
|
||||
boot_disk {
|
||||
initialize_params {
|
||||
image = "debian-cloud/debian-11"
|
||||
}
|
||||
}
|
||||
|
||||
network_interface {
|
||||
network = google_compute_network.vpc.name
|
||||
# variables.tf
|
||||
variable "project_id" { type = string }
|
||||
variable "region" { type = string; default = "us-central1" }
|
||||
variable "environment" {
|
||||
type = string
|
||||
validation {
|
||||
condition = contains(["dev", "staging", "production"], var.environment)
|
||||
error_message = "Must be dev, staging, or production."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
## Project Setup and State Bucket
|
||||
|
||||
- Use service accounts for authentication
|
||||
- Store state in GCS
|
||||
- Use labels consistently
|
||||
- Implement least-privilege IAM
|
||||
```bash
|
||||
gcloud storage buckets create gs://my-project-tf-state \
|
||||
--location=us-central1 --uniform-bucket-level-access --public-access-prevention
|
||||
gcloud storage buckets update gs://my-project-tf-state --versioning
|
||||
|
||||
terraform init
|
||||
terraform plan -var="project_id=my-project" -var="environment=production" -out=tfplan
|
||||
terraform apply tfplan
|
||||
```
|
||||
|
||||
```hcl
|
||||
resource "google_project_service" "apis" {
|
||||
for_each = toset([
|
||||
"compute.googleapis.com", "container.googleapis.com",
|
||||
"sqladmin.googleapis.com", "servicenetworking.googleapis.com",
|
||||
"cloudfunctions.googleapis.com", "run.googleapis.com",
|
||||
"secretmanager.googleapis.com", "artifactregistry.googleapis.com",
|
||||
])
|
||||
project = var.project_id
|
||||
service = each.value
|
||||
disable_dependent_services = false
|
||||
disable_on_destroy = false
|
||||
}
|
||||
```
|
||||
|
||||
## Networking Module
|
||||
|
||||
```hcl
|
||||
# modules/networking/main.tf
|
||||
resource "google_compute_network" "vpc" {
|
||||
name = "${var.environment}-vpc"
|
||||
auto_create_subnetworks = false
|
||||
routing_mode = "REGIONAL"
|
||||
}
|
||||
|
||||
resource "google_compute_subnetwork" "main" {
|
||||
name = "${var.environment}-main-subnet"
|
||||
ip_cidr_range = var.subnet_cidr
|
||||
region = var.region
|
||||
network = google_compute_network.vpc.id
|
||||
private_ip_google_access = true
|
||||
log_config { aggregation_interval = "INTERVAL_5_SEC"; flow_sampling = 0.5 }
|
||||
}
|
||||
|
||||
resource "google_compute_subnetwork" "gke" {
|
||||
name = "${var.environment}-gke-subnet"
|
||||
ip_cidr_range = var.gke_subnet_cidr
|
||||
region = var.region
|
||||
network = google_compute_network.vpc.id
|
||||
private_ip_google_access = true
|
||||
secondary_ip_range { range_name = "pods"; ip_cidr_range = var.pods_cidr }
|
||||
secondary_ip_range { range_name = "services"; ip_cidr_range = var.services_cidr }
|
||||
}
|
||||
|
||||
resource "google_compute_firewall" "allow_iap" {
|
||||
name = "${var.environment}-allow-iap"
|
||||
network = google_compute_network.vpc.name
|
||||
allow { protocol = "tcp"; ports = ["22", "3389"] }
|
||||
source_ranges = ["35.235.240.0/20"]
|
||||
}
|
||||
|
||||
resource "google_compute_router" "router" {
|
||||
name = "${var.environment}-router"
|
||||
region = var.region
|
||||
network = google_compute_network.vpc.id
|
||||
}
|
||||
|
||||
resource "google_compute_router_nat" "nat" {
|
||||
name = "${var.environment}-nat"
|
||||
router = google_compute_router.router.name
|
||||
region = var.region
|
||||
nat_ip_allocate_option = "AUTO_ONLY"
|
||||
source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
|
||||
log_config { enable = true; filter = "ERRORS_ONLY" }
|
||||
}
|
||||
|
||||
output "vpc_id" { value = google_compute_network.vpc.id }
|
||||
output "gke_subnet_id" { value = google_compute_subnetwork.gke.id }
|
||||
```
|
||||
|
||||
## GKE Cluster Module
|
||||
|
||||
```hcl
|
||||
# modules/gke/main.tf
|
||||
resource "google_container_cluster" "primary" {
|
||||
name = "${var.environment}-cluster"
|
||||
location = var.region
|
||||
|
||||
release_channel { channel = var.release_channel }
|
||||
workload_identity_config { workload_pool = "${var.project_id}.svc.id.goog" }
|
||||
network = var.vpc_name
|
||||
subnetwork = var.gke_subnet_name
|
||||
|
||||
ip_allocation_policy {
|
||||
cluster_secondary_range_name = "pods"
|
||||
services_secondary_range_name = "services"
|
||||
}
|
||||
private_cluster_config {
|
||||
enable_private_nodes = true
|
||||
master_ipv4_cidr_block = "172.16.0.0/28"
|
||||
}
|
||||
network_policy { enabled = true }
|
||||
logging_config { enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"] }
|
||||
monitoring_config {
|
||||
enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
|
||||
managed_prometheus { enabled = true }
|
||||
}
|
||||
|
||||
remove_default_node_pool = true
|
||||
initial_node_count = 1
|
||||
}
|
||||
|
||||
resource "google_container_node_pool" "primary" {
|
||||
name = "primary-pool"
|
||||
cluster = google_container_cluster.primary.name
|
||||
location = var.region
|
||||
|
||||
initial_node_count = var.initial_node_count
|
||||
autoscaling { min_node_count = var.min_nodes; max_node_count = var.max_nodes }
|
||||
management { auto_repair = true; auto_upgrade = true }
|
||||
|
||||
node_config {
|
||||
machine_type = var.machine_type
|
||||
disk_size_gb = 100
|
||||
oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
|
||||
shielded_instance_config { enable_secure_boot = true; enable_integrity_monitoring = true }
|
||||
metadata = { disable-legacy-endpoints = "true" }
|
||||
}
|
||||
}
|
||||
|
||||
output "cluster_name" { value = google_container_cluster.primary.name }
|
||||
output "cluster_endpoint" { value = google_container_cluster.primary.endpoint; sensitive = true }
|
||||
```
|
||||
|
||||
## Cloud SQL Module
|
||||
|
||||
```hcl
|
||||
# modules/cloud-sql/main.tf
|
||||
resource "google_sql_database_instance" "main" {
|
||||
name = "${var.environment}-db"
|
||||
database_version = var.database_version
|
||||
region = var.region
|
||||
|
||||
settings {
|
||||
tier = var.tier
|
||||
availability_type = var.environment == "production" ? "REGIONAL" : "ZONAL"
|
||||
disk_type = "PD_SSD"
|
||||
disk_size = var.disk_size
|
||||
disk_autoresize = true
|
||||
|
||||
backup_configuration {
|
||||
enabled = true
|
||||
start_time = "02:00"
|
||||
point_in_time_recovery_enabled = true
|
||||
backup_retention_settings { retained_backups = var.environment == "production" ? 30 : 7 }
|
||||
}
|
||||
ip_configuration {
|
||||
ipv4_enabled = false
|
||||
private_network = var.vpc_id
|
||||
require_ssl = true
|
||||
}
|
||||
database_flags { name = "max_connections"; value = var.max_connections }
|
||||
}
|
||||
|
||||
deletion_protection = var.environment == "production"
|
||||
depends_on = [var.private_vpc_connection]
|
||||
}
|
||||
|
||||
resource "google_sql_database" "app" { name = var.database_name; instance = google_sql_database_instance.main.name }
|
||||
resource "google_sql_user" "app" { name = var.db_user; instance = google_sql_database_instance.main.name; password = random_password.db.result }
|
||||
resource "random_password" "db" { length = 32; special = true }
|
||||
|
||||
output "connection_name" { value = google_sql_database_instance.main.connection_name }
|
||||
output "private_ip" { value = google_sql_database_instance.main.private_ip_address }
|
||||
```
|
||||
|
||||
## IAM and Service Accounts
|
||||
|
||||
```hcl
|
||||
resource "google_service_account" "gke_nodes" {
|
||||
account_id = "${var.environment}-gke-nodes"
|
||||
display_name = "GKE Node Pool SA"
|
||||
}
|
||||
|
||||
resource "google_project_iam_member" "gke_nodes" {
|
||||
for_each = toset([
|
||||
"roles/logging.logWriter", "roles/monitoring.metricWriter",
|
||||
"roles/artifactregistry.reader",
|
||||
])
|
||||
project = var.project_id
|
||||
role = each.value
|
||||
member = "serviceAccount:${google_service_account.gke_nodes.email}"
|
||||
}
|
||||
|
||||
resource "google_service_account" "app" {
|
||||
account_id = "${var.environment}-app"
|
||||
display_name = "Application SA"
|
||||
}
|
||||
|
||||
resource "google_service_account_iam_member" "workload_identity" {
|
||||
service_account_id = google_service_account.app.name
|
||||
role = "roles/iam.workloadIdentityUser"
|
||||
member = "serviceAccount:${var.project_id}.svc.id.goog[myapp/app-ksa]"
|
||||
}
|
||||
```
|
||||
|
||||
## Root Module Composition
|
||||
|
||||
```hcl
|
||||
module "networking" {
|
||||
source = "./modules/networking"
|
||||
project_id = var.project_id
|
||||
environment = var.environment
|
||||
region = var.region
|
||||
}
|
||||
|
||||
module "gke" {
|
||||
source = "./modules/gke"
|
||||
project_id = var.project_id
|
||||
environment = var.environment
|
||||
region = var.region
|
||||
vpc_name = module.networking.vpc_id
|
||||
gke_subnet_name = module.networking.gke_subnet_id
|
||||
node_sa_email = google_service_account.gke_nodes.email
|
||||
depends_on = [module.networking]
|
||||
}
|
||||
|
||||
module "database" {
|
||||
source = "./modules/cloud-sql"
|
||||
project_id = var.project_id
|
||||
environment = var.environment
|
||||
region = var.region
|
||||
vpc_id = module.networking.vpc_id
|
||||
database_version = "POSTGRES_16"
|
||||
tier = "db-custom-4-16384"
|
||||
private_vpc_connection = module.networking.private_vpc_connection
|
||||
depends_on = [module.networking]
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
```hcl
|
||||
# environments/production.tfvars
|
||||
project_id = "my-company-prod"
|
||||
environment = "production"
|
||||
region = "us-central1"
|
||||
```
|
||||
|
||||
```bash
|
||||
terraform plan -var-file=environments/production.tfvars -out=tfplan
|
||||
terraform apply tfplan
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
```bash
|
||||
terraform init -input=false
|
||||
terraform validate && terraform fmt -check
|
||||
terraform plan -var-file=environments/${ENV}.tfvars -out=tfplan -input=false
|
||||
terraform apply -input=false tfplan
|
||||
|
||||
# Import existing resources
|
||||
terraform import google_compute_network.vpc projects/${PROJECT_ID}/global/networks/prod-vpc
|
||||
|
||||
# State management
|
||||
terraform state list
|
||||
terraform state mv google_compute_instance.old google_compute_instance.new
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| `Error 403: Access Not Configured` | API not enabled | Add API to `google_project_service` resources |
|
||||
| `Error acquiring the state lock` | Concurrent run or stale lock | Run `terraform force-unlock LOCK_ID` after verification |
|
||||
| `Resource already exists` | Created outside Terraform | Import with `terraform import` |
|
||||
| `Quota exceeded` | Project quota too low | Request increase in Cloud Console > Quotas |
|
||||
| Plan shows destroy/recreate | Changed force-new attribute | Use `moved` blocks or `terraform state mv` |
|
||||
| `Backend initialization required` | Changed backend config | Run `terraform init -migrate-state` |
|
||||
| Cycle in resource graph | Circular references | Refactor with data sources; split applies |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- **gcp-networking** - VPC and firewall resources managed by Terraform
|
||||
- **gcp-gke** - GKE cluster provisioning with Terraform modules
|
||||
- **gcp-cloud-sql** - Cloud SQL instance management via Terraform
|
||||
- **gcp-compute** - Compute Engine resources defined in Terraform
|
||||
- **gcp-cloud-functions** - Serverless function deployment with Terraform
|
||||
|
||||
Reference in New Issue
Block a user