DevOps-Security-Agent-Skills/infrastructure/servers/ssh-configuration/SKILL.md

---
name: ssh-configuration
description: Configure SSH servers and clients securely. Manage keys, tunnels, and config files. Use when setting up secure remote access.
license: MIT
metadata:
  author: devops-skills
  version: "1.0"
---

# SSH Configuration

Secure SSH server and client configuration for production environments, including key management, hardened sshd settings, bastion host architecture, tunneling, and multiplexing.

## When to Use

- Setting up secure remote access to Linux or Unix servers
- Hardening SSH daemon configuration to meet compliance requirements
- Configuring bastion / jump hosts for private network access
- Creating SSH tunnels for secure port forwarding
- Managing SSH keys for teams or automated deployments
- Troubleshooting connection, authentication, or performance issues

## Prerequisites

- OpenSSH client installed locally (`ssh -V` to verify)
- OpenSSH server installed on target (`sshd`)
- Root or sudo access on the server for sshd_config changes
- Firewall rules allowing TCP port 22 (or custom SSH port)

## Key Generation and Management

```bash
# Generate an Ed25519 key (recommended -- fast, secure, short)
ssh-keygen -t ed25519 -C "jane@example.com" -f ~/.ssh/id_ed25519

# Generate an RSA 4096-bit key (for legacy compatibility)
ssh-keygen -t rsa -b 4096 -C "jane@example.com" -f ~/.ssh/id_rsa_legacy

# Generate a key with a custom comment and no passphrase (CI/CD use only)
ssh-keygen -t ed25519 -C "ci-deploy-key" -f ~/.ssh/ci_deploy -N ""

# Copy public key to a remote server
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server

# Manually append a public key (when ssh-copy-id is unavailable)
cat ~/.ssh/id_ed25519.pub | ssh user@server "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

# List fingerprints of keys on the agent
ssh-add -l

# Start the SSH agent and add a key
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

# Add a key with a lifetime (auto-removed after 8 hours)
ssh-add -t 28800 ~/.ssh/id_ed25519

# Remove all keys from the agent
ssh-add -D

# Convert an OpenSSH key to PEM format (for tools that need it)
ssh-keygen -p -m PEM -f ~/.ssh/id_rsa_legacy

# Show the public key fingerprint (SHA256)
ssh-keygen -lf ~/.ssh/id_ed25519.pub

# Rotate a key: generate new, deploy, then revoke old
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_new -C "jane@example.com rotated $(date +%Y-%m)"
ssh-copy-id -i ~/.ssh/id_ed25519_new.pub user@server
# After verifying the new key works, remove the old public key from authorized_keys on the server
```

## SSH Client Configuration (~/.ssh/config)

```text
# Global defaults applied to all hosts
Host *
  AddKeysToAgent yes
  IdentitiesOnly yes
  ServerAliveInterval 60
  ServerAliveCountMax 3
  TCPKeepAlive yes
  Compression yes

# Production servers via bastion
Host bastion
  HostName bastion.example.com
  User ops
  IdentityFile ~/.ssh/id_ed25519
  Port 22

Host prod-web-*
  User deploy
  IdentityFile ~/.ssh/id_ed25519
  ProxyJump bastion
  Port 22

Host prod-web-1
  HostName 10.0.1.10

Host prod-web-2
  HostName 10.0.1.11

# Staging accessed directly
Host staging
  HostName staging.example.com
  User deploy
  IdentityFile ~/.ssh/id_ed25519_staging

# Database tunnel through bastion
Host db-tunnel
  HostName 10.0.2.50
  User dba
  ProxyJump bastion
  LocalForward 5432 localhost:5432

# GitHub deploy key
Host github-deploy
  HostName github.com
  User git
  IdentityFile ~/.ssh/github_deploy_key
  IdentitiesOnly yes

# Connection multiplexing for faster repeated connections
Host fast-*
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 600
```

```bash
# Create the sockets directory for multiplexing
mkdir -p ~/.ssh/sockets
chmod 700 ~/.ssh/sockets
```

## Hardened Server Configuration (/etc/ssh/sshd_config)

```bash
# /etc/ssh/sshd_config -- hardened configuration
# -----------------------------------------------

# Listen on a non-default port (obscurity, not security -- combine with firewall)
Port 22

# Protocol and key exchange
Protocol 2
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

# Authentication
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
MaxAuthTries 3
MaxSessions 5
LoginGraceTime 30

# Restrict users and groups
AllowGroups ssh-users ops-team
# AllowUsers deploy admin

# Disable unused authentication methods
ChallengeResponseAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no

# Forwarding controls
AllowTcpForwarding yes
AllowAgentForwarding no
X11Forwarding no
PermitTunnel no

# Security hardening
ClientAliveInterval 300
ClientAliveCountMax 2
UsePAM yes
UseDNS no
PermitEmptyPasswords no
PermitUserEnvironment no

# Logging
SyslogFacility AUTH
LogLevel VERBOSE

# SFTP subsystem
Subsystem sftp /usr/lib/openssh/sftp-server -f AUTH -l INFO

# Match block: restrict deploy user to SFTP only
Match User sftponly
  ForceCommand internal-sftp
  ChrootDirectory /home/%u
  AllowTcpForwarding no
  AllowAgentForwarding no
  X11Forwarding no
```

```bash
# Validate configuration before restarting
sshd -t

# Restart sshd to apply changes
systemctl restart sshd

# Always keep an existing session open while testing
# Open a NEW terminal to verify you can still connect before closing the old one
```

## Bastion Host Setup

```bash
# On the bastion server, restrict forwarding to internal subnets only
# /etc/ssh/sshd_config addition on bastion:
AllowTcpForwarding yes
PermitOpen 10.0.0.0/8:22 10.0.0.0/8:5432

# Disable shell access for jump-only users
Match User jump-user
  PermitTTY no
  ForceCommand /usr/sbin/nologin
  AllowTcpForwarding yes

# Connect through the bastion from a client in one command
ssh -J ops@bastion.example.com deploy@10.0.1.10

# Equivalent using ProxyCommand (older SSH versions)
ssh -o ProxyCommand="ssh -W %h:%p ops@bastion.example.com" deploy@10.0.1.10

# Multi-hop: client -> bastion -> app-server -> db-server
ssh -J ops@bastion,deploy@10.0.1.10 dba@10.0.2.50
```

## SSH Tunneling

```bash
# Local port forward: access remote service on localhost
# Access remote PostgreSQL (10.0.2.50:5432) via bastion at localhost:5432
ssh -L 5432:10.0.2.50:5432 ops@bastion.example.com -N

# Remote port forward: expose local service to the remote network
# Make local dev server (localhost:3000) available on server port 8080
ssh -R 8080:localhost:3000 user@server -N

# Dynamic SOCKS proxy: route all traffic through the server
ssh -D 1080 user@server -N
# Then configure browser or apps to use SOCKS5 proxy at localhost:1080

# Tunnel with a background process
ssh -fN -L 5432:10.0.2.50:5432 ops@bastion.example.com
# Find and kill the tunnel later
ps aux | grep "ssh -fN" | grep -v grep
kill <pid>

# Autossh for persistent tunnels (auto-reconnects)
autossh -M 0 -f -N -L 5432:10.0.2.50:5432 ops@bastion.example.com \
  -o "ServerAliveInterval=30" -o "ServerAliveCountMax=3"
```

## Agent Forwarding (Use with Caution)

```bash
# Enable agent forwarding for a single connection
ssh -A user@bastion

# From the bastion, your local keys are available to authenticate further
ssh deploy@10.0.1.10   # Uses your local key via the agent

# SECURITY WARNING: Agent forwarding exposes your keys to anyone with root
# on the intermediate host. Prefer ProxyJump instead.

# Safer alternative: ProxyJump does not expose the agent
ssh -J ops@bastion deploy@10.0.1.10
```

## SSH Key Restrictions in authorized_keys

```text
# Restrict a key to a specific command only (backup key)
command="/usr/local/bin/run-backup.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-ed25519 AAAA... backup@example.com

# Restrict a key to specific source IPs
from="10.0.0.0/24,192.168.1.0/24" ssh-ed25519 AAAA... admin@example.com

# Read-only SFTP key with chroot
command="internal-sftp",no-port-forwarding,no-pty ssh-ed25519 AAAA... sftp-upload@example.com
```

## Troubleshooting

| Symptom | Diagnostic Command | Common Fix |
|---|---|---|
| Connection refused | `ss -tlnp \| grep 22` on server | Ensure sshd is running; check firewall rules |
| Permission denied (publickey) | `ssh -vvv user@server` | Verify key is in authorized_keys, permissions 600/700 |
| Host key verification failed | `ssh-keygen -R server` | Remove stale host key; verify server identity |
| Connection timeout | `ssh -o ConnectTimeout=5 user@server` | Check network path, security groups, NACLs |
| Slow SSH login | Check `UseDNS` in sshd_config | Set `UseDNS no`; check reverse DNS |
| Broken pipe / dropped sessions | Add `ServerAliveInterval 60` to config | Configure keepalive on both client and server |
| Agent forwarding not working | `ssh-add -l` on bastion | Ensure `-A` flag used and agent has keys loaded |
| Tunnel port already in use | `ss -tlnp \| grep <port>` | Kill existing tunnel or use a different local port |

## Related Skills

- `linux-administration` -- General Linux system administration
- `user-management` -- Managing the users who connect via SSH
- `systemd-services` -- Managing sshd as a systemd service
- `performance-tuning` -- Network tuning for SSH performance