V2

infrastructure/servers/linux-administration/SKILL.md

@@ -9,57 +9,343 @@ metadata:
 
 # Linux Administration
 
-Core Linux system administration skills.
+Core Linux system administration skills for managing production servers, development environments, and infrastructure hosts across Debian/Ubuntu and RHEL/CentOS distributions.
+
+## When to Use
+
+- Provisioning and maintaining Linux servers in any environment
+- Installing, updating, or removing software packages
+- Managing filesystems, disk usage, and mount points
+- Investigating runaway processes or high resource consumption
+- Scheduling recurring tasks with cron or systemd timers
+- Analyzing system and application logs for troubleshooting
+
+## Prerequisites
+
+- Root or sudo access on the target system
+- SSH access configured (see `ssh-configuration` skill)
+- Familiarity with a terminal text editor (vim, nano)
+- Package manager available (`apt` on Debian/Ubuntu, `dnf` on RHEL 8+/Fedora)
 
 ## Package Management
 
-```bash
-# Debian/Ubuntu
-apt update && apt upgrade -y
-apt install nginx
-apt remove nginx
-apt autoremove
+### Debian / Ubuntu (apt)
 
-# RHEL/CentOS
-dnf update
-dnf install nginx
+```bash
+# Update package index and upgrade all installed packages
+apt update && apt upgrade -y
+
+# Search for a package by keyword
+apt search nginx
+
+# Show detailed package info including dependencies
+apt show nginx
+
+# Install a specific version of a package
+apt install nginx=1.24.0-1ubuntu1
+
+# Install multiple packages in one command
+apt install -y nginx certbot python3-certbot-nginx
+
+# Remove a package but keep its config files
+apt remove nginx
+
+# Remove a package and purge all config files
+apt purge nginx
+
+# Remove unused dependency packages
+apt autoremove -y
+
+# List all installed packages
+dpkg -l | grep nginx
+
+# Pin a package to prevent automatic upgrades
+cat <<'EOF' > /etc/apt/preferences.d/pin-nginx
+Package: nginx
+Pin: version 1.24.0-1ubuntu1
+Pin-Priority: 1001
+EOF
+
+# Add an external repository (example: Docker CE)
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
+echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
+  > /etc/apt/sources.list.d/docker.list
+apt update
+```
+
+### RHEL / CentOS / Fedora (dnf)
+
+```bash
+# Update all packages
+dnf update -y
+
+# Search for a package
+dnf search nginx
+
+# Show package details
+dnf info nginx
+
+# Install a package
+dnf install -y nginx
+
+# Install a specific version
+dnf install nginx-1.24.0-1.el9
+
+# Remove a package
 dnf remove nginx
+
+# List installed packages
+dnf list installed | grep nginx
+
+# Enable a module stream (example: Node.js 20)
+dnf module enable nodejs:20
+dnf install -y nodejs
+
+# Add an external repository
+dnf install -y epel-release
+
+# View repository list
+dnf repolist --all
+
+# Clean cached package data
+dnf clean all
 ```
 
 ## System Information
 
 ```bash
-uname -a           # Kernel info
-hostnamectl        # System info
-lscpu              # CPU info
-free -h            # Memory usage
-df -h              # Disk usage
-ip addr            # Network interfaces
+# Kernel and OS release
+uname -a
+cat /etc/os-release
+
+# Hostname and system metadata
+hostnamectl
+
+# CPU information
+lscpu
+nproc                      # Number of processing units
+
+# Memory usage (human-readable)
+free -h
+
+# Disk usage summary
+df -hT                     # Include filesystem type
+du -sh /var/log/*          # Summarize directory sizes
+
+# Network interfaces and IP addresses
+ip addr show
+ip route show              # Routing table
+
+# Uptime and load average
+uptime
+w                          # Who is logged in and load
 ```
 
-## Log Management
+## Filesystem Management
 
 ```bash
-journalctl -u nginx          # Service logs
-journalctl -f                # Follow logs
-tail -f /var/log/syslog      # System logs
-dmesg                        # Kernel messages
+# List block devices and partitions
+lsblk
+fdisk -l
+
+# Create a new ext4 filesystem on a partition
+mkfs.ext4 /dev/sdb1
+
+# Mount a filesystem temporarily
+mount /dev/sdb1 /mnt/data
+
+# Add a persistent mount via fstab
+echo '/dev/sdb1  /mnt/data  ext4  defaults,noatime  0  2' >> /etc/fstab
+mount -a                   # Mount everything in fstab
+
+# Check and repair a filesystem (unmount first)
+umount /dev/sdb1
+fsck.ext4 -y /dev/sdb1
+
+# Monitor disk I/O in real time
+iostat -xz 2
+
+# Find files larger than 100 MB
+find / -xdev -type f -size +100M -exec ls -lh {} \;
+
+# Check inode usage (out-of-inodes can mimic out-of-disk)
+df -i
 ```
 
 ## Process Management
 
 ```bash
-ps aux | grep nginx
-top / htop
+# List all processes with full details
+ps auxf
+
+# Interactive process viewer (prefer htop if installed)
+top
+htop
+
+# Find processes by name
+pgrep -la nginx
+
+# Show process tree
+pstree -p
+
+# Send graceful stop signal (SIGTERM)
+kill <pid>
+
+# Force kill an unresponsive process (SIGKILL)
 kill -9 <pid>
-pgrep nginx
+
+# Kill all processes matching a name
 pkill nginx
+
+# Show open files for a process
+lsof -p <pid>
+
+# Show which process is listening on a port
+ss -tlnp | grep :80
+lsof -i :80
+
+# Run a process immune to hangups (persists after logout)
+nohup /opt/myapp/start.sh > /var/log/myapp.log 2>&1 &
+
+# Limit CPU usage of a running process with cgroups v2
+systemd-run --scope -p CPUQuota=25% --unit=limit-myapp /opt/myapp/start.sh
 ```
 
-## Best Practices
+## Cron Job Management
 
-- Regular updates
-- Minimal installed packages
-- Proper file permissions
-- Log rotation configuration
-- Automated backups
+```bash
+# Edit the current user's crontab
+crontab -e
+
+# List current user's cron jobs
+crontab -l
+
+# Example crontab entries
+# ┌───── minute (0-59)
+# │ ┌───── hour (0-23)
+# │ │ ┌───── day of month (1-31)
+# │ │ │ ┌───── month (1-12)
+# │ │ │ │ ┌───── day of week (0-7, 0 and 7 = Sunday)
+# * * * * * command
+
+# Run a backup every day at 2:30 AM
+30 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1
+
+# Run a cleanup every Sunday at midnight
+0 0 * * 0 /usr/local/bin/cleanup.sh
+
+# Run a health check every 5 minutes
+*/5 * * * * /usr/local/bin/healthcheck.sh
+
+# Place system-wide cron scripts in drop-in directories
+ls /etc/cron.daily/
+ls /etc/cron.weekly/
+
+# Restrict cron access to specific users
+echo "deploy" >> /etc/cron.allow
+```
+
+## Log Management
+
+```bash
+# Follow systemd journal for a specific service
+journalctl -u nginx -f
+
+# Show logs since last boot
+journalctl -b
+
+# Show logs from a specific time range
+journalctl --since "2025-01-15 08:00" --until "2025-01-15 12:00"
+
+# Show only error-level and above
+journalctl -p err
+
+# Tail traditional syslog
+tail -f /var/log/syslog          # Debian/Ubuntu
+tail -f /var/log/messages        # RHEL/CentOS
+
+# Kernel ring buffer messages
+dmesg -T                          # Human-readable timestamps
+dmesg --level=err,warn
+
+# Check disk usage of log directory
+du -sh /var/log/*
+
+# Configure logrotate for a custom application
+cat <<'EOF' > /etc/logrotate.d/myapp
+/var/log/myapp/*.log {
+    daily
+    missingok
+    rotate 14
+    compress
+    delaycompress
+    notifempty
+    create 0640 myapp myapp
+    sharedscripts
+    postrotate
+        systemctl reload myapp > /dev/null 2>&1 || true
+    endscript
+}
+EOF
+
+# Force a logrotate run for testing
+logrotate -f /etc/logrotate.d/myapp
+
+# Centralized logging: forward journal to a remote syslog
+# In /etc/systemd/journal-upload.conf:
+# URL=http://logserver.example.com:19532
+```
+
+## Networking Essentials
+
+```bash
+# Test connectivity
+ping -c 4 8.8.8.8
+
+# DNS lookup
+dig example.com
+nslookup example.com
+
+# Trace route to host
+traceroute example.com
+
+# List listening ports and associated processes
+ss -tlnp
+
+# Show active connections
+ss -tunap
+
+# Firewall management (UFW on Ubuntu)
+ufw allow 22/tcp
+ufw allow 80/tcp
+ufw allow 443/tcp
+ufw enable
+ufw status verbose
+
+# Firewall management (firewalld on RHEL/CentOS)
+firewall-cmd --permanent --add-service=http
+firewall-cmd --permanent --add-service=https
+firewall-cmd --reload
+firewall-cmd --list-all
+```
+
+## Troubleshooting
+
+| Symptom | Diagnostic Command | Common Fix |
+|---|---|---|
+| Disk full | `df -h` and `du -sh /var/log/*` | Clear old logs, run `logrotate -f`, remove temp files |
+| Out of inodes | `df -i` | Delete many small files, check `/tmp` and mail spools |
+| High CPU usage | `top`, `ps aux --sort=-%cpu` | Identify and restart or kill the offending process |
+| High memory / swapping | `free -h`, `vmstat 1` | Tune `vm.swappiness`, add RAM, identify memory leak |
+| Service won't start | `systemctl status <svc>`, `journalctl -u <svc>` | Check config syntax, file permissions, port conflicts |
+| DNS resolution fails | `dig @8.8.8.8 example.com`, `cat /etc/resolv.conf` | Fix nameserver entries, restart `systemd-resolved` |
+| Package dependency error | `apt --fix-broken install` or `dnf distro-sync` | Resolve held or conflicting packages |
+| SSH connection refused | `ss -tlnp \| grep 22`, `systemctl status sshd` | Ensure sshd is running and firewall allows port 22 |
+
+## Related Skills
+
+- `ssh-configuration` -- Secure remote access to Linux servers
+- `user-management` -- Create and manage users, groups, and sudo
+- `systemd-services` -- Write and manage systemd unit files
+- `performance-tuning` -- Kernel and application performance optimization
+- `backup-recovery` -- Protect server data with automated backups