dsivov/environment-troubleshooting

Environment Troubleshooting Expertise

When This Applies

Apply this guidance when:

• Diagnosing system or infrastructure issues
• Resolving permission, network, or resource problems
• Debugging service crashes or connectivity failures
• Analyzing logs for infrastructure issues

Diagnostic Process

Step 1: Gather Information

# System overview
uname -a               # OS and kernel
df -h                  # Disk space
free -h                # Memory usage
top -bn1 | head -20    # CPU and process overview

# Network
ip addr                # Network interfaces
ss -tlnp               # Listening ports
curl -v <url>          # Test connectivity
dig <hostname>         # DNS resolution

# Services
systemctl status <service>  # Service status
journalctl -u <service> -n 50  # Recent logs
docker ps -a           # Container status

Step 2: Identify the Category

| Symptom | Category | Common Causes | |---------|----------|---------------| | Service won't start | Process | Port conflict, missing config, permission | | Connection refused | Network | Firewall, wrong port, service not running | | Permission denied | Access | File permissions, user/group, SELinux | | Out of memory | Resource | Memory leak, insufficient allocation | | Disk full | Resource | Logs, temp files, unrotated data | | Slow response | Performance | CPU saturation, disk I/O, network latency | | DNS failure | Network | DNS config, resolv.conf, network partition |

Step 3: Resolve

Apply the fix, then verify the original issue is resolved.

Common Issues and Solutions

Port Conflicts

# Find what's using a port
ss -tlnp | grep :3000
# or
lsof -i :3000

Solution: Stop the conflicting process or change the port.

Permission Errors

# Check file ownership and permissions
ls -la /path/to/file
# Check the running user
whoami
id

Solution: chmod / chown to grant appropriate access. Never use 777.

Disk Space

# Find largest directories
du -sh /* 2>/dev/null | sort -rh | head -10
# Find large files
find / -type f -size +100M 2>/dev/null

Solution: Clean logs, remove unused images/containers, rotate old data.

Container Issues

# Check container logs
docker logs <container> --tail 100
# Check container resource usage
docker stats --no-stream
# Inspect container
docker inspect <container>

Memory Issues

# Check per-process memory
ps aux --sort=-%mem | head -10
# Check for OOM kills
dmesg | grep -i "out of memory"

Log Analysis

Where to Look

| Service Type | Log Location | |-------------|-------------| | System | /var/log/syslog or journalctl | | Application | Application-specific log dir or stdout | | Docker | docker logs <container> | | Web server | /var/log/nginx/ or /var/log/apache2/ | | Database | DB-specific log directory |

What to Look For

1. Timestamps — When did the problem start?
2. Error patterns — Are errors repeating? Increasing frequency?
3. Stack traces — What component is failing?
4. Resource numbers — Memory usage, connection counts, disk I/O

Escalation

When the issue is outside your scope:

• Code bug causing crashes → Send to Developer via queue
• Architecture causing scaling issues → Send to Architect
• Needs code commit to fix → Send to Integrator after implementing fix
• Production incident → Send to Production Engineer