aldengolab/k8s-hostnetwork-port-conflict

Kubernetes hostNetwork Port Conflict Debugging

Problem

A pod with hostNetwork: true shares the host's network namespace. If a system-level service on the node already binds the same port, the container fails with "Address already in use" and enters CrashLoopBackOff. The error is misleading because nothing inside Kubernetes appears to hold the port.

Context / Trigger Conditions

• Container logs show failed to create listening socket for port <N>: Address already in use or similar bind errors
• The pod spec includes hostNetwork: true
• kubectl get pods shows CrashLoopBackOff with restarts climbing
• No other pod in the cluster appears to use the conflicting port
• The same container image works in a non-hostNetwork pod

Solution

1. Confirm the port conflict from inside the cluster

From any running container on the same node with hostNetwork (or the crashing pod's surviving sidecar):

kubectl exec -n <ns> <pod> -c <container> -- cat /proc/net/udp   # UDP
kubectl exec -n <ns> <pod> -c <container> -- cat /proc/net/tcp   # TCP

Ports are in hex in column 2 (local_address). Convert: e.g. 0045 = 69. Look for 00000000:<hex-port> (bound on all interfaces).

2. Identify the host process

Use kubectl debug node/ with the sysadmin profile to get root access:

kubectl debug node/<node-name> -it --image=ubuntu --profile=sysadmin -- bash

Inside the debug pod, enter the host namespaces:

nsenter -t 1 -m -u -i -n -p -- ss -tulnp | grep :<port>

This shows the process name and PID holding the port.

Note: Without --profile=sysadmin, nsenter fails with "cannot open /proc/1/ns/ipc: Permission denied".

3. Resolve the conflict

Option A — Stop the host service (if it's redundant):

# systemd-based hosts:
nsenter -t 1 -m -u -i -n -p -- systemctl stop <service>
nsenter -t 1 -m -u -i -n -p -- systemctl disable <service>

# Non-systemd hosts (e.g., minimal k3s distros):
nsenter -t 1 -m -u -i -n -p -- kill <pid>
# Then find and disable the init mechanism (inetd.conf, xinetd.d/, init.d/)

Option B — Change the container's port (if the service must stay): Reconfigure the containerized application to use an alternative port. Note that for protocol-standard ports (TFTP/69, DHCP/67-68), clients may expect the default port, making Option A preferable.

4. Prevent recurrence

• Audit the node for existing services before deploying hostNetwork pods: ss -tulnp | grep :<port> on the node
• For bare-metal/on-prem clusters, document which ports are reserved by host services so future hostNetwork pods avoid them
• Consider whether hostNetwork is truly required; a NodePort or LoadBalancer Service avoids the conflict entirely

Verification

After removing the conflicting service, either wait for CrashLoopBackOff retry or delete the pod to force immediate recreation:

kubectl delete pod -n <ns> <pod-name>

Confirm the new pod reaches Running + Ready with 0 restarts.

Notes

• CrashLoopBackOff exponential backoff can delay retries significantly. Deleting the pod forces an immediate fresh attempt.
• On minimal OS distributions (common with k3s), systemctl may not exist. Fall back to kill and investigate the init system.
• The /proc/net/udp technique works from any container sharing the host network namespace — it doesn't require privileges.