scitix/service-debug

Service Connectivity Diagnosis

When a Service is unreachable, returns connection refused, or has no backends, follow this flow to identify the root cause.

Scope: This skill is for diagnosis only. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify the Service, Endpoints, or pod specs — that should be left to the user.

Diagnostic Flow

1. Check the Service

kubectl get svc <service> -n <ns> -o wide

Confirm the Service exists and note:

• TYPE — ClusterIP, NodePort, LoadBalancer, or ExternalName
• CLUSTER-IP — should not be None unless it is a headless service
• PORT(S) — the port(s) the Service exposes
• SELECTOR — the label selector used to find backend pods

2. Check Endpoints

kubectl get endpoints <service> -n <ns>

• If ENDPOINTS column shows <none> or is empty, no pods match the Service's selector — go to step 3.
• If endpoints exist, note the IPs and ports — go to step 4.

For more detail (including notReadyAddresses):

kubectl describe endpoints <service> -n <ns>

3. Investigate selector and pods

The Service's selector must match labels on running, ready pods.

Get the Service selector:

kubectl get svc <service> -n <ns> -o jsonpath='{.spec.selector}'

List pods that match the selector:

kubectl get pods -n <ns> -l <key>=<value> -o wide

(Replace <key>=<value> with each selector pair from the previous command.)

If no pods match:

• The selector may be wrong (typo, outdated label)
• The pods may be in a different namespace
• The pods may not exist yet

If pods exist but none are Ready:

• Pods may be starting, crashing, or failing readiness probes
• Use pod-crash-debug or check readiness probes

4. Check port configuration

Verify that the Service port, targetPort, and the container's actual listening port all align.

kubectl get svc <service> -n <ns> -o jsonpath='{range .spec.ports[*]}port={.port} targetPort={.targetPort} protocol={.protocol}{"\n"}{end}'

kubectl get pod <backend-pod> -n <ns> -o jsonpath='{range .spec.containers[*].ports[*]}containerPort={.containerPort} protocol={.protocol}{"\n"}{end}'

The Service's targetPort must match one of the container's containerPort values (or the named port). If they don't match, traffic will be sent to the wrong port.

Note: if the container does not declare containerPort, the traffic may still work — containerPort is informational. The real check is whether the application is actually listening on the targetPort.

5. Match patterns and conclude

---

Endpoints empty — No pods match selector

The Service has no endpoints because no running, ready pods match its label selector.

Possible causes:

• Selector mismatch — the Service selector labels don't match the pod labels (typo or outdated after a rename)
• No pods exist — the backing Deployment/StatefulSet has 0 replicas or hasn't been created
• Pods not ready — pods exist but are failing readiness probes, so they are excluded from Endpoints

Advise the user to compare the Service selector with the pod labels and ensure pods are running and ready.

---

Connection refused — Port mismatch or application not listening

Endpoints exist, but connections to the Service are refused.

• The Service targetPort may not match the port the application is actually listening on
• The application may not have started yet or may have crashed

Check the application logs:

kubectl logs <backend-pod> -n <ns> --tail=50

---

Service type ExternalName not resolving — DNS-based service issue

ExternalName services return a CNAME record. If the target hostname is unreachable or doesn't resolve, the service will appear broken.

kubectl get svc <service> -n <ns> -o jsonpath='{.spec.externalName}'

Verify the external name resolves from within the cluster (see dns-debug skill).

---

NodePort / LoadBalancer not reachable from outside — External access issue

For NodePort: verify the node's firewall allows the allocated port.

kubectl get svc <service> -n <ns> -o jsonpath='{.spec.ports[*].nodePort}'

For LoadBalancer: check the external IP assignment.

kubectl get svc <service> -n <ns>

If EXTERNAL-IP shows <pending>, the cloud load balancer provisioner may have failed:

kubectl describe svc <service> -n <ns>

Check events for errors from the cloud controller manager.

---

Headless Service (ClusterIP: None) — Expected behavior

Headless services do not get a cluster IP. DNS returns the individual pod IPs directly. This is by design for StatefulSets and services that need direct pod addressing.

If clients are getting connection refused, verify the individual pod IPs are correct and the pods are ready.

Notes

• For services using sessionAffinity: ClientIP, connections from the same source IP are routed to the same pod — if that pod becomes unhealthy, the session sticks to it until the timeout.
• EndpointSlices (default in K8s 1.21+) replace Endpoints for large-scale services. You can check them with: kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<service>.
• If the cluster uses a service mesh (Istio, Linkerd), traffic routing may be controlled by the mesh — check the mesh's VirtualService or ServiceProfile resources.
• If endpoints exist and ports match but connections still time out, a NetworkPolicy may be blocking traffic — use networkpolicy-debug to check.