Kubernetes Troubleshooting Runbook: Diagnose Pods, Services, Ingress, and Deployments

Overview

Kubernetes failures often appear at a different layer from their root cause. A deployment may be healthy while its pods fail readiness checks. A service may have a correct selector but no usable endpoints. An ingress may report a route while the backend listens on the wrong port. The safest troubleshooting process follows the request path and separates observation from remediation.

Use this order as a default:

1. Confirm the cluster context and namespace.
2. Inspect workload status and recent events.
3. Check container state, logs, probes, and configuration.
4. Verify scheduling, networking, and service discovery.
5. Apply the smallest reversible change, then verify the result.

Before running commands, record the time, affected namespace, workload name, recent deployment or configuration change, and the user-visible symptom. Avoid deleting a pod or rolling back immediately if you still need its logs and events. A replacement pod can remove useful evidence, and a rollback can hide the change that caused the failure.

Start with cluster and namespace context

kubectl config current-context
kubectl config get-contexts
kubectl get namespaces
kubectl get pods -n <namespace> -o wide
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Use an explicit -n <namespace> on every command while investigating. If the issue may be cluster-wide, repeat the relevant checks across namespaces or ask the platform owner to confirm node, control-plane, and admission-system health. Command availability and output can vary by Kubernetes and client version, so confirm syntax with kubectl help when a command behaves differently than expected.

Checklist by scenario

CrashLoopBackOff or a repeatedly restarting container

CrashLoopBackOff describes a restart pattern, not a single root cause. The process may exit, a probe may kill the container, or a startup dependency may be unavailable.

1. Identify the pod, container, restart count, and last termination reason.
2. Read current and previous container logs.
3. Inspect the pod description and recent events.
4. Compare the command, arguments, environment, mounts, and image with the intended configuration.
5. Check startup, liveness, and readiness probe settings.

kubectl get pod <pod> -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> -c <container>
kubectl logs <pod> -n <namespace> -c <container> --previous
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState}'

Pay attention to exit codes, termination messages, out-of-memory indications, missing files, invalid flags, and connection errors. If the container exits too quickly to inspect interactively, use its image and configuration to reproduce the startup command in a controlled environment rather than repeatedly restarting production workloads.

ImagePullBackOff, ErrImagePull, or an image that will not start

Check the exact image reference first. A tag typo, unavailable digest, private registry permission, missing image-pull secret, or node-to-registry connectivity problem can produce similar symptoms.

kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].image}'
kubectl get serviceaccount <service-account> -n <namespace> -o yaml
kubectl get secret <pull-secret> -n <namespace>

• Compare the image name, registry host, tag, and digest with the release record.
• Confirm that the pod uses the intended service account.
• Verify that the pull secret exists in the same namespace as the pod.
• Check whether other workloads on the same node can pull from the registry.
• Do not paste registry credentials into tickets, shell history, or chat.

If the image was just published, verify that the deployment references the artifact that actually exists. Release and artifact checks should be part of the delivery workflow; the related artifact repository requirements checklist can help structure that review.

Pending pods or pods that are not being scheduled

A pending pod is usually a scheduling, capacity, or admission problem rather than an application problem. The scheduler event is often the most useful first clue.

kubectl get pod <pod> -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>
kubectl get nodes -o wide
kubectl describe node <node>
kubectl get resourcequota -n <namespace>
kubectl get limitrange -n <namespace>

Look for insufficient CPU or memory, node selectors, affinity rules, taints without matching tolerations, volume constraints, quota limits, and unsatisfied topology requirements. Compare the pod's resource requests with allocatable node capacity, not merely current process usage. If a recent manifest added a constraint, temporarily reverting that specific constraint may be safer than changing every node or disabling scheduling safeguards.

Failed readiness, liveness, or startup probes

Probe failures can prevent traffic, restart a healthy-but-slow process, or block startup during dependency initialization. Inspect the probe path, port, scheme, headers, timeout, delay, period, and failure threshold.

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> -c <container>
kubectl exec -n <namespace> <pod> -c <container> -- sh -c 'command -v wget || command -v curl || true'

Test from an appropriate location and with the same port and path used by the probe. A probe that succeeds inside the container does not prove that a service, ingress controller, or external client can reach the application. Confirm that the application binds to the expected interface and that the probe does not depend on an unavailable external system unless that dependency is intentionally part of health semantics.

Service has no traffic or routes to the wrong pods

Trace the path from service to selector to endpoints. A service can exist and have a cluster IP while still selecting no pods or targeting an incorrect port.

kubectl get svc <service> -n <namespace> -o yaml
kubectl describe svc <service> -n <namespace>
kubectl get pods -n <namespace> --show-labels
kubectl get endpoints <service> -n <namespace>
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>

• Compare the service selector with the labels on running pods.
• Compare port and targetPort with the container port and application listener.
• Check whether selected pods are ready; readiness can affect usable endpoints.
• Test DNS and connectivity from a temporary diagnostic pod when permitted.
• Check network policies if selectors and ports are correct but traffic is blocked.

Ingress returns 404, 502, 503, or TLS errors

Separate ingress-controller behavior from backend behavior. First confirm that the ingress object is accepted and associated with the intended controller. Then validate the host, path, service name, service port, and TLS secret.

kubectl get ingress <ingress> -n <namespace> -o yaml
kubectl describe ingress <ingress> -n <namespace>
kubectl get svc <service> -n <namespace>
kubectl get pods -n <controller-namespace> -l <controller-label>
kubectl logs -n <controller-namespace> <controller-pod>

Use the intended host header when testing. A 404 may indicate a host or path mismatch, while a gateway error often points farther downstream at service endpoints or application availability. Confirm the ingress class and controller conventions used by your cluster; do not assume that a manifest written for one controller has identical behavior in another.

What to double-check

• Scope: Are you querying the correct context, cluster, namespace, pod, and container?
• Timeline: Did the first failure occur after an image, manifest, secret, certificate, node, or network change?
• Ownership: Is the object managed by a Deployment, StatefulSet, Job, Helm release, operator, or another controller? Edit the source of truth rather than making a temporary change that the controller will overwrite.
• Configuration: Are referenced ConfigMaps and Secrets present, current, correctly mounted, and in the same namespace?
• Resources: Do requests, limits, quotas, and node capacity support the workload's actual startup and runtime behavior?
• Traffic path: Can you trace client, ingress, service, endpoint, pod, and application port independently?
• Evidence: Have you saved relevant describe output, events, logs, manifests, and timestamps before restarting or deleting anything?

Use labels and field selectors to reduce noisy output, but first confirm that your selectors match the labels actually applied. When collecting diagnostics, remove credentials and sensitive payloads before sharing output.

Common mistakes

• Looking only at application logs: Scheduling failures, image-pull errors, probe failures, and admission problems may appear only in pod descriptions or events.
• Running commands in the wrong namespace: A successful command against an empty or unrelated namespace can create false confidence.
• Deleting first: Deletion may remove the most useful evidence and create a new, different failure.
• Changing several variables at once: Updating the image, probes, resources, and service simultaneously makes verification difficult.
• Ignoring controllers: A manual pod edit is usually temporary when a higher-level controller owns the object.
• Assuming a healthy pod means a healthy service: Readiness, selectors, ports, endpoints, network policy, and ingress rules still need independent checks.
• Using destructive cleanup as diagnosis: Removing caches, volumes, or secrets can worsen the incident and eliminate evidence.

When to revisit

Keep this runbook beside your incident response process and update it whenever the cluster's operating model changes. Revisit it before seasonal planning cycles, major release periods, and capacity reviews, when teams may change resource requests, autoscaling, ingress rules, registry workflows, or observability coverage.

Update the command examples after changes to Kubernetes versions, admission controls, workload controllers, ingress implementations, service mesh behavior, or internal naming conventions. Verify that labels, namespace names, controller selectors, and diagnostic images still match your environment. A command that depends on a particular controller or client feature should be marked with its assumption.

After each meaningful incident, add one concise symptom-to-command entry: what the user saw, which layer failed, which command confirmed it, and what verification proved the fix. Before closing an incident, run this final checklist:

1. Capture the final evidence and the exact configuration change.
2. Confirm pod readiness, service endpoints, ingress behavior, and relevant logs.
3. Check that the fix survives a normal reconciliation or rollout.
4. Record a rollback or containment step for the next occurrence.
5. Schedule a runbook review if the failure exposed a missing alert, unclear ownership, or outdated command.

Use this workflow as a starting point, not a substitute for your cluster's access controls, change process, and incident procedures. The most reliable Kubernetes troubleshooting is deliberate: preserve evidence, test one hypothesis at a time, and verify the entire request path after every change.