Kubernetes routing failures often look similar from the outside but originate at different layers. This guide provides a repeatable workflow for diagnosing 404 responses, 502 errors, TLS failures, and unreachable endpoints through Ingress or Gateway API resources. It also gives you a recurring checklist for checking Services, EndpointSlices, DNS, controllers, load balancers, certificates, network policies, and HTTP routes after deployments, migrations, and cluster upgrades.
Overview
A request to a Kubernetes application usually travels through several components: a client resolves a DNS name, connects to a load balancer or gateway, reaches an Ingress or Gateway controller, matches an HTTP route, selects a Service, and is finally forwarded to a ready Pod. A failure at any point can produce a misleading symptom.
For example, a 404 may mean that the controller received the request but no host or path rule matched. It may also be an application-generated 404 after routing worked correctly. A 502 commonly indicates that the proxy could not establish a usable connection to the backend, but the precise cause may be an empty EndpointSlice, an incorrect Service port, a failed readiness check, or a network policy. TLS errors can occur before HTTP routing is evaluated, so debugging the route first may waste time.
Use a layered approach. Start with the exact request that fails, then verify the public entry point, controller, route, Service, endpoints, and Pods in that order. Avoid changing multiple resources at once; each isolated check should narrow the fault domain.
First, record the failing request
Write down the hostname, URL path, method, port, response code, TLS error, source network, and approximate time. Test both the public name and, where appropriate, an internal address. A request such as curl -vk https://app.example.test/orders can show certificate details, redirects, response headers, and whether the response came from a controller or the application.
Check the response body and headers before assuming that every 404 or 502 is produced by Kubernetes. A custom application error page, application-specific header, or known server signature can indicate that the request reached the workload.
What to track
1. Ingress or Gateway resources
Confirm that the resource exists in the expected namespace and that its host, path, backend reference, and status are correct:
kubectl get ingress -A
kubectl describe ingress -n APP_NAMESPACE APP_INGRESS
kubectl get gateway,httproute -A
kubectl describe gateway -n APP_NAMESPACE APP_GATEWAY
kubectl describe httproute -n APP_NAMESPACE APP_ROUTEFor Ingress, inspect the ingressClassName and confirm that the intended controller watches that class. Older configurations may rely on annotations or a controller-wide default class. Do not assume that an Ingress without an explicit class is being processed by the controller you expect.
For Gateway API, inspect listener status, route references, and conditions such as whether a listener is programmed or a route is accepted. A route can exist and still be rejected because its parent reference, hostname, section name, or cross-namespace permission does not match the Gateway configuration.
Review annotations carefully. An annotation supported by one Ingress controller may be ignored by another, and controller behavior or annotation names can change between releases. Prefer the controller's current documentation and a small, explicit configuration over copying annotations from an unrelated example.
2. Controller status and logs
Find the controller Deployment or Pod and inspect its logs during a fresh request:
kubectl get pods -A | grep -Ei 'ingress|gateway|controller'
kubectl logs -n CONTROLLER_NAMESPACE deploy/CONTROLLER_NAME --since=15m
kubectl get events -A --sort-by=.lastTimestampLook for messages about rejected configuration, invalid backends, missing Secrets, failed Service references, unsupported annotations, listener conflicts, or configuration reloads. If the controller has multiple replicas, check enough Pods to determine whether the problem affects all replicas or only one.
Also check the controller Service and its external address:
kubectl get svc -n CONTROLLER_NAMESPACE CONTROLLER_SERVICE -o wide
kubectl get endpointslice -n CONTROLLER_NAMESPACE -l kubernetes.io/service-name=CONTROLLER_SERVICEAn external address that is pending, stale, or attached to the wrong load balancer can make an otherwise valid route unreachable.
3. Services, ports, and EndpointSlices
A Kubernetes Service has at least two important port concepts: the port exposed by the Service and the targetPort used for the Pod. The Ingress or HTTPRoute must reference the Service port, not necessarily the container port. Check the full mapping:
kubectl get svc -n APP_NAMESPACE APP_SERVICE -o yaml
kubectl describe svc -n APP_NAMESPACE APP_SERVICE
kubectl get endpointslice -n APP_NAMESPACE -l kubernetes.io/service-name=APP_SERVICE -o yamlConfirm that the Service selector matches the Pod labels, the named port resolves to the intended port, and EndpointSlices contain the expected addresses. An empty EndpointSlice usually means that no matching Pod is considered ready for that Service. If endpoints exist but the proxy returns 502, verify that the target port is listening and that the protocol expected by the proxy matches the application.
4. Pods and readiness
Inspect the selected Pods, not only the Deployment:
kubectl get pods -n APP_NAMESPACE -l app=APP_LABEL -o wide
kubectl describe pod -n APP_NAMESPACE POD_NAME
kubectl logs -n APP_NAMESPACE POD_NAME --since=15m
kubectl get pod -n APP_NAMESPACE POD_NAME -o jsonpath='{.status.conditions}'A running container is not automatically a routable backend. Readiness probes control whether a Pod is included in Service endpoints. Check the probe path, port, scheme, startup timing, and any recent application changes. For a broader probe review, use the Kubernetes health checks checklist.
5. DNS and external load balancing
Verify that the public name resolves to the intended address from the same network as the affected client:
dig +short app.example.test
nslookup app.example.test
curl -vk --resolve app.example.test:443:LOAD_BALANCER_IP https://app.example.test/healthCompare the DNS result with the address shown by the controller Service or cloud load balancer integration. During a migration, stale records, split-horizon DNS, or a record pointing to an old cluster can make the Kubernetes configuration appear broken when the request never reaches it. Confirm the load balancer listener, health checks, security group or firewall rules, and whether it forwards the expected TCP port.
6. TLS Secrets and certificates
TLS is commonly terminated at the load balancer, Ingress controller, Gateway listener, or application. Identify the termination point before troubleshooting. For an Ingress, check the referenced Secret and its namespace:
kubectl get secret -n APP_NAMESPACE TLS_SECRET
kubectl describe secret -n APP_NAMESPACE TLS_SECRET
kubectl describe ingress -n APP_NAMESPACE APP_INGRESSDo not print private key data into shared logs or tickets. Instead, check that the Secret contains the expected certificate and key fields, that the hostname appears in the certificate's subject alternative names, and that the controller can access the Secret. With Gateway API, inspect the listener's certificate references and the resulting status conditions.
If cert-manager is used, check the Certificate, CertificateRequest, Issuer, and related events. Certificate automation may be healthy while DNS still points to another endpoint, so validate both the certificate resource and the actual certificate presented by the public hostname using curl -vk or an approved TLS inspection tool.
7. Network policies and protocol details
When resources and endpoints look correct but the controller still returns 502, inspect NetworkPolicy objects and the network plugin's logs. Confirm that traffic is allowed from the controller's namespace or Pod labels to the application namespace and port. Also check whether the backend expects HTTP, HTTPS, HTTP/2, or a particular Host header. A proxy connecting with the wrong protocol can fail even when the target port is open.
For container-level checks, test from a temporary diagnostic Pod in the relevant namespace rather than assuming that a successful test from your laptop proves in-cluster connectivity:
kubectl run netcheck -n APP_NAMESPACE --rm -it --image=curlimages/curl -- sh
curl -v http://APP_SERVICE.APP_NAMESPACE.svc.cluster.local:SERVICE_PORT/healthSymptom-to-check reference
| Symptom | Most useful first checks |
|---|---|
| 404 from controller | Host, path, IngressClass, Gateway listener, HTTPRoute acceptance, and path matching rules. |
| 404 from application | Confirm routing reached the intended Pod; then check application routes, rewrite behavior, and the forwarded path. |
| 502 or upstream connection failure | Service port, targetPort, EndpointSlices, readiness, NetworkPolicy, backend protocol, and controller logs. |
| TLS handshake or certificate error | DNS target, TLS termination point, listener, Secret, certificate names, and certificate automation events. |
| Timeout or unreachable endpoint | DNS, load balancer address, firewall rules, listener ports, controller Service, and network policies. |
| Works internally but not publicly | External DNS, load balancer health, public firewall rules, TLS configuration, and route host matching. |
Cadence and checkpoints
Treat routing configuration as something to review, not a one-time setup. A lightweight monthly or quarterly check can catch drift before an incident.
- After each deployment: test the public hostname, important paths, redirects, TLS, and application health endpoint. Confirm that the intended Service has ready endpoints.
- During a monthly review: list Ingresses, Gateways, HTTPRoutes, controller versions, external addresses, TLS Secrets, and recent warning events. Look for unused routes, duplicate hosts, and resources with unclear ownership.
- Before a cluster or controller upgrade: compare supported API versions, Gateway API features, annotations, admission behavior, and certificate integrations. Render or validate manifests in a non-production environment when possible.
- Before a domain migration: lower DNS TTL only according to your change plan, verify the new load balancer and certificate, test the exact Host header, and keep a rollback record for DNS and routing changes.
- After an incident: save the relevant resource descriptions, controller logs, events, DNS observations, and timestamps. Add the successful diagnostic commands to the team's runbook.
Use a simple tracking record with columns for hostname, namespace, route resource, controller or Gateway, Service, TLS Secret, external address, last successful test, and owner. This turns an informal routing inventory into a useful operational checkpoint.
How to interpret changes
Compare observations across layers instead of treating a single command as proof. If DNS resolves correctly but controller access logs show nothing, investigate the load balancer or network path. If the controller logs a matched route but reports no endpoints, focus on selectors, readiness, and Service ports. If the controller forwards successfully and the application returns 404, stop changing Ingress rules and inspect application routing or path rewrites.
Configuration changes should also be interpreted with timing. A new Deployment can temporarily remove all ready endpoints while probes initialize. A certificate renewal can update a Secret without changing DNS. A controller upgrade can expose previously ignored configuration or alter how an annotation is handled. Record the change time and compare it with controller events, Pod rollout status, certificate events, and external monitoring.
For production systems, add synthetic checks for the most important host and path combinations. A useful check validates DNS resolution, TLS hostname verification, expected status code, and a small response assertion. Monitor at least one path that exercises the real backend rather than only checking that the load balancer accepts a TCP connection.
Keep the troubleshooting process safe. Prefer read-only commands first, avoid editing live resources during diagnosis unless a change is approved, and redact credentials and private key material from shared output. For broader incident coordination, pair this routing checklist with the DevOps incident response runbook template.
When to revisit
Revisit this guide whenever a route returns a new status code, a hostname changes, a certificate is renewed or replaced, a Service or readiness probe changes, or a load balancer, Ingress controller, Gateway implementation, or network policy is upgraded. It is also worth reviewing quarterly even when nothing appears broken: unused routes and stale DNS records are easier to remove before they become confusing during an incident.
Before closing a routing incident, run the checklist from the outside in: resolve the hostname, inspect the TLS handshake, confirm the controller or Gateway accepted the route, verify the Service port and EndpointSlices, test backend connectivity, and confirm the application response. Save the commands and evidence that identified the fault. On the next deployment or infrastructure change, replay the same checks as a controlled checkpoint rather than waiting for the next 404 or 502.
If the cluster contains many applications, link this workflow from your service ownership documentation and maintain one short record per public hostname. That combination—a repeatable diagnostic sequence and a current routing inventory—makes Kubernetes Ingress troubleshooting faster without relying on controller-specific guesses.
