A Kubernetes cluster upgrade is a change-management exercise, not just a package update. This reusable checklist covers inventory, version compatibility, backups, deprecated APIs, staging tests, node draining, workload validation, and rollback decisions so you can approach each control-plane or node-pool upgrade with a recorded plan.
Overview
A safe Kubernetes version upgrade starts before the upgrade command. First establish the target version, the supported path from your current version, and the recovery options available if a component does not behave as expected. Managed Kubernetes services, self-managed clusters, distributions, admission controllers, storage plugins, ingress controllers, and service meshes can all impose their own requirements.
Use this article as a readiness worksheet. Record the answers rather than relying on memory:
- Current state: Kubernetes version, node operating system and runtime, cluster provider, node pools, and critical add-ons.
- Target state: control-plane version, node versions, add-on versions, and the order in which each component will change.
- Impact: maintenance window, workload owners, expected disruptions, communication channel, and escalation contacts.
- Recovery: tested backups, infrastructure definitions, application redeploy procedures, and a clearly defined stop or rollback decision.
- Validation: health, capacity, networking, storage, ingress, security, and application checks with named owners.
Do not assume that a control-plane upgrade automatically upgrades every node or add-on. Depending on the platform, node pools may require a separate action, replacement strategy, or maintenance setting. Confirm the provider-specific procedure and the compatibility guidance for every cluster component before scheduling production work.
Version-skew reference worksheet
Version skew describes the supported relationship between Kubernetes components during a transition. The exact rules can change, so use the official documentation for your Kubernetes release and provider as the authority. This table is a planning aid, not a substitute for that check:
| Component | Record before upgrade | Question to verify |
|---|---|---|
| Control plane | Current and target versions | Is the proposed upgrade path supported directly, or must you take an intermediate step? |
| kubelet on nodes | Version per node pool | What version difference is allowed during and after the control-plane change? |
| kubectl and CI clients | Versions used by operators and pipelines | Can clients authenticate and manage the target cluster without an unsupported skew? |
| API server clients | Controllers, operators, agents, and automation | Do any clients depend on removed or changed APIs? |
| Cloud and cluster add-ons | Ingress, DNS, CNI, CSI, metrics, policy, and autoscaling versions | Is each release compatible with the target Kubernetes version? |
Checklist by scenario
Before any production upgrade
- Open the provider or distribution upgrade guide and record prerequisites, supported paths, maintenance behavior, and exclusions.
- Inventory nodes, namespaces, deployments, stateful workloads, daemonsets, jobs, persistent volumes, ingress objects, custom resources, webhooks, and admission policies.
- Review capacity. Confirm that remaining nodes can run workloads while one node is cordoned and drained. Check requests, limits, disruption budgets, and autoscaler behavior. For background, see the Kubernetes resource requests and limits guide.
- Confirm backups for cluster state where applicable, persistent data, and application configuration. Perform a restore test or document why a particular data source is recoverable by another method.
- Scan manifests and deployed resources for APIs removed or changed in the target release. Review custom resources and the versions served by every operator.
- Check webhook certificates, service accounts, image pull credentials, network policies, DNS, storage classes, and cloud permissions.
- Define success criteria and a stop condition. Examples include elevated error rates, failed readiness checks, unavailable replicas, storage attach failures, or loss of ingress traffic.
When a staging cluster is available
- Reproduce the intended upgrade sequence in staging, including add-on upgrades and node replacement behavior.
- Deploy representative applications, not only a minimal smoke-test pod. Include a stateful workload, an ingress route, scheduled jobs, autoscaling, and any admission or policy controls used in production.
- Exercise ordinary operations: rollout, rollback, scale-out, pod rescheduling, node drain, secret rotation, persistent-volume attach, and log or metric collection.
- Test the deprecated-API remediation in the same delivery pipeline that will deploy production. A manifest that passes linting may still fail if a live custom resource or third-party controller uses an old API.
- Capture timings, warnings, changed defaults, and provider-specific messages. Turn findings into a production runbook rather than relying on a verbal handoff.
For a node-pool or worker upgrade
- Confirm spare capacity and verify that pod disruption budgets do not prevent safe eviction.
- Cordon the selected node so new workloads are not scheduled there, then drain it using a deliberate eviction policy:
kubectl cordon NODE_NAME
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data --timeout=10mReview the output rather than treating a completed command as proof that the node was safe to remove. EmptyDir data is temporary and may be lost; use the flag only when that loss is acceptable. Pods protected by disruption budgets may remain pending, which is a signal to resolve capacity or availability constraints before continuing.
- Replace or upgrade the node according to the platform procedure, then confirm it returns Ready with the expected labels, taints, runtime, storage, and network configuration.
- Uncordon only after node-level checks pass:
kubectl uncordon NODE_NAME
kubectl get nodes -o wideRepeat in small batches. Keep enough healthy capacity to serve production traffic and leave time to investigate before the next batch.
What to double-check
Deprecated and removed APIs
API compatibility is one of the most common upgrade hazards. Search source repositories, Helm charts, GitOps definitions, admission policies, and live objects. Useful approaches include rendering charts before scanning them and querying the cluster for API versions in use. Tools that inspect deprecated APIs can help, but compare their output with release notes and the resources actually installed in your environment.
Pay special attention to ingress, policy, autoscaling, batch, storage, and custom resource definitions. Update manifests before the cluster upgrade, deploy them in staging, and verify that controllers understand the new version. Do not delete an old API version or CRD until you understand whether existing objects can be converted and recovered.
Networking, ingress, and storage
Check service discovery from inside and outside the cluster, load-balancer health, TLS renewal, network policies, and external DNS behavior. Validate both a new connection and an existing long-lived connection where your applications depend on one. Use the Kubernetes ingress and Gateway troubleshooting guide as a companion checklist.
For storage, confirm CSI driver compatibility, volume attachment, expansion, snapshots, reclaim policies, and failover behavior. A pod reaching Running is not enough: write and read test data in a non-production validation path and verify that recovery procedures remain usable.
Workload health and observability
Before starting, record baseline latency, error rate, saturation, restart counts, pending pods, and node health. After each phase, compare the same signals. Confirm that metrics, logs, traces, alerts, and dashboards still work; an upgrade that hides telemetry is not a successful upgrade. Review readiness, liveness, and startup probes, especially for workloads that restart or become temporarily unavailable during node movement. The Kubernetes health checks checklist provides a useful validation reference.
Common mistakes
- Upgrading directly in production: A staging rehearsal exposes incompatibilities while the cost of correction is lower.
- Ignoring add-ons: The control plane may be compatible while an ingress, CNI, CSI, operator, or policy component is not.
- Draining without capacity: Requests, disruption budgets, and affinity rules can leave evicted pods pending.
- Confusing backup with recovery: A successful backup job does not prove that data or cluster configuration can be restored.
- Checking only pod status: Running pods can still have broken routes, failed probes, missing metrics, or inaccessible storage.
- Changing too many variables: Avoid combining a Kubernetes upgrade with unrelated application, network, database, or infrastructure changes unless the dependency is explicit.
- Skipping the record: Keep the version matrix, command output, validation results, incidents, and final decision in the team’s operational documentation.
For troubleshooting during or after the change, use the Kubernetes troubleshooting runbook. If the upgrade becomes an incident, switch from ad hoc debugging to a documented incident response process.
When to revisit
Revisit this checklist before every control-plane, node-pool, or major add-on upgrade. Schedule a formal review during seasonal planning cycles and whenever the provider changes its upgrade workflow, maintenance policy, supported versions, node images, or rollback behavior. Also update it when your cluster adopts a new ingress controller, service mesh, storage driver, policy engine, operator, CI/CD path, or GitOps tool.
At the start of each upgrade, make a dated copy of the worksheet and assign owners for compatibility, backup, capacity, application validation, and communications. Afterward, record what differed from the plan and convert successful fixes into automated checks. The practical finish line is not simply a new version number: it is a cluster that serves traffic, stores data, emits reliable telemetry, and can be operated by the next person who follows the runbook.