Jackson WilliamsKubernetes stands out as one of the most dynamic projects on Github, boasting an impressive 80,000+ commits and 550+ releases. Setting up a highly available (HA) Kubernetes cluster on-premises or in the cloud is a well-documented process that typically requires minimal effort. Tools like Kops or Kubespray can further simplify this process, making it more efficient. For expert strategies on cloud architecture and maximizing efficiency, you can visit https://computerstechnicians.com.
However, periodic updates are necessary to ensure the cluster stays up-to-date with the latest security patches and bug fixes, as well as to leverage new features released continuously. This is particularly important when running an outdated version (such as v1.9) or when automating the process to always be on the latest supported version.
When operating an HA Kubernetes Cluster, the upgrade process involves two distinct tasks that should not overlap: upgrading the Kubernetes Cluster and, if necessary, upgrading the etcd cluster, which serves as the distributed key-value backing store of Kubernetes. Let's explore how to accomplish these tasks with minimal disruptions.
Navigating Kubernetes Upgrade Options
This upgrade process specifically applies to manually deploying Kubernetes in the cloud or on-premises. It does not cover managed Kubernetes environments (where upgrades are automatically handled by the platform) or Kubernetes services on public clouds (such as AWS' EKS or Azure Kubernetes Service), which have their own upgrade processes.
For this tutorial, we assume a healthy 3-node Kubernetes and Etcd Clusters have been provisioned. A setup can be created using six DigitalOcean Droplets plus one for the worker node.
Let's consider the following Kubernetes master nodes, all running v1.13:
| Name | Address | Hostname |
| kube-1 | 10.0.11.1 | kube-1.example.com |
| kube-2 | 10.0.11.2 | kube-2.example.com |
| kube-3 | 10.0.11.3 | kube-3.example.com |
Additionally, we have one worker node running v1.13:
| Name | Address | Hostname |
| worker | 10.0.12.1 | worker.example.com |
The protocol for upgrading Kubernetes master nodes is meticulously detailed on the official Kubernetes documentation website. The current upgrade trajectories are as follows:
- Transitioning from v1.12 to v1.13 HA
- Upgrading from v1.12 to v1.13
- Advancing from v1.13 to v1.14
- Progressing from v1.14 to v1.15
Although there is only one documented version for HA Clusters, the steps can be tailored for other upgrade trajectories. In this example, we will delve into the upgrade path from v1.13 to v1.14 HA. It is vital to note that omitting a version – for instance, upgrading from v1.13 to v1.15 – is strongly advised against.
Prior to initiating, it is imperative to review the release notes of the intended upgrade version, as they may underscore breaking changes.
Upgrading Kubernetes: A Thorough, Step-by-Step Manual
Let’s proceed with the upgrade steps:
1. Log In to the First Node and Upgrade the kubeadm Tool Only:
Upgrading Kubernetes: A Step-by-Step Guide
To upgrade Kubernetes, we need to follow a series of steps to ensure a smooth transition. The process involves upgrading kubeadm, verifying the upgrade plan, applying the plan, updating kubelet, and upgrading kubectl and kubeadm on other nodes.
1. Upgrade Kubeadm on the First Master Node
$ ssh admin@10.0.11.1$ apt-mark unhold kubeadm && \
$ apt-get update && apt-get install -y kubeadm=1.13.0-00 && apt-mark hold kubeadmThe reason for running apt-mark unhold and apt-mark hold is to prevent automatic upgrades of other components, such as kubelet, to the latest version (v1.15) by default. By using hold, we mark a package as held back, preventing automatic installation, upgrade, or removal.
2. Verify the Upgrade Plan:
$ kubeadm upgrade plan
...
COMPONENT CURRENT AVAILABLE
API Server v1.13.0 v1.14.0
Controller Manager v1.13.0 v1.14.0
Scheduler v1.13.0 v1.14.0
Kube Proxy v1.13.0 v1.14.0
...3. Apply the Upgrade Plan:
$ kubeadm upgrade plan apply v1.14.04. Update Kubelet and Restart the Service:
$ apt-mark unhold kubelet && apt-get update && apt-get install -y kubelet=1.14.0-00 && apt-mark hold kubelet
$ systemctl restart kubelet5. Apply the Upgrade Plan to the Other Master Nodes:
$ ssh admin@10.0.11.2$ kubeadm upgrade node experimental-control-plane
$ ssh admin@10.0.11.3$ kubeadm upgrade node experimental-control-plane6. Upgrade kubectl on all Master Nodes:
$ apt-mark unhold kubectl && apt-get update && apt-get install -y kubectl=1.14.0-00 && apt-mark hold kubectl7. Upgrade kubeadm on First Worker Node:
$ ssh worker@10.0.12.1$ apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.14.0-00 && apt-mark hold kubeadm8. Login to a Master Node and Drain First Worker Node:
$ ssh admin@10.0.11.1$ kubectl drain worker --ignore-daemonsets9. Enhance kubelet Configuration on Worker Node:
$ ssh worker@10.0.12.1$ kubeadm upgrade node config --kubelet-version v1.14.010. Update kubelet on Worker Node and Restart the Service:
$ apt-mark unhold kubelet && apt-get update && apt-get install -y kubelet=1.14.0-00 && apt-mark hold kubelet
$ systemctl restart kubelet11. Reactivate Worker Node:
$ ssh admin@10.0.11.1$ kubectl uncordon worker
Step 12: Repeat steps 7-11 for the remaining worker nodes.
Step 13: Validate the cluster's health:
$ kubectl get nodesEtcd Upgrade Paths
As you’re well aware, etcd serves as the highly distributed key-value store backing Kubernetes, essentially functioning as the single source of truth. When operating a highly available Kubernetes cluster, it’s equally essential to run a highly available etcd cluster, providing a failsafe in the event of node failures.
Typically, we would maintain a minimum of three etcd nodes running the latest supported version. The process of upgrading etcd nodes is thoroughly documented in the etcd repository. The current upgrade paths are as follows:
- Migrate from 2.3 to 3.0
- Upgrade from 3.0 to 3.1
- Transition from 3.1 to 3.2
- Upgrade from 3.2 to 3.3
- Migrate from 3.3 to 3.4
- Upgrade from 3.4 to 3.5
When planning etcd upgrades, it’s crucial to adhere to the following plan:
- Determine the version in use. For instance:
$ ./etcdctl endpoint status - Refrain from bypassing multiple minor versions in a single leap. For example, avoid upgrading directly from 3.3 to 3.5. Instead, progress from 3.3 to 3.4, and then from 3.4 to 3.5.
- Leverage the pre-built Kubernetes etcd image. The Kubernetes team provides a custom etcd image located here, which includes etcd and etcdctl binaries for multiple etcd versions, as well as a migration operator utility for upgrading and downgrading etcd. This will facilitate the automation of migrating and upgrading etcd instances.
Among these paths, the most critical change is the transition from 2.3 to 3.0, as it involves a substantial API revamp that is documented here. Additionally, note that:
-
- Etcd v3 is capable of handling requests for both v2 and v3 data. For instance, we can utilize the ETCDCTL_API environment variable to specify the API version:
$ ETCDCTL_API=2 ./etcdctl endpoint status- It's essential to note that running etcd version 3 against a data directory formatted for version 2 does not trigger an automatic upgrade to the version 3 format.
- When using the version 2 API with etcd version 3, only the version 2 data stored in etcd is updated.
You may be wondering which Kubernetes versions are compatible with each etcd version. The documentation provides a concise overview, stating:
- Kubernetes v1.0: exclusively supports etcd2
- Kubernetes v1.5.1: introduces etcd3 support, with new clusters defaulting to etcd2
- Kubernetes v1.6.0: new clusters created with kube-up.sh default to etcd3, and kube-apiserver defaults to etcd3
- Kubernetes v1.9.0: announces the deprecation of the etcd2 storage backend
- Kubernetes v1.13.0: removes the etcd2 storage backend, with kube-apiserver refusing to start with –storage-backend=etcd2, citing the message etcd2 is no longer a supported storage backend
Based on this information, if you are running Kubernetes v1.12.0 with etcd2, you will need to upgrade etcd to version 3 when you upgrade Kubernetes to v1.13.0, as –storage-backend=etcd3 is not supported. If you have Kubernetes v1.12.0 and below, you can run both etcd2 and etcd3 concurrently.
Before proceeding with any steps, it's crucial to perform routine maintenance tasks, such as taking periodic snapshots and performing smoke rollbacks. Ensure you verify the health of the cluster:
Let's assume we have the following etcd cluster nodes:
Node Name Address Hostname etcd-1 10.0.11.1 etcd-1.example.com etcd-2 10.0.11.2 etcd-2.example.com etcd-3 10.0.11.3 etcd-3.example.com $ ./etcdctl cluster-health member 6e3bd23ae5f1eae2 is functioning correctly: received a healthy response from http://10.0.1.1:22379 member 924e2e83f93f2565 is functioning correctly: received a healthy response from http://10.0.1.2:22379 member 8211f1d0a64f3269 is functioning correctly: received a healthy response from http://10.0.1.3:22379 cluster is functioning correctlyUpgrading etcd
Based on the above considerations, a typical etcd upgrade procedure involves the following steps:
1. Access the First Node and Terminate the Existing etcd Process:
$ ssh 10.0.1.1 $ kill `pgrep etcd`2. Create a Backup of the etcd Data Directory to Provide a Downgrade Path in Case of Errors:
$ ./etcdctl backup \ --data-dir %data_dir% \ [--wal-dir %wal_dir%] \ --backup-dir %backup_data_dir% \ [--backup-wal-dir %backup_wal_dir%]3. Download the New Binary from the etcd Releases Page and Launch the etcd Server Using the Same Configuration:
ETCD_VER=v3.3.15 # choose either URL GOOGLE_URL=https://storage.googleapis.com/etcd GITHUB_URL=https://github.com/etcd-io/etcd/releases/download DOWNLOAD_URL=${GOOGLE_URL} rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz rm -rf /usr/local/etcd && mkdir -p /usr/local/etcd curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local/etcd --strip-components=1 rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz /usr/local/etcd/etcd --version ETCDCTL_API=3 /usr/local/etcd/etcdctl version # start etcd server /usr/local/etcd/etcd -name etcd-1 -listen-peer-urls http://10.0.1.1:2380 -listen-client-urls http://10.0.1.1:2379,http://127.0.0.1:2379 -advertise-client-urls http://10.0.1.1:2379,http://127.0.0.1:23794. Replicate the Process for All Remaining Members.
5. Validate the Cluster's Correct Functionality:
$ ./etcdctl endpoint health 10.0.1.1:12379 is operating as expected: successfully ratified proposal: took = 10.0.1.2:12379 is operating as expected: successfully ratified proposal: took = 10.0.1.3:12379 is operating as expected: successfully ratified proposal: took =Crucial Note: If you experience difficulties establishing a connection to the cluster, you may need to provide transport layer security certificates via HTTPS; for instance:
$ ./etcdctl --ca-file=/etc/kubernetes/pki/etcd/ca.crt --cert-file=/etc/kubernetes/pki/etcd/server.crt --key-file=/etc/kubernetes/pki/etcd/server.key endpoint healthFor added convenience, you can leverage the following environment variables:
ETCD_CA_FILE=/etc/kubernetes/pki/etcd/ca.crt ETCD_CERT_FILE=/etc/kubernetes/pki/etcd/server.crt ETCD_KEY_FILE=/etc/kubernetes/pki/etcd/server.keyFinal Thoughts
In this article, we presented a detailed, step-by-step guide on upgrading both Kubernetes and Etcd clusters. These vital maintenance procedures are essential for the smooth day-to-day operations in a typical business environment. All stakeholders involved in HA Kubernetes deployments should acquaint themselves with the aforementioned steps.
- Etcd v3 is capable of handling requests for both v2 and v3 data. For instance, we can utilize the ETCDCTL_API environment variable to specify the API version:
