Michael LevanThere are a lot of Kubernetes options out there. To be blunt, perhaps too many. All of the various options to run Kubernetes are great and were created with amazing intentions, but it ends up/ended up confusing engineers because due to the various options available, no one knows which they should choose and why.
In this blog post, you’ll learn about the difference between AKS, EKS, and GKE to help you see what you’ll get based on which option you choose.
What Are Managed Kubernetes Services
As we’ll mention a few times throughout this blog post, regardless of where Kubernetes is running, the core of it doesn’t change. Plugins are Plugins. Schedulers are Schedulers. The API server is the API server. The core makeup of Kubernetes does not change.
What changes is how you’re running Kubernetes.
Without a Managed Kubernetes Service, you’re responsible for everything. How and where the Control Plane components (Scheduler, Controllers, API Server, Etcd/Cluster Store) and the Worker Nodes components (Container Runtime, Kubelet, Kube-Proxy) run is up to you. For some organizations/engineers, this is a very good thing. For example, if you’re running large Kubernetes clusters, having Etcd on a separate server and managing the scaling/high availability yourself is sometimes recommended. However, at this point in time, for many teams/engineers, not having to worry about things like the Control Plane components is a breath of fresh air.
In short, Managed Kubernetes Services abstract away the Control Plane components for you. The only thing you’re responsible for is Kubernetes version upgrades (v1.24 to v1.25 for example). Everything about the Control Plane is abstracted and managed by the cloud provider.
When you create a Managed Kubernetes Service, the Worker Nodes are created for you as well. The key difference is the Worker Nodes are created on standard VMs on the cloud provider, so you would be responsible for managing them, but perhaps not for long. One of the new-ish architectures that have been coming out over the past 1-3 years is “Serverless Kubernetes”. Services like GKE Autopilot and EKS Fargate Profiles (which you’ll learn about in this blog post) abstract away the Worker Nodes for you. That means both the Control Plane and the Worker Node(s) are managed 100% by the cloud provider.
AKS
If you want to create an AKS cluster with Terraform, you can find all of the Terraform code and scripts here: https://github.com/AdminTurnedDevOps/Kubernetes-Quickstart-Environments/tree/main/azure
Azure Kubernetes Service (AKS) is the Managed k8s service in Azure. It gives you the ability to run Kubernetes workloads, along with some cool features like built-in Service Mesh and GitOps options.
If you decided to create an AKS cluster, you can use the following to connect to it.
az aks get-credentials -n name_of_aks_cluster -g resource_group_for_aks_cluster
Once connected, run the kubectl get pods command and specify all Kubernetes Namespaces to see what’s running.
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system azure-ip-masq-agent-6dg7q 1/1 Running 0 44m
kube-system azure-ip-masq-agent-lsll9 1/1 Running 0 43m
kube-system azure-npm-28vwk 1/1 Running 0 43m
kube-system azure-npm-d27x5 1/1 Running 0 43m
kube-system cloud-node-manager-48bxp 1/1 Running 0 43m
kube-system cloud-node-manager-s9pb2 1/1 Running 0 44m
kube-system coredns-59b6bf8b4f-p5lmr 1/1 Running 0 45m
kube-system coredns-59b6bf8b4f-p9mpk 1/1 Running 0 43m
kube-system coredns-autoscaler-5655d66f64-sgrcc 1/1 Running 0 45m
kube-system csi-azuredisk-node-2ncwb 3/3 Running 0 43m
kube-system csi-azuredisk-node-8n4mf 3/3 Running 0 44m
kube-system csi-azurefile-node-2mw47 3/3 Running 0 44m
kube-system csi-azurefile-node-cgsww 3/3 Running 0 43m
kube-system konnectivity-agent-7845c8b94c-mvwqn 1/1 Running 0 9m46s
kube-system konnectivity-agent-7845c8b94c-sctk5 1/1 Running 0 9m43s
kube-system kube-proxy-947jn 1/1 Running 0 44m
kube-system kube-proxy-p5lq7 1/1 Running 0 43m
kube-system metrics-server-8655f897d8-9mjkq 2/2 Running 1 (42m ago) 43m
kube-system metrics-server-8655f897d8-kk6kw 2/2 Running 1 (41m ago) 43m
Notice how you see things like specific CSI’s for Azure storage options, Azure IP options, and even the Kubernetes Metrics Server deployed by default.
If you log into the Azure portal and go to the AKS cluster, you’ll see an output similar to the screenshot below which specifies the resource group that it exists in, the Kubernetes version, networking options, and a few other configurations.
At the time of writing this, the following Kubernetes versions are available with AKS:
- 1.23.12
- 1.23.15
- 1.24.6
- 1.24.9
- 1.25.4
- 1.25.5
- 1.26.0 (in preview)
One of the best built-in features with AKS is the ability to use Azure Active Directory. Active Directory has been the de facto standard in enterprise environments for authentication and authorization for 20+ years.
You can use Azure AD for Kubernetes RBAC.
To enable Azure AD with AKS, you can use the following Terraform block.
azure_active_directory_role_based_access_control {
managed = true
azure_rbac_enabled = true
}
AKS also has a ton of options around Service Mesh, GitOps, and automated application deployments.
The last thing to touch on is the ability to use other CNI’s. AWS EKS has this as well, but just as an example, you can specify in the AKS resource when creating it with Terraform what network plugin you want to use. In the code below, you can see that Cilium is selected.
resource "azurerm_kubernetes_cluster" "k8squickstart" {
name = var.name
location = var.location
resource_group_name = var.resource_group_name
dns_prefix = "${var.name}-dns01"
network_profile {
network_plugin = "azure"
network_plugin_mode = "Overlay"
ebpf_data_plane = "cilium"
}
EKS
If you want to create an EKS cluster with Terraform, you can find all of the Terraform code and scripts here: https://github.com/AdminTurnedDevOps/Kubernetes-Quickstart-Environments/tree/main/aws
AWS Elastic Kubernetes Service (EKS) is the AWS version of Managed k8s services for Kubernetes. Much like AKS, it gives you the ability to utilize a managed Kubernetes cluster, but in AWS.
If you decided to create an EKS cluster, you can connect with it using the below.
aws eks update-kubeconfig --region name_of_region --name name_of_eks_cluster
Once the EKS cluster is created, use the kubectl get pods command and specify all of the Namespaces.
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-4chvr 1/1 Running 0 23m
kube-system aws-node-bz7jt 1/1 Running 0 23m
kube-system aws-node-s7rd4 1/1 Running 0 23m
kube-system coredns-79989457d9-dw8ql 1/1 Running 0 27m
kube-system coredns-79989457d9-ldqxl 1/1 Running 0 27m
kube-system ebs-csi-controller-545b8d5cf9-mjzpt 6/6 Running 0 25m
kube-system ebs-csi-controller-545b8d5cf9-sk5bd 6/6 Running 0 25m
kube-system ebs-csi-node-gfsrv 3/3 Running 0 23m
kube-system ebs-csi-node-m5lcw 3/3 Running 0 23m
kube-system ebs-csi-node-v5vm7 3/3 Running 0 23m
kube-system kube-proxy-7bwq4 1/1 Running 0 23m
kube-system kube-proxy-bcr8t 1/1 Running 0 23m
kube-system kube-proxy-lb566 1/1 Running 0 23m
Notice the difference in comparison with the AKS cluster. In the case of EKS, Pods are deployed for the EBS CSI, which is specific to AWS. Also notice that there is no Metrics Server.
One of the cool things about Managed Kubernetes Services is the built-in monitoring and observability. AKS has App Insights and EKS has Container Insights via CloudWatch. CloudWatch is a great monitoring and observability tool in general, so combining it with Container Insights is a great match for EKS.
If you’d like to try out enabling Container Insights, you can do so with the below script.
ClusterName=k8squickstart-cluster
RegionName=us-east-1
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${RegionName}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' | kubectl apply -f -
At the time of writing, the following Kubernetes versions are available in EKS:
- 1.22
- 1.23
- 1.24
- 1.25
Fargate Profiles
Wrapping up the EKS section, let’s talk about Fargate Profiles.
In the What Are Managed Kubernetes Services section, you learned about “Serverless Kubernetes”. Fargate Profiles for EKS is how you can enable that feature. When Fargate Profiles are enabled, you no longer have to manage the Control Plane or the Worker Nodes.
resource "aws_eks_fargate_profile" "k8senv-eks-serverless" {
cluster_name = aws_eks_cluster.k8senv-eks.name
fargate_profile_name = "k8senv-serverless-eks"
pod_execution_role_arn = aws_iam_role.eks-fargate.arn
subnet_ids = [var.private_subnet_id_1]
selector {
namespace = "default"
}
GKE
If you want to create a GKE cluster with Terraform, you can find all of the Terraform code and scripts here: https://github.com/AdminTurnedDevOps/Kubernetes-Quickstart-Environments/tree/main/Google
💡 If you’ve already used GKE, you say see an error around the auth plug-in with Kubernetes v1.26. If so, check out these instructions: [https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke](https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke)Google Kubernetes Engine (GKE), much like AKS and EKS, is the Managed k8s service in GCP. One of the great things about GKE is it certainly feels a bit more robust and mature compared to other Managed k8s services. The assumption as to “why” is because Google made Kubernetes and it was the first Managed k8s service, so it’s had more time to evolve.
If you decided to create a GKE cluster, you can use the following to connect to it.
gcloud container clusters get-credentials name_of_gke_cluster --region us-east1 --project gcp_project_name
GKE is incredibly robust and has a ton of amazing features. You’ll have to go through everything that’s available, but GKE has everything from managing multiple clusters with Anthos to Service Mesh to a service that helps you migrate VM-based apps to containers.
At the time of writing this, there are a plethora of versions of Kubernetes for GKE starting from 1.21.x and going to 1.25.x.
GKE Autpilot
Wrapping up the GKE section, let’s talk about GKE Autopilot.
In the What Are Managed Kubernetes Services section, you learned about “Serverless Kubernetes”. GKE Autopilot is how you enable Serverless Kubernetes in GKE. When Autopilot is enabled, you no longer have to manage the Control Plane or the Worker Nodes.
resource "google_container_cluster" "primary" {
name = var.cluster_name
location = var.region
ip_allocation_policy {}
network = var.vpc_name
subnetwork = var.subnet_name
enable_autopilot = true
}
Virtual Kubelets
The topic around virtual kubelets and cloud providers comes up often as the extendability can make for a more robust scaling methodology. Before diving into the virtual kubelet, let’s talk about what a Kubelet is.
The Kubelet is an agent that runs on every single Kubernetes Worker Node. The Kubelet has a few jobs, like registering the Worker Nodes CPU and memory to see what resources are available, but the primary job of the Kubelet agent is to listen to the Kubernetes API server for new tasks. Essentially, the Kubelet ensures the containers that are defined in your Pod Spec (within your Kubernetes Manifest) are running and healthy.
At this point, you may be asking yourself “where does the Virtual Kubelet come into play”. Because the Kubelet must run on a Kubernetes Worker Node, the process of spinning up new Worker Nodes or scaling out Worker Nodes can take time, time which you may not have. For example, your Worker Nodes may be running out of resources (CPU, memory, etc.), and you need just a little more CPU/memory to run one application. It wouldn’t be cost-effective to create an entirely new Worker Node for one app. Instead, you can deploy Pods to locations that act like Worker Nodes.
A great example of this is Azure Container Instances (ACI). ACI is a bite-sized way to run containers. It doesn’t require a Kubernetes cluster or any other tooling. ACI by itself wouldn’t be the best option for production workloads, but ACI combined with AKS would. AKS can “burst” (scale/push out) containers out to ACI. Instead of creating an entirely new Worker Node, the containers just run on ACI as the Virtual Kubelet is compatible with ACI.
AWS does the same thing with EC2 instances. With the Virtual Kubelet, Elastic Kubernetes Service (EKS) will “burst” out to EC2 instances. Essentially, the Pods just run on the EC2 instances without any of the Kubernetes Worker Node components other than the Virtual Kubelet.
So, What’s The Difference?
Throughout this blog post, Managed Kubernetes Services as a whole, the differentiators, and third-party components like the Virtual Kubelet were defined.
This leaves us with one last question - what’s the difference?
The difference is going to ultimately come down to what the cloud provider does with the Managed Kubernetes Service.
Kubernetes is Kubernetes. What it is at the “core” doesn’t change. CNI’s are CNI’s. CSI’s are CSI’s. How the Scheduler works is how the Scheduler works. The core attributes of Kubernetes do not change. However, Kubernetes is open source. That means you, me, and anyone else can fork the repo or create a Pull Request (PR) and make changes (of course, the PR process requires extensive review from the Kubernetes Release Team). What this means is a cloud provider can morph Kubernetes into what they need. For example, as we discussed previously, AKS has the Metrics Server on by default. EKS does not. AKS has Azure Storage and Azure Fileshares as a default CSI. EKS has EBS as a default CSI.
With the above being said, the difference between the Managed Kubernetes Services isn’t a core Kubernetes difference. It’s a difference around modification.
One important factor to wrap this blog post up with is you shouldn’t change cloud providers for Kubernetes Services. If you have 100% of your workloads running in Azure, but want to move 5% of your workloads to AWS so you can use EKS, there’s almost never a reason to do this at the technical level. If you do want to go multi-cloud just for Kubernetes, ensure that you have a compelling reason prior as it’ll end up being a heavy lift.
