Michael LevanGetting a Service Mesh up and running in your environment is the key to network, security and reliability success. It’s the make or break between truly understanding what’s happening at a network layer inside of your cluster and deploying applications while flying blind.
Once the Service Mesh is up, the next Day Two operation that you must focus on is monitoring. Without monitoring, you could have the best Service Meshed network in the world, but it won’t mean much if the network is down.
In this blog post, you’ll learn how to combine monitoring with Linkerd to ensure network and observability success.
Prerequisites
For the purposes of this blog post, you should have:
- A Kubernetes cluster up and running. It could be a cluster running locally (like Minikube or Kind) or a Managed Kubernetes Service cluster (Azure Kubernetes Service, Elastic Kubernetes Service, Google Kubernetes Engine, etc.)
- Linkerd installed. You can install Linkerd using the documentation here: https://linkerd.io/2.12/getting-started/
Linkerd and App Deployment
Before having the ability to monitor an application, you first need one deployed. To get a full view and understanding of how monitoring would work with several microservices, you can deploy the popular demo Sockshop app. The Sockshop app has several microservices and various Kubernetes Resources that you can deploy to see a full architecture vs just deploying a Hello World or Nginx web app.
You can find the Sockshop app here: https://microservices-demo.github.io/docs/quickstart.html
Let’s learn how to deploy it.
First, clone down the Sockshop application repository.
git clone https://github.com/microservices-demo/microservices-demo
Next, change the directory (cd) into the microservices-demo directory.
cd microservices-demo
Change the directory to the Kubernetes deployment directory.
cd deploy/kubernetes
Create the expected sock-shop Kubernetes Namespace that the Sockshop application is expecting the Kubernetes Resources to be deployed into.
kubectl create namespace sock-shop
Lastly, deploy the Sockshop application.
kubectl apply -f complete-demo.yaml
You should see an output on the terminal similar to the screenshot below.
Give the application a few minutes to get up and running.
After a few minutes, you can run the following Kubectl command to see all of the resources up and running.
kubectl get all -n sock-shop
Inject Linkerd’s sidecar into the Deployment Pods so they’re fully managed with the Service Mesh.
kubectl get deploy -o yaml -n sock-shop | linkerd inject - | kubectl apply -f -
After a few minutes, you should see the container count in all of the Pods raise by one container (the Linkerd sidecar).
Monitoring With Viz
Now that the Sockshop microservice is deployed, you can monitor it using Viz. Viz is a built-in monitoring system for Linkerd. It gives you the ability to see all of the Service Mesh’d resources and gives you various pieces of information about each Pod (which you’ll see in an upcoming section).
In this section, you’ll do the following:
- Install Viz
- View the Sockshop app in Viz
Installing Viz
to install Viz, you can use the Linkerd CLI with the following command:
linkerd viz install | kubectl apply -f -
Next, run Viz locally.
linkerd viz dashboard &
Monitoring
Once Viz is installed and you run the dashboard command above, you can begin to monitor and observe the Service Mesh’d resources.
First, within Linkerd, click the Namespace section and choose the sock-shop Namespace.
Under the HTTP Metrics section, you’ll see the sock-shop Namespace.
Click the Namespace.
You can now see that all of the Deployments inside of the sock-shop Namespace are successfully Mesh’d. Notice how you can see information around the success of the application itself and latency information.
As an example, click on the Carts Deployment.
Within the Deployment, you’re able to see various information around success rates and latency for the application running inside of the Pods.
Scrolling down a bit on the same page, you can see inbound and outbound routing to the Deployment itself. This is useful if you want a true method of seeing what’s happening inside of the Pod network for your application. Ultimately, the ability to see, from a networking perspective, what’s “talking to” your app is crucial for troubleshooting and understanding Pod communication.
To dive a little bit deeper, click on the Cart Pod.
Once you click on the Pod, you can see what’s accessing your Pod (to the Pod and where the Pod is going) from an ingress and egress perspective. Also notice on the right side that you can see mTLS is active and your connections to the Pod is secure.
If you’d like a deeper look into the various routes for your application, you can use the linkerd viz routes command.
The Viz dashboard has a lot of great features in terms of Service Mesh, but in a production environment, you’ll most likely want to also use (in addition) a monitoring solution that can monitor your entire environment.
That’s where Prometheus and Grafana come into play.
Prometheus and Viz
When Viz gets installed, Prometheus automatically gets installed as well. Prometheus is a great way to visualize the Metrics portion of observability practices.
To view the Prometheus dashboard, you can use the following command for port forwarding.
kubectl port-forward -n linkerd-viz service/prometheus 68484:9090
Open up a web browser and go to the URL that displays when you run the above command.
Once in the Prometheus instance, go to Status → Targets.
You’ll see within the targets all of the Linkerd resources are automatically observed via Prometheus. It’ll show you the endpoint for the metrics, labels, and the status of the endpoint to ensure that it’s up and operational.
Grafana and Linkerd
The final piece to monitoring Linkerd is by using one of the most popular monitoring UIs in the Kubernetes space, Grafana.
However, Grafana isn’t installed yet. To get Grafana up and running, you’ll have to ensure that Grafana is installed. You can install Grafana by using the official Helm Chart.
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana -n grafana --create-namespace grafana/grafana
Once you run the above installation, you’ll see an output similar to the screenshot below.
Wait a few minutes for the full Grafana installation to be complete.
To confirm that the installation was a success, check that all of the resources in the Namespace were created successfully.
kubectl get all -n grafana
To reach the Grafana UI, run the following:
kubectl port-forward -n grafana service/grafana :80
The UI should now be available locally and you can reach it via output from the above command.
Once the UI is available, you’ll need the password. The Grafana password, when installed via Helm, is saved as a Kubernetes Secret.
To retrieve the password for the login, run the following command (the username is admin):
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Once you’re logged into Grafana, go to the Dashboards section and click the blue New button.
Choose the Import option
At this point, you can import a Grafana dashboard.
For Linkerd, there’s already a dashboard that you can use. You can download the JSON for the dashboard here: https://grafana.com/api/dashboards/3986/revisions/1/download
Once downloaded, upload the JSON dashboard and you should see a similar screen as shown in the screenshot below and click the blue Import button.
Once uploaded, you’ll see a Linkerd dashboard showing statistics about the Service Mesh.
A Quick Note On Buoyant Cloud
Buoyant Cloud is fully automated Linkerd and is the enterprise/paid version of the platform. The UI gives you the ability to see:
- The health of Linkerd itself
- The health and behavior of your application.
It’s multi-user, fully hosted, cross-cluster, and gives you the enterprise feel of a Service Mesh if multiple members of your team need to access it.
