Imagine you’ve just set up your Kubernetes cluster; the apps are running smoothly, and everything seems good. Do you think your work is finished?
We wish it was that simple. While the apps are running, you need to continuously observe how things are performing, which requires data for such insights.
Your ability to recognize the potential problem patterns and understand how Kubernetes handles certain workloads will make the difference between a quick resolution of the problem or a website down for the entire day.
Here comes the Kubernetes observability.
What is Kubernetes Observability?
Kubernetes observability means monitoring and analyzing the state and behavior of a Kubernetes cluster and the applications running on it.
This process involves collecting various data types from different sources within the cluster, such as metrics, logs, and traces, and using tools and techniques to visualize and make sense of this data.
Observability is critical to ensure the availability, reliability, and performance of the Kubernetes environment.
By monitoring key metrics such as CPU usage, memory consumption, and network traffic, operators can proactively identify and address issues before they evolve into bigger problems.
But how do you do that?
Two tools that can help you with this are – Prometheus and Grafana. Both of these tools are open source. Prometheus best gathers metrics from various sources, while Grafana visualizes complex time-series data.
The combination of both could be very powerful.
Using Prometheus and Grafana
We’ll now see how to set up Prometheus and Grafana on an existing Kubernetes cluster. We’ll also set up a dashboard in Grafana to visualize the data coming from the cluster. After that, you can choose any Kubernetes installation hosted on a cloud server or your local machine.
Installing Prometheus
Before installing Prometheus, you must add the appropriate Helm repository using the Heml repo-add command and update the repository metadata using Helm repo update. For this guide, we’ll use the Bitnami Helm repository, which provides a comprehensive Helm chart for Prmetheus with a wide range of configuration options.
If you’re unfamiliar with Helm, we recommend checking out this guide — Getting Started with Helm.
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
To ensure a smooth installation of Prometheus, it is essential to check the various configuration options available. With numerous options ranging from exposing each component (e.g., load balancer and type of ingress) to data storage, evaluating these options before installation is important.
While it is critical to properly configure these options for production installations, this article will assume the installation of Prometheus with default configuration into a Kubernetes cluster without any specific requirements. The default configuration can be installed using the Helm install command, simplifying the installation process.
helm install prometheus bitnami/kube-prometheus
You’ll see a new pod after a few seconds:
alertmanager-prometheus-prometheus-oper-alertmanager-0 2/2 Running 0 25m
prometheus-kube-state-metrics-68cb46fdd4-gk4jh 1/1 Running 0 25m
prometheus-node-exporter-rkg84 1/1 Running 0 25m
prometheus-prometheus-oper-operator-745f4b599c-xjjsn 1/1 Running 0 25m
prometheus-prometheus-prometheus-oper-prometheus-0 3/3 Running 1 25m
Prometheus can be accessed in various ways, depending on the Kubernetes cluster’s configuration. According to the Prometheus documentation, one typical method is to expose the server through a reverse proxy like Nginx. However, the default configuration of the Prometheus Helm chart only makes it accessible to other pods within the Kubernetes cluster. Therefore, a recommended alternative approach is to utilize the kubectl port-forward command. After executing the following command, open a new terminal, enter the command, and keep it open:
kubectl port-forward –namespace default svc/prometheus-kube-prometheus-prometheus 9090:9090
Excellent! The command mentioned above will redirect all traffic to the Prometheus-server pod’s 9090 port to your local machine. You can now access Prometheus by opening your web browser and visiting http://localhost:9090.
Installing Grafana
Similar to Prometheus, Grafana also offers a convenient Helm chart for installation. You will find various configuration options to customize the installation according to your requirements. However, for this tutorial, you can simply install it with the default configuration to witness its functionalities. The bitnami/grafana Helm chart can be installed by running the following command:
helm install grafana bitnami/grafana
You’ll see new pods in a few seconds.
grafana-66c98bcb86-xpd5t 1/1 Running 0 2d23h
You will need to take an additional step before proceeding. Installing the Grafana Helm chart generates a randomized password and stores it as a secret in your Kubernetes cluster. To retrieve this password, you will need to execute the following command:
echo “$(kubectl get secret grafana-admin –namespace default -o jsonpath=”{.data.GF_SECURITY_ADMIN_PASSWORD}” | base64 –decode)”
Note down the generated password, as you will need it shortly. To access the service outside of the Kubernetes cluster, use the kubectl port-forward command in a new terminal:
export POD_NAME=$(kubectl get pods –namespace default -l “app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana” -o jsonpath=”{.items[0].metadata.name}”)
kubectl –namespace default port-forward $POD_NAME 3000
After executing the command, Grafana will be accessible at http://localhost:3000. When accessing the URL, you will be prompted for a username set to “admin” by default, along with the password you obtained in the previous step. However, upon logging in, you might notice that the Grafana dashboard appears incomplete.
That’s because, while Prometheus is automatically collecting metrics from your Kubernetes cluster, Grafana is not yet aware of your Prometheus installation. Grafana can easily be configured to use Prometheus as a data source and seamlessly communicate with the Prometheus server.
Summary
In this article, we’ve discussed the importance of Kubernetes observability and introduced two open-source tools, Prometheus and Grafana, to monitor and analyze the state and behavior of a Kubernetes cluster and applications running on it.
Then, we went through a step-by-step guide on how to set up Prometheus and Grafana on an existing Kubernetes cluster to create a dashboard in Grafana to visualize the collected data.
Read more:
Getting started with Kubernetes Observability with Prometheus and Grafana