Mastering Observability Talk

This document provides a guide for setting up a Google Kubernetes Engine (GKE) cluster and installing the necessary components to collect and forward telemetry data.

Prerequisites

Before we begin, make sure the following tools are installed:

kubectl - Kubernetes command-line tool

helm - Kubernetes package manager

Google Cloud SDK (gcloud CLI)

Terraform-CLI

A Google Cloud account with billing enabled

Google Kubernetes Engine (GKE) Cluster Setup

Authenticate with Google Cloud

gcloud  auth  login
gcloud  config  set  project  liquid-mastering-obs-543

Create a cluster via Terraform

At first we need a basic cluster, where we can host our demo application, to produce telemetry data. We created a very basic Terraform file for this purpose, which create a GKE autopilot cluster.

cd  terraform
terraform  init
terraform  apply
cd  ..

Get credentials for kubectl

In order to access the cluster we need a valid kubeconfig, which is generated by the following command.

gcloud  container  clusters  get-credentials  mastering-observability  --zone=europe-west9

Check access to K8s

Now we can check the access.

kubectl  get  service

Demo Application

Last but not least, we will install a demo application called EasyTrade. EasyTrade is a Dynatrace demo application that simulates a Stock broker with deliberately built-in performance problems to demonstrate Dynatrace's monitoring capabilities. It provides a realistic environment for users to experience how Dynatrace detects and diagnoses issues across the full stack, from frontend user experience to backend services and infrastructure.

kubectl create ns easytrade
kubectl apply -k https://github.com/Dynatrace/easytrade.git//kubernetes-manifests/base -n easytrade

Installation of Telemetry Collectors

Setting up Grafana Cloud Kubernetes Monitoring

The Grafana Kubernetes Monitoring solution provides comprehensive monitoring for Kubernetes clusters by collecting metrics, logs, traces, and profiles using Grafana Agent. Follow these steps to set up monitoring with Grafana Cloud:

1. Prerequisites

• A Grafana Cloud account and API keys for:

• Metrics (Prometheus)

• Logs (Loki)

• Traces (Tempo)

• Profiles (Pyroscope)

2. Update Configuration File

Edit the chart/grafana-chart-values.yaml file to update your Grafana Cloud-specific credentials:

• Replace all empty password values with your respective API keys

• Update the URLs if needed (based on your Grafana Cloud instance region)

• Update the cluster.name value to match your cluster name

cluster:
  name: my-cluster  # Change to your cluster name

destinations:
  - name: grafana-cloud-metrics
    type: prometheus
    url: https://prometheus-prod-XX-prod-eu-west-2.grafana.net/api/prom/push
    auth:
      password: "your-prometheus-api-key"  # Add your API key

  - name: grafana-cloud-logs
    type: loki
    url: https://logs-prod-XXX.grafana.net/loki/api/v1/push
    auth:
      password: "your-loki-api-key"  # Add your API key

  - name: grafana-cloud-traces
    type: otlp
    url: https://tempo-prod-XX-prod-eu-west-2.grafana.net:443
    auth:
      password: "your-tempo-api-key"  # Add your API key

  - name: grafana-cloud-profiles
    type: pyroscope
    url: https://profiles-prod-XXX.grafana.net:443
    auth:
      password: "your-pyroscope-api-key"  # Add your API key

3. Add the Grafana Helm Repository

helm  repo  add  grafana  https://grafana.github.io/helm-charts
helm  repo  update

4. Install the Grafana Kubernetes Monitoring Chart

helm  upgrade  --install  k8s-monitoring  grafana/k8s-monitoring  \
--namespace monitoring \
--create-namespace  \
-f chart/grafana-chart-values.yaml

5. Verify the Installation

kubectl  get  pods  -n  monitoring

6. Access Your Metrics in Grafana Cloud

Once installed, your cluster's metrics, logs, traces, and profiles will be available in your Grafana Cloud instance. You can access them through:

• Grafana Dashboard: Browse the Kubernetes dashboards in your Grafana Cloud instance

• Explore: Use Explore to query your metrics, logs, traces, and profiles

7. Features Enabled

The configuration enables:

• Cluster metrics collection (including OpenCost)

• Kubernetes events

• Pod logs

• Application observability (OTLP receivers for traces)

• Auto-instrumentation

• Profiling

• Annotation autodiscovery

For more information on customizing your setup, refer to the Grafana Kubernetes Monitoring documentation.

Dynatrace Setup

1. Prerequisites

• A Dynatrace Environment with:

  • an Operator Token

  • a Data Ingest Token

Tokens can be obtained from:http://<dynatrace-url>/ui/apps/dynatrace.classic.deploy.oneagent/ui/deployment/oneagent/kubernetes

2. Install Dynatrace Operator

helm install dynatrace-operator oci://public.ecr.aws/dynatrace/dynatrace-operator \
--create-namespace \
--namespace dynatrace \
--atomic

3. Apply Dynakube

Edit the dynatrace/dynakube.yaml with your apiToken and dataIngestToken obtained from the previous step and apply the dynakube.yaml:

kubectl apply -f dynatrace/dynakube.yaml