Skip to content

Lab: Monitoring and Logging

Duration: 25 minutes

Objectives

  • Install a Prometheus and Grafana monitoring stack
  • Deploy an application that exposes Prometheus metrics
  • Configure Prometheus scraping with a ServiceMonitor
  • Query application metrics with PromQL
  • Explore Kubernetes metrics in Grafana
  • Install Loki and query application logs
  • Create and test a Prometheus alert

Prerequisites

  • Kind cluster running
  • kubectl configured and working
  • Helm 3 installed
  • At least 8GB RAM available for the cluster

Tasks

Task 1: Install the Monitoring Stack

Install kube-prometheus-stack into a monitoring namespace.

Requirements:

  • Add the Prometheus Community Helm repository
  • Create the monitoring namespace
  • Install the chart with release name monitoring
  • Configure Prometheus to discover ServiceMonitors and PrometheusRules outside the Helm release labels
  • Verify that Grafana, Prometheus, AlertManager, node-exporter, and kube-state-metrics are running
Hint

Add the chart repository and create the namespace:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring

Install the stack with ServiceMonitor and PrometheusRule selector overrides:

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set grafana.service.type=ClusterIP \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false

Check the Pods:

kubectl get pods -n monitoring
Solution 
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set grafana.service.type=ClusterIP \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false

Wait for the main components:

kubectl wait --for=condition=Available deployment/monitoring-grafana \
  -n monitoring --timeout=180s

kubectl wait --for=condition=Ready pod \
  -l app.kubernetes.io/name=prometheus \
  -n monitoring --timeout=180s

kubectl get pods -n monitoring

Expected output:

NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-monitoring-kube-prometheus-alertmanager-0   2/2     Running   0          2m
monitoring-grafana-xxxxxxxxxx-xxxxx                      3/3     Running   0          2m
monitoring-kube-prometheus-operator-xxxxxxxxxx-xxxxx     1/1     Running   0          2m
monitoring-kube-state-metrics-xxxxxxxxxx-xxxxx           1/1     Running   0          2m
monitoring-prometheus-node-exporter-xxxxx                1/1     Running   0          2m
prometheus-monitoring-kube-prometheus-prometheus-0        2/2     Running   0          2m

Task 2: Deploy a Metrics-Enabled Application

Deploy podinfo into a dedicated namespace and expose both its HTTP traffic and metrics endpoint.

Requirements:

  • Namespace: observability-lab
  • Deployment name: podinfo
  • Image: ghcr.io/stefanprodan/podinfo:6.5.3
  • Replicas: 2
  • Label: app=podinfo
  • Container port name: http
  • Container port: 9898
  • CPU request: 100m, Memory request: 64Mi
  • CPU limit: 250m, Memory limit: 128Mi
  • Service name: podinfo
  • Service port http: 80 → target port http
  • Service port metrics: 9797 → target port http
Hint

Create the namespace first:

kubectl create namespace observability-lab

Your Service should expose two named ports:

ports:
- name: http
  port: 80
  targetPort: http
- name: metrics
  port: 9797
  targetPort: http

Apply your manifest and verify the Deployment:

kubectl apply -f podinfo.yaml
kubectl wait --for=condition=Available deployment/podinfo \
  -n observability-lab --timeout=90s

Port-forward both service ports:

kubectl port-forward -n observability-lab svc/podinfo 9898:80 9797:9797

In another terminal:

curl http://localhost:9898
curl http://localhost:9797/metrics | head
Solution 
kubectl create namespace observability-lab

Create podinfo-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: observability-lab
  labels:
    app: podinfo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
    spec:
      containers:
      - name: podinfo
        image: ghcr.io/stefanprodan/podinfo:6.5.3
        ports:
        - name: http
          containerPort: 9898
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 250m
            memory: 128Mi

Create podinfo-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: podinfo
  namespace: observability-lab
  labels:
    app: podinfo
spec:
  selector:
    app: podinfo
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: metrics
    port: 9797
    targetPort: http

Apply and verify:

kubectl apply -f podinfo-deployment.yaml
kubectl apply -f podinfo-service.yaml

kubectl wait --for=condition=Available deployment/podinfo \
  -n observability-lab --timeout=90s

kubectl get pods,svc -n observability-lab

Expected output:

NAME                           READY   STATUS    RESTARTS   AGE
pod/podinfo-xxxxxxxxxx-xxxxx   1/1     Running   0          30s
pod/podinfo-xxxxxxxxxx-xxxxx   1/1     Running   0          30s

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)            AGE
service/podinfo   ClusterIP   10.96.xxx.xxx   <none>        80/TCP,9797/TCP    30s

Test metrics endpoint:

kubectl port-forward -n observability-lab svc/podinfo 9898:80 9797:9797
curl http://localhost:9797/metrics | head
# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter

Task 3: Configure Prometheus Scraping

Create a ServiceMonitor so Prometheus discovers the podinfo metrics endpoint.

Requirements:

  • ServiceMonitor name: podinfo
  • Namespace: observability-lab
  • Select the Service with label app=podinfo
  • Scrape the metrics service port
  • Scrape path: /metrics
  • Scrape interval: 15s
  • Confirm the target appears as UP in Prometheus
Hint

A ServiceMonitor selects a Service by labels and scrapes a named Service port:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: podinfo
  namespace: observability-lab
spec:
  selector:
    matchLabels:
      app: podinfo
  endpoints:
  - port: metrics

Port-forward Prometheus:

kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

Open http://localhost:9090/targets and find the observability-lab/podinfo target.

Solution

Create podinfo-servicemonitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: podinfo
  namespace: observability-lab
  labels:
    app: podinfo
spec:
  selector:
    matchLabels:
      app: podinfo
  namespaceSelector:
    matchNames:
    - observability-lab
  endpoints:
  - port: metrics
    path: /metrics
    interval: 15s

Apply and inspect:

kubectl apply -f podinfo-servicemonitor.yaml
kubectl get servicemonitor -n observability-lab
kubectl describe servicemonitor podinfo -n observability-lab

Port-forward Prometheus:

kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

Open http://localhost:9090/targets.

Expected result:

observability-lab/podinfo/0 (2/2 up)

Task 4: Query Application Metrics

Use Prometheus to inspect the podinfo target and request metrics.

Requirements:

  • Confirm Prometheus can scrape podinfo
  • Generate successful and failed HTTP requests
  • Query request rate grouped by status code
  • Identify which status code appears after requesting a missing path
Hint

Start with the scrape health:

up{namespace="observability-lab"}

Then query request rate by HTTP status:

sum(rate(http_requests_total{namespace="observability-lab"}[1m])) by (status)

Generate traffic through a port-forward:

kubectl port-forward -n observability-lab svc/podinfo 9898:80

In another terminal:

for i in $(seq 1 30); do curl -s http://localhost:9898 > /dev/null; done
for i in $(seq 1 5); do curl -s http://localhost:9898/not-found > /dev/null; done
Solution

Generate traffic:

kubectl port-forward -n observability-lab svc/podinfo 9898:80

In another terminal:

for i in $(seq 1 30); do curl -s http://localhost:9898 > /dev/null; done
for i in $(seq 1 5); do curl -s http://localhost:9898/not-found > /dev/null; done

Run these PromQL queries in Prometheus:

up{namespace="observability-lab"}

Expected result: Two podinfo targets with value 1

sum(rate(http_requests_total{namespace="observability-lab"}[1m])) by (status)

Expected result: Request-rate series grouped by status, including 200 and 404 after the generated traffic.

Task 5: Explore Metrics in Grafana

Open Grafana and inspect both built-in Kubernetes dashboards and your application metrics.

Requirements:

  • Retrieve the Grafana admin password from the monitoring-grafana Secret
  • Port-forward Grafana to local port 3000
  • Open a Kubernetes namespace dashboard for observability-lab
  • Use Explore to run a PromQL query for podinfo request rate
Hint

Decode the Grafana password:

kubectl get secret -n monitoring monitoring-grafana \
  -o jsonpath="{.data.admin-password}" | base64 --decode
echo ""

Port-forward Grafana:

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

Open http://localhost:3000.

  • Username: admin
  • Password: the decoded secret value

Try this query in Grafana Explore:

sum(rate(http_requests_total{namespace="observability-lab"}[1m])) by (status)
Solution 
kubectl get secret -n monitoring monitoring-grafana \
  -o jsonpath="{.data.admin-password}" | base64 --decode
echo ""

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

Open http://localhost:3000.

  • Username: admin
  • Password: decoded from the monitoring-grafana Secret

Open Dashboards and choose Kubernetes / Compute Resources / Namespace (Pods). Select observability-lab.

In Explore, select the Prometheus data source and run:

sum(rate(http_requests_total{namespace="observability-lab"}[1m])) by (status)

Expected result: Grafana displays podinfo request rate grouped by HTTP status code.

Task 6: Install Loki and Query Logs

Install Loki with Promtail and query podinfo logs from Grafana.

Requirements:

  • Add the Grafana Helm repository
  • Install grafana/loki-stack into the monitoring namespace
  • Enable Promtail
  • Do not install a second Grafana instance
  • Add Loki as a Grafana data source
  • Query logs for the observability-lab namespace
Hint

Install Loki and Promtail without installing a second Grafana:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install loki grafana/loki-stack \
  --namespace monitoring \
  --set promtail.enabled=true \
  --set grafana.enabled=false

Use this data source URL in Grafana:

http://loki:3100

Try these LogQL queries:

{namespace="observability-lab"}

{namespace="observability-lab", app="podinfo"}
Solution 
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install loki grafana/loki-stack \
  --namespace monitoring \
  --set promtail.enabled=true \
  --set grafana.enabled=false

Verify Loki and Promtail:

kubectl wait --for=condition=Ready pod \
  -l app=loki \
  -n monitoring --timeout=120s

kubectl get pods -n monitoring | grep promtail

Add a Grafana data source:

Type: Loki
URL:  http://loki:3100

Generate logs:

kubectl port-forward -n observability-lab svc/podinfo 9898:80

In another terminal:

for i in $(seq 1 10); do curl -s http://localhost:9898/version; done
for i in $(seq 1 3); do curl -s http://localhost:9898/headers; done

Run these LogQL queries in Grafana Explore:

{namespace="observability-lab"}

{namespace="observability-lab", app="podinfo"}

Expected result: Recent podinfo access logs appear in Grafana Explore.

Task 7: Create and Test an Alert

Create a PrometheusRule that fires when Prometheus cannot find a podinfo target.

Requirements:

  • PrometheusRule name: podinfo-alerts
  • Namespace: observability-lab
  • Alert name: PodinfoTargetMissing
  • Expression should detect absence of the podinfo up metric
  • Alert should become pending/firing after podinfo is scaled to zero
  • Restore podinfo to 2 replicas after the test
Hint

The expression can use absent():

absent(up{namespace="observability-lab", service="podinfo"}) == 1

Test the alert:

kubectl apply -f podinfo-alert.yaml
kubectl scale deployment podinfo -n observability-lab --replicas=0

Open http://localhost:9090/alerts and wait for PodinfoTargetMissing.

Restore the application:

kubectl scale deployment podinfo -n observability-lab --replicas=2
kubectl wait --for=condition=Available deployment/podinfo \
  -n observability-lab --timeout=90s
Solution

Create podinfo-alert.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: podinfo-alerts
  namespace: observability-lab
  labels:
    app: podinfo
spec:
  groups:
  - name: podinfo.rules
    rules:
    - alert: PodinfoTargetMissing
      expr: absent(up{namespace="observability-lab", service="podinfo"}) == 1
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: "Podinfo target is missing"
        description: "Prometheus cannot find a podinfo metrics target."

Apply it:

kubectl apply -f podinfo-alert.yaml
kubectl get prometheusrule -n observability-lab

Open http://localhost:9090/rules and verify that PodinfoTargetMissing is loaded.

Scale the app down:

kubectl scale deployment podinfo -n observability-lab --replicas=0

Open http://localhost:9090/alerts.

Expected result: PodinfoTargetMissing moves from pending to firing after one minute.

Restore the app:

kubectl scale deployment podinfo -n observability-lab --replicas=2
kubectl wait --for=condition=Available deployment/podinfo \
  -n observability-lab --timeout=90s

Verification

Check your work:

kubectl get pods -n monitoring
kubectl get pods,svc,servicemonitor,prometheusrule -n observability-lab
kubectl get endpoints podinfo -n observability-lab

Expected outcomes:

  • observability-lab/podinfo target is UP
  • up{namespace="observability-lab"} returns podinfo samples
  • http_requests_total{namespace="observability-lab"} appears after traffic is generated
  • PodinfoTargetMissing appears on the Prometheus alerts page
  • Grafana can query Prometheus metrics and Loki logs

Cleanup

kubectl delete namespace observability-lab
helm uninstall loki -n monitoring
helm uninstall monitoring -n monitoring
kubectl delete namespace monitoring

Bonus Challenges

  • Create a Grafana dashboard panel for request rate by HTTP status.
  • Add an alert that fires when podinfo has fewer than two available replicas.
  • Use LogQL to count podinfo log lines over a five-minute window.
  • Compare kubectl logs output with the same logs in Loki.

Key takeaways

  1. Prometheus scrapes metrics via a pull model; ServiceMonitor resources define which Services to scrape and how
  2. PromQL enables powerful metric queries — range vectors and aggregation functions unlock deep analysis
  3. Grafana unifies Prometheus metrics and Loki logs in a single observability dashboard
  4. Loki provides log aggregation without indexing full log content, keeping storage costs low
  5. absent() in alert rules catches missing metrics — useful when a target disappearing is itself the problem
  6. Label selectors connect ServiceMonitors to Services; a mismatch silently produces no data
  7. Service targetPort must match the container port where metrics are actually exposed — a misconfigured port means Prometheus scrapes the wrong endpoint and returns no data

Next section

Once you've reviewed the content and completed the lab, proceed to the next section.