Overview¶

Architecture Overview

Metrics Collection Deployment Steps

Deploy a Prometheus Operator using YAML files
Deploy a Thanos using bitnami helm chart (Receiver, Store Gateway, Querier, Compactor, )
Deploy a Thanos ruler using Prometheus Operator
Deploy a Prometheus Alertmanager using Prometheus Operator
Deploy a OpenTelemetry Operator
Deploy a OpenTelemetry Collector using OpenTelemetry Operator
Deploy a Grafana using helm chart

cd ~/Projects/retail-lakehouse/observability
bash ./install.sh

The install.sh script in the observability directory is as follows:

install.sh

#!/bin/bash

set -euo pipefail

NAMESPACE=observability

kubectl create namespace $NAMESPACE || true

# Install Prometheus Operator using YAML files
# https://prometheus-operator.dev/docs/getting-started/installation/#install-using-yaml-files
TMPDIR=$(mktemp -d)
LATEST=v0.85.0
curl -s "https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags/$LATEST/kustomization.yaml" > "$TMPDIR/kustomization.yaml"
curl -s "https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags/$LATEST/bundle.yaml" > "$TMPDIR/bundle.yaml"
(cd $TMPDIR && kustomize edit set namespace $NAMESPACE) && kubectl create -k "$TMPDIR"
sleep 5
kubectl wait --for=condition=Ready pods -l  app.kubernetes.io/name=prometheus-operator -n $NAMESPACE --timeout=180s

# Deploy Jaeger using Otel Operator
# 
cd ~/Projects/retail-lakehouse/observability/jaeger
CERT_MANAGER_VERSION=v1.18.2
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/$CERT_MANAGER_VERSION/cert-manager.yaml
OTEL_OPERATOR_VERSION=v0.132.0
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/download/$OTEL_OPERATOR_VERSION/opentelemetry-operator.yaml
kubectl create namespace jaeger || true
kubectl apply -f otel-collector-jaeger.yaml -n  jaeger

# Deploy Thanos using kube-thanos jsonnet
# https://github.com/thanos-io/kube-thanos
cd ~/Projects/retail-lakehouse/observability/thanos

kubectl create namespace thanos || true
jb install github.com/thanos-io/kube-thanos/jsonnet/kube-thanos@main
rm -f manifests/thanos-*
jsonnet -J vendor -m manifests/ thanos.jsonnet | xargs -I{} sh -c "cat {} | yq -P > {}.yaml; rm -f {}" -- {}

kubectl apply -f manifests/

Manually¶

kubectl apply -f manifests/prometheus-alertmanager.yaml
kubectl get all -n thanos

Result

NAME                                  READY   STATUS    RESTARTS   AGE
pod/alertmanager-alertmanager-sre-0   2/2     Running   0          2m2s
pod/alertmanager-alertmanager-sre-1   2/2     Running   0          2m2s

NAME                            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
service/alertmanager-operated   ClusterIP   None         <none>        9093/TCP,9094/TCP,9094/UDP   2m2s

NAME                                             READY   AGE
statefulset.apps/alertmanager-alertmanager-sre   2/2     2m2s

kubectl apply -f manifests/memcached.yaml
k get all -n thanos

Result

NAME                                  READY   STATUS    RESTARTS   AGE
pod/alertmanager-alertmanager-sre-0   2/2     Running   0          6m22s
pod/alertmanager-alertmanager-sre-1   2/2     Running   0          6m22s
pod/memcached-0                       1/1     Running   0          44s
pod/memcached-1                       1/1     Running   0          33s
pod/memcached-2                       1/1     Running   0          22s

NAME                            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
service/alertmanager-operated   ClusterIP   None         <none>        9093/TCP,9094/TCP,9094/UDP   6m22s
service/memcached               ClusterIP   None         <none>        11211/TCP                    44s

NAME                                             READY   AGE
statefulset.apps/alertmanager-alertmanager-sre   2/2     6m22s
statefulset.apps/memcached                       3/3     44s

Traces Collection Deployment Steps

Deploy a Jaeger
Deploy a OpenTelemetry Collector
Trino jmx