Skip to content

ADR 004

Collect telemetry with OpenTelemetry into Mimir and Loki

OpenTelemetry Collectors send metrics to Mimir and logs to Loki. Notifications and the dead-man's switch run off-site, so alerts still arrive when the lab is down.

Status
Accepted
Rollout
0 of 7 gates
Related
006 007 011 012

Accepted, not yet implemented.

Context

One operator runs the platform. Metrics, logs and alerts have to show whether the cluster is healthy, whether an upgrade worked, whether backups are fresh and which component is breaking. They also have to survive one Proxmox host failure, as the rest of the platform does (ADR 007).

Observability stays in the lab. Only the notification endpoint and the dead-man's switch run outside it, on the Oracle Cloud VM, because they must work when the lab is down (ADR 012).

Decision

Collection

Use the OpenTelemetry Collector everywhere. In the cluster, the OpenTelemetry Operator's Target Allocator reads ServiceMonitor and PodMonitor resources and spreads the scrape targets across the collector pods. Proxmox sends OTLP natively. Mimir and Loki accept OTLP, so the collectors send to them directly.

One small Alloy pod runs only mimir.rules.kubernetes and loki.rules.kubernetes. Its only job is to load rules from the cluster into the Mimir and Loki rulers. It collects no telemetry.

Metrics

Run Mimir in classic architecture with the mimir-distributed Helm chart:

  • Set ingest_storage.enabled: false and kafka.enabled: false.
  • Pin the chart version and follow its documented classic recipe, which also sets ingester.push_grpc_method_enabled: true.
  • Run three ingesters with replication factor 3, so one node can fail.
  • Run two replicas of each stateless component and one compactor.
  • Keep per-tenant retention in the chart's runtime overrides. The analytics tenant keeps its data longer than the default.

With three ingesters and replication factor 3, every series lives on every ingester. Ingester memory per node is therefore not divided by three. Size each ingester for the full series count. The gain over one Prometheus is that scraping, queries and rule evaluation run in other components, and one node can fail without losing metrics.

Mimir ingest storage is not used. It needs a Kafka-compatible broker. The chart's bundled Kafka is for demonstration and testing only, so production would need a separate Kafka-compatible service to run, upgrade and back up. For one operator that costs more than the classic write path. Classic is still supported. Revisit ingest storage when Grafana announces the deprecation date for classic.

Logs

Run Loki in single-binary mode with three replicas. Traces are out of scope.

Object storage

Mimir and Loki store blocks and chunks in Garage on TrueNAS.

If TrueNAS is down, ingesters keep recent data on their local disks and rules on recent data keep firing. Queries older than the ingester window fail, and the local disks fill up eventually. Size the ingester disks for a supported outage window, decided at build time. A blackbox probe and node metrics watch TrueNAS, and TrueNAS's own alerts send email.

Alerting

Keep rules in Git as PrometheusRule resources. PromQL rules and LogQL rules use separate selectors, so each ruler loads only its own rules.

Run a standalone Alertmanager with two replicas, defined by a prometheus-operator Alertmanager resource and AlertmanagerConfig resources. Receiver URLs come from OpenBao through External Secrets. The Mimir ruler and the Loki ruler both send to it through their alertmanager_url settings.

The Alertmanager resource takes its global routing from one AlertmanagerConfig through spec.alertmanagerConfiguration. The operator adds a namespace matcher to the routes of every namespaced AlertmanagerConfig, so those routes only match alerts that carry their namespace label. Watchdog and host alerts don't carry one, so the top-level route must come from spec.alertmanagerConfiguration.

Turn off Mimir's own Alertmanager and delete terraform/alerting.tf. One Alertmanager serves both rulers, and its routing is a Kubernetes resource in Git instead of Terraform state.

Notifications

Send alerts to Discord, and to ntfy on the Oracle Cloud VM for alerts that must be seen. ntfy runs with:

  • a persistent message cache;
  • a deny-by-default ACL, write-only publisher users and a read-only phone user;
  • its /metrics endpoint scraped by the lab.

Alertmanager posts to ntfy with ?template=alertmanager. Gotify is retired.

Dead-man's switch

Run Uptime Kuma on the Oracle VM. These ping its push monitors:

  • the Watchdog alert from Alertmanager;
  • the jobs that run outside the cluster: the Docker host Git backup and the daily Terraform state copy.

The Oracle VM joins the Ansible inventory and the docs. The lab checks that the VM itself is alive, because Oracle reclaims idle free-tier VMs.

Not in scope

Traces, profiling, service maps and SLO automation. Each would need its own decision.

Consequences

  • Mimir has more components than one Prometheus. In exchange it tolerates a node failure and keeps long-term data in object storage.
  • Three-way replication triples ingester memory and disk across the cluster. The capacity rule in ADR 007 has to include it.
  • Mimir and Loki depend on TrueNAS for anything older than the ingester window. A long TrueNAS outage fills the ingester disks.
  • Alloy stays only for rule sync.
  • Mimir and Loki majors are in the manual Renovate set (ADR 011).

Implementation status

Not built. Today Alertmanager routing lives in Mimir's built-in Alertmanager, managed by Terraform in terraform/alerting.tf with the mimirtool provider.

References

Rollout gates

  • On staging, Mimir, Loki, the collectors, the rule-sync Alloy pod and the standalone Alertmanager run as production will run them. open
  • A Mimir alert, a Loki alert and Watchdog each arrive at their receivers on staging. open
  • A staging drill with Garage stopped, including a ruler restart, confirms that a notification arrives. open
  • ntfy delivers to the phone with the screen off, on Wi-Fi and on mobile data. open
  • The Uptime Kuma push monitors and the lab-side check of the Oracle VM alert when their source stops. open
  • The ingester disk size and the outage window it covers are recorded. open
  • The rules and routing run on production's Alertmanager, and terraform/alerting.tf is deleted. open

docs/decisions/004-opentelemetry-observability.md