Operate the host OpenTelemetry Collector

Use this runbook for collectors installed by ansible/playbooks/opentelemetry.yml. The role sends host metrics and new journald entries to the OTLP endpoint in otelcol_exporter_endpoint.

Kubernetes node telemetry does not use this role. Flux manages that collector.

Verify a rollout

Run the playbook against one host before you add more hosts:

cd ansible
ansible-playbook playbooks/opentelemetry.yml -e target_hosts=my-host

The role validates the generated configuration, checks journal access, restarts the service, and checks the local health endpoint. Then inspect the service:

ansible my-host -b -m ansible.builtin.command \
  -a "systemctl --no-pager --full status otelcol-contrib"
ansible my-host -b -m ansible.builtin.command \
  -a "journalctl -u otelcol-contrib --since=-10m --no-pager"

Check the backend for host.name=my-host. Confirm that it has both recent host metrics and a recent journal record. The local health check does not prove that the backend accepted telemetry.

If the exporter reports TLS, authentication, or connection errors, stop the rollout. Fix the endpoint or its secret-backed settings before you add another host.

Roll back a configuration or package update

Revert the role to its last working commit. Run the playbook against one host:

cd ansible
ansible-playbook playbooks/opentelemetry.yml -e target_hosts=my-host

Confirm that the package version in the restored role has checksums for every supported architecture. Repeat the rollout checks before you continue.

Remove the collector from a host

Remove the opentelemetry or otel-collector tag from the host in NetBox first. This prevents a later playbook run from reinstalling the collector.

Stop and disable the service:

cd ansible
ansible my-host -b -m ansible.builtin.systemd \
  -a "name=otelcol-contrib state=stopped enabled=false"

Remove the package and the systemd override:

ansible my-host -b -m ansible.builtin.apt \
  -a "name=otelcol-contrib state=absent"
ansible my-host -b -m ansible.builtin.file \
  -a "path=/etc/systemd/system/otelcol-contrib.service.d state=absent"
ansible my-host -b -m ansible.builtin.systemd -a "daemon_reload=true"

Remove the collector account from systemd-journal:

ansible my-host -b -m ansible.builtin.command \
  -a "gpasswd --delete otelcol-contrib systemd-journal"

Package removal keeps /etc/otelcol-contrib and /var/lib/otelcol-contrib. Keep the state directory if you might reinstall the collector. It contains the journald cursor and queued telemetry. Delete it only when losing that state is acceptable:

ansible my-host -b -m ansible.builtin.file \
  -a "path=/var/lib/otelcol-contrib state=absent"

docs/runbooks/opentelemetry-host-collector.md