Skip to content

Copy production Grafana into staging

Use this procedure to create an independent test copy. Keep production Grafana running and leave its deployment, PVC, DNS record, and OAuth client unchanged. Do not use this procedure for a production cutover.

Check the target before copying data

  1. Confirm that talos-test is the intended staging context. Verify the node addresses against the Talos lifecycle runbook.
  2. Confirm that production Grafana still responds at https://grafana.local.isabelsoler.es/api/health.
  3. Review the staging resources under kubernetes/apps/staging/grafana/ and kubernetes/infrastructure/staging/. Keep Grafana on the source image digest during the copy. Do not combine the restore with a version upgrade.
  4. Verify that GF_UNIFIED_ALERTING_ENABLED=false is present before the first staging start. The database retains copied alert rules and contact points, but staging must not start its alerting engine or notification dispatcher.
  5. Keep the source PVC and database untouched. Never mount the source volume read-write in an export helper, and never copy a live SQLite file with cp.

Prepare staging storage

Follow ADR 008. Use the dedicated worker disk, not the Talos OS disk or an emptyDir volume.

  1. Review Terraform changes for worker talos-test-worker-01 only. The optional Longhorn settings add scsi1 without changing scsi0.
  2. If a new schematic defers configuration data sources for all nodes, apply the schematic creation separately. Re-plan before any VM mutation. Reject control-plane or production changes.
  3. Attach the disk through the reviewed Terraform plan. Inspect both Proxmox and Talos disk inventories before applying the user-volume configuration.
  4. Check that the selected disk is empty and that the selector matches its exact observed identity. The staging worker uses its SCSI by-path symlink because Talos does not report the serial assigned to the hotplugged Proxmox disk. Do not infer a disk identity from /dev/sdb.
  5. Obtain the exact NoCloud installer from Terraform. Follow the worker-only maintenance procedure in Talos lifecycle management. Keep the Talos version unchanged for an extension-only operation.
  6. Verify iscsi-tools, util-linux-tools, the iscsid service, and node readiness.
  7. Apply the reviewed worker configuration. Verify that u-longhorn is ready, that its ext4 filesystem is mounted at /var/mnt/longhorn, and that kubelet has the shared bind mount at /var/mnt/longhorn.
  8. Reconcile the dedicated longhorn Flux Kustomization. Use longhorn-staging-r1-retain explicitly for the Grafana PVC.
  9. Verify CSI registration, disk readiness, PVC binding, and persistent data across a test pod recreation before importing the database.

The staging class has one replica on one worker. It has no availability redundancy. Retain protects against automatic volume deletion, not data loss, and does not replace an off-cluster backup.

Correct an initial disk-path mismatch

Do not import data if Longhorn reports the OS filesystem or OS-disk capacity. The host, kubelet mount destination, and Longhorn data path must all be /var/mnt/longhorn. Changing the Helm default alone does not relocate an already registered Longhorn disk or its replicas.

  1. Keep Grafana behind its restore gate and verify that its application container has never started. Confirm that no database or verification marker exists.
  2. Apply the reviewed Terraform mount correction and Flux chart change.
  3. Use Longhorn disk management to register the dedicated path and disable new scheduling on the OS-backed disk. Preserve any volume containing user data.
  4. Relocate the replica, or replace only a separately verified empty staging volume after an explicit data-safe recovery decision. Do not treat an empty deployment directory as proof that every Longhorn volume is disposable.
  5. Require an ext4 host mount with the dedicated disk capacity, a healthy replica on that disk, and a PVC persistence test before restoring the gated pod. Longhorn can report filesystemType: ext2/ext3 for this ext4 mount. Verify the actual filesystem through Talos read /proc/1/mountinfo and get volumestatus u-longhorn; do not rewrite controller-owned status to force an ext4 label.
  6. For an empty-PVC replacement, commit replicas: 0 and wait for Flux to remove the pod. Recheck the exact PVC UID, bound PV, Retain policy, and absence of all files other than empty lost+found through a temporary helper. Delete only that claim with a UID precondition. Preserve the released PV and Longhorn volume while Flux provisions the replacement.
  7. Write a uniquely named test file on the new claim, recreate the helper pod, read and verify its contents, then remove only that test file and helper. Commit replicas: 1 only after storage checks pass. Keep the verification marker absent so the Grafana application remains unstarted.

Reserve ingress without conflicting with production

The staging ingress reservation is 10.0.20.90/32. Production MetalLB owns 10.0.20.100-10.0.20.199; the reported DHCP range is 10.0.20.200-250.

Before reusing or changing the staging address, check the current VM inventory, Kubernetes address pools, DNS records, and a direct VLAN-local ARP probe. An unanswered ping does not establish availability. Offline, undocumented assignments remain outside these checks.

Manage the grafana-staging.local.isabelsoler.es record in Terraform. Do not change production's grafana record or edit BIND manually. The staging-only Traefik overlay removes the inherited production VIP and uses externalTrafficPolicy: Cluster for Cilium L2 announcements.

Use the namespaced DNS-01 Issuer to obtain a new staging certificate. Do not copy the production certificate private key.

Transfer configuration through OpenBao

Follow ADR 009. Obtain authorized operator access before reading or transferring secret values.

  1. Copy the effective source grafana.ini into kv/apps/staging/grafana as config. Preserve every setting. Override only the staging hostname and alerting behavior in the staging Deployment.
  2. Store the source's effective encryption key as encryption_key at the same path. If the source has no override, read the default from the pinned source image. A different key prevents decryption of copied datasource credentials.
  3. Store the required DNS-01 credential in kv/infra/staging/ingress as cloudflare_api_token. Keep it separate from Grafana's mounted configuration.
  4. Apply the reviewed Terraform JWT auth backend, roles, and policies. The Grafana and ingress service accounts have separate read-only policies and short-lived tokens. Do not distribute the operator token to either workload.
  5. Verify both SecretStores and ExternalSecrets. Never print their secret values.

The staging service-account public signing key is pinned in terraform/files/talos-staging-service-account.pub. After a staging cluster recreation or signing-key rotation, verify the new issuer and key before updating this file and the OpenBao trust configuration.

Restore and release the startup gate

  1. Let Flux create grafana-staging-data and the Deployment. Its init container waits for /var/lib/grafana/.staging-copy-verified.
  2. Create a consistent SQLite snapshot with the SQLite online backup API from a read-only source mount. Keep production running.
  3. Archive the snapshot and plugin/data files in a protected operator directory. Exclude the live database, WAL, and shared-memory files from the filesystem archive. Include the consistent snapshot as grafana.db instead.
  4. Verify the archive checksum and snapshot integrity before import. Keep the source and destination cluster identities distinct.
  5. Refuse to overwrite an existing restored database or verification marker. Investigate a partial previous restore before retrying.
  6. Extract safely into the staging PVC. Reject absolute paths, parent traversal, and unsafe links. Give the restored files to UID and GID 472.
  7. Compare the restored database's integrity, table counts and content hashes, and plugin inventory with the snapshot. Write the verification marker last.
  8. Verify the Deployment becomes ready. Remove only the temporary helper pods created by this operation. Preserve the private snapshot until the restore has passed the checks below and its retention is agreed.

Verify the copy

  • Confirm that Flux applied the intended commit and that its Kustomizations are Ready.
  • Verify HTTPS trust and /api/health, including database: ok.
  • Compare dashboards, folders, users, organisations, datasource definitions, plugins, and stored alert rules with the snapshot inventory.
  • Test each datasource. Staging-only Services and EndpointSlices preserve the copied Mimir and Loki DNS names and forward to their production load-balancer addresses. No Mimir or Loki data is copied. Verify these endpoints after any production backend move. Other datasource definitions remain unchanged.
  • Test Authentik login and ensure redirects stay on the staging hostname. A public OAuth endpoint returning HTTP 403 does not prove that SSO works. If a new callback requires an identity-provider change, obtain approval rather than silently changing the production OAuth client.
  • Confirm that the alerting engine remains disabled and no duplicate notifications were sent.
  • Recreate the staging pod and verify that its data persists.
  • Recheck production health and confirm that its deployment and PVC identities are unchanged.

Stop or recover the test copy

To stop the copy, set its replica count to zero through Git and Flux. Keep the staging PVC while investigating. Its prune-protection annotation and the StorageClass's Retain policy do not authorize deleting the data.

If the staged copy fails, continue using production. Do not copy the modified staging database back into production, merge the databases, or switch production DNS as part of this procedure.

docs/runbooks/grafana-staging-copy.md