Skip to content

Terraform

OpenTofu manages one main root and local modules for each infrastructure domain. A separate recovery bootstrap root manages the GCS state bucket. Both roots and CI use OpenTofu 1.13.1.

All Terraform source lives under terraform/. Run the commands on this page from that directory unless a command says otherwise.

Repository layout

  • Root *.tf files: single Terraform root. Allow direnv, then use plain tofu commands in this directory
  • modules/dns/: local BIND/RFC2136 records and Cloudflare records
  • modules/k8s_nodes/: Proxmox cloud-init VMs for Kubernetes nodes
  • modules/talos_vms/: Talos VM, machine configuration, bootstrap, and client configuration lifecycle
  • modules/openbao_gcp/: GCP Cloud KMS resources for OpenBao auto-unseal
  • modules/pangolin/: Pangolin public resources and backend targets
  • modules/proxmox_access/: Proxmox user, API token, and least-privilege ACLs for Hermes
  • modules/proxmox_vms/: standalone named Proxmox VMs
  • bootstrap/gcs-state/: isolated OpenTofu GCS bucket bootstrap with local state that must remain outside the bucket
  • alerting.tf: Mimir Alertmanager routing managed with the OVH mimirtool Terraform provider

Inputs and secrets

Keep non-secret inventory in checked-in *.auto.tfvars files, as with k8s_nodes.auto.tfvars and records.auto.tfvars. OpenTofu reads provider secrets from OpenBao using the HashiCorp Vault provider. The GCS backend uses Google Application Default Credentials.

Use this workflow:

export VAULT_ADDR="https://vault.local.isabelsoler.es"
bao login -method=oidc
cd terraform
direnv allow
tofu init -reconfigure -input=false -lockfile=readonly
tofu plan

.envrc sets VAULT_TOKEN for both bao and the Vault provider in this order:

  1. VAULT_TOKEN, if already set.
  2. The admin token that bao login writes to ~/.vault-token, if bao token lookup accepts it. Override the path with OPENBAO_ADMIN_TOKEN_FILE.
  3. BAO_TOKEN, copied into VAULT_TOKEN.
  4. The OpenBao Agent sink, $XDG_RUNTIME_DIR/openbao-token by default. Override it with OPENBAO_AGENT_TOKEN_FILE.

If none of these exist, .envrc leaves VAULT_TOKEN unset, and bao and the provider fall back to their own token file. direnv watches the admin token file and the sink, so the environment reloads after a login or when the agent writes a new token.

On the vscode VM, the user service openbao-agent.service keeps the sink filled with a token for the vscode-vm AppRole, so bao login is not needed there. That token can only read and list, which is enough for tofu plan. For an apply that changes OpenBao resources, log in first:

bao login -method=oidc

direnv then switches tofu to the admin token. The OIDC token expires after an hour, but direnv only rechecks it when it reloads. If a plan then fails with 403 permission denied on auth/token/lookup-self, run direnv reload to fall back to the agent token, or log in again. From the repository root, direnv exec terraform tofu -chdir=terraform ... checks the token on every run.

Don't export VAULT_TOKEN yourself. It takes priority over both token files. If you did, run unset VAULT_TOKEN.

On the VM, ~/.bashrc sets BAO_TOKEN_PATH to the sink, so bao reads the agent token by default. A bao shell function sends bao login to ~/.vault-token instead, and bao-admin runs any command with that admin token. Without the function, a plain bao login would overwrite the agent token with your admin token. If the sink ever has the wrong token, run systemctl --user restart openbao-agent.service.

.envrc also sets TERRAFORM_VAULT_SKIP_CHILD_TOKEN=true. The provider then uses the token directly instead of creating a child token, which the AppRole policy does not allow.

vscode_vm_openbao.tf manages the approle/ auth mount, the vscode-vm role, and the vscode-vm policy. They were created by hand and imported. The mount and the role have prevent_destroy. Replacing the role would change its role_id, and deleting the mount deletes every AppRole credential. Either way the agent stops logging in. If a plan wants to replace either one, stop and find out why. prevent_destroy does not cover deleting the resource block. To stop managing these objects, use a removed block with destroy = false or tofu state rm.

Terraform does not manage the secret_id, so it stays out of state. The role_id is in state. To rotate the secret_id, use bao with an admin token and update the agent's secret_id file on the VM.

The agent token cannot change its own policy. Apply changes to these resources with an OIDC token, as described above.

The vscode-vm policy grants read and list on every path. That's deliberate. The VM is trusted and the token can't write, and a per-secret list would need an update each time this root, .envrc, or interactive bao on the VM reads a new secret. If you ever narrow it, check that the new policy still covers every path tofu plan and .envrc read, or the next plan with the agent token fails.

By default, secrets are read from these OpenBao KV v2 paths:

  • kv/infra/cloudflare: Cloudflare credentials, default keys api_token and zone_id
  • kv/infra/dns/local.isabelsoler.es: RFC2136 DNS credentials, default key key_secret
  • kv/infra/proxmox/C1/terraform: Proxmox credentials, default keys api_token_id, api_token_secret, and ssh_keys
  • kv/infra/pangolin/terraform: Pangolin Integration API credentials, default key api_key
  • kv/infra/alerting: Alert notification credentials, default key discord_webhook_url

Hermes Proxmox access

The Proxmox access module creates the hermes@pve!hermes API token for the hermes@pve user. It keeps the existing api-ro-users membership and its PVEAuditor ACL on / unchanged. Token privilege separation stays disabled.

The module also manages the backup permissions below:

Subject Path Role Privileges Propagate
hermes@pve /vms BackupCreator VM.Backup yes
hermes@pve /storage/pbs BackupStorageWriter Datastore.AllocateSpace yes

Both custom roles contain only the listed privilege. Direct user ACLs for PVEAuditor are also present at /vms and /storage/pbs. Proxmox can give a direct user ACL precedence over an inherited group ACL, so these entries keep Hermes' existing audit access on the paths where the new direct grants apply.

This access is intentionally write-capable. VM.Backup includes VM restore operations, and backup retention or prune/delete actions can remove recovery points. Review backup jobs and retention settings before enabling them. The module grants access to the existing PBS storage path; it does not manage PBS retention policy.

Stop-mode backups can interrupt guests. Prefer snapshot mode where supported, and wait for a successful backup task before modifying a guest. These guest backups do not replace the application-specific recovery methods in ADR 006.

The initial permissions were applied through pveum and adopted into Terraform state. On September 25, 2026, a targeted read-only plan refreshed all six resources and reported no changes. The root file proxmox_access_imports.tf retains the six import declarations as a record of their origin. Terraform ignores them while the destinations remain in state. A clean plan does not prove that backup, restore, and pruning work as intended; test those separately.

For normal revocation, remove the grants in Terraform and review the plan. If Terraform is unavailable and you must revoke access immediately, first check whether another ACL uses either custom role. Then run these commands on a Proxmox node as an administrator:

pveum acl delete /vms --users hermes@pve --roles BackupCreator,PVEAuditor
pveum acl delete /storage/pbs --users hermes@pve --roles BackupStorageWriter,PVEAuditor
pveum role delete BackupCreator
pveum role delete BackupStorageWriter

This restores the inherited group audit permissions without changing the group. Remove the corresponding Terraform resources and import declarations before the next apply, or Terraform will restore the grants.

If Terraform state loses these resources but they still exist on Proxmox, review an import plan before applying. The retained import declarations already identify the six resources. For manual state recovery instead, use:

tofu import 'module.proxmox_access.proxmox_virtual_environment_role.hermes_backup' BackupCreator
tofu import 'module.proxmox_access.proxmox_virtual_environment_role.hermes_backup_storage' BackupStorageWriter
tofu import 'module.proxmox_access.proxmox_acl.hermes_vms_auditor' '/vms?hermes@pve?PVEAuditor'
tofu import 'module.proxmox_access.proxmox_acl.hermes_backup' '/vms?hermes@pve?BackupCreator'
tofu import 'module.proxmox_access.proxmox_acl.hermes_storage_auditor' '/storage/pbs?hermes@pve?PVEAuditor'
tofu import 'module.proxmox_access.proxmox_acl.hermes_backup_storage' '/storage/pbs?hermes@pve?BackupStorageWriter'

After the first apply, copy the sensitive token output into Hermes' .env:

tofu output -raw hermes_proxmox_api_token_id
tofu output -raw hermes_proxmox_api_token_secret

If a secret uses different key names, override the corresponding *_key variable rather than moving secrets into a Terraform-specific path.

Required non-secret values that are intentionally environment-specific include proxmox_api_url, openbao_addr, and openbao_skip_tls_verify.

Alerting

alerting.tf manages Mimir Alertmanager tenant routing for the anonymous tenant using the OVH-maintained ovh/mimirtool Terraform provider. The routes mirror the current Grafana contact points: Discord remains the default receiver, severity="critical" alerts go to a must-see receiver that sends both Discord and Gotify, and alerts labelled investigate="true" or incident="true" go to the Hermes webhook for automated investigation. The Hermes webhook uses X-Gitlab-Token with the route secret because Alertmanager cannot emit Grafana-style HMAC signatures. warning/info alerts keep quieter repeat timing, and lower-severity duplicates are inhibited when a critical alert with the same alertname and namespace is firing.

State backend

For slow plans, follow Investigate slow Terraform plans to distinguish backend access from resource refresh delays.

The main root stores state in the GCS bucket homelab-igresc-terraform-state under prefix homelab. Static backend settings are in versions.tf. Authenticate with Google Application Default Credentials before initializing a checkout:

gcloud auth application-default login
tofu init -reconfigure -input=false -lockfile=readonly

Use -reconfigure when initializing an existing checkout after pulling the backend and CLI changes. Do not use -migrate-state or -force-copy to adopt this configuration. GCS already holds the authoritative state.

ADR 010 records the separate backend migration. The GCS runbook covers operator authentication, the manual B2 copy after each apply, and state recovery.

The Talos module also generates cluster machine secrets and configurations, applies them to the nodes, bootstraps the first control plane, and retrieves talosconfig and kubeconfig. Remote Terraform state stores these sensitive values. Keep the Talos version, Kubernetes version, boot ISO, and installer image pinned to matching releases.

The generated machine configuration is uploaded to PVE5 as a Proxmox snippet and attached as NoCloud user-data. The BPG provider therefore needs direct SSH access to PVE5. By default, .envrc reads $HOME/.ssh/id_rsa.priv.pem. Override PROXMOX_SSH_PRIVATE_KEY_FILE when using another key.

Talos system extensions

talos_image_factory.tf declares the system extensions in the immutable Talos image. Terraform creates the content-addressed Image Factory schematic and derives the matching installer image and NoCloud ISO URL. Adding or removing an extension creates a new schematic ID.

Changing the schematic does not mutate a running Talos installation. Reconcile each node to the new image with a health-gated talosctl upgrade, worker first and then control plane. The talos_installer_image and talos_iso_url outputs provide the exact artifacts for that operation and future recovery.

Talos Cilium and Flux bootstrap

The flux_operator_bootstrap module runs its job on the host network and installs Cilium only when the release is absent. This lets Flux controllers start on Talos without kube-proxy. The minimal values in cilium-bootstrap-values.yaml configure kube-proxy replacement, KubePrism, Kubernetes IPAM, VXLAN routing, Talos cgroups, and required Linux capabilities.

See Talos Cilium and Flux bootstrap for the pre-CNI API access requirement, failure diagnostics, recovery procedure, and bootstrap resource lifecycle.

After bootstrap, Flux adopts the cilium Helm release and reconciles its complete configuration from kubernetes/infrastructure/networking/cilium. Flux also adopts the gateway-api-crds release before enabling Cilium Gateway API, L2 announcements, and standalone Envoy. Terraform retains only the temporary bootstrap transport resources and managed Git pull secret.

The private Flux deploy key and pinned Gitea host keys are read from kv/infra/gitea/flux-staging through the Vault provider. Override the path with TF_VAR_flux_secret_path.

LoadBalancer IP pools and L2 announcement policies remain cluster-specific GitOps resources. Do not add an IP pool until its address range is reserved outside DHCP and checked against existing static assignments.

After an apply, run these checks:

tofu output -raw talos_kubeconfig > /tmp/talos-test.kubeconfig
KUBECONFIG=/tmp/talos-test.kubeconfig kubectl get nodes
KUBECONFIG=/tmp/talos-test.kubeconfig helm -n kube-system status cilium

OpenTofu compatibility checks

Keep provider versions unchanged when changing IaC CLIs. After the GCS backend cutover, initialize with the committed lock file and review a read-only plan:

tofu init -reconfigure -input=false -lockfile=readonly
TERRAFORM_VAULT_SKIP_CHILD_TOKEN=true tofu plan -input=false -lock=false

The environment setting prevents the Vault provider from creating a child token during this read-only check. It uses the existing operator token.

Require no changes before approving the first OpenTofu apply. Before that apply, record the GCS generation of homelab/default.tfstate. The first OpenTofu apply rewrites provider addresses in state from registry.terraform.io to registry.opentofu.org, even when infrastructure does not change.

To roll back to Terraform after an OpenTofu state write:

  1. Freeze every Terraform and OpenTofu writer. Preserve the current GCS generation and an encrypted recovery copy.
  2. Check whether infrastructure changed after the recorded pre-OpenTofu generation. If it did, stop. Reconcile the newest state instead of restoring the older snapshot.
  3. Restore the pre-migration configuration and Terraform lock from 683a513. Keep the GCS backend. Use Terraform 1.15.8.
  4. Get explicit owner approval to restore the recorded pre-OpenTofu generation of gs://homelab-igresc-terraform-state/homelab/default.tfstate as a new live generation. Use gcloud storage cp with an --if-generation-match precondition set to the current destination generation. Stop if that generation changed. Never use the old Garage copy.
  5. Check restored contents, lineage, resource inventory, and original provider addresses. Initialize a fresh working directory with terraform init -reconfigure -input=false -lockfile=readonly.
  6. Run a read-only refreshed plan with TERRAFORM_VAULT_SKIP_CHILD_TOKEN=true terraform plan -input=false -refresh=true -lock=false -detailed-exitcode. Require no unexpected changes before lifting the freeze.

The OpenTofu migration guide also requires restoring a backup when rolling back after state changes.

To update provider checksums for all supported operator and CI platforms, run:

tofu providers lock -platform=linux_amd64 -platform=linux_arm64 \
  -platform=darwin_amd64 -platform=darwin_arm64 -platform=windows_amd64

The provider lock command checks package availability and records checksums from the origin registry.

docs/platform/terraform.md