Investigate slow Terraform plans
Use this procedure to separate backend access, provider startup, and resource
refresh time. Run commands from the repository root with the existing OpenBao
login and allowed direnv environment described in the
Terraform guide.
Measure a complete plan
Check for another Terraform writer before starting. Keep state locking enabled. Use the JSON UI event stream to record timestamps and resource addresses without printing resource values or complete diagnostics:
(
set -o pipefail
unset TF_LOG TF_LOG_CORE TF_LOG_PROVIDER TF_LOG_PATH
time direnv exec terraform tofu -chdir=terraform plan \
-input=false -json -detailed-exitcode |
jq --unbuffered -c '
select(.type == "version" or .type == "refresh_start" or
.type == "refresh_complete" or .type == "change_summary" or
.type == "diagnostic") |
{time: .["@timestamp"], event: .type,
resource: .hook.resource.addr,
severity: .diagnostic.severity, summary: .diagnostic.summary,
changes: .changes}'
)
Exit code 0 means no changes, 2 means a successful plan with changes, and 1
means an error. This command neither saves nor applies a plan. Existing
provider-specific logging can still write local files, including proxmox.log.
Do not share those files or raw debug logs without checking for secrets.
Match each refresh_start with refresh_complete for the same resource address.
Refreshes overlap, so adding their durations does not give total plan time.
A long interval before the first refresh needs separate investigation. Resource
events do not identify which backend or provider-startup operation caused it.
Isolate the delayed phase
Measure a backend read without refreshing resource APIs:
If this is slow or fails, follow the
backend runbook. The main root
reads its state from GCS, as configured in terraform/versions.tf.
For a slow resource, repeat the filtered plan command with -target=<address>.
Targeting is a diagnostic tool here. Its result does not establish that the
complete root is clean, and it can still incur backend and provider-startup costs.
For module.openbao_gcp.google_project_service.cloudkms, compare these read-only
Google CLI requests:
time gcloud projects describe homelab-igresc \
--format='value(projectId,lifecycleState)'
time gcloud services list --enabled --project=homelab-igresc \
--format='value(config.name)'
The Google CLI and Terraform can use different credentials. These checks help separate API reachability from Terraform timing, but do not prove that Terraform's Application Default Credentials are healthy.
If the delay is intermittent, compare timestamps from a slow run and a fast run. For deeper tracing, collect only backend operation names, HTTP status codes, durations, provider startup events, and retry counts. Filter before displaying or retaining output. Raw Terraform traces can contain credentials and secret values.
Do not use -refresh=false as a permanent workaround. It omits drift checks.
Do not increase parallelism until the measurements show that queued work causes
the delay. Neither setting repairs a slow backend or an individual API request.