Change flow and promotion

Context

One operator runs the homelab, with AI agents and Renovate doing most of the changes. Pushing to main deploys staging today. The Gitea checks run only on PRs, and agents push straight to main, so nothing requires them. Production will move to Talos and sync from this repo too, so one bad commit could reach both clusters. The flow has to stop that without making the owner approve every update.

Decision

Forge

Use GitHub Free with private repos. Free has no protected branches, required checks or CODEOWNERS on private repos. GitHub Pro is optional and only buys the PR gate below. GitLab.com Free is the fallback. Every forge-specific step belongs to the forge migration, so picking a different forge changes that migration and not this design. Gitea is retired.

Pull requests and required checks

This section applies only if the owner buys GitHub Pro. Without Pro, changes go straight to main as they do today, and the checks run after the push.

Every change to main goes through a PR with required checks. Agents and Renovate work as the GitHub App igresc-homelab and merge their own PRs once the checks pass.

CODEOWNERS requires the owner's approval for these paths:

• docs/decisions/**
• .github/**
• CODEOWNERS
• renovate.json
• AGENTS.md

The owner is on the bypass list, so they can merge their own PRs.

Agents are trusted. CODEOWNERS controls what merges, not what runs. Any same-repo PR workflow can run code and write commit statuses before anyone approves it, so an agent could forge a status. We accept that. The checks guard against mistakes, not against a hostile agent.

Identities

• Agents and Renovate use the private GitHub App igresc-homelab, owned by igresc and installed only on selected repos. There is no machine account. They mint installation tokens that expire after 1 hour from the App private key in OpenBao at kv/infra/github/homelab-app (keys app_id and private_key). Commits show as igresc-homelab[bot]. With Pro, the App can be a ruleset bypass actor.
• Flux gets one read-only deploy key per cluster.
• Flux notification-controller gets a fine-grained PAT owned by igresc that can only write commit statuses.
• GHCR pulls use a classic PAT on igresc with only read:packages. It is stored in OpenBao at kv/infra/github/ghcr-pull (keys username and token), and External Secrets turns it into image pull secrets. GHCR accepts only classic PATs outside Actions, so App tokens and fine-grained PATs don't work for it.
• The Docker host Git backup uses a read-only PAT kept outside the NFS backup tree.
• Recovery codes and a second authenticator live in Bitwarden cloud, and the recovery email is outside the lab.

GitOps promotion

Both clusters sync main, each from its own directory: kubernetes/clusters/staging and kubernetes/clusters/prod. Chart versions, image tags and values that change behavior live in the environment directories. No shared base changes both clusters at once. That means:

• the shared Traefik, cert-manager and External Secrets pins in kubernetes/infrastructure/controllers move into per-environment patches;
• the Flux selector in each FluxInstance is per environment;
• the bootstrap Cilium pin in terraform/cilium.tf is per environment.

CRD upgrades stay manual.

Renovate

Renovate runs as a scheduled workflow on the self-hosted runners. It uses installation tokens from the igresc-homelab App, so its PRs trigger CI. Staging and production have separate rules, split with matchFileNames and additionalBranchPrefix.

• Staging PRs merge themselves after the checks pass.
• Production PRs open once the release is 3 days old.
• Production PRs need the staging-soak check.

staging-soak passes only when both of these are true:

• staging runs the same candidate: the same chart version, image digest and values that change behavior;
• Flux has reported that candidate healthy on staging for 24 hours.

The 24 hours start at the successful staging reconcile. They reset when the candidate changes. The check rejects failed or superseded evidence. A scheduled workflow re-runs it on open production PRs, so a PR passes once its soak is done without anyone pushing to it.

Patch and minor app updates merge themselves. Majors and the platform set stay manual. The platform set is Talos, Kubernetes, Cilium, Flux, Longhorn, CNPG, OpenBao, cert-manager, External Secrets, Mimir and Loki.

Renovate also covers Terraform providers, Ansible collections and workflow actions.

Promotion evidence

Flux notification-controller posts commit statuses from the staging Kustomizations after their health checks pass. It uses the status-only PAT, which keeps Flux's credential narrow. That PAT doesn't stop an agent from writing a status. The trusted-agent assumption covers that.

CI runners

Actions Runner Controller on production runs CI later, with an ephemeral pod per job, no Docker socket, and rootless BuildKit for images. Until production exists, runner containers on the Docker host run the jobs. GitHub Free has no account-level self-hosted runners, so each repo registers its own. igresc/homelab has four, one job each, so its jobs run in parallel. Repos that only build an image use GitHub-hosted runners instead, which fit in the 2,000 free minutes a month. The Gitea runners are retired.

There is no separate privileged runner. The repos are private and only the owner and agents can push, so there is no fork threat to isolate.

Registry

Images go to GHCR and stay private.

Terraform in CI

CI plans the app root on every PR and applies it on main. It logs into OpenBao with the GitHub Actions OIDC token through a JWT auth mount.

• tf-plan is read-only and accepts any branch of igresc/homelab.
• tf-apply accepts only refs/heads/main with event_name=push, so a main-ref token never runs PR code.
• Both roles also bind the repository, the audience and the Terraform workflow file (workflow_ref).
• GCS access uses Workload Identity Federation with the same conditions.
• PR plans run with -lock=false.
• The vault provider sets skip_child_token = true in CI, so tf-plan doesn't need token-create rights.

No Terraform secrets are stored in GitHub. The first apply, which creates these roles, runs locally.

Consequences

Most updates reach staging and then production with no owner action. With Pro, CODEOWNERS paths need the owner's approval. Majors and the platform set stay manual. A bad staging change still reaches staging, but production waits for 24 healthy hours on the same candidate.

A Renovate production PR can't merge until staging has run the same candidate healthy for 24 hours. Moving every pin into the environment directories is the cost.

A compromised agent could forge staging-soak and merge a production change. CODEOWNERS doesn't cover terraform/**, so any merge on main, including one an agent pushed through with a forged status, triggers a tf-apply of the app root. That root holds the OpenBao policies and auth, and Grafana. This design accepts both risks under the trusted-agent assumption, in exchange for not reviewing every PR.

References

• ADR 001: Keep the homelab in a monorepo
• ADR 009: Use OpenBao as the secret authority
• ADR 010: Store Terraform state in Google Cloud Storage
• ADR 012: What to self-host
• Flux notification-controller, Git commit status
• Renovate minimumReleaseAge

Rollout gates

• If the owner bought Pro, the pilot in a throwaway private repo passes with real App installation tokens. Direct and force pushes to main fail. Missing or failing checks block merges. Passing checks let the App merge without approval. CODEOWNERS paths need the owner. The owner's bypass works. open
• If the owner bought Pro, branch protection and CODEOWNERS are on for igresc/homelab. open
• CI runs on Actions and renders both environments on every PR. open
• ARC runs the CI jobs, and the Docker-host runner no longer does. open
• Flux on both clusters syncs from GitHub with its own deploy key. open
• No shared base pins a version or behavior-changing value for both clusters. Each FluxInstance selector and the Cilium bootstrap pin are per environment. open
• Renovate runs on a schedule with split staging and production rules. open
• Flux posts commit statuses from the staging Kustomizations, and staging-soak blocks a production PR until its candidate has 24 healthy hours on staging. The check has failed at least once on a changed or failed candidate. open
• The scheduled staging-soak re-run updates open production PRs. open
• The tf-plan and tf-apply roles and the GCS federation exist. A PR plan and a main apply have run in CI, and a PR token has failed to get tf-apply. open
• Images pull from GHCR through the External Secrets pull secret. open

docs/decisions/011-change-flow-and-promotion.md