Back up Git repositories from the Docker host

Use this procedure to install the daily Git backup on docker.local.isabelsoler.es. The job is prepared in Git but is not installed or running yet. The Flux SSH endpoint, Kuma monitor, B2 inclusion, and first live restore still need owner setup. ADR 006 defines the recovery policy.

Prepare the NAS storage

Nebula only provides NFS storage, ZFS snapshots, and the existing B2 sync. Do not install scripts, credentials, cron entries, or SSH Git commands there.

In the TrueNAS UI, create a dedicated dataset such as /mnt/tank/backups/github. Use POSIX permissions and ACLs. Keep it owned by root with mode 0700 until you configure the optional Flux reader.
Create an NFS share for that dataset. Restrict access to the Docker host's address. Enable NFSv3 and map the client's root user to root for this share. Leave Mapall unset so the optional Flux account keeps its own identity. The job runs as root on the Docker host and needs to create and verify files on this dataset. Do not grant root access to other exports.
Include the dataset in the existing ZFS snapshot policy and encrypted TrueNAS to B2 task. The backup tree contains mirrors, bundles, history, and cached ISOs. Credentials stay on the Docker host.

Install the job on the Docker host

Use an Ansible-installed systemd timer with the host's Python 3 and Git. It needs no custom container image or scheduler process. Systemd catches up a missed daily run after downtime and requires the NFS mount before starting. The script also checks that the destination is an NFS mount before any write. See systemd timers and mount dependencies.

This fits the existing docker_hosts inventory and docker_host role in ansible/playbooks/docker.yml. The runner stack uses Docker Compose, but a host timer avoids another image and Docker network. The owner must perform installation with a privileged account. igresc has no usable sudo access on this host.

From the repository checkout, preview the opt-in tasks with an SSH account that has root access or working sudo. Replace the account placeholder. Add --ask-become-pass if that account requires a sudo password:

cd ansible
ansible-playbook playbooks/docker.yml --tags git_backup --check --diff \
  -e 'ansible_user=<PRIVILEGED_SSH_USER>' \
  -e docker_host_git_backup_enabled=true \
  -e docker_host_git_backup_nfs_source=nebula:/mnt/tank/backups/github

Review the file copies, NFS mount, and systemd units before applying. The mount and NAS permissions need an independent review before installation. Run the same command without --check --diff to install. The timer stays disabled until the manual checks pass. No code is copied to nebula. 2. Confirm the mount on the Docker host from a privileged shell:

```bash
findmnt --mountpoint /srv/git-backup -o TARGET,SOURCE,FSTYPE,OPTIONS
```

Expect nebula:/mnt/tank/backups/github and nfs. Ansible records the mount in /etc/fstab with hard,_netdev,nofail and NFSv3. Keep network locking enabled. Do not add nolock, soft, or a local fallback directory. NAS unavailability can block I/O. It must not produce a successful local backup. Kuma detects a missed heartbeat, including mount failures. 3. Review /etc/git-backup/config.json. It lists the 12 requested active repositories and archived MyInvestor-API, all owned by igresc. The list is explicit. A new GitHub repository does not join automatically. To audit the list from your laptop, run:

```bash
gh repo list igresc --limit 100 --json name,isPrivate,isArchived
```

Keep talos_versions aligned with talos_version in terraform/variables_talos.tf. CI checks the committed default. Reapply the Ansible tasks after changes to scripts/git-backup/. Keep both versions in the configured list during an upgrade if recovery still needs the old version. Installed scripts live in /usr/local/lib/git-backup.

The job caches the upstream metal-amd64.iso and sha256sum.txt for each configured Talos version. It verifies the SHA-256 checksum on every run and removes other cached versions only after all requested versions pass. This is the stock recovery ISO, not the custom NoCloud image with system extensions used by terraform/talos_image_factory.tf. The custom installer still comes from Image Factory. See the Talos release assets.

Create the read-only GitHub token

On GitHub, open Settings > Developer settings > Personal access tokens > Fine-grained tokens > Generate new token.
Name the token Docker host Git backup. Select resource owner igresc.
Select All repositories, or select every repository in config.json. Grant repository permissions Contents: Read-only and Metadata: Read-only. Leave every other permission unset.
Set Expiration to No expiration for this unattended personal-account job. GitHub allows infinite lifetimes. Organization or enterprise policies can impose a maximum lifetime. The default is 30 days if left unchanged. If you choose an expiry, set a renewal reminder before that date. Review the token during the yearly recovery drill and revoke it when retiring the job. See GitHub's current token documentation.

In a root shell on the Docker host, write the token through a hidden prompt:

/usr/bin/python3 - <<'PY'
import getpass, os
from pathlib import Path
os.umask(0o077)
path = Path('/etc/git-backup/private/github-token')
with path.open('w') as stream:
    stream.write(getpass.getpass('GitHub read-only token: ') + '\n')
path.chmod(0o600)
PY

The root systemd service reads the token through askpass.py. The token never enters Git arguments, remote URLs, bare-repository configuration, or log messages. Do not put the token in config.json, a shell command, or a Git credential store. To rotate it, replace the file through the same hidden prompt and run the job once before revoking the old token.

Configure the external failure monitor

Create a Push monitor in the external Uptime Kuma instance. Use a 24-hour heartbeat interval and zero retries so freshness does not extend through a second missed run. Enable its notification route.
Write its HTTPS push URL to /etc/git-backup/private/kuma-url, using the hidden-prompt command above with that path and a Kuma push URL: prompt.
Keep the URL file root-owned and mode 0600. A missing file logs a warning and skips the ping. Other ping failures make the job fail. The job sends status=up only after all repositories and the Talos cache pass. It sends status=down with a short failure count otherwise.

The Kuma push endpoint accepts status and msg. A successful Git fetch does not prove offsite freshness. Configure a separate monitor for the TrueNAS to B2 job.

Run once and enable the timer

Run these commands from a root shell on the Docker host.

Run the installed service and inspect its result and logs:

systemctl start git-backup.service
systemctl show git-backup.service -p Result -p ExecMainStatus
journalctl -u git-backup.service --no-pager -n 100

Expect a verified bundle for every repository, a verified Talos ISO, Result=success, and ExecMainStatus=0. Confirm that Kuma receives an up heartbeat. Transport error details are suppressed because they can expose credentials. For Git failures, check repository access, the token file's ownership and mode, and GitHub reachability. The unit requires the mount, and the script refuses a local directory even when run manually. 2. Confirm that /mnt/tank/backups/github is included in the existing encrypted TrueNAS to B2 task. Confirm that the next successful B2 run contains a new bundle. Credentials under /etc/git-backup/private never enter that dataset. Local Git bundles are not individually encrypted. 3. Restore one bundle with the next procedure. Then rerun the Ansible command from the install step with both docker_host_git_backup_enabled=true and docker_host_git_backup_timer_enabled=true. Retain those non-secret settings in the owner's inventory overrides. With the feature enabled, later runs with timer_enabled=false stop and disable the timer. Setting only git_backup_enabled=false skips management and does not uninstall it. 4. On the Docker host, check the schedule:

```bash
systemctl list-timers git-backup.timer
systemctl cat git-backup.timer
```

The timer runs daily at 02:15 UTC, with Persistent=true to catch up a missed run after downtime. After the first scheduled run, check all 13 mirrors, bundle dates, Kuma, and B2 task success. Until those checks pass, the backup is not operational.

The lock is /srv/git-backup/.lock on NFS. An overlapping run exits nonzero. Do not delete the lock file to release it. The kernel releases its lock when the process exits. Keep the NAS snapshot policy enabled for the dataset.

Each run writes a self-contained bundle of all Git refs. Retention keeps the newest bundle from each of the latest 30 dates and the first bundle from each of the latest 12 months. A bundle can satisfy both rules. Dates use UTC. Before accepting any bundle, the job restores it into a temporary bare repository and runs git fsck --full. Allow scratch space for one repository restore in addition to its mirror and retained bundles. Header verification alone cannot detect packed-object corruption. Git LFS payloads, issues, releases, Actions artifacts, and packages are outside Git bundle coverage.

If main loses history or is deleted, the job keeps a verified old-main bundle under history/<repo>/ and reports failure. The mirror and its new dated bundle still reflect upstream. Inspect the retained history before accepting the rewrite. History bundles are never pruned automatically. A failed fetch or an interrupted run can also leave a history bundle. Keep it until you have checked the repository. Other repositories still run after a failure.

To disable the job, run systemctl disable --now git-backup.timer on the Docker host and set docker_host_git_backup_timer_enabled=false in the owner's overrides. Let an active service finish before changing files. To roll back, restore the previous script, config, and unit versions from Git and reapply the role. Test the service before enabling the timer again. Preserve the NFS mount, mirrors, bundles, and history. Do not delete backup data as part of rollback.

Restore a bundle

Copy a selected bundle to a recovery workstation. Use an empty directory for verification, since git bundle verify needs a Git repository:

    git init --bare verify.git
    git -C verify.git bundle verify /absolute/path/to/selected.bundle
    git clone --mirror /absolute/path/to/selected.bundle recovered.git
    git -C recovered.git fsck --full
    git -C recovered.git log -1 --oneline main

Check the commit against the intended recovery point. Clone recovered.git into a working checkout when you need files. To restore upstream, review the refs first and obtain approval before pushing. A mirror push can delete refs. For an old-main protection bundle, only main is restored.

Prepare the read-only Flux SSH endpoint

Staging's Flux deploy key exists. Production Talos gets its key when it is built. This procedure does not create keys or change the current Flux source. Production k3s still uses the separate gitops repository.

From a root shell on the Docker host, create a dedicated account and root-owned SSH directories. Keep SSH password authentication disabled. The account needs /bin/sh so SSH can execute the forced command:

useradd --system --home-dir /var/lib/flux-git --shell /bin/sh flux-git
usermod --password '*' flux-git
install -d -o root -g root -m 0755 /var/lib/flux-git /var/lib/flux-git/.ssh
install -o root -g root -m 0644 /dev/null /var/lib/flux-git/.ssh/authorized_keys

Run the creation commands only for a new account. Do not truncate an existing authorized-keys file. Give flux-git no sudo access or membership in the docker group. 2. Grant the account traversal on the mount and mirrors directory, then read-only access to the homelab mirror. Run on the Docker host after the first successful backup:

```bash
setfacl -m u:flux-git:--x /srv/git-backup /srv/git-backup/mirrors
setfacl -R -m u:flux-git:r-X /srv/git-backup/mirrors/homelab.git
find /srv/git-backup/mirrors/homelab.git -type d -exec setfacl -m d:u:flux-git:r-x {} +
```

The dataset must support POSIX ACLs through NFS. Confirm the ACLs with getfacl and test access after another backup creates new pack files. Keep all paths root-owned with no write ACL for flux-git. Do not grant access to other mirrors, bundles, history, or /etc/git-backup/private. 3. Check that /usr/local/lib/git-backup/serve.py and every parent directory are root-owned and not writable by flux-git. Add one forced entry per cluster to the root-owned authorized-keys file. Replace only the public-key placeholders:

```text
restrict,command="/usr/bin/python3 /usr/local/lib/git-backup/serve.py" ssh-ed25519 <STAGING_FLUX_DEPLOY_PUBLIC_KEY> staging
restrict,command="/usr/bin/python3 /usr/local/lib/git-backup/serve.py" ssh-ed25519 <PRODUCTION_TALOS_FLUX_DEPLOY_PUBLIC_KEY> production
```

The staging public key is identity.pub at kv/infra/github/flux-staging in OpenBao. Add the production line once that key exists. Do not add an unrestricted copy. The forced command permits only git-upload-pack for the exact homelab path. It rejects pushes, shells, other repositories, and path traversal. It trusts only that root-owned mirror through Git's safe.directory option. 4. Get the public host key fingerprint through the trusted Docker host console or an existing verified administrator connection:

```bash
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub -E sha256
```

From the recovery workstation, collect the public key and compare its fingerprint with that trusted value before accepting it:

```bash
ssh-keyscan -t ed25519 docker.local.isabelsoler.es > docker-known-hosts
ssh-keygen -lf docker-known-hosts -E sha256
```

Test git ls-remote and a clone using the Flux identity and verified known hosts. Confirm that SSH shell access and git-receive-pack are denied. The source URL is:
```
ssh://flux-git@docker.local.isabelsoler.es/srv/git-backup/mirrors/homelab.git
```

Use a DNS name or address reachable from the cluster and include that exact host in known hosts. Put identity, identity.pub, and known_hosts in the Flux SSH Secret through the approved secret workflow. Never commit a private deploy key. See Flux SSH authentication. 6. Once the Flux source URL is a bootstrap input, set that input to the Docker host URL and select the matching Secret. Keep branch main and the cluster's existing sync path. Today kubernetes/clusters/staging/flux-system/flux-instance.yaml sets the source, and Flux reconciles that file from Git. Follow Cilium and Flux bootstrap when that input exists. Do not patch the live GitRepository while its owner can revert the change. 7. With GitHub blocked in a controlled staging drill, confirm a successful reconcile from the Docker host. Restore the normal bootstrap source afterwards and confirm it reconciles. Record the drill evidence in the commit message.

To roll back the endpoint, restore the previous bootstrap source input and remove the forced public-key entries. Preserve all backup files.

docs/runbooks/git-backup.md