Skip to content

Operate FDM Monster on Talos

Use this runbook for the FDM Monster instance on the talos-test staging cluster. Flux reconciles kubernetes/apps/staging/fdm-monster/ from this repository's main branch. The separate production cluster is outside this deployment.

Check the deployment

  1. Confirm that both Talos nodes are Ready.
  2. Check the application and its dependencies.
kubectl --context talos-test get nodes
flux --context talos-test get kustomizations
kubectl --context talos-test -n 3d-printing get deployment,pod,pvc,service,ingress \
    -l app.kubernetes.io/name=fdm-monster-staging
kubectl --context talos-test -n 3d-printing get certificate fdm-monster-staging-tls

Require fdm-monster-staging to be Ready in Flux, one ready application pod, a Bound PVC, and a Ready certificate. The application runs in 3d-printing. Its certificate uses the shared staging-ingress-letsencrypt ClusterIssuer, managed by staging-certificates. The issuer obtains its DNS-01 credentials through External Secrets in cert-manager.

  1. Verify the HTTPS endpoint and login settings.
curl --fail --silent --show-error \
    https://fdm-monster.staging.local.isabelsoler.es/api/v2/auth/login-required

Require loginRequired=true, registration=false, and isDemoMode=false. The first visit shows the setup wizard. Create your administrator account there, keep login required, and keep public registration disabled. Add printer addresses and credentials through the authenticated UI. Prefer printer IPs or unicast DNS names because the standard container does not provide LAN mDNS.

The Deployment runs upstream image 2.1.1 at its recorded digest, directly with Node rather than PM2. Kubernetes owns restart and health checks. The process runs as UID 1000 with a read-only root filesystem.

Diagnose DNS and ingress

The DNS declaration is in terraform/records.auto.tfvars. The intended A record is fdm-monster.staging.local.isabelsoler.es at the existing staging Traefik address, 10.0.20.90. Apply DNS through the reviewed Terraform workflow in Terraform. Do not edit BIND manually.

If DNS is unavailable, check the certificate and application directly through Traefik. This command preserves hostname and certificate verification.

curl --fail --silent --show-error \
    --resolve fdm-monster.staging.local.isabelsoler.es:443:10.0.20.90 \
    https://fdm-monster.staging.local.isabelsoler.es/api/v2/auth/login-required

For browser access before the DNS record is applied, add the hostname and 10.0.20.90 to your workstation's hosts file. Remove the temporary entry after authoritative DNS resolves correctly.

Check the shared ClusterIssuer and its ExternalSecret if certificate issuance fails. Inspect status and events without printing secret values.

kubectl --context talos-test -n 3d-printing describe certificate fdm-monster-staging-tls
kubectl --context talos-test get clusterissuer staging-ingress-letsencrypt
kubectl --context talos-test -n cert-manager get externalsecret staging-ingress-dns01

Connect the Prusa camera

  1. In FDM Monster's authenticated camera settings, enter the direct go2rtc URL.
http://192.168.1.6:1984/api/stream.mjpeg?src=prusa_fdm

The browser connects directly to Home Assistant's go2rtc API on port 1984. This URL does not use Home Assistant's ingress session. No Kubernetes camera proxy is configured.

  1. Keep go2rtc's API listening on :1984 and its prusa_fdm stream configured to output MJPEG. Manage the camera's RTSP credentials in Home Assistant, outside this repository.
  2. Check the direct API from the LAN if the camera fails to load.
curl --silent --show-error --max-time 5 --dump-header /tmp/prusa-mjpeg.headers \
    --output /dev/null 'http://192.168.1.6:1984/api/stream.mjpeg?src=prusa_fdm'
cat /tmp/prusa-mjpeg.headers

A working stream returns HTTP 200 and Content-Type: multipart/x-mixed-replace; boundary=frame. The bounded curl request times out because a live MJPEG stream has no end. A 200 response with an empty body or application/octet-stream does not prove that video works. For codecs not matched: video:H264 ... => video:JPEG, check go2rtc's FFmpeg MJPEG source. If FFmpeg treats a stream name as a filename, use the camera's RTSP URL as the FFmpeg input in Home Assistant's configuration.

HTTP 401 from /api/hassio_ingress/ means that Home Assistant's ingress session is missing. Use the direct API URL above rather than copying session cookies. HTTP 401 from FDM Monster's /api/v2/camera-stream/ concerns its own login; sign in again before saving camera settings. stream.html is a viewer page, not the MJPEG URL that FDM Monster needs.

Preserve data and recover safely

Keep one replica and the Recreate strategy. SQLite needs one writer. fdm-monster-staging-data holds the database at /data/database and media at /data/media. The application generates its JWT signing key and persists it in SQLite. No shared default signing key or demo account is configured.

The 5 GiB PVC uses longhorn-staging-r1-retain on the dedicated staging disk. Flux pruning is disabled for the PVC, and the storage class retains the PV. Removing the application Flux Kustomization prunes its application resources while preserving the protected PVC.

Single-worker storage has no availability redundancy. The staging Longhorn backup target is unavailable, so this instance has no verified off-cluster backup. Keep irreplaceable data elsewhere until its Longhorn backup to B2 runs and a restore test passes. ADR 006 sets the backup and its max age.

Before an upgrade or recovery operation:

  1. Commit replicas: 0 and wait for Flux to stop the application pod.
  2. Take a consistent backup of the whole PVC, including database and media.
  3. Verify the backup outside the cluster before changing the application image.
  4. Commit the reviewed image change and restore replicas: 1.
  5. Check HTTPS, authentication, printer inventory, and the persistent files.

If deployment fails, commit replicas: 0 to stop only this instance while preserving its PVC. Do not delete the PVC or a Longhorn volume to retry startup. Reverting a fresh deployment can remove its application resources, but the protected PVC remains. Database migrations run at startup. After an upgrade, restore the matching database and media backup before running an older image. An image-only downgrade is not a proven database rollback.

Move the instance to 3d-printing

Both PVCs are named fdm-monster-staging-data: the retained source in monitoring and the active target in 3d-printing. The startup guard requires the persisted migration receipt and a nonempty database. Include the receipt when backing up or restoring this volume.

The migration uses fdm-monster.staging.local.isabelsoler.es as its only final hostname. The shared staging-ingress-letsencrypt ClusterIssuer replaces the dependency on Grafana's namespaced Issuer. The original source PVC is retained in monitoring for recovery. The following procedure explains the stopped-volume migration.

  1. Apply the reviewed Terraform changes for the staging-cluster-issuer OpenBao role and the fdm-monster.staging DNS record. Keep the old record until the new endpoint passes validation.
  2. Require staging-certificates to be Ready in Flux and staging-ingress-dns01 to be Ready in cert-manager. ClusterIssuer readiness alone does not prove that the DNS-01 token is available.
  3. Commit the application namespace change to 3d-printing, its new hostname, issuerRef.kind: ClusterIssuer, and replicas: 0. Keep PVC pruning disabled. Change the Flux dependency from grafana-staging to staging-certificates. Protect the new namespace from pruning.
  4. Wait for the old Deployment and every pod using its PVC to disappear. Require the new PVC to be Bound and the new certificate to be Ready. Both application instances must stay stopped throughout the copy.
  5. Run the migration helper from the repository root.
./scripts/migrate-fdm-monster-namespace.sh
kubectl --context talos-test -n monitoring wait --for=delete \
    pod/fdm-monster-migration-source --timeout=60s
kubectl --context talos-test -n 3d-printing wait --for=delete \
    pod/fdm-monster-migration-target --timeout=60s

The helper requires distinct, Bound source and target volumes and an empty target. It mounts the source read-only and streams the stopped database and media directly between pods. It compares paths, permissions, and contents and checks that the source did not change. A successful copy creates /data/.fdm-monster-namespace-migration-verified on the target. It never starts either application and never deletes either PVC.

If the copy fails, leave both applications stopped. The helper refuses to overwrite a partial copy on a retry. If extraction finished but failed while restoring the storage-owned root's metadata, use ./scripts/migrate-fdm-monster-namespace.sh --verify-existing-copy. This mode compares the existing contents without copying or overwriting data files. It creates the verification marker only if the source and target hashes match. If verification fails, inspect the failure and prepare a new empty target volume through a reviewed recovery change. Keep the source.

  1. Require the verification marker and a nonempty SQLite database in an init container before application startup. Commit replicas: 1 only after both helper pods have terminated.
  2. Verify the new DNS record, trusted TLS, HTTPS UI, Socket.IO, authentication settings, and persistent data. Check that the old Ingress no longer exists.
  3. Remove the old DNS record through a reviewed Terraform plan. Retain the original PVC in monitoring until a verified backup and restore exist.

Before the target has accepted writes, rollback can restore the original manifests in monitoring and start the retained source volume. After the target has accepted writes, the original copy is stale. Stop the target and copy its verified current contents back through a reviewed recovery procedure before starting the source. Never start both SQLite writers or discard the current target volume to recover the old hostname.

Upstream references

docs/runbooks/fdm-monster.md