lotus-land-story: mirror restic backups to lolk
Also disables dhcpcd's resolv.conf hook on lolk, where trixie's ifupdown runs it and it fights tailscaled over the file.

Assisted-by: GLM-5.3 via pi
change xllrkztrwzorpuqxozoklonwwltwsyos
commit 074939ecc0636395b4004b2863614edbf2cdab43
author Alpha Chen <alpha@kejadlen.dev>
date
parent ntusqskz
diff --git a/AGENTS.md b/AGENTS.md
index 35e40c0..c28fb57 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -7,6 +7,7 @@
   Prepare the change, then hand it off.
 - Don't run ad-hoc commands against hosts. Ask for the fact instead.
 - Keep evergreen documentation, meaning how things work now, in `docs/`.
-  Update it in the same change that makes it stale. Dated write-ups go in
-  `docs/<host>/YYYY-MM-DD-slug.md`.
+  Update it in the same change that makes it stale. Prefer living documents
+  over dated write-ups; `docs/<host>/YYYY-MM-DD-slug.md` is for when the
+  history itself matters, like a one-time migration record.
 - Commit scope is the directory the change lives in (`lotus-land-story:`).
diff --git a/docs/backup.md b/docs/backup.md
new file mode 100644
index 0000000..31f37cf
--- /dev/null
+++ b/docs/backup.md
@@ -0,0 +1,73 @@
+# Backups
+
+A daily cron script backs lotus-land-story up with restic into a primary
+repository on mountain-of-faith, then mirrors every snapshot into a secondary
+repository on legacy-of-lunatic-kingdom. Losing either machine loses at most
+one day of backups.
+
+## The daily run
+
+`/etc/cron.daily/backup` on lotus-land-story (templated from
+`lotus-land-story/templates/backup.sh`, owned by the play in
+`lotus-land-story/backup.yml`) runs as root. cron has no MTA, so the script
+sends its own output to the journal with the `backup` tag. Each run:
+
+1. Pings healthchecks.io `/start`, distinguishing a run that died early from
+   one that never fired.
+2. Backs up `/mnt/lotus-land-story/`, excluding `docker/`.
+3. Backs up `pg_dumpall` output via stdin as `lotus-land-story.sql`.
+4. Runs `forget --prune` against the primary with retention of daily for one
+   month, weekly for one year, monthly forever.
+5. Mirrors every snapshot onto the secondary with
+   `restic copy --from-repo <primary>`.
+
+An exit trap pings healthchecks.io with success or failure and pushes
+`backup_*` metrics to VictoriaMetrics. Telemetry failures never change the
+script's own exit status, but the copy in step 5 does count toward it: a
+broken mirror fails the run.
+
+## Repositories
+
+- Primary: `sftp:restic@mountain-of-faith:/mnt/mountain-of-faith/restic`
+- Secondary: `sftp:restic@legacy-of-lunatic-kingdom:/mnt/legacy-of-lunatic-kingdom/restic`
+
+Both live under a `restic` system user on the host, reachable by SSH keys for
+lotus-land-story (the backup) and 1password (manual restores). The
+repositories share one password, stored in `host_vars/lotus-land-story.sops.yml`.
+
+Restic resolves the source repository's password separately from
+`RESTIC_PASSWORD`, so the script exports `RESTIC_FROM_PASSWORD` for the copy.
+The repository URLs are defined once as play vars in `lotus-land-story/backup.yml`
+and templated into the script.
+
+## What the playbooks handle
+
+- The backup play trusts lolk's host key by reading it over Ansible and
+  writing it to known_hosts; the script's sftp connection fails closed on an
+  unknown host key.
+- The play probes the secondary with `restic cat config` and, if missing,
+  initializes it with `init --from-repo <primary> --copy-chunker-params`.
+  Copy requires an existing destination, and matching chunker parameters let
+  the mirror deduplicate against the primary instead of storing a second copy
+  of every pack file.
+- lolk's playbook (`legacy-of-lunatic-kingdom/main.yml`) hosts the repository
+  (user, keys, directory), so it has to be applied before the backup play can
+  first reach `restic@lolk`.
+
+## The secondary is append-only
+
+`forget --prune` runs against the primary only, so the secondary accumulates
+every snapshot since the mirror started. If the NAS ever becomes the primary,
+drop the copy step from `lotus-land-story/templates/backup.sh`, point
+`RESTIC_REPOSITORY` at the NAS repository, and run a one-time
+`forget --prune` there with the same retention flags.
+
+## Verifying a run
+
+```sh
+ssh lotus-land-story sudo /etc/cron.daily/backup
+ssh lotus-land-story journalctl -t backup
+```
+
+The journal shows the restic output including the copy; healthchecks.io
+should show the success ping for the same run.
diff --git a/legacy-of-lunatic-kingdom/README.md b/legacy-of-lunatic-kingdom/README.md
index 2f7ff75..eb6c821 100644
--- a/legacy-of-lunatic-kingdom/README.md
+++ b/legacy-of-lunatic-kingdom/README.md
@@ -62,4 +62,3 @@ through Tailscale.
 
 - jellyfin
 - immich
-- restic
diff --git a/legacy-of-lunatic-kingdom/main.yml b/legacy-of-lunatic-kingdom/main.yml
index d2d0702..a01ae51 100644
--- a/legacy-of-lunatic-kingdom/main.yml
+++ b/legacy-of-lunatic-kingdom/main.yml
@@ -85,7 +85,52 @@
 - import_playbook: ../playbooks/unattended-upgrades.yml # noqa: name[play]
 - import_playbook: ../playbooks/journal-upload.yml      # noqa: name[play]
 
+- name: Host a Restic repository
+  tags: restic
+  hosts: legacy-of-lunatic-kingdom
+  become: true
+  tasks:
+
+    - name: Create Restic user
+      ansible.builtin.user:
+        name: restic
+        system: true
+        state: present
+
+    - name: Add SSH key for restic user
+      ansible.posix.authorized_key:
+        user: restic
+        key: "{{ item }}"
+      loop:
+        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILrlVay1GlpCuhYSUup/K/gRYmltycViPBAdZ8t9OKtu # lotus-land-story
+        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPgX0tA28NO3djYaV++JxggR0t47xou+R0HQVwVjzj8s # 1password
+
+    - name: Create Restic directory
+      ansible.builtin.file:
+        path: "{{ (data_dir, 'restic') | path_join }}"
+        state: directory
+        owner: restic
+        group: restic
+        mode: "0755"
+
+- name: Keep dhcpcd out of resolv.conf
+  # Trixie's ifupdown uses dhcpcd for the dhcp method, and it rewrites
+  # /etc/resolv.conf on every lease event, stomping the file tailscaled owns
+  # until its next check restores MagicDNS. Takes effect at the next dhcpcd
+  # start, i.e. a reboot or a restart of the networking service.
+  tags: dns
+  hosts: legacy-of-lunatic-kingdom
+  become: true
+  tasks:
+
+    - name: Disable dhcpcd resolv.conf hook
+      ansible.builtin.lineinfile:
+        path: /etc/dhcpcd.conf
+        line: nohook resolv.conf
+        state: present
+
 # TODO
-# - restic
+# - jellyfin
+# - immich
 
 # vim: ft=yaml.ansible
diff --git a/lotus-land-story/backup.yml b/lotus-land-story/backup.yml
index 5d1270f..7389b55 100644
--- a/lotus-land-story/backup.yml
+++ b/lotus-land-story/backup.yml
@@ -1,5 +1,8 @@
 - name: Backup
   hosts: lotus-land-story
+  vars:
+    primary_repository: sftp:restic@mountain-of-faith:/mnt/mountain-of-faith/restic
+    secondary_repository: sftp:restic@legacy-of-lunatic-kingdom:/mnt/legacy-of-lunatic-kingdom/restic
   tasks:
 
     - name: Install Restic
@@ -15,6 +18,39 @@
         path: ~/.ssh/id_ed25519
         type: ed25519
 
+    # The backup script sftp's to this host as restic@, which fails closed on
+    # an unknown host key. Ansible's own connections don't populate known_hosts.
+    - name: Read legacy-of-lunatic-kingdom host key
+      ansible.builtin.slurp:
+        src: /etc/ssh/ssh_host_ed25519_key.pub
+      delegate_to: legacy-of-lunatic-kingdom
+      register: lolk_host_key
+
+    - name: Trust legacy-of-lunatic-kingdom host key
+      ansible.builtin.known_hosts:
+        name: legacy-of-lunatic-kingdom
+        key: "legacy-of-lunatic-kingdom {{ lolk_host_key.content | b64decode }}"
+
+    # restic copy fails on a missing destination, and --copy-chunker-params
+    # keeps dedup working across the two repositories.
+    - name: Check for secondary Restic repository
+      ansible.builtin.command:
+        cmd: restic --repo {{ secondary_repository }} cat config
+      environment:
+        RESTIC_PASSWORD: "{{ restic.password }}"
+      register: secondary_repo
+      changed_when: false
+      failed_when: false
+
+    - name: Initialize secondary Restic repository
+      ansible.builtin.command: >-
+        restic --repo {{ secondary_repository }}
+        init --from-repo {{ primary_repository }} --copy-chunker-params
+      environment:
+        RESTIC_PASSWORD: "{{ restic.password }}"
+        RESTIC_FROM_PASSWORD: "{{ restic.password }}"
+      when: secondary_repo.rc != 0
+
     - name: Set up daily backup
       ansible.builtin.template:
         src: templates/backup.sh
diff --git a/lotus-land-story/templates/backup.sh b/lotus-land-story/templates/backup.sh
index ccae3d9..c7045db 100644
--- a/lotus-land-story/templates/backup.sh
+++ b/lotus-land-story/templates/backup.sh
@@ -2,7 +2,10 @@
 
 set -euo pipefail
 
-export RESTIC_REPOSITORY=sftp:restic@mountain-of-faith:/mnt/mountain-of-faith/restic
+readonly PRIMARY_REPOSITORY={{ primary_repository }}
+readonly SECONDARY_REPOSITORY={{ secondary_repository }}
+
+export RESTIC_REPOSITORY=${PRIMARY_REPOSITORY}
 export RESTIC_PASSWORD={{ restic.password }}
 
 readonly HEALTHCHECK=https://hc-ping.com/{{ healthchecks_io.backup_sh }}
@@ -78,3 +81,13 @@ restic --verbose forget \
   --keep-within-weekly 1y \
   --keep-within-monthly 100y \
   --prune
+
+# Mirror onto the NAS repo while both are dual-write targets. Restic resolves
+# the source repository's password separately from RESTIC_PASSWORD, so pass
+# it explicitly; the repos share one.
+export RESTIC_REPOSITORY=${SECONDARY_REPOSITORY}
+export RESTIC_FROM_PASSWORD=${RESTIC_PASSWORD}
+restic --verbose copy --from-repo "${PRIMARY_REPOSITORY}"
+
+# Snapshots forgotten above are never removed from the secondary; a one-time
+# forget --prune against it happens at the cutover.