lotus-land-story: mirror restic backups to lolk
Also disables dhcpcd's resolv.conf hook on lolk, where trixie's ifupdown runs it and it fights tailscaled over the file.
Assisted-by: GLM-5.3 via pi
diff --git a/AGENTS.md b/AGENTS.md
index 35e40c0..c28fb57 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -7,6 +7,7 @@
Prepare the change, then hand it off.
- Don't run ad-hoc commands against hosts. Ask for the fact instead.
- Keep evergreen documentation, meaning how things work now, in `docs/`.
- Update it in the same change that makes it stale. Dated write-ups go in
- `docs/<host>/YYYY-MM-DD-slug.md`.
+ Update it in the same change that makes it stale. Prefer living documents
+ over dated write-ups; `docs/<host>/YYYY-MM-DD-slug.md` is for when the
+ history itself matters, like a one-time migration record.
- Commit scope is the directory the change lives in (`lotus-land-story:`).
diff --git a/docs/backup.md b/docs/backup.md
new file mode 100644
index 0000000..31f37cf
--- /dev/null
+++ b/docs/backup.md
@@ -0,0 +1,73 @@
+# Backups
+
+A daily cron script backs lotus-land-story up with restic into a primary
+repository on mountain-of-faith, then mirrors every snapshot into a secondary
+repository on legacy-of-lunatic-kingdom. Losing either machine loses at most
+one day of backups.
+
+## The daily run
+
+`/etc/cron.daily/backup` on lotus-land-story (templated from
+`lotus-land-story/templates/backup.sh`, owned by the play in
+`lotus-land-story/backup.yml`) runs as root. cron has no MTA, so the script
+sends its own output to the journal with the `backup` tag. Each run:
+
+1. Pings healthchecks.io `/start`, distinguishing a run that died early from
+ one that never fired.
+2. Backs up `/mnt/lotus-land-story/`, excluding `docker/`.
+3. Backs up `pg_dumpall` output via stdin as `lotus-land-story.sql`.
+4. Runs `forget --prune` against the primary with retention of daily for one
+ month, weekly for one year, monthly forever.
+5. Mirrors every snapshot onto the secondary with
+ `restic copy --from-repo <primary>`.
+
+An exit trap pings healthchecks.io with success or failure and pushes
+`backup_*` metrics to VictoriaMetrics. Telemetry failures never change the
+script's own exit status, but the copy in step 5 does count toward it: a
+broken mirror fails the run.
+
+## Repositories
+
+- Primary: `sftp:restic@mountain-of-faith:/mnt/mountain-of-faith/restic`
+- Secondary: `sftp:restic@legacy-of-lunatic-kingdom:/mnt/legacy-of-lunatic-kingdom/restic`
+
+Both live under a `restic` system user on the host, reachable by SSH keys for
+lotus-land-story (the backup) and 1password (manual restores). The
+repositories share one password, stored in `host_vars/lotus-land-story.sops.yml`.
+
+Restic resolves the source repository's password separately from
+`RESTIC_PASSWORD`, so the script exports `RESTIC_FROM_PASSWORD` for the copy.
+The repository URLs are defined once as play vars in `lotus-land-story/backup.yml`
+and templated into the script.
+
+## What the playbooks handle
+
+- The backup play trusts lolk's host key by reading it over Ansible and
+ writing it to known_hosts; the script's sftp connection fails closed on an
+ unknown host key.
+- The play probes the secondary with `restic cat config` and, if missing,
+ initializes it with `init --from-repo <primary> --copy-chunker-params`.
+ Copy requires an existing destination, and matching chunker parameters let
+ the mirror deduplicate against the primary instead of storing a second copy
+ of every pack file.
+- lolk's playbook (`legacy-of-lunatic-kingdom/main.yml`) hosts the repository
+ (user, keys, directory), so it has to be applied before the backup play can
+ first reach `restic@lolk`.
+
+## The secondary is append-only
+
+`forget --prune` runs against the primary only, so the secondary accumulates
+every snapshot since the mirror started. If the NAS ever becomes the primary,
+drop the copy step from `lotus-land-story/templates/backup.sh`, point
+`RESTIC_REPOSITORY` at the NAS repository, and run a one-time
+`forget --prune` there with the same retention flags.
+
+## Verifying a run
+
+```sh
+ssh lotus-land-story sudo /etc/cron.daily/backup
+ssh lotus-land-story journalctl -t backup
+```
+
+The journal shows the restic output including the copy; healthchecks.io
+should show the success ping for the same run.
diff --git a/legacy-of-lunatic-kingdom/README.md b/legacy-of-lunatic-kingdom/README.md
index 2f7ff75..eb6c821 100644
--- a/legacy-of-lunatic-kingdom/README.md
+++ b/legacy-of-lunatic-kingdom/README.md
@@ -62,4 +62,3 @@ through Tailscale.
- jellyfin
- immich
-- restic
diff --git a/legacy-of-lunatic-kingdom/main.yml b/legacy-of-lunatic-kingdom/main.yml
index d2d0702..a01ae51 100644
--- a/legacy-of-lunatic-kingdom/main.yml
+++ b/legacy-of-lunatic-kingdom/main.yml
@@ -85,7 +85,52 @@
- import_playbook: ../playbooks/unattended-upgrades.yml # noqa: name[play]
- import_playbook: ../playbooks/journal-upload.yml # noqa: name[play]
+- name: Host a Restic repository
+ tags: restic
+ hosts: legacy-of-lunatic-kingdom
+ become: true
+ tasks:
+
+ - name: Create Restic user
+ ansible.builtin.user:
+ name: restic
+ system: true
+ state: present
+
+ - name: Add SSH key for restic user
+ ansible.posix.authorized_key:
+ user: restic
+ key: "{{ item }}"
+ loop:
+ - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILrlVay1GlpCuhYSUup/K/gRYmltycViPBAdZ8t9OKtu # lotus-land-story
+ - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPgX0tA28NO3djYaV++JxggR0t47xou+R0HQVwVjzj8s # 1password
+
+ - name: Create Restic directory
+ ansible.builtin.file:
+ path: "{{ (data_dir, 'restic') | path_join }}"
+ state: directory
+ owner: restic
+ group: restic
+ mode: "0755"
+
+- name: Keep dhcpcd out of resolv.conf
+ # Trixie's ifupdown uses dhcpcd for the dhcp method, and it rewrites
+ # /etc/resolv.conf on every lease event, stomping the file tailscaled owns
+ # until its next check restores MagicDNS. Takes effect at the next dhcpcd
+ # start, i.e. a reboot or a restart of the networking service.
+ tags: dns
+ hosts: legacy-of-lunatic-kingdom
+ become: true
+ tasks:
+
+ - name: Disable dhcpcd resolv.conf hook
+ ansible.builtin.lineinfile:
+ path: /etc/dhcpcd.conf
+ line: nohook resolv.conf
+ state: present
+
# TODO
-# - restic
+# - jellyfin
+# - immich
# vim: ft=yaml.ansible
diff --git a/lotus-land-story/backup.yml b/lotus-land-story/backup.yml
index 5d1270f..7389b55 100644
--- a/lotus-land-story/backup.yml
+++ b/lotus-land-story/backup.yml
@@ -1,5 +1,8 @@
- name: Backup
hosts: lotus-land-story
+ vars:
+ primary_repository: sftp:restic@mountain-of-faith:/mnt/mountain-of-faith/restic
+ secondary_repository: sftp:restic@legacy-of-lunatic-kingdom:/mnt/legacy-of-lunatic-kingdom/restic
tasks:
- name: Install Restic
@@ -15,6 +18,39 @@
path: ~/.ssh/id_ed25519
type: ed25519
+ # The backup script sftp's to this host as restic@, which fails closed on
+ # an unknown host key. Ansible's own connections don't populate known_hosts.
+ - name: Read legacy-of-lunatic-kingdom host key
+ ansible.builtin.slurp:
+ src: /etc/ssh/ssh_host_ed25519_key.pub
+ delegate_to: legacy-of-lunatic-kingdom
+ register: lolk_host_key
+
+ - name: Trust legacy-of-lunatic-kingdom host key
+ ansible.builtin.known_hosts:
+ name: legacy-of-lunatic-kingdom
+ key: "legacy-of-lunatic-kingdom {{ lolk_host_key.content | b64decode }}"
+
+ # restic copy fails on a missing destination, and --copy-chunker-params
+ # keeps dedup working across the two repositories.
+ - name: Check for secondary Restic repository
+ ansible.builtin.command:
+ cmd: restic --repo {{ secondary_repository }} cat config
+ environment:
+ RESTIC_PASSWORD: "{{ restic.password }}"
+ register: secondary_repo
+ changed_when: false
+ failed_when: false
+
+ - name: Initialize secondary Restic repository
+ ansible.builtin.command: >-
+ restic --repo {{ secondary_repository }}
+ init --from-repo {{ primary_repository }} --copy-chunker-params
+ environment:
+ RESTIC_PASSWORD: "{{ restic.password }}"
+ RESTIC_FROM_PASSWORD: "{{ restic.password }}"
+ when: secondary_repo.rc != 0
+
- name: Set up daily backup
ansible.builtin.template:
src: templates/backup.sh
diff --git a/lotus-land-story/templates/backup.sh b/lotus-land-story/templates/backup.sh
index ccae3d9..c7045db 100644
--- a/lotus-land-story/templates/backup.sh
+++ b/lotus-land-story/templates/backup.sh
@@ -2,7 +2,10 @@
set -euo pipefail
-export RESTIC_REPOSITORY=sftp:restic@mountain-of-faith:/mnt/mountain-of-faith/restic
+readonly PRIMARY_REPOSITORY={{ primary_repository }}
+readonly SECONDARY_REPOSITORY={{ secondary_repository }}
+
+export RESTIC_REPOSITORY=${PRIMARY_REPOSITORY}
export RESTIC_PASSWORD={{ restic.password }}
readonly HEALTHCHECK=https://hc-ping.com/{{ healthchecks_io.backup_sh }}
@@ -78,3 +81,13 @@ restic --verbose forget \
--keep-within-weekly 1y \
--keep-within-monthly 100y \
--prune
+
+# Mirror onto the NAS repo while both are dual-write targets. Restic resolves
+# the source repository's password separately from RESTIC_PASSWORD, so pass
+# it explicitly; the repos share one.
+export RESTIC_REPOSITORY=${SECONDARY_REPOSITORY}
+export RESTIC_FROM_PASSWORD=${RESTIC_PASSWORD}
+restic --verbose copy --from-repo "${PRIMARY_REPOSITORY}"
+
+# Snapshots forgotten above are never removed from the secondary; a one-time
+# forget --prune against it happens at the cutover.