Assisted-by: Claude Opus 5.5 via Claude Code

diff --git a/docs/imperishable-night/2026-10-16-migrating-from-lotus-land-story.md b/docs/imperishable-night/2026-10-16-migrating-from-lotus-land-story.md
new file mode 100644
index 0000000..5500b4f
--- /dev/null
+++ b/docs/imperishable-night/2026-10-16-migrating-from-lotus-land-story.md
@@ -0,0 +1,268 @@
+# Migrating from lotus-land-story to imperishable-night
+
+lotus-land-story is a shared Linode 4 GB running Debian 11. It has two
+problems that an in-place fix handles badly:
+
+- It is short on memory. Over 14 days, available memory averaged 0.59 GB
+  and bottomed out at 60 MB, with half a gigabyte in swap.
+- Its OS is out of support. Bullseye's LTS ended on 2026-08-31, and the
+  planned move to podman needs two OS upgrades first
+  ([podman migration](../lotus-land-story/2026-09-04-podman-migration.md),
+  [bookworm upgrade](../lotus-land-story/2026-10-16-bullseye-to-bookworm.md)).
+
+imperishable-night replaces it: a shared Linode 8 GB (4 vCPU, 8 GB RAM,
+160 GB disk) running NixOS, with every service as a podman container
+through `virtualisation.oci-containers`. Services move over one at a time
+while lotus-land-story keeps serving the rest. There is no single
+cutover.
+
+The move is lift-and-shift. Each container keeps its image, tag, and
+environment. Replacing containers with native NixOS modules is out of
+scope.
+
+## Bootstrap the host
+
+The root tofu configuration creates the instance in `us-west` from a
+stock Debian image as a managed `linode_instance` resource, and a second
+step replaces Debian with NixOS. The NixOS configuration lives in
+a flake under `imperishable-night/`.
+
+sops-nix decrypts secrets with an age key that `ssh-to-age` derives from
+the host's SSH key. Add that age key as a recipient on the secrets the
+host needs. Those secrets are the same ones Ansible reads from
+`host_vars/lotus-land-story.sops.yml` today.
+
+Where the Nix tooling runs is undecided. The two options differ in
+whether the workstation needs Nix installed.
+
+### Option A: Nix only on the host
+
+The workstation needs no Nix. A tofu `remote-exec` provisioner runs
+nixos-infect on the Debian instance, which converts it to NixOS in place.
+nixos-infect is a community script and less maintained than
+nixos-anywhere. Read the host's SSH key after the conversion to derive
+its age key.
+
+To deploy, push the repository, SSH in, and run the following on the
+host:
+
+```sh
+nixos-rebuild switch --flake <repository>#imperishable-night
+```
+
+Evaluation and builds both happen on the host, so a mistake in the
+configuration surfaces there instead of on the workstation.
+
+### Option B: Nix on the workstation
+
+The workstation has Nix installed, though not NixOS. nixos-anywhere
+replaces Debian over SSH. Pass a pre-generated SSH host key with
+`--extra-files` so the age key exists before first boot.
+
+To deploy, run the following from the repository root:
+
+```sh
+nix run nixpkgs#nixos-rebuild -- switch \
+  --flake ./imperishable-night#imperishable-night \
+  --target-host imperishable-night --build-host imperishable-night
+```
+
+The workstation evaluates the configuration and the host builds it, so
+macOS needs no Linux builder. nixos-rebuild on macOS may also need
+`--fast` or `--no-reexec`, which is unverified.
+
+### Base configuration
+
+The base configuration covers the following:
+
+- Tailscale, joined as `imperishable-night`
+- node-exporter, added to the scrape config in
+  `lotus-land-story/templates/scrape.yml`
+- PostgreSQL and the restic backup job (see
+  [Back up both hosts](#back-up-both-hosts))
+- Caddy, running the same `ghcr.io/kejadlen/caddy-kejadlen` image with
+  no sites yet (see [Move ingress with each service](#move-ingress-with-each-service))
+- A firewall that trusts `tailscale0` and opens 80 and 443 publicly
+
+## Move ingress with each service
+
+Each site moves to imperishable-night's Caddy along with its service.
+lotus-land-story's Caddy doesn't proxy to imperishable-night's
+containers. That would mean publishing every container port on the host,
+and several collide: gitea, quire, docket, dockhand, and grafana all
+listen on 3000. With a Caddy on each host, every upstream stays a
+container name on that host's own network.
+
+How a site moves depends on how Caddy serves it:
+
+- A tailnet site is its own tailscale node through caddy-tailscale
+  (`bind tailscale/rss`). Remove the site from lotus-land-story's
+  Caddyfile, delete its machine in the tailscale admin console, and let
+  imperishable-night's Caddy register the same name. If the old machine
+  still exists, tailscale registers the new one as `rss-1` instead.
+- A public site needs its A record in `dns.tf` pointed at
+  imperishable-night instead of `local.lotus_land_story`. Lower
+  the record's TTL a day ahead. Copy the site's certificate from
+  lotus-land-story's Caddy data directory first, so imperishable-night
+  serves it without waiting on ACME. While the record propagates, point
+  the old site at imperishable-night's Caddy on 443 so that clients
+  still resolving the old address reach the new host:
+
+  ```caddy
+  hey.kejadlen.dev {
+  	reverse_proxy https://imperishable-night {
+  		header_up Host {host}
+  		transport http {
+  			tls_server_name {host}
+  		}
+  	}
+  }
+  ```
+
+  This is one Caddy-to-Caddy hop on 443, so no container ports are
+  involved. Remove the site from lotus-land-story once the old TTL has
+  passed.
+
+A tailnet site is down between the old Caddy dropping it and the new
+Caddy registering it. A public site with the bridge above has no gap
+beyond the service's own downtime.
+
+Two Caddyfile values are host-specific on imperishable-night:
+`trusted_proxies` comes from the podman network's subnet instead of the
+docker network's, and `TS_AUTHKEY` needs a reusable key so that
+registering each new site doesn't take a new key.
+
+## Move a service
+
+Follow these steps for each service:
+
+1. Define the container and its Caddy site in the flake, with the image
+   tag and environment from its playbook in `lotus-land-story/`.
+2. Deploy imperishable-night with the container stopped and the site
+   left out.
+3. Stop the container on lotus-land-story, and remove its site from
+   lotus-land-story's Caddyfile.
+4. Copy the service's data:
+   - For files, rsync `/mnt/lotus-land-story/<service>/`.
+   - For a database, `pg_dump` that one database and restore it on
+     imperishable-night.
+5. Start the container on imperishable-night.
+6. Move the site, as described in
+   [Move ingress with each service](#move-ingress-with-each-service).
+7. Verify the service, then remove it from lotus-land-story's
+   `main.yml`.
+
+Restoring the data from restic instead of rsyncing it also works, and it
+tests the backups along the way.
+
+### Shared state
+
+miniflux and ketchup share lotus-land-story's PostgreSQL 13
+cluster. Each database moves with its service. Moving one database at a
+time also spreads the jump from 13 to the newer nixpkgs PostgreSQL across
+services instead of doing it all at once.
+
+paperless-ngx is the only redis user, so redis moves with it.
+
+## Order of moves
+
+Move the services in the following order, starting with those whose
+breakage costs least:
+
+1. Personal services: ketchup, docket, quire, irc (soju), and homebox.
+2. PostgreSQL-backed services: miniflux.
+3. Family-facing services: calibre-web, paperless-ngx with redis, and
+   pro-tacts.
+4. Public services: gitea with anubis, and gotosocial. gitea also has a
+   pihole record in the root `main.tf` that points `git.kejadlen.dev` at
+   lotus-land-story's tailnet address, so repoint it at the same time.
+5. authelia.
+6. Observability: VictoriaMetrics, VictoriaLogs, VictoriaTraces, Grafana,
+   Vector, and the OpenTelemetry collector.
+7. Sites that only Caddy serves: `setup`, `gists`, `files`, `birbs`,
+   `pihole`, and the two Home Assistant proxies. The file-serving sites
+   need their directories under `/mnt/lotus-land-story/caddy/data`
+   copied along with them.
+
+dockhand and cadvisor talk to the Docker socket. Check whether each
+works against podman's Docker-compatible socket before moving it.
+
+## Move gotosocial
+
+gotosocial federates as `hey.kejadlen.dev`, and other servers know the
+account by that host and by the signing keys in gotosocial's database.
+`GTS_HOST` stays the same and the SQLite database moves intact, so the
+account's identity is unchanged. This is a server relocation, not an
+ActivityPub account move, and followers don't need to do anything.
+
+The open question is what federation traffic does during the outage:
+
+- Inbound deliveries fail while gotosocial is down. Remote servers retry
+  failed deliveries with backoff; Mastodon retries for days, but other
+  servers' retry windows are unverified.
+- Outbound deliveries that gotosocial has queued but not sent may be
+  lost when it stops, if its queue lives only in memory. Whether it does
+  is unverified, so avoid posting right before the move.
+
+To keep the outage short, follow these steps:
+
+1. Lower the TTL on `hey.kejadlen.dev` a day ahead.
+2. Copy `/mnt/lotus-land-story/gotosocial/` to imperishable-night while
+   gotosocial is still running, so the final copy only transfers what
+   changed.
+3. Stop gotosocial, then back up the database with
+   `sqlite3 ... '.backup ...'` as a rollback point, as
+   `lotus-land-story/gotosocial.yml` already does before upgrades.
+4. Rsync the data directory again, then start gotosocial on
+   imperishable-night.
+5. Switch the old site to the Caddy-to-Caddy bridge, then point the A
+   record at imperishable-night.
+
+Logging in to gotosocial goes through authelia over
+`https://auth.kejadlen.dev`, which is public, so gotosocial can move
+before authelia does.
+
+## Move authelia
+
+authelia is the OIDC provider for Tailscale login, so an authelia
+outage blocks anyone from signing in to the tailnet. Its move has three
+constraints:
+
+- The issuer URL stays `https://auth.kejadlen.dev`.
+- The OIDC signing key and storage encryption key stay the same.
+- The SQLite database, `users_database.yml`, and secrets move intact,
+  copied while the container is stopped.
+
+Moving `auth.kejadlen.dev` and `auth.chislan.family` takes an A record
+change, so lower their TTLs a day ahead to keep the login outage short.
+
+## Back up both hosts
+
+lotus-land-story's daily restic run covers only its own volume. Before
+the first stateful service moves, imperishable-night needs its own
+restic job against the repository on legacy-of-lunatic-kingdom, with a
+distinct host tag. Use NixOS's `services.restic.backups`. It has the
+same phases as `lotus-land-story/templates/backup.sh`, with
+`podman exec` replacing `docker exec` for the pro-tacts dump. Update
+[backups](../backup.md) when it lands.
+
+## Point observability at the old stack
+
+Until the observability stack moves, imperishable-night's Vector ships
+logs to VictoriaLogs on lotus-land-story, and lotus-land-story's
+VictoriaMetrics scrapes imperishable-night. When the stack moves, either
+copy the VictoriaMetrics data directory or let the old history age out.
+
+## Retire lotus-land-story
+
+Once its Caddyfile serves no sites, lotus-land-story is idle. After a
+quiet period, take a final restic snapshot of it. Then remove it from
+`hosts.yml` and the scrape config.
+
+The live tofu configuration doesn't manage lotus-land-story. The root
+`main.tf` only looks it up with data sources, and the separate
+configuration in `lotus-land-story/` has had no state changes since
+April 2023, so it still describes a Nanode with a 10 GB volume. Delete the
+instance and its volume in Linode directly. Then remove the
+`lotus-land-story/` tofu files and the `lotus_land_story` data sources and
+local from the root `main.tf`.