double-dealing-character: fix exit node return path
The tailnet aggregate has to live in the main table: mwan3rtmon mirrors main into table 1 and deletes anything main lacks, so a route written straight to table 1 is reaped on the next sync. Assisted-by: Claude Opus 5 via Claude Code
diff --git a/double-dealing-character/2026-08-07-exit-node-return-path.md b/double-dealing-character/2026-08-07-exit-node-return-path.md
index b12a797..355b67b 100644
--- a/double-dealing-character/2026-08-07-exit-node-return-path.md
+++ b/double-dealing-character/2026-08-07-exit-node-return-path.md
@@ -1,5 +1,10 @@
# Exit node return path hijacked by mwan3
+Fixed on 2026-08-10 by adding `100.64.0.0/10 dev tailscale0` to the `main`
+table; the two sections on removing mwan3 record approaches that failed and
+should not be retried. Verified end to end — a peer's egress IP became the
+Mango's public address.
+
Symptom: a peer (the Apple TV) selects `double-dealing-character` as its exit
node and gets no internet. SSH into the Mango works, and `curl https://google.com/`
from the Mango works, so the box itself is fine.
@@ -60,38 +65,140 @@ own encrypted packets can never re-enter the tunnel. A bare table 52 lookup at
priority 1000 sits above that guard and breaks the invariant. Any rule added
above 5210 has to carry the same fwmark exclusion.
-## Fixing it
+## Stopping mwan3 wedges the box too
-Preferred: stop mwan3. This is a single-WAN box, and mwan3 only exists for
-GL.iNet's failover between ethernet, repeater, and tethering. Removing it drops
-rules 1001, 2001, 2061, and 2062, and lets Tailscale's 5210-5270 set work as
-designed.
+The obvious next move is to remove mwan3 outright. Nothing on this box needs
+it: `mwan3 status` reports `wan` (wired, `eth0.2`, 192.168.1.26) online and
+`wwan`, `tethering`, and all three v6 interfaces offline, so GL.iNet's failover
+between ethernet, repeater, and tethering has nothing to fail over to. Removing
+it drops rules 1001, 2001, 2061, and 2062, and lets Tailscale's 5210-5270 set
+work as designed.
```sh
+# DO NOT DO THIS OVER THE TAILNET
/etc/init.d/mwan3 stop && /etc/init.d/mwan3 disable
```
-If failover is worth keeping, add the tailnet prefix to mwan3's table instead
-of reordering rules. This leaves tailscaled's own traffic path untouched,
-because locally generated packets have `iif lo` and never match rule 1001:
+Tried on 2026-08-08. The SSH session died the instant the command ran, which on
+its own is expected — `mwan3 stop` flushes conntrack, and the session was
+established through it. The box never came back.
+
+The failure signature differs from the table-52 attempt. WireGuard stayed up:
+`tailscale status` kept reporting the node active with `tx` and `rx` both
+climbing by disco-sized increments, though the direct connection degraded to
+the `nyc` relay within a minute. Nothing over the tunnel worked — no ping, no
+SSH to `100.110.110.52`. Handshakes fine, tun path dead. The earlier attempt
+lost handshakes instead.
+
+What the two attempts share is the trigger, not the symptom. Both mutated the
+`ip rule` table underneath a running tailscaled, and both ended in a power
+cycle with no mechanism confirmed, because the box became unreachable before it
+could be inspected. Suspect tailscaled reacting badly to its routing
+environment changing beneath it, rather than anything specific to the rule that
+changed.
+
+## `mwan3 disable` does not disable mwan3
+
+The reboot that followed brought mwan3 straight back. `/etc/init.d/mwan3
+enabled` reports disabled and `/etc/rc.d` holds no symlink, yet `mwan3rtmon`
+and `mwan3track` are running and rule 1001 is present.
+
+`/etc/hotplug.d/iface/15-mwan3` starts it. The WAN's DHCP `ifup` on boot fires
+the hotplug event, the event starts mwan3, and the rc.d symlink `disable`
+removed never enters into it. Three more scripts in that directory reference
+mwan3 as well (`16-mwan3`, `16-mwan3-user`, `17-dual-sim`).
+
+So the whole "remove mwan3" line of attack is a dead end on GL.iNet firmware
+short of deleting the hotplug scripts, which a firmware upgrade would restore.
+Leave mwan3 running and work with it.
+
+## The fix
+
+Put the tailnet aggregate in the `main` table and let mwan3 copy it down:
```sh
-ip route add 100.64.0.0/10 dev tailscale0 table 1
+ip route add 100.64.0.0/10 dev tailscale0
```
-mwan3 flushes table 1 on WAN events, so this needs re-adding from a hotplug
-script to survive.
+The `mwan3rtmon` daemon continuously mirrors `main` into every interface table.
+It diffs the two and reconciles in both directions, adding routes main has and
+deleting routes main lacks (the `ipv4.main` loop in `/lib/mwan3/mwan3.sh`).
+That has two consequences. A route added straight to table 1 is reaped on the
+next sync, so the hotplug script this note previously recommended for it would
+have been racing a monitor loop. A route added to `main` lands in table 1
+within seconds and stays there across mwan3's rebuilds, for free.
+
+Table 52 holds a per-peer `/32` for every node rather than an aggregate, so
+there is no single tailscale-managed route to mirror; the `/10` covers the
+tailnet in one entry.
+
+This does not repeat the mistake the table-52 hoist made. Rules 5210-5250 exist
+to keep tailscaled's own encrypted packets out of the tunnel, and those packets
+are addressed to peers' public endpoints, never into `100.64.0.0/10`. A route
+for that prefix in `main` cannot catch them.
+
+Persist it from a hotplug script keyed on the tailscale interface, since the
+route needs `tailscale0` to exist and tailscaled starts last at `START=99`:
+
+```sh
+# /etc/hotplug.d/iface/30-tailnet-route
+[ "$INTERFACE" = "tailscale" ] || exit 0
+case "$ACTION" in
+ ifup) ip route replace 100.64.0.0/10 dev tailscale0 ;;
+ ifdown) ip route del 100.64.0.0/10 dev tailscale0 2>/dev/null ;;
+esac
+```
+
+`INTERFACE` is the UCI section name, `tailscale`, not the device name. To
+confirm the event fires for it without disturbing anything, look for
+`network.tailscale.up='1'` in `/var/state/network` — `00-netstate` writes that
+line from the same hotplug event.
+
+Note that `ifup tailscale` is not a test of this. netifd no-ops `ifup` on an
+interface that is already up, so no event fires and the script never runs.
+
+Installed by `mango-tailscale-setup.sh configure-routing`, and listed in
+`/etc/sysupgrade.conf` so it survives a firmware upgrade.
+
+## Applying changes without losing the box
+
+Two of the three attempts here wedged the Mango and cost a power cycle, both
+while mutating routing state under a running tailscaled. Arm a dead-man's
+switch before the change and the failure mode costs ten minutes instead of a
+trip to the hardware:
+
+```sh
+nohup sh -c 'sleep 600; ip route del 100.64.0.0/10 dev tailscale0' >/dev/null 2>&1 &
+ip route add 100.64.0.0/10 dev tailscale0
+```
+
+Verify, then `kill` the pending `sleep` once the change proves good.
+
+Adding a route turned out to be far safer than the earlier attempts, which
+deleted or reordered `ip rule` entries and flushed conntrack. Nothing dropped,
+including the SSH session.
+
+## Recovering a wedged box
+
+Power-cycle it. Nothing tried here survives a reboot — including `mwan3
+disable`, for the reasons above.
-Neither fix is in `mango-tailscale-setup.sh` yet.
+A box that comes back still broken is by definition not reachable over the
+tailnet. Reach it on its own LAN or wifi at 192.168.8.1. Its WAN-side address
+on the home LAN (192.168.1.26) is not a second way in; the `wan` zone defaults
+to `input=REJECT` on GL.iNet firmware, and this has not been tested from a host
+on that subnet.
## Verifying
From the Mango, check that a reply destined for a peer routes into the tunnel
-rather than back out the WAN:
+rather than back out the WAN. With the fix in place the answer comes from table
+1, not table 52 — mwan3's rule 1001 still wins, it just now resolves correctly:
```sh
ip route get 100.86.9.54 from 8.8.8.8 iif eth0.2
-# want: 100.86.9.54 from 8.8.8.8 dev tailscale0 table 52
+# want: 100.86.9.54 from 8.8.8.8 dev tailscale0 table 1
+# broken: 100.86.9.54 from 8.8.8.8 via 192.168.1.1 dev eth0.2 table 1
```
From a peer, confirm the egress IP actually changes:
diff --git a/double-dealing-character/mango-tailscale-setup.sh b/double-dealing-character/mango-tailscale-setup.sh
index 1b02d81..809af97 100755
--- a/double-dealing-character/mango-tailscale-setup.sh
+++ b/double-dealing-character/mango-tailscale-setup.sh
@@ -432,6 +432,60 @@ cmd_configure_firewall() {
log "Firewall configured"
}
+# Keep exit-node return traffic inside the tunnel despite mwan3.
+#
+# GL.iNet firmware runs mwan3 for WAN failover, and mwan3 installs an `ip rule`
+# at priority 1001 that sends everything arriving on the WAN to route table 1.
+# Table 1 carries a default route but no tailnet routes, so replies headed back
+# to an exit-node peer leave via the upstream router instead of tailscale0.
+# Only forwarded traffic breaks - packets addressed to the Mango itself match
+# the local table at priority 0 and never reach rule 1001, which is why the box
+# looks perfectly healthy while the exit node does not work at all.
+#
+# The aggregate goes in the MAIN table, not table 1. mwan3's mwan3rtmon daemon
+# continuously mirrors main into each interface table and deletes anything main
+# does not have, so a route written straight to table 1 is reaped on the next
+# sync; one written to main is copied down within seconds and stays. Table 52
+# holds a per-peer /32 rather than an aggregate, so there is no single
+# tailscale-managed route to mirror instead.
+#
+# This does not disturb the guard at rules 5210-5250, which keeps tailscaled's
+# own encrypted packets out of the tunnel: those are addressed to peers' public
+# endpoints and never into 100.64.0.0/10.
+#
+# Do NOT try to remove mwan3 instead. `/etc/init.d/mwan3 disable` does not hold
+# because /etc/hotplug.d/iface/15-mwan3 restarts it on the WAN's DHCP ifup, and
+# stopping it live has wedged this box. See 2026-08-07-exit-node-return-path.md.
+cmd_configure_routing() {
+ log "Installing tailnet route hotplug script"
+
+ # Keyed on the tailscale interface because the route needs tailscale0 to
+ # exist, and tailscaled starts last at START=99. INTERFACE is the UCI
+ # section name, not the device name.
+ cat > /etc/hotplug.d/iface/30-tailnet-route <<'EOF'
+#!/bin/sh
+# See cmd_configure_routing in mango-tailscale-setup.sh for why this route
+# belongs in the main table rather than mwan3's table 1.
+[ "$INTERFACE" = "tailscale" ] || exit 0
+
+case "$ACTION" in
+ ifup) ip route replace 100.64.0.0/10 dev tailscale0 ;;
+ ifdown) ip route del 100.64.0.0/10 dev tailscale0 2>/dev/null ;;
+esac
+EOF
+ chmod +x /etc/hotplug.d/iface/30-tailnet-route
+
+ # Apply now if the interface is already up; otherwise the hotplug script
+ # picks it up on the next ifup. Note that `ifup tailscale` is NOT a way to
+ # force that - netifd no-ops it when the interface is already up.
+ if [ -d /sys/class/net/tailscale0 ]; then
+ ip route replace 100.64.0.0/10 dev tailscale0
+ log "Tailnet route added to main table"
+ else
+ log "tailscale0 not up yet; route will be added on next ifup"
+ fi
+}
+
# Install a procd init script for tailscaled.
#
# procd is OpenWrt's service supervisor, the equivalent of systemd or runit
@@ -574,15 +628,18 @@ cmd_up() {
# already preserved automatically; what we add here is the extra stuff that
# lives outside /etc/config:
#
-# - /etc/init.d/tailscale : custom procd script
-# - /etc/modules.d/30-tun : auto-load tun module on boot
-# - /etc/dropbear/authorized_keys : SSH public keys (NOT in /etc/config)
+# - /etc/init.d/tailscale : custom procd script
+# - /etc/modules.d/30-tun : auto-load tun module on boot
+# - /etc/dropbear/authorized_keys : SSH public keys (NOT in /etc/config)
+# - /etc/hotplug.d/iface/30-tailnet-route : tailnet route past mwan3's rule 1001
#
# Note that /mnt/sda1/* is on USB and unaffected by sysupgrade entirely.
cmd_persist_upgrade_config() {
log "Adding paths to /etc/sysupgrade.conf"
[ -f /etc/sysupgrade.conf ] || touch /etc/sysupgrade.conf
- for path in /etc/init.d/tailscale /etc/modules.d/30-tun /etc/dropbear/authorized_keys; do
+ for path in /etc/init.d/tailscale /etc/modules.d/30-tun \
+ /etc/dropbear/authorized_keys \
+ /etc/hotplug.d/iface/30-tailnet-route; do
if grep -qxF "$path" /etc/sysupgrade.conf; then
log " $path (already present)"
else
@@ -624,6 +681,16 @@ cmd_status() {
echo
echo "=== Firewall (tailscale zone & forwarding) ==="
uci show firewall 2>/dev/null | grep -E "tailscale" || echo "(no tailscale zone configured)"
+ echo
+ echo "=== Exit node return path ==="
+ if ip route show table main 2>/dev/null | grep -q '^100\.64\.0\.0/10'; then
+ echo "tailnet route in main: present"
+ else
+ echo "tailnet route in main: MISSING (run 'configure-routing')"
+ fi
+ # Should resolve to tailscale0. Anything via the WAN gateway means mwan3's
+ # rule 1001 is still winning and the exit node will not work.
+ ip route get 100.100.100.100 from 8.8.8.8 iif eth0.2 2>/dev/null | head -1
}
# Pull the current stable Tailscale and restart the service.
@@ -650,6 +717,8 @@ cmd_all() {
cmd_install_tailscale
cmd_configure_firewall
cmd_install_init
+ # After install_init, so tailscaled has created tailscale0 to route to.
+ cmd_configure_routing
cmd_configure_ssh "--ssh-key=$SSH_PUBKEY"
cmd_persist_upgrade_config
@@ -677,6 +746,7 @@ Stages (each idempotent, can be re-run):
install-tailscale Download current stable mipsle Tailscale
configure-firewall Tailscale UCI iface + zone + forward to wan
install-init /etc/init.d/tailscale, enable, start
+ configure-routing Tailnet route in main table, past mwan3 rule 1001
configure-ssh --ssh-key='ssh-ed25519 AAAA... user'
OR --ssh-key-file=/path/to/key.pub
disable-password-ssh Lock dropbear to key auth (do AFTER testing keys)
@@ -717,6 +787,7 @@ case "$cmd" in
install-tailscale) cmd_install_tailscale ;;
configure-firewall) cmd_configure_firewall ;;
install-init) cmd_install_init ;;
+ configure-routing) cmd_configure_routing ;;
configure-ssh) cmd_configure_ssh "$@" ;;
disable-password-ssh) cmd_disable_password_ssh ;;
up) cmd_up ;;