By the end of part 4 the box did its job. I still had to look at it to know it was working, and nothing protected the data if the disk died. This part covers both.
Four Dashboards
Dockge manages the compose stacks I'd been running by hand. It reads the same files in /srv/stacks, so nothing moved, but the folder has to be mounted at the same path inside the container.
A web UI for the compose stacks under /srv/stacks, working through the Docker socket.
- The stacks folder is /srv/stacks, at the same path inside the container
- Bound to 127.0.0.1:5001
Beszel graphs CPU, memory, disk and per-container use. Its hub and agent both run on host networking. On the bridge network, the hub would have to cross the firewall's hairpin to reach the agent, and the system would stay red.
Resource graphs for the box and its containers, from a hub and an agent that both run on host networking.
- Hub and agent both on host networking
- The system added as localhost, port 45876
- The key written into the compose file
- Served by Caddy as metrics.dev.tailnet.example
Uptime Kuma checks each service and sends a message when one goes down. It also runs on host networking, because the services listen on 127.0.0.1 and a bridge container can't reach the host's loopback. I wrote every monitor as 127.0.0.1, since localhost tries IPv6 first.
Up and down checks for every service, probing 127.0.0.1 from host networking, with alerts that actually arrive.
- Host networking
- Monitors on 127.0.0.1, not localhost
- Email alerts through the shared relay
- Served by Caddy as status.dev.tailnet.example
Homepage is the front page, one tile per service with its container status. I wrote its config files before the first start, because it fills an empty folder with its own placeholder groups.1
One landing page with a tile per service and live container status, configured from files you own.
- Config under /srv/stacks/homepage/config
- Config written before the first start
- HOMEPAGE_ALLOWED_HOSTS set
- The socket mounted read-only
- 127.0.0.1:3002 outside, 3000 inside
Backups, Twice
The data disk is one copy. restic makes two more, one on a machine at home over SFTP and one in Cloudflare R2. restic encrypts before it uploads, so neither target sees anything readable. A systemd timer runs it daily, and one target failing doesn't stop the other.
The backup runs as root, so root's SSH key has to be on the backup host, not mine. A plain ssh test can pass by asking for a password, which a timer can't answer.
I restored a file from the off-site copy before I called it done. An untested backup is a guess.
Encrypted daily backups to a machine at home and to object storage off-site, run as root by a systemd timer, with a tested restore.
- Two repositories, one script
- Runs as root
- Secrets in a root-only env file
- HOME set in the script
- The Docker data root excluded
- Keep 7 daily, 4 weekly, 6 monthly
- Daily at 03:30
Mail When Something Breaks
Forgejo, Uptime Kuma and a failed backup all send mail through one SMTP account. A shell script can't speak SMTP, so msmtp hands systemd's mail to the relay. A small template unit sends the journal lines for any unit that fails.
To test it, I pointed the backup at a host that doesn't exist and waited for the email.
One outbound mail account for Forgejo, Uptime Kuma, and an email whenever a systemd job fails.
- One relay account
- Forgejo's mailer set by environment variables
- Uptime Kuma's SMTP notification on the same account
- A notify-email@ template unit
The Reboot Test
Last, I rebooted the box and changed nothing else. I checked that /srv mounted, Tailscale reconnected, and every container came back. The runner had to show online and finish a test job. The firewall had to still be answering only on the tailnet. My tmux session had to restore, a git push over port 2222 had to work, and Uptime Kuma had to show green with a test alert arriving.
Then I reached everything from my phone on cellular, off my home network entirely. That is the only test that shows the box is reachable from anywhere.
Each month I update the host and pull new images for each stack, and I prune old images when Beszel shows /srv filling up. I keep Forgejo updates manual so a major version bump doesn't happen on its own.
Since It’s Just Sitting There
Given that the machine is rarely overtaxed on CPU, I decided to add an external 4-bay drive container I had from an old NAS and added Jellyfin, CalibreWeb-NextGen, and TimeMachine to the server as well. I’ll write up those reference cards sometime in the future.
Every card from this series is at /reference.
No comments yet