Fall 2026brianmars.comBuilt at EGBT Technologies
ReferenceNotifications

msmtp and failure alerts

One outbound mail account for Forgejo, Uptime Kuma, and an email whenever a systemd job fails.

Names and addresses are placeholders. The list says what each one stands for.

What it does here

Three things send mail: Forgejo, Uptime Kuma and a failed backup. They share one SMTP account. A shell script can't speak SMTP, so msmtp sits between systemd and the relay.

From the manual

Canonical setup

msmtp as the system's sendmail, configured in /etc/msmtprc, and a systemd template unit that OnFailure= can call.

Upstream: msmtp documentation, systemd.unit.

Edited for this build

In this build

  1. One relay accountForwardEmail's outbound SMTP, smtp.forwardemail.net on port 465 with implicit TLS, sending as noreply@tailnet.example.
  2. Forgejo's mailer set by environment variables
  3. Uptime Kuma's SMTP notification on the same account
  4. A notify-email@ template unitAttached to the backup service with an OnFailure= drop-in, and reusable for any other unit.
  5. Errors first, then the failed run's logThe mail leads with the !! and error lines, then shows only the run that failed (journalctl -I), not the unit's last 40 lines.

Enable outbound SMTP for the domain at the provider first, add the DNS records it lists (SPF, DKIM, DMARC, return path), create the sending alias and generate its password.

Forgejo mailer (add to the forgejo service's environment)
      - FORGEJO__mailer__ENABLED=true
      - FORGEJO__mailer__PROTOCOL=smtps
      - FORGEJO__mailer__SMTP_ADDR=smtp.forwardemail.net
      - FORGEJO__mailer__SMTP_PORT=465
      - FORGEJO__mailer__FROM=Forgejo <noreply@tailnet.example>
      - FORGEJO__mailer__USER=noreply@tailnet.example
      - FORGEJO__mailer__PASSWD=the-generated-password

Test from Site Administration, then Configuration, then Send Test Email. In Uptime Kuma, add an Email (SMTP) notification with the same host, port and account, TLS on, and attach it to every monitor.

/etc/msmtprc (root-only)
sudo apt install -y msmtp msmtp-mta
sudo install -m 600 /dev/null /etc/msmtprc
sudo tee /etc/msmtprc >/dev/null <<'EOF'
defaults
auth on
tls on
tls_starttls off
account forwardemail
host smtp.forwardemail.net
port 465
from noreply@tailnet.example
user noreply@tailnet.example
password the-generated-password
account default : forwardemail
EOF
printf 'Subject: msmtp test\n\nhello\n' | sudo msmtp you@tailnet.example
Failure hook and template unit
sudo tee /usr/local/bin/notify-failure.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
UNIT="$1"
# -I is the unit's latest invocation, i.e. the run that just failed
LOG=$(journalctl -u "$UNIT" -I --no-pager -o short)
{
  echo "Subject: [devbox] ${UNIT} FAILED"
  echo "To: you@tailnet.example"
  echo
  echo "${UNIT} failed on $(hostname) at $(date)."
  echo
  echo "== Error lines =="
  grep -E '!!|FAILED|Fatal|[Ee]rror' <<<"$LOG" || echo "(none matched; see the full log below)"
  echo
  echo "== Full log of the failed run (last 200 lines) =="
  tail -n 200 <<<"$LOG"
} | msmtp -C /etc/msmtprc you@tailnet.example
EOF
sudo chmod 700 /usr/local/bin/notify-failure.sh
sudo bash -n /usr/local/bin/notify-failure.sh

sudo tee /etc/systemd/system/notify-email@.service >/dev/null <<'EOF'
[Unit]
Description=Send failure email for %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure.sh %i
EOF

sudo mkdir -p /etc/systemd/system/devbox-backup.service.d
sudo tee /etc/systemd/system/devbox-backup.service.d/onfailure.conf >/dev/null <<'EOF'
[Unit]
OnFailure=notify-email@%n.service
EOF
sudo systemctl daemon-reload
Where it breaks

Seams

Where this meets the rest of the build. Each one leads with what you will actually see.

A normal user meets /etc/msmtprc

Symptomno configuration file available

The file holds the password, so it's root-only and a normal user can't read it. Test with sudo msmtp, which matches how the failure hook runs.

A bridge container meets the mail host

SymptomForgejo's test email can't resolve smtp.forwardemail.net

Forgejo runs on a bridge network and needs the pinned DNS in the host's daemon.json. See the Docker card.

The relay meets its approval

Symptommail is refused before the provider has enabled outbound SMTP

Outbound SMTP is enabled per domain and can take up to a day to approve. Wait for it before testing anything else.

A 40-line tail meets a two-target backup

Symptomthe failure mail shows a successful prune and no error

One target's forget --prune prints about 40 lines. If the home copy fails and R2 then succeeds, journalctl -n 40 shows only R2's output, and the !! backup to ... FAILED line is cut off. Grep for the error lines first and read the failed run with journalctl -u UNIT -I, as in the hook above.

Verify

Point SFTP_REPO at a host that doesn't exist, run sudo systemctl start devbox-backup.service, and check the failure email arrives with the error line at the top. Then put SFTP_REPO back.

Test the hook on its own
sudo systemctl start notify-email@devbox-backup.service.service
systemctl status notify-email@devbox-backup.service.service   # status=0/SUCCESS
Corrections and updates
  1. 17 Oct 2026First written.
  2. 23 Oct 2026The failure mail leads with the error lines and shows only the failed run. Added the seam for a 40-line tail that cuts off the error.