How I back up my homelab: snapshots, database dumps and an offline copy

Backups are the least exciting part of a homelab, until the day you need one. I built mine in steps, mostly after “what if…” moments, and it ended up as plain bash scripts, rsync and systemd timers: no backup software with its own database that I’d have to restore first. This article walks through the whole setup: what I back up, how the snapshots work, why databases are dumped instead of copied, how the third copy is protected against a hacked server, how I know every night that it worked, and how a restore looks.

1. Three copies, two disks, one of them out of reach

The classic rule is 3-2-1: three copies of your data, on two different media, one of them offsite. Here is my version of it, and where it falls short:

Copy	Where	Protects against
1. Live data	The server’s SSD	nothing – this is what we protect
2. Nightly snapshots	A USB disk on the server, 14–30 days of history	deleted or overwritten files, broken updates, a dead server SSD
3. Pulled copy	An encrypted SSD that a laptop fills itself	a dead backup disk, a hacked server, ransomware, mistakes with the snapshots

Honestly, the third copy is in the same house. Against fire or burglary it doesn’t help; for that, the SSD would have to live somewhere else, or an encrypted cloud copy would be the next step. For everything else, these three layers have covered every real problem I’ve had so far.

2. What gets backed up

Every service has its own small script and its own time slot, so a failing job never blocks the others, and the heavy Nextcloud job runs last:

Service	What is saved	Kept	Downtime
WordPress sites (web host)	Files via tar over SSH + MySQL dump	30 days	none
Paperless-ngx	PostgreSQL dump + all documents + search index	30 days	none
Authentik (single sign-on)	PostgreSQL dump + compose folder incl. .env	30 days	none
Infrastructure	Traefik (incl. certificates), CrowdSec config, Portainer, SSH config, the backup scripts themselves	60 days	Portainer paused ~1 s
Zigbee2MQTT	The whole data folder (network key, device database)	30 days	paused a few seconds
BookStack wiki	MariaDB dump + uploads + .env	30 days	none
PostgreSQL for Home Assistant	Dump of every database + roles	14 days	none
Vaultwarden	Consistent SQLite copy + keys + attachments	30 days	none
Nextcloud	All user files + PostgreSQL dump + config volume	14 days	maintenance mode during the run

Not backed up on purpose: logs, caches, message queues, CrowdSec’s ban list (it rebuilds itself), and the live database folders. Those are useless while the database runs; the dumps are the backup.

3. Snapshots with rsync hardlinks

Every run creates a folder named after the date, like 2026-10-20_0030. Each of these folders looks like a complete copy: you can browse it, copy single files out of it, or restore all of it. But files that didn’t change since the previous night aren’t copied again. They are hardlinks to the same file in the previous snapshot, so they take no extra disk space. That’s what rsync’s --link-dest option does:

/backup/paperless_backup/
├── 2026-10-19_0030/
│   ├── paperless-db.dump   database dump
│   ├── media/              all documents (mostly hardlinks)
│   ├── data/               search index, classifier
│   ├── compose/            docker-compose.yml, .env
│   └── rsync.log
├── 2026-10-20_0030/
├── latest -> 2026-10-20_0030
└── backup-paperless.sh    the script copies itself here

• Deleting an old snapshot is safe. A file only disappears from the disk when the last snapshot pointing to it is deleted.
• The filesystem must support hardlinks: ext4, XFS or btrfs. Not exFAT or NTFS – the typical format of a new USB disk.
• Hardlinks are not extra copies. Thirty snapshots of an unchanged document are still one file on the disk. If that disk sector dies, all thirty are affected. That’s why the third copy exists.
• The script copies itself next to the snapshots, so the instructions are on the backup disk even when the server is gone.

4. Databases are dumped, never copied

Copying the files of a running database gives you a snapshot of a moving target: half-written pages, a write-ahead log that doesn’t match. It may restore, or it may not. A dump asks the database itself for a consistent export, while it keeps running. And every dump is checked right after it is written:

Database	Dump	Check
PostgreSQL	pg_dump -Fc (compressed custom format)	pg_restore --list must be able to read it
MariaDB / MySQL	mariadb-dump / mysqldump, gzipped	gzip -t and the line “Dump completed” at the end
SQLite (Vaultwarden)	sqlite3 … ".backup" (consistent online copy)	PRAGMA integrity_check

A dump that fails its check stops the whole job, and the night’s snapshot never becomes latest (next section). A corrupt backup is worse than none, because you only find out when you need it.

5. One backup script, explained

All my scripts follow the same pattern. Here is the Paperless one, slightly shortened:

#!/usr/bin/env bash
set -Eeuo pipefail

PROJECT=/srv/docker/paperless          # compose folder
DEST=/backup/paperless_backup
KEEP_DAYS=30
STAMP=$(date +%F_%H%M)
SNAP=$DEST/$STAMP
WORK=$SNAP.partial

fail() { echo "ERROR: $*"; exit 1; }

# 1. never fill the system disk if the backup disk is missing
mountpoint -q /backup || fail "/backup is not mounted"

mkdir -p "$DEST" && chmod 700 "$DEST"
rm -rf "$DEST"/*.partial               # leftovers of a failed run
mkdir -p "$WORK"

LINK=()
[ -d "$DEST/latest" ] && LINK=(--link-dest="$DEST/latest/media")

# 2. database dump + check
cd "$PROJECT"
docker compose exec -T db sh -c 'pg_dump -U "$POSTGRES_USER" -Fc "$POSTGRES_DB"' > "$WORK/paperless-db.dump"
docker compose exec -T db pg_restore --list < "$WORK/paperless-db.dump" > /dev/null 
  || fail "dump check failed"

# 3. files, unchanged ones as hardlinks to the last snapshot
rsync -aHAX --numeric-ids "${LINK[@]}" "$PROJECT/data/media/" "$WORK/media/"

# 4. only now does the snapshot become real
mv "$WORK" "$SNAP"
ln -sfn "$STAMP" "$DEST/latest"
cp -f "$0" "$DEST/"

# 5. retention
find "$DEST" -mindepth 1 -maxdepth 1 -type d -name '20??-??-??_????' 
  -mtime +"$KEEP_DAYS" -exec rm -rf {} +

• set -Eeuo pipefail: any failing command stops the script. A backup script that carries on after an error produces backups that look fine and aren’t.
• mountpoint -q: if the USB disk isn’t mounted, /backup is just an empty folder on the system disk. Without this check, the first night after a disk failure fills your system SSD.
• The .partial folder: the snapshot is built under a temporary name and only renamed when everything succeeded. A failed run never becomes latest, so the next run still links against the last good one.
• rsync -aHAX --numeric-ids: keeps permissions, hardlinks, ACLs, extended attributes and the numeric owner IDs. Containers often run as user IDs that don’t exist on the host; with names instead of numbers, a restore would get the owners wrong.
• Retention by folder age: find -mtime +30 removes snapshots older than 30 days. Thanks to hardlinks, that’s always safe.

6. Scheduling with systemd timers

I use systemd timers instead of cron. They log every run in the journal, they catch up on runs that were missed while the server was off (Persistent=true), and a service can wait until the backup disk is mounted:

# /etc/systemd/system/backup-paperless.service
[Unit]
Description=Paperless backup
RequiresMountsFor=/backup
After=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup-paperless.sh

# /etc/systemd/system/backup-paperless.timer
[Unit]
Description=Nightly Paperless backup

[Timer]
OnCalendar=*-*-* 00:30
Persistent=true

[Install]
WantedBy=timers.target

Enable it with systemctl enable --now backup-paperless.timer. systemctl list-timers shows when each job ran last and runs next, and journalctl -u backup-paperless shows its output. My jobs start between 23:30 and 03:00, at least 15 minutes apart, so they never compete for the disk.

7. Services that need special care

• Nextcloud: the job switches on maintenance mode, dumps the database, copies the files and switches it off again. Otherwise someone could upload a file between the dump and the file copy, and the two wouldn’t match. The copy runs with an rsync bandwidth limit (--bwlimit) so the disks aren’t saturated, and at 03:00, when nobody uploads anything. On a normal night that’s a few minutes of maintenance mode.
• Zigbee2MQTT: its folder holds the network key and the database of all paired devices. Without it, every device has to be paired again. The container is paused (docker pause, not stopped) for the few seconds of the copy, so the database file is consistent.
• Authentik and BookStack: the database alone is not enough. Both keep a secret key in their .env that signs sessions or encrypts data. Restore the dump together with the .env from the same night, or logins and encrypted fields stop working.
• The reverse proxy: Traefik’s acme.json holds all certificates and the Let’s Encrypt account key (see my Traefik article). Without it, Traefik requests every certificate again and can run into rate limits.
• Sites on a web host: the server logs in with an SSH key, streams the WordPress folder as a tar archive and dumps the database on the host. The host’s own backups are a bonus, not a plan.

8. The third copy: pulled, read-only, encrypted

The obvious way to make a second copy is to let the server push it somewhere. The problem: if the server is hacked or hit by ransomware, the attacker has the same access and can delete or encrypt that copy too. So my third copy works the other way round. A laptop pulls the newest snapshot of every job onto an encrypted USB SSD, and the server has no way to reach it:

Decision	Why
Pull instead of push	The server has no login on the laptop at all. A compromised server cannot touch the copy.
Read-only key on the server	A dedicated user may only run rrsync -ro on the backup folder. The laptop can read, never write or delete.
New dated snapshot per run, never --delete	If snapshots disappear on the server (by mistake or on purpose), the older copies on the SSD stay.
Same folder layout	The restore script works on the SSD unchanged.
LUKS encryption with a keyfile	Unlocked only during a run, closed afterwards. A lost SSD is unreadable.

The rrsync restriction is one line in the server’s authorized_keys:

command="sudo rrsync -ro /backup",restrict ssh-ed25519 AAAA… offsite-pull

The pull job on the laptop runs every hour but only does something when it makes sense: at most once a day, only on mains power, never while the server’s own backups run (23:15–04:30), and only when the SSD is plugged in and the server answers. It also checks what it copies:

• Spot checks: dumps not empty, gzip archives readable, the Vaultwarden database passes its integrity check, the certificate file isn’t empty.
• Ransomware alarm: before copying Nextcloud, a dry run counts how many files changed. If it’s more than 20 %, something is very wrong (that’s what mass encryption looks like). The copy is kept, but old generations are no longer deleted until I look at it.
• Longer history than on the server: the newest 7 copies, then one per week for 8 weeks and one per month for 12 months. Nextcloud, at about 800 GB, keeps 10 generations.

9. Knowing it worked

A backup that silently stopped three weeks ago is the most common way to lose data. So a small Python script on the server collects the state of every job every five minutes and writes it to a JSON file: last run, result, age of the newest snapshot, whether the timer is enabled, whether the backup disk is mounted.

• A small status page on the server shows it as a table.
• Home Assistant reads the same file and creates a sensor per job (ok, failed, stale, …). If any job has a problem for ten minutes, I get a push notification and a Telegram message. “Stale” means the newest snapshot is older than 26 hours, which also catches a timer that simply never ran.
• Coverage check: the script also lists every running container and compares it with a list of covered services. A new container without a backup job shows up as a warning. That closed the gap I worried about most: adding a service on a Sunday afternoon and forgetting its backup.

The coverage list comes with a house rule: every new container gets its backup job, its restore step and its coverage entry in the same change. “Later” is how backups get forgotten.

10. Restoring

First, the cheap way: most “I deleted something” moments are solved in the app itself. Nextcloud has deleted files and versions, Paperless a trash for 30 days. Only when that doesn’t help do I go to the snapshots, and a single file is just a copy out of the right date folder.

For the bad day, when the server is gone or several services are broken, there is an interactive restore script on the backup disk itself:

1. Choose jobs: one, several or all, in a fixed order: infrastructure first (reverse proxy, certificates), then single sign-on, then the databases and apps.
2. Choose the snapshot: the newest, one date for all, or per job.
3. Choose steps per job: files, database, containers.
4. Dry run: the script prints every command it would run, and only continues when I type RESTORE.

It never deletes anything: files it overwrites are moved into a .pre-restore-<time> folder, and every database is dumped before it is replaced. The backup timers are paused during the restore, so no backup job runs against a half-restored system.

A restore script you never ran is a hope, not a plan. Mine has a test bench: fake docker, systemctl and sshd commands and fake snapshots, so it can run every scenario without touching real data. It’s 211 checks, and it has to pass after every change to the restore script. It still doesn’t replace a real restore on a spare machine once in a while: that’s the only proof that counts.

What I learned

• Boring tools win. rsync, bash and systemd will still be there in ten years, and I can restore with nothing but a shell.
• Dump databases and check the dump. A copied database folder is a gamble; an unchecked dump is a surprise waiting to happen.
• Snapshots aren’t copies. Hardlinks give you history, not redundancy. You still need a second disk.
• Pull, don’t push, the copy that has to survive an attack. Whatever the server can reach, an attacker on the server can reach too.
• Monitor the backups, not just the services. The alert for “no new snapshot in 26 hours” is worth more than any feature of the backup itself.
• Secrets belong to the data. Keys and .env files from the same night as the database, or the restore looks fine and nothing works.