Incremental Backups with rsync

Linux

Mastering the rsync file synchronization tool can be immensely useful for any kind of local or remote file sync task from the command line or from scripts. This is not only because of the feature variety, but also because of its ubiquity. Most Linux distributions ship it by default and even macOS comes with an alternative implementation out-of-the-box (openrsync), that is fairly compatible (rsync is also available via homebrew).

I use it to publish this website and to create backups of my home directory onto an external hard drive.

My Backup Scripts

I wanted to backup my laptop’s /home/<username> directory in regular intervals, but I didn’t want to rely on specialized backup solutions that are not easily available from e.g. a USB live system, in case I need to restore from backup. For this purpose I have an external LUKS2-encrypted hard drive, formatted with XFS, that contains my backup scripts and backups.

In addition to rsync I only depend on GNU make to backup my files. On the backup drive I have subdirectories per computer, that include the Makefile and backup-excludes.txt files. The home backup itself lives in another username-specific subdirectory. I could certainly just use a shell script, but running make <something> is so much easier for me to remember, than anything else (make is actually a build tool and not a task runner).

So in order to create a new incremental backup I connect my USB drive, unlock it, mount it and then open my preferred terminal konsole to cd into the backup directory for the machine, that includes the Makefile. Then, it only takes a make backup command to perform the backup.

The Makefile is pretty short and mainly invokes rsync with the correct parameters.

RSYNC = /usr/bin/rsync

MAKEFILE_DIR := $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))
EXCLUDES_FILE = backup-excludes.txt

SOURCE_PATH = $(HOME)
DESTINATION_PATH = $(MAKEFILE_DIR)/

backup:
	$(RSYNC) \
		--archive \
		--compress \
		--recursive \
		--xattrs \
		--delete \
		--stats \
		--human-readable \
		--exclude-from="$(EXCLUDES_FILE)" \
		$(SOURCE_PATH) \
		$(DESTINATION_PATH)
.PHONY: backup
  • beware of the required tab indentation
  • MAKEFILE_DIR detects the absolute path to the containing directory; I wish there was a more elegant solution to achieve this
  • the --xattrs parameter is primarily useful on macOS and ensures that extended attributes of files, like tags, are fully backed up and not discarded
  • the --delete parameter deletes any file from the backup that does not exist on the machine; just be aware
  • the --progress parameter can optionally be added to get some visual progress
  • the .PHONY line marks the backup target as a phony target that does not produce an output file; this causes make to behave like a task runner instead of a build tool

The backup-excludes.txt file is very specific for the computer, because it contains all the directories in my home directory, that should be skipped from backups, e.g. temporary cache files. I exclude browser caches, my Steam library, my Downloads and other redundant directories, that only bloat up the backup.

.cache
.var/app/io.gitlab.librewolf-community/cache
.var/app/com.heroicgameslauncher.hgl/cache
.var/app/com.vivaldi.Vivaldi/cache
.var/app/dev.zed.Zed/data/zed/node
.local/share/Trash
.local/share/umu
Downloads
Games/Steam

After make backup concludes, a human-readable summary will be presented:

Number of files: 513.615 (reg: 445.674, dir: 43.132, link: 24.806, special: 3)
Number of created files: 31.529 (reg: 24.279, dir: 3.931, link: 3.319)
Number of deleted files: 30.380 (reg: 23.223, dir: 3.856, link: 3.300, special: 1)
Number of regular files transferred: 27.938
Total file size: 459,61G bytes
Total transferred file size: 10,64G bytes
Literal data: 10,64G bytes
Number of 4 KiB logical blocks touched: 2.613.216
Matched data: 0 bytes
File list size: 5,24M
File list generation time: 0,001 seconds
File list transfer time: 0,000 seconds
Total bytes sent: 5,50G
Total bytes received: 5,67M

sent 5,50G bytes  received 5,67M bytes  8,21M bytes/sec
total size is 459,61G  speedup is 83,44

Testing and Restoring Backups

A backup is only useful when you know how to restore it. I validate this from time to time when I provision a freshly-wiped secondary computer with the restored backup of my primary machine. In those cases I skip files and backup contents that are not relevant for the new machine. This is why I don’t have a global make restore target yet. If I ever need it, I will update this article.

Offsite Backups

I don’t rely exclusively on my backup drive. I also have another encrypted drive that is stored offsite. But this one is only refreshed on a quarterly basis. The Makefile setup is similar. Important projects and documents are redundantly stored on git servers and in cloud drives.

At the moment I feel good with this backup strategy, but these things tend to evolve and change over time. Maybe one day I will look into borg backup (resistance might be futile).