drosera.ch Logo

Simple Snapshot-style backups using an rsync server

Table of content:

  1. Concept
  2. Motivation and Acknowledgments
  3. Components and Configuration
    1. rsync server
    2. pre and post transfer scripts
    3. And finally the backup script

Concept (^)

This document describes how to make rotating snapshots of arbitrary files and directories using an rsync server (rsync running in daemon mode). The number of kept snapshots is easily configurable by defining a "lifespan". The proposed method makes sure, that no snapshot can be overwritten and uses hardlinks to minimize disk space usage on the server side. Enhanced security is achieved by the fact, that no passwords have to be advertized and access is restrictred to specific clients. Key points are:

Motivation and Acknowledgments (^)

The main motivation and starting point for this has been Mike Rubel's "classical" Easy Automated Snapshot-Style Backups with Linux and Rsync (thanks for that, Mike). However I had some problems with it:

  1. I require, that backup client (the machines to be backed up) and the backup server are different machines and are as separated as possible. Therefore I didn't like the idea of fiddling around with ssh keys to ensure a password-free rsync operation. Once someone would get root access to the rsync a simple misconfiguration on the backup server could allow a security breach.
  2. The mount/remount/umount mechanisms proposed in Mike Rubel's article seemed an unnecessary overhead and too to error prone to me.
  3. I wanted to be flexible regarding the number and intervals of the snapshots and have the possibility of intermediate snapshots between two regular backups. This is not possible with the original method, as invoking the script between regular runs induces the rotation and removes older but still needed backups.
  4. Last but not least my backup server runs opensolaris. root in opensolaris is not a user but an RBAC role. SSHing (respectievely rsyncing) as root to such a host is not easily possible w/o interfering with the RBAC configuration, which I didn't want to do. Therefore I needed an SSH-less method for rsync. The server runs several services anyway, so why not running one more: rsync --daemon?

Components and Configuration (^)

The pieces of the puzzle are:

  1. an rsync server accepting only requests from specified host
  2. rsync pre-xfer and post-xfer scripts that ensure correct boundary conditions
  3. a simple backup script on the client side which is run by cron or manually

rsync server

The server is started with rsync --daemon and is configured by /etc/rsyncd.conf. This is an anonymized version of my /etc/rsyncd.conf:

comment        = My Backup
path           = /local/backup/client1
numeric ids    = false
log file       = /local/backup/client1/rsyncd.log
list           = false
uid            = root
read only      = false
clients allow  = client1.domain.com
pre-xfer exec  = /local/backup/client1/pre-xfer.sh
post-xfer exec = /local/backup/client1/post-xfer.sh
use chroot     = false
Caveat 1:
I found the numeric ids option on a webpage that I don't remember any more. However this option is not mentioned in the rsync manpage. You should probably rather define it in the backup script
Caveat 2:
On my opensolaris (2009.06 snv_111b X86) box, the provided rsync (rsync SUNWrsync@2.6.9-0.111) didn't work at all with these settings. The pre-xfer exec script apparently always returned failure. I had to manually compile and install rsync 3.0.6.

pre and post transfer scripts

rsync offers the possibility to define two scripts that are executed before and after the transfer (pre-xfer exec and post-xfer exec in /etc/rsyncd.conf). If the pre-xfer script doesn't exit with 0, rsync is aborted. The post-xfer script is executed after the transfer (even if pre-xfer fails!).

I use the pre-xfer script to create a sane environment for the file transfer and make sure, no valid data is overwritten. The post-xfer script is used to remove snapshots that have reached their lifespan and to rotate the other directories. Both scripts share a common settings file xfer.ini.


xfer.ini contains shared settings for pre-xfer and post-xfer scripts and include a small function to write information to a custom logfile:

# settings for pre/post-xfer.sh

# ------- V A R I A B L E S -------

# current timestamp
NOW=`date +"%F, %X"`               # e.g. "2010-01-05, 12:58:25 PM"

# various pathes, files and filenames
BASE=/local/backup/client1         # base directory
LOG=$BASE/backup.log               # my own logfile
TIMESTAMP_FILE=TIMESTAMP           # name (!) of timestamp files
TIMESTAMP_LIST=${BASE}/TIMESTAMPS  # full path (!) to timestamp list

# how long to keep snapshots (in seconds)
MAXKEEP=1209600                    # two weeks = 60 * 60 * 24 * 14

# ------- F U N C T I O N S -------

WriteLog () {
  /usr/bin/echo `date +"%F, %X"`: "$1" >> ${LOG}

pre-xfer script

I used the pre-xfer exec script to create a sane environment for the synchronization:

#! /bin/ksh

# pre-xfer script for module bu.client1
#  Author: Frank Thommen, http://www.drosera.ch/frank/
#  License/Copyright: None, free to use
#  Warranty: None :-)
# Please retain the originator informations when using or distributing this script

. ./xfer.ini


# Create target directory it it doesn't exist yet
if [ ! -d ${BACKUP} ]; then
  mkdir ${BACKUP}

# Create link-dest directory if it doesn't exist yet
if [ ! -d ${BACKUP0} ]; then
  mkdir ${BACKUP0}

# Make sure the destination directory is pristine
if [ ! -z "`ls -A ${BACKUP}`" ]; then
  WriteLog "**** ERROR: Backup aborted because ${BACKUP} is not empty ****"
  exit 1

# Allow rsync only in the predefined module/path
if [ ${RSYNC_REQUEST} != "bu.client1/backup" ]; then
  WriteLog "**** ERROR: Backup aborted because of wrong rsync request (${RSYNC_REQUEST}) ****"
  exit 1

# write start timestamp
WriteLog "---- Backup Started ----"

post-xfer script

The post-xfer script does cleanup, rotation and keeps track of the backup dates of all currently present snapshots in a file TIMESTAMPS:

  1. Check the age of each snapshot and remove all snapshots that have reached their lifespan ($MAXKEEP).
  2. Rotate the remaining snapshots one up
  3. Recreate a base snapshot directory backup
  4. Recreate TIMESTAMPS with the timestamps of all currently present snapshots
#! /bin/ksh

#  post-xfer.sh fuer Module bu.radagast

. ./xfer.ini

if [ $RSYNC_EXIT_STATUS -ne 0 ]
  WriteLog "---- Stopping Backup due to failure ----"
  exit 1

MAX=$(ls -d $BASE/backup.*(\d) | grep -v log | cut -d. -f2 | sort -n | tail -1)
DIRS=$(ls -d $BASE/backup.*(\d) | grep -v log | cut -d. -f2 | sort -nr)
NOWS=`date +"%s"`  # current time in seconds since epoch

# Clean up and rotate directories and recreate the first one
for DIR in $DIRS
  if [ $(($NOWS - `stat --printf="%Y" backup.${DIR}`)) -gt $MAXKEEP ]
    rm -rf backup.${DIR}
    WriteLog "${NOW}: Removed backup.${DIR} since it is older than MAXKEEP"
    mv backup.${DIR} backup.$((DIR+1))

# Create correct start conditions

mv backup backup.0
echo "STOP: $NOW" >> backup.0/${TIMESTAMP_FILE}
mkdir backup

# Rewrite the list of timestamps from the currently present snapshots
# in chronological order
DIRS2=$(ls -d $BASE/backup.*(\d) | grep -v log | cut -d. -f2 | sort -n)
cp /dev/null ${TIMESTAMP_LIST}     # start from an empty list
for DIR in $DIRS2
    /usr/bin/echo "backup.${DIR}: \c";
   (grep START backup.${DIR}/${TIMESTAMP_FILE} | cut -d" " -f2-)
echo "" >> ${TIMESTAMP_LIST}            # final end-of-line

WriteLog "---- Backup Finished ----"

And finally the backup script

The core of the backup script finally looks like:

rsync -avR --link-dest=/backup.0                    \
      ...whatever I want to back up ...             \

Contact me via the webform
Last Update: 09-NOV-2015, ft