|
|
Simple Snapshot-style backups using an rsync server |
This document describes how to make rotating snapshots of arbitrary files and directories using an rsync server (rsync running in daemon mode). The number of kept snapshots is easily configurable by defining a "lifespan". The proposed method makes sure, that no snapshot can be overwritten and uses hardlinks to minimize disk space usage on the server side. Enhanced security is achieved by the fact, that no passwords have to be advertized and access is restrictred to specific clients. Key points are:
The main motivation and starting point for this has been Mike Rubel's "classical" Easy Automated Snapshot-Style Backups with Linux and Rsync (thanks for that, Mike). However I had some problems with it:
The pieces of the puzzle are:
The server is started with rsync --daemon and is configured by /etc/rsyncd.conf. This is an anonymized version of my /etc/rsyncd.conf:
[bu.client1] comment = My Backup path = /local/backup/client1 numeric ids = false log file = /local/backup/client1/rsyncd.log list = false uid = root read only = false clients allow = client1.domain.com pre-xfer exec = /local/backup/client1/pre-xfer.sh post-xfer exec = /local/backup/client1/post-xfer.sh use chroot = false
rsync offers the possibility to define two scripts that are executed before and after the transfer (pre-xfer exec and post-xfer exec in /etc/rsyncd.conf). If the pre-xfer script doesn't exit with 0, rsync is aborted. The post-xfer script is executed after the transfer (even if pre-xfer fails!).
I use the pre-xfer script to create a sane environment for the file transfer and make sure, no valid data is overwritten. The post-xfer script is used to remove snapshots that have reached their lifespan and to rotate the other directories. Both scripts share a common settings file xfer.ini.
xfer.ini contains shared settings for pre-xfer and post-xfer scripts and include a small function to write information to a custom logfile:
#
# settings for pre/post-xfer.sh
#
# ------- V A R I A B L E S -------
# current timestamp
NOW=`date +"%F, %X"` # e.g. "2010-01-05, 12:58:25 PM"
# various pathes, files and filenames
BASE=/local/backup/client1 # base directory
LOG=$BASE/backup.log # my own logfile
TIMESTAMP_FILE=TIMESTAMP # name (!) of timestamp files
TIMESTAMP_LIST=${BASE}/TIMESTAMPS # full path (!) to timestamp list
# how long to keep snapshots (in seconds)
MAXKEEP=1209600 # two weeks = 60 * 60 * 24 * 14
# ------- F U N C T I O N S -------
WriteLog () {
/usr/bin/echo `date +"%F, %X"`: "$1" >> ${LOG}
}
I used the pre-xfer exec script to create a sane environment for the synchronization:
#! /bin/ksh
#
# pre-xfer script for module bu.client1
#
# Author: Frank Thommen, http://www.drosera.ch/frank/
# License/Copyright: None, free to use
# Warranty: None :-)
#
# Please retain the originator informations when using or distributing this script
#
. ./xfer.ini
BACKUP0=${BASE}/backup.0
BACKUP=${BASE}/backup
#
# Create target directory it it doesn't exist yet
#
if [ ! -d ${BACKUP} ]; then
mkdir ${BACKUP}
fi
#
#
# Create link-dest directory if it doesn't exist yet
#
if [ ! -d ${BACKUP0} ]; then
mkdir ${BACKUP0}
fi
#
# Make sure the destination directory is pristine
#
if [ ! -z "`ls -A ${BACKUP}`" ]; then
WriteLog "**** ERROR: Backup aborted because ${BACKUP} is not empty ****"
exit 1
fi
#
# Allow rsync only in the predefined module/path
#
if [ ${RSYNC_REQUEST} != "bu.client1/backup" ]; then
WriteLog "**** ERROR: Backup aborted because of wrong rsync request (${RSYNC_REQUEST}) ****"
exit 1
fi
#
# write start timestamp
#
echo "START: ${NOW}" > ${BACKUP}/${TIMESTAMP_FILE}
WriteLog "---- Backup Started ----"
The post-xfer script does cleanup, rotation and keeps track of the backup dates of all currently present snapshots in a file TIMESTAMPS:
#! /bin/ksh
#
# post-xfer.sh fuer Module bu.radagast
#
. ./xfer.ini
if [ $RSYNC_EXIT_STATUS -ne 0 ]
then
WriteLog "---- Stopping Backup due to failure ----"
exit 1
fi
MAX=$(ls -d $BASE/backup.*(\d) | grep -v log | cut -d. -f2 | sort -n | tail -1)
DIRS=$(ls -d $BASE/backup.*(\d) | grep -v log | cut -d. -f2 | sort -nr)
NOWS=`date +"%s"` # current time in seconds since epoch
#
# Clean up and rotate directories and recreate the first one
#
for DIR in $DIRS
do
if [ $(($NOWS - `stat --printf="%Y" backup.${DIR}`)) -gt $MAXKEEP ]
then
rm -rf backup.${DIR}
WriteLog "${NOW}: Removed backup.${DIR} since it is older than MAXKEEP"
else
mv backup.${DIR} backup.$((DIR+1))
fi
done
# Create correct start conditions
mv backup backup.0
echo "STOP: $NOW" >> backup.0/${TIMESTAMP_FILE}
mkdir backup
#
# Rewrite the list of timestamps from the currently present snapshots
# in chronological order
#
DIRS2=$(ls -d $BASE/backup.*(\d) | grep -v log | cut -d. -f2 | sort -n)
cp /dev/null ${TIMESTAMP_LIST} # start from an empty list
for DIR in $DIRS2
do
(
/usr/bin/echo "backup.${DIR}: \c";
(grep START backup.${DIR}/${TIMESTAMP_FILE} | cut -d" " -f2-)
) >> ${TIMESTAMP_LIST}
done
echo "" >> ${TIMESTAMP_LIST} # final end-of-line
WriteLog "---- Backup Finished ----"
The core of the backup script finally looks like:
rsync -avR --link-dest=/backup.0 \
...whatever I want to back up ... \
my.backup.server::bu.my_hostname/backup