royong
10-11-2007, 12:20
Background information ::
- File based backup required for a Linux machine
- 7 days versioning a.k.a rotating backups required
- Backup hard disk mounted on the Linux machine as /backup
- Rsync seems to be a efficient tool for this purpose.
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
http://samba.anu.edu.au/ftp/rsync/rsync.html
(a) Prep Work
# mkdir /backup/daily
# mkdir /backup/daily/backup.0
# mkdir /backup/daily/backup.1
# mkdir /backup/daily/backup.2
# mkdir /backup/daily/backup.3
# mkdir /backup/daily/backup.4
# mkdir /backup/daily/backup.5
# mkdir /backup/daily/backup.6
# mkdir /backup/daily/backup.7
# mkdir /usr/local/bin/rsync-dailylocalbackup
# touch /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
# chmod +x /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
# touch /var/log/rsync-dailylocalbackup.log.0
# touch /var/log/rsync-dailylocalbackup.log.1
# touch /var/log/rsync-dailylocalbackup.log.2
# touch /var/log/rsync-dailylocalbackup.log.3
# touch /var/log/rsync-dailylocalbackup.log.4
# touch /var/log/rsync-dailylocalbackup.log.5
# touch /var/log/rsync-dailylocalbackup.log.6
(a) The Main Script
# cat > /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh <<"EOF"
#!/bin/bash
# /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
# --------------------------------------------------------------------
# Obivously needs constant work to make it better but the basic idea
# is it makes rotating backup-snapshots of / whenever called except
# directories, folders and files specified on the EXLUCDES file
# --------------------------------------------------------------------
# Captures the results in a log file
# Emails the log file to root@localhost
# --------------------------------------------------------------------
ID=/usr/bin/id;
ECHO=/bin/echo;
MOUNT=/bin/mount;
RM=/bin/rm;
MV=my_mv;
CP=/bin/cp;
TOUCH=/bin/touch;
RSYNC=/usr/bin/rsync;
function my_mv() {
REF=/tmp/rsyncsnapshot-mymv-$$;
touch -r $1 $REF;
/bin/mv $1 $2;
touch -r $REF $2;
/bin/rm $REF;
}
# ------------- file locations ---------------------------------------
SNAPSHOT_RW=/backup;
EXCLUDES=/usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup-exclude;
# ------------- the script itself ------------------------------------
# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root. Exiting..."; exit; } fi
#rotating log files
if [ -f /var/log/rsync-dailylocalbackup.log.7 ] ; then rm -fr /var/log/rsync-dailylocalbackup.log.7 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.6 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.6 /var/log/rsync-dailylocalbackup.log.7 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.5 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.5 /var/log/rsync-dailylocalbackup.log.6 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.4 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.4 /var/log/rsync-dailylocalbackup.log.5 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.3 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.3 /var/log/rsync-dailylocalbackup.log.4 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.2 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.2 /var/log/rsync-dailylocalbackup.log.3 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.1 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.1 /var/log/rsync-dailylocalbackup.log.2 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.0 ] ; then cp -a /var/log/rsync-dailylocalbackup.log.0 /var/log/rsync-dailylocalbackup.log.1 ; fi;
echo "" > /var/log/rsync-dailylocalbackup.log.0
#rotating snapshots
if [ -d $SNAPSHOT_RW/daily/backup.7 ] ; then $RM -rf $SNAPSHOT_RW/daily/backup.7;fi ;
if [ -d $SNAPSHOT_RW/daily/backup.6 ] ; then $MV $SNAPSHOT_RW/daily/backup.6 $SNAPSHOT_RW/daily/backup.7 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.5 ] ; then $MV $SNAPSHOT_RW/daily/backup.5 $SNAPSHOT_RW/daily/backup.6 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.4 ] ; then $MV $SNAPSHOT_RW/daily/backup.4 $SNAPSHOT_RW/daily/backup.5 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.3 ] ; then $MV $SNAPSHOT_RW/daily/backup.3 $SNAPSHOT_RW/daily/backup.4 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.2 ] ; then $MV $SNAPSHOT_RW/daily/backup.2 $SNAPSHOT_RW/daily/backup.3 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.1 ] ; then $MV $SNAPSHOT_RW/daily/backup.1 $SNAPSHOT_RW/daily/backup.2 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.0 ] ; then $CP -al $SNAPSHOT_RW/daily/backup.0 $SNAPSHOT_RW/daily/backup.1 ; fi;
# step 4: rsync from the system into the latest snapshot
# (notice that rsync behaves like cp --remove-destination by default,
# so the destination is unlinked first.
# If it were not so, this would copy over the other snapshot(s) too!
echo "RSYNC Daily Local Backup STARTED on `date`" >> /var/log/rsync-dailylocalbackup.log.0
$RSYNC -va --delete --delete-excluded --exclude-from="$EXCLUDES" // $SNAPSHOT_RW/daily/backup.0 >> /var/log/rsync-dailylocalbackup.log.0
# step 5: update the mtime of backup.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/daily/backup.0
# and thats it.
echo "RSYNC Daily Local Backup COMPLETED on `date`" >> /var/log/rsync-dailylocalbackup.log.0
# some customization before we finalize the email
echo "Current Disk Space Usage" >> /var/log/rsync-dailylocalbackup.log.0
df -Th >> /var/log/rsync-dailylocalbackup.log.0
# E-mail log file
cat /var/log/rsync-dailylocalbackup.log.0 | mail -s "RSYNC Daily Local Backup Status for `hostname`" root@localhost
EOF
(c) Some directories that we chose to exclude
# cat > /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup-exclude <<"EOF"
#!/bin/bash
# --------------------------------------------------------------------
# Part of a script located at the following:
# /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
# --------------------------------------------------------------------
# Directories, folders and files specified thereafter will be
# excluded from the Rotating Filesystem Backup
# --------------------------------------------------------------------
/backup
/backup/
/dev
/dev/
/lost+found
/lost+found/
/media
/media/
/mnt
/mnt/
/proc
/proc/
/sys
/sys/
EOF
(d) Adding the job to the system crontab
# echo "05 08 * * * root /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh" >> /etc/crontab
(e) Running the backup job for the 1st time
# /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
PS - The above script has been amended from the original for use. Certain tweaks and changes have been made. I used to have the URL link to the original site and coder but lost it during a machine migration. I will try and get it back and properly credit the original site and coder.
- File based backup required for a Linux machine
- 7 days versioning a.k.a rotating backups required
- Backup hard disk mounted on the Linux machine as /backup
- Rsync seems to be a efficient tool for this purpose.
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
http://samba.anu.edu.au/ftp/rsync/rsync.html
(a) Prep Work
# mkdir /backup/daily
# mkdir /backup/daily/backup.0
# mkdir /backup/daily/backup.1
# mkdir /backup/daily/backup.2
# mkdir /backup/daily/backup.3
# mkdir /backup/daily/backup.4
# mkdir /backup/daily/backup.5
# mkdir /backup/daily/backup.6
# mkdir /backup/daily/backup.7
# mkdir /usr/local/bin/rsync-dailylocalbackup
# touch /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
# chmod +x /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
# touch /var/log/rsync-dailylocalbackup.log.0
# touch /var/log/rsync-dailylocalbackup.log.1
# touch /var/log/rsync-dailylocalbackup.log.2
# touch /var/log/rsync-dailylocalbackup.log.3
# touch /var/log/rsync-dailylocalbackup.log.4
# touch /var/log/rsync-dailylocalbackup.log.5
# touch /var/log/rsync-dailylocalbackup.log.6
(a) The Main Script
# cat > /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh <<"EOF"
#!/bin/bash
# /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
# --------------------------------------------------------------------
# Obivously needs constant work to make it better but the basic idea
# is it makes rotating backup-snapshots of / whenever called except
# directories, folders and files specified on the EXLUCDES file
# --------------------------------------------------------------------
# Captures the results in a log file
# Emails the log file to root@localhost
# --------------------------------------------------------------------
ID=/usr/bin/id;
ECHO=/bin/echo;
MOUNT=/bin/mount;
RM=/bin/rm;
MV=my_mv;
CP=/bin/cp;
TOUCH=/bin/touch;
RSYNC=/usr/bin/rsync;
function my_mv() {
REF=/tmp/rsyncsnapshot-mymv-$$;
touch -r $1 $REF;
/bin/mv $1 $2;
touch -r $REF $2;
/bin/rm $REF;
}
# ------------- file locations ---------------------------------------
SNAPSHOT_RW=/backup;
EXCLUDES=/usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup-exclude;
# ------------- the script itself ------------------------------------
# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root. Exiting..."; exit; } fi
#rotating log files
if [ -f /var/log/rsync-dailylocalbackup.log.7 ] ; then rm -fr /var/log/rsync-dailylocalbackup.log.7 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.6 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.6 /var/log/rsync-dailylocalbackup.log.7 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.5 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.5 /var/log/rsync-dailylocalbackup.log.6 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.4 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.4 /var/log/rsync-dailylocalbackup.log.5 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.3 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.3 /var/log/rsync-dailylocalbackup.log.4 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.2 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.2 /var/log/rsync-dailylocalbackup.log.3 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.1 ] ; then mv -f /var/log/rsync-dailylocalbackup.log.1 /var/log/rsync-dailylocalbackup.log.2 ; fi;
if [ -f /var/log/rsync-dailylocalbackup.log.0 ] ; then cp -a /var/log/rsync-dailylocalbackup.log.0 /var/log/rsync-dailylocalbackup.log.1 ; fi;
echo "" > /var/log/rsync-dailylocalbackup.log.0
#rotating snapshots
if [ -d $SNAPSHOT_RW/daily/backup.7 ] ; then $RM -rf $SNAPSHOT_RW/daily/backup.7;fi ;
if [ -d $SNAPSHOT_RW/daily/backup.6 ] ; then $MV $SNAPSHOT_RW/daily/backup.6 $SNAPSHOT_RW/daily/backup.7 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.5 ] ; then $MV $SNAPSHOT_RW/daily/backup.5 $SNAPSHOT_RW/daily/backup.6 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.4 ] ; then $MV $SNAPSHOT_RW/daily/backup.4 $SNAPSHOT_RW/daily/backup.5 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.3 ] ; then $MV $SNAPSHOT_RW/daily/backup.3 $SNAPSHOT_RW/daily/backup.4 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.2 ] ; then $MV $SNAPSHOT_RW/daily/backup.2 $SNAPSHOT_RW/daily/backup.3 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.1 ] ; then $MV $SNAPSHOT_RW/daily/backup.1 $SNAPSHOT_RW/daily/backup.2 ; fi;
if [ -d $SNAPSHOT_RW/daily/backup.0 ] ; then $CP -al $SNAPSHOT_RW/daily/backup.0 $SNAPSHOT_RW/daily/backup.1 ; fi;
# step 4: rsync from the system into the latest snapshot
# (notice that rsync behaves like cp --remove-destination by default,
# so the destination is unlinked first.
# If it were not so, this would copy over the other snapshot(s) too!
echo "RSYNC Daily Local Backup STARTED on `date`" >> /var/log/rsync-dailylocalbackup.log.0
$RSYNC -va --delete --delete-excluded --exclude-from="$EXCLUDES" // $SNAPSHOT_RW/daily/backup.0 >> /var/log/rsync-dailylocalbackup.log.0
# step 5: update the mtime of backup.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/daily/backup.0
# and thats it.
echo "RSYNC Daily Local Backup COMPLETED on `date`" >> /var/log/rsync-dailylocalbackup.log.0
# some customization before we finalize the email
echo "Current Disk Space Usage" >> /var/log/rsync-dailylocalbackup.log.0
df -Th >> /var/log/rsync-dailylocalbackup.log.0
# E-mail log file
cat /var/log/rsync-dailylocalbackup.log.0 | mail -s "RSYNC Daily Local Backup Status for `hostname`" root@localhost
EOF
(c) Some directories that we chose to exclude
# cat > /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup-exclude <<"EOF"
#!/bin/bash
# --------------------------------------------------------------------
# Part of a script located at the following:
# /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
# --------------------------------------------------------------------
# Directories, folders and files specified thereafter will be
# excluded from the Rotating Filesystem Backup
# --------------------------------------------------------------------
/backup
/backup/
/dev
/dev/
/lost+found
/lost+found/
/media
/media/
/mnt
/mnt/
/proc
/proc/
/sys
/sys/
EOF
(d) Adding the job to the system crontab
# echo "05 08 * * * root /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh" >> /etc/crontab
(e) Running the backup job for the 1st time
# /usr/local/bin/rsync-dailylocalbackup/rsync-dailylocalbackup.sh
PS - The above script has been amended from the original for use. Certain tweaks and changes have been made. I used to have the URL link to the original site and coder but lost it during a machine migration. I will try and get it back and properly credit the original site and coder.