Thursday, February 19, 2009

The Backup Tool

In my previous blog entry I wrote an overview of an in-house backup solution which seems to be a good enough replacement for over 90% of backups currently done by Netbackup in our environment. I promised to show some examples on how it actually works. I can't give you output from a live system so will show some examples from a test one. Let's go thru couple of examples then.

Please keep in mind that it is still more like a working prototype than a finished product (and it will most certainly stay that way to some extend).


To list all backups (I run this on an empty system)
# backup -l
CLIENT NAME REFER USED RATIO RETENTION

Let's run a backup for a client mk-archive-1.test

# backup -c mk-archive-1.test
Creating new file system archive-2/backup/mk-archive-1.test
Using generic rules file: /archive-2/conf/standard-os.rsync.rules
Using client rules file: /archive-2/conf/mk-archive-1.test.rsync.rules
Starting rsync
Creating snapshot archive-2/backup/mk-archive-1.test@rsync-2009-02-19_15:11--2009-02-19_15:14
Log file: /archive-2/logs/mk-archive-1.test.rsync.2009-02-19_15:11--2009-02-19_15:14
#

Above you can see that it uses to config files - one is a global file describing includes/excludes which are run for all clients and the second file which describes an include/exclude file for that specific client. In many cases you don't need to create that file - the tool will create an empty one for you.

Let's list all our backups then.

# backup -lv
CLIENT NAME REFER USED RATIO RETENTION
mk-archive-1.test 1.15G 1.15G 1.75x 35 (global)
mk-archive-1.test@rsync-2009-02-19_15:11--2009-02-19_15:14 1.15G 0 1.75x
#

The snapshot definies a backup and I put the start and end date of the backup in its name.

If you want to schedule a backup from a cron you do not need any verbose output - there is an option "-q" which keeps the tool quiet.

# backup -q -c mk-archive-1.test
#
# backup -lv
CLIENT NAME REFER USED RATIO RETENTION
mk-archive-1.test 1.15G 1.16G 1.75x 35 (global)
mk-archive-1.test@rsync-2009-02-19_15:11--2009-02-19_15:14 1.15G 6.63M 1.75x
mk-archive-1.test@rsync-2009-02-19_15:16--2009-02-19_15:16 1.15G 0 1.75x
#


Now lets change the retention policy for the client to 15 days.

# backup -c mk-archive-1.test -e 15
#
# backup -lv
CLIENT NAME REFER USED RATIO RETENTION
mk-archive-1.uk.test 1.15G 1.16G 1.75x 15 (local)
mk-archive-1.uk.test@rsync-2009-02-19_15:11--2009-02-19_15:14 1.15G 6.63M 1.75x
mk-archive-1.uk.test@rsync-2009-02-19_15:16--2009-02-19_15:16 1.15G 0 1.75x
#


To start an expiry process of old backups (not that there is something to expire on this empty system...):

# backup -E

Expiry started on : 2009-02-19_17:21
Expiry finished on : 2009-02-19_17:21
Global retention policy : 35
Total number of deleted backups : 0
Total number of preserved backups : 0
Log file : /archive-2/logs/retention_2009-02-19_17:21--2009-02-19_17:21

You can also expire all backups or for a specific client according to a global and a client specific retention policies, you can generate reports, list all currently active backups, etc. The current usage information for the tool looks like:


# backup -h

usage: backup {-c client_name} [-r rsync_destination] [-hvq]
backup [-lvF]
backup [-Lv]
backup {-R date} [-v]
backup {-E} [-v] [-n] [-c client_name]
backup {-e days} {-c client_name}
backup {-D backup_name} [-f]
backup {-A} {-c client_name} [-n] [-f] [-ff]

This script starts remote client backup using rsync.

OPTIONS:
-h Show this message
-r Rsync destination. If not specified then it will become Client_name/ALL/
-c Client name (FQDN)
-v Verbose
-q Quiet (no output)
-l list all clients in a backup
-v will also include all backups for each client
-vF will list only backups which are marked as FAILED
-e sets a retention policy for a client
if number of days is zero then client retention policy is set to global
if client_name is "global" then set a global retention policy
-L list all running backups
-v more verbose output
-vv even more verbose output
-R Show report for backups from a specified date ("today" "yesterday" are also allowed)
-v list failed backups
-vv list failed and successful backups
-E expire (delete) backups according to a retention policy
-c client_name expires backup only for specified client
-v more verbose output
-n simulate only - do not delete anything
-D deletes specified backup
-f forces deletion of a backup - this is required to delete a backup if
there are no more successful backups for the client
-A archive specified client - only one backup is allowed in order to achive client
-c client_name - valid client name, this option is mandatory
-n simulate only - do not archive anything
-f deletes all backup for a client except most recent one and archives client
-ff archives a client along with all backups
-I Initializes file systems within a pool (currently: archive-1)

EXAMPLES:

BACKUP
In order to immediatelly start a backup for a given client:

backup -c XXX.yyy.zz
backup -r XXX.yyy.zz/ALL/ -c XXX.yyy.zz

Above two commands are doing exactly the same - the first version is preffered.
The 2nd version is useful when doing backups over ssh tunnel or via a dedicated backup interface
when it is required to connect to different address that a client name. For example, in order
to start a backup for a client XXX.yyy.zz t via ssh tunnel at localhost:5001 issue:

backup -r localhost:5001/ALL/ -c XXX.yyy.zz

RETENTION POLICY

backup -E - expire backups according to retention policy
backup -e 30 -c global - sets global retention policy to 30 days
backup -l - list all clients in backup including their retention policy



6 comments:

gagendel said...

Reminds me of dirvish, which I've been happily backing up all my Linux/Solaris systems with for years. For Windows, I use rsync.net backup agent. For Mac I use time machine. All to my OpenSolaris server ZFS Raid-Z array. I couldn't be happier. It's saved me more times than I can remember.

Kim Nørgaard said...

This looks very interesting.

Is the source code something you would like to share with the community so that others can benefit from it?

milek said...

Yes, I would like to do it but I need first get a green light from my employer.

Anonymous said...

It's been awhile, so I'm assuming your employer doesn't want you to release the code? :\

milek said...

Unfortunately I change employers in the mean time... I will check with them if it would be possible to open source it.

Chris Nagele said...

I would love to use this as well. Please keep us posted if you get permission to open source it.