dobackup.pl
===========

Legal:

  Copyright 1997-2004 Webcon, Inc.

  This code is available for public use under the terms of the GNU Public
  License (GPL). See http://www.gnu.org/copyleft/gpl.html for license
  details. 

  This software is OSI Certified Open Source Software.
  OSI Certified is a certification mark of the Open Source Initiative.

  Webcon reserves the right to modify the licensing terms for any future
  version of this product.

Contact:

  website: http://www.webcon.ca/opensource/dobackup/
  email:   dobackup@webcon.ca

Quick overview:

  A flexible Perl script to handle unattended incremental backups of
  multiple servers. Handles multiple media sets with automatic media
  preparation and rotation, configurable 'what-to-backup', global exclusion
  patterns, user settable 'don't-backup-this-directory' metafiles. Design
  goal: zero-maintenance, nothing to do except change the media when told.

Getting started:

  Media preparation:
  ------------------

  Each unit of media used for dobackup must be uniquely identifiable. To
  this end, dobackup looks for a file named "MEDIAUNIT=<tag>" on the disk
  (in the mount-point really) before starting the backup. The reasons are
  two-fold: 1) to ensure that a media unit is really intended for backups,
  so you don't wipe out something important by accident, and 2) to verify
  that proper rotation of media units has occurred.

  To prepare each unit, format and mount it as usual, then:

  # touch MEDIAUNIT=<tag>
  # chmod 000 MEDIAUNIT=<tag>

  where <tag> is equal to the unit name, e.g. "JAZ01".

  If you have 4 JAZ cartridges, use names like JAZ01, JAZ02, JAZ03, JAZ04,
  then put each of these names in /etc/dobackup.conf, in the appropriate
  section. The names are really arbitrary. You could call the units "Larry",
  "Curly", and "Moe", if you wanted to, but they must be listed in the
  config file in the order in which they will be rotated. For example:

  [MediaSet=JazSetOne]
  Units= <<EOU
  JAZ01
  JAZ02
  JAZ03
  JAZ04
  EOU


  Scheduling:
  -----------

  Setup dobackup to run in root's crontab (/etc/crontab):

  Example 1: Backups to a permanent drive, daily incremental with a full
             backup on the 1st day of each month.

  #Daily Backups
  #(1:15am each day, full on the 1st of the month)
  15 1 1    * * root /usr/bin/perl /usr/sbin/dobackup.pl --fullbackup
  15 1 2-31 * * root /usr/bin/perl /usr/sbin/dobackup.pl

  Example 2: Backups to removable media, daily incremental with weekly media
             rotation. Full backup at the start of each week. Unmount device
             after each backup for extra security.

  #Daily Backups
  #(1:15am each day, eject Monday morning, ready for full backup Monday night)
  15 1 * * 0,3-6 root /usr/bin/perl /usr/sbin/dobackup.pl; umount /jaz
  15 1 * * 1     root /usr/bin/perl /usr/sbin/dobackup.pl --rotate
  15 1 * * 2     root /usr/bin/perl /usr/sbin/dobackup.pl --fullbackup; umount /jaz

Syntax:

  -h, --help       Prints short usage instructions

  -d=INT           Enabled debugging mode. INT = verbosity level.
  --debug=INT      This adds lots of diagnostic output from many parts of
                   the program. Use this to track unexplained failures to a
                   specific part of the code.

  -v, --version    Prints version information

  --test           Part of the process of a normal backup is to verify that
                   all the pre-flight checks are good before beginning a
                   backup.  This includes such things as ensuring all
                   external programs are available, backup media is properly
                   configured, hosts to backup are alive, etc..

                   This option tells the program to exit after performing
                   the pre-flight checks, and do NOT actually perform the
                   backup or update any config file parameters.

  --config=STRING  Use an alternate configuration file named STRING.
                   (Default = /etc/dobackup.conf)

  --timestamp=REAL Normally, each backup collects files modified since the
                   the time of the previous backup. This option allows you
                   to override the timestamp of the previous backup with the
                   supplied value, REAL. The value is in standard Unix epoch
                   time.

  --host=STRING    Normally, each host in the config file is backed up in
                   turn. This option allows a single specified host (STRING)
                   from the config file to be backed up. All configuration
                   values for other hosts are left untouched.

  --rotate         Upon completion of this backup, eject the media (if
                   applicable) and prepare for the next media unit in the
                   set for the next backup.

                   When used with --test, a forced media rotation can be
                   accomplished without performing a backup.

Description:

  General:
  --------

  This program is designed to be a flexible backup program. It will work
  with reliable removable media such as JAZ drives, potentially unreliable
  removable media like tape drives, dedicated drive partitions or simply a
  directory on an existing partition.

  We currently prefer to backup to dedicated drive partitions since they
  can be unmount'ed, offering greater data security and integrity and
  cannot be easily stolen like removable media (unless they swipe the
  whole server).

  The Program determines what to backup by comparing timestamps:
    1. If a Host's TimeStamp = 0, a full backup occurs.
    2. If a Host's TimeStamp exists and is not 0, then a backup of all
       files modified after the TimeStamp occurs.

    NEW: (v4.21)   A new "timewarp" mode has been added that acts much like
    the old incremental backups, but instead backs up everything that
    changed since the last FULL backup. This way only 2 files are ever
    required to perform a full restore to any given date. See the included
    sample config file for the config lines necessary to enable this mode.

  (TimeStamp is in seconds since Unix Epoch, 0 = 1970/01/01 00:00:00)


  Backup Exclusions:
  ------------------

  Rather than depending on filesystem-dependent flags or attributes, this
  program takes advantage of hidden "dot" (.) files for per-directory
  user-controllable exclusions.

  Any user can create a ".nobackup" file in any directory that they own. The
  contents of this file are read as one pattern per line to match files or
  directories that are not to be backed up under the current path.

  e.g. If the contents of /home/bob/.nobackup are:
	stuff
	private_thing

  then the items
	/home/bob/stuff
	/home/bob/private_things

  will be skipped during backup.

  More complex things can be excluded by using perl regex. For example, the
  following entries in /home/bob/.nobackup:

	\.mozilla/.*/Cache$
	.*\.(?:lha|zip|t?gz|Z|tar)$

  would prevent bob's Mozilla cache directory from being backed up, as well
  as any common archive formats.

  This makes it easy for users to exclude large files which don't need to be
  backed up without bothering the sysadmin.

  For backward-compatibility, by creating a zero-byte ".nobackup" file in
  any directory, that directory (and anything below it) are excluded from
  the backup.

  Note 1: Any pattern in a .nobackup file staring with "/" or ".." will be
          ignored.
  Note 2: The .nobackup file must have the same owner as the directory it
          resides in, or be owned by root.

  These regex patterns are also used in the "Excludes" per-host section of
  the config file for adding global exclusion patterns to your backups. See
  the sample config file for examples.


  Reliability:
  ------------

  The program is designed to be reliable. It makes use of many external
  tools, and since all paths are configurable and are potentially
  different on every system, all necessary external programs are verified
  prior to beginning the backup process, thereby ensuring the backup is
  able to complete successfully once started.

  By default, a backup summary report is e-mailed to the sysadmin (or any
  configured e-mail address) upon the completion of a successful backup, or
  upon encountering any internal failure.

  Verbose logs are stored by the program in a configurable log directory.
  Each log file is timestamped and host-stamped for easy tracking of
  individual backups. You can see exactly what files were backed up
  without having to dig out the actual backup media.

  Quite intentionally, the program does NOT check for enough remaining
  media space before starting the backup. In our opinion, it is better to
  have a partial backup written out, than none at all. Of course, your
  chosen archive tool (afio, by default) must be able to deal with
  partial files (afio does).

Preconditions:

  Program is intended to be run in root's crontab. All directories to be
  considered for backup should be listed one per line in the appropriate
  section of the config file. The system will recursively include any
  subdirectories under the ones listed.

Dependencies:

  On Redhat 9, dobackup uses tools from several packages. Not necessarily
  all of these will be required, depending on your specific configuration.

	afio (>= 2.5 for working large file support)
	eject
	fileutils
	grep
	iputils
	mount
	mt-st
	perl
	sendmail
	sh-utils
	textutils

  You'll also need these perl modules installed:

	Config::IniFiles

To-do:

  Please see the TODO file, part of this distribution.

Uncategorized notes that need to be expanded on:

  -Device used for backups should have an appropriate entry in /etc/fstab
   (Disk & Filesystem modes)

   Example:
	/dev/sda1 /jaz ext2 defaults,noauto 0 0

  -Tape mode had been unused for years, but has recently been put back into
   use by a few people. With a few minor updates to the code, tape mode seems
   to be back to working like a charm.

  -Multiple (remote) hosts can be backed up from one system. Appropriate NFS,
   SMB or other network filesystem mounts are left to the user. See sample
   config file.
