Backup statistics - using HAMMER on my LAN backup box

[Matthew Dillon]

    HAMMER is stable enough now that I am able to run it on my
    LAN backup box.  I'm using it to test that the snapshots work
    as expected as well as to test the long term effects of reblocking
    and pruning.  The LAN backup box NFS mounts all the other boxes primary
    partitions and uses that to create a daily snapshots from 5 machines
    (apollo, crater, leaf, pkgbox, and my office workstation), covering
    around 90G of backed-up data.  The box mirrors the latest daily
    snapshot off-site once a week so I can afford to lose the data if
    I hit a bug.

    With UFS I had to use the hardlink trick (w/ cpdup) to generate backups.
    It took 4-6 hours every day for the backup box to create the snapshots,
    and I couldn't use more then half the 700G of backup space because
    using more resulted in having too many inodes (> 40 million) for UFS's
    fsck to be able to fsck without running out of memory.

STARTING MIRRORS Mon Mar 31 01:15:00 PDT 2008 level 2
DONE MIRRORING   Mon Mar 31 05:01:51 PDT 2008		~4 hrs
STARTING MIRRORS Wed Apr  2 01:15:00 PDT 2008 level 2
DONE MIRRORING   Wed Apr  2 07:02:00 PDT 2008		~6 hrs
STARTING MIRRORS Fri Apr  4 01:15:00 PDT 2008 level 2
DONE MIRRORING   Fri Apr  4 07:32:33 PDT 2008		~6 hrs
STARTING MIRRORS Sat Apr  5 01:15:00 PDT 2008 level 2
DONE MIRRORING   Sat Apr  5 05:09:13 PDT 2008		~4 hrs

    With HAMMER I don't have to use the hardlink trick.  I can just
    cpdup straight out and then create a @@ softlink to the snapshot.
    It takes less then an hour to do a daily backup that way.

STARTING MIRRORS Tue May  6 01:15:01 PDT 2008 level 2
DONE MIRRORING   Tue May  6 02:11:17 PDT 2008		~56 min
STARTING MIRRORS Sat May 10 01:15:00 PDT 2008 level 2
DONE MIRRORING   Sat May 10 02:17:36 PDT 2008		~62 min
STARTING MIRRORS Sun May 11 01:15:01 PDT 2008 level 2
DONE MIRRORING   Sun May 11 02:09:02 PDT 2008		~54 min

    So far the integrity of the snapshots is good.  I am doing a
    tar cf - <softlink>/. | md5 on each fixed snapshot and will check
    to see if the value changes over time.  I already see that I might
    want to create a mount option to update mtime as a record update
    instead of as an in-place update, to guarantee that it does not
    change from the point of view of a snapshot.  Being able to
    integrity-check a snapshot will likely become an important aspect of
    the filesystem.

    I am seeing a certain degree of fragmentation, particularly when
    listing directories.  It will be interesting to see what kind of
    effect reblocking has on that.

    Please note that HAMMER is not ready for production use yet, there
    is still the filesystem-full handling to implement and much more serious
    testing of the reblocking and pruning code is required, not to mention
    the crash recovery code.  I expect to find a few more bugs, but I'm
    really happy with the results so far.

						-Matt