HAMMER Update 31-Jul-2008 - Streaming Mirroring on HEAD (2.0 will be synced Saturday)
dillon at apollo.backplane.com
Thu Jul 31 16:20:20 PDT 2008
Today's commit to HEAD fixes all known issues except the still-open
process blocking issue. I am awaiting further testing and cores on
that one. I hope to have the blocking issue resolved by the 2.0 sync
The big feature available in this new commit is streaming mirroring
between two HAMMER filesystems! Streaming mirroring is in a mid-alpha
state. The mirroring code should continue to be considered experimental
until we can get some really good large-scale testing in using the
new streaming feature.
I have done moderate testing by streaming /usr/obj from one test box
to another while doing a buildworld, and so far it appears to work,
but I fully expect to find a few bugs in the code considering the
complexity of the code.
Note that this commit is to HEAD. I will MFC it to 2.0 on Saturday.
HOW TO TEST
Testing the data integrity of a mirror is very easy as long as the
master is in normal mode (and not no-history mode). Lets say you
had a HAMMER /usr/obj on two machines, MASTER and SLAVE, and wanted
to mirror /usr/obj on the master to /usr/obj/pfs1 on the slave.
Here is how you would set it up:
MASTER# hammer pfs-status /usr/obj | fgrep shared-uuid
SLAVE# hammer pfs-slave /usr/obj/pfs1 shared-uuid=.....
MASTER# hammer -v mirror-stream /usr/obj SLAVE:/usr/obj/pfs1
* Reports mirroring state in real time and doesn't exit
* Can be killed and restarted on a whim, though at the moment
it restarts at the last full sync point so a large initial
load will restart at the beginning (this is a function of
the hammer utility and will be fixed soon).
* Maintains a mirroring stream until the pipe is broken. Can be
used in a scripted loop along with the -b bandwidth option to
fully automate mirroring even if your network is unreliable.
* The slave will be approximately 30-60 seconds lagged from the
master (longer if you limit the bandwidth with the -b option).
Ok, that's it, you are now continuously mirroring. Now lets say you
wanted to test the mirror by running stuff that modifies /usr/obj
on the MASTER. Here is an example:
(on another xterm):
MASTER# cd /usr/src
MASTER# make -j 16 buildworld
Note that it may take a sync or two to get the mirroring going,
and the mirror will tend to only trigger when the master does a
filesystem sync (upwards of 30 seconds), or if you sync manually.
CHECKING DATA INTEGRITY
AND HAVING SOME FUN
The special PFS softlink on the slave always points to the latest fully
synchronized TID, and HAMMER munges atime/mtime/st_dev so they are
consistent. Thus you can do a directory-relative (tar | md5) sequence
and you should get the same MD5 on both the master and slave for that
In addition, you can do this test WHILE the master is being modified
(for example by an ongoing buildworld).
SLAVE# cd /usr/obj/pfs1
/usr/obj/@@0x00000001efa9ec90:00001 <<--- example output
SLAVE# tar cf - . | md5
MASTER# cd /usr/obj/@@0x00000001efa9ec90 <--- just the TID part
MASTER# tar cf - . | md5
5fc2efe93cb7e53a610cd46c31fd35bf <--- same MD5
... wait a while ...
SLAVE# cd /usr/obj/pfs1 <--- re-CD updates TID
... repeat same procedure
If your master is 'nohistory' or contains files marked for 'nohistory'
your snapshots on the slave will not be consistent with the master,
but if you let the master go idle (i.e. ^Z the buildworld) and let
the slave catch up, then you can still do an integrity check using the
More information about the Kernel