HAMMER Update 31-Jul-2008 - Streaming Mirroring on HEAD (2.0 will be synced Saturday)

[Matthew Dillon]

    Today's commit to HEAD fixes all known issues except the still-open
    process blocking issue.  I am awaiting further testing and cores on
    that one.  I hope to have the blocking issue resolved by the 2.0 sync
    on Saturday.

    The big feature available in this new commit is streaming mirroring
    between two HAMMER filesystems!  Streaming mirroring is in a mid-alpha
    state.  The mirroring code should continue to be considered experimental
    until we can get some really good large-scale testing in using the
    new streaming feature.

    I have done moderate testing by streaming /usr/obj from one test box
    to another while doing a buildworld, and so far it appears to work,
    but I fully expect to find a few bugs in the code considering the
    complexity of the code.

    Note that this commit is to HEAD.  I will MFC it to 2.0 on Saturday.

				    --
				HOW TO TEST

    Testing the data integrity of a mirror is very easy as long as the
    master is in normal mode (and not no-history mode).   Lets say you
    had a HAMMER /usr/obj on two machines, MASTER and SLAVE, and wanted
    to mirror /usr/obj on the master to /usr/obj/pfs1 on the slave.

    Here is how you would set it up:

	MASTER# hammer pfs-status /usr/obj | fgrep shared-uuid
	shared-uuid=......

	SLAVE# hammer pfs-slave /usr/obj/pfs1 shared-uuid=.....

	MASTER# hammer -v mirror-stream /usr/obj SLAVE:/usr/obj/pfs1

    Features:

	* Reports mirroring state in real time and doesn't exit

	* Can be killed and restarted on a whim, though at the moment
	  it restarts at the last full sync point so a large initial
	  load will restart at the beginning (this is a function of
	  the hammer utility and will be fixed soon).

	* Maintains a mirroring stream until the pipe is broken.  Can be
	  used in a scripted loop along with the -b bandwidth option to
	  fully automate mirroring even if your network is unreliable.

	* The slave will be approximately 30-60 seconds lagged from the
	  master (longer if you limit the bandwidth with the -b option).

    Ok, that's it, you are now continuously mirroring.  Now lets say you
    wanted to test the mirror by running stuff that modifies /usr/obj
    on the MASTER.  Here is an example:

	(on another xterm):
	MASTER# cd /usr/src
	MASTER# make -j 16 buildworld

	Note that it may take a sync or two to get the mirroring going,
	and the mirror will tend to only trigger when the master does a
	filesystem sync (upwards of 30 seconds), or if you sync manually.

				  --
			CHECKING DATA INTEGRITY
			  AND HAVING SOME FUN

    The special PFS softlink on the slave always points to the latest fully
    synchronized TID, and HAMMER munges atime/mtime/st_dev so they are
    consistent.  Thus you can do a directory-relative (tar | md5) sequence
    and you should get the same MD5 on both the master and slave for that
    snapshot TID.

    In addition, you can do this test WHILE the master is being modified
    (for example by an ongoing buildworld).

	SLAVE# cd /usr/obj/pfs1
	SLAVE# pwd
	/usr/obj/@@0x00000001efa9ec90:00001		<<--- example output
	SLAVE# tar cf - . | md5
	5fc2efe93cb7e53a610cd46c31fd35bf

	MASTER# cd /usr/obj/@@0x00000001efa9ec90	<--- just the TID part
	MASTER# tar cf - . | md5
	5fc2efe93cb7e53a610cd46c31fd35bf		<--- same MD5


	... wait a while ...


	SLAVE# cd /usr/obj/pfs1				<--- re-CD updates TID
	SLAVE# pwd
	/usr/obj/@@0x000000021da1af40:00001
	... repeat same procedure

    If your master is 'nohistory' or contains files marked for 'nohistory'
    your snapshots on the slave will not be consistent with the master,
    but if you let the master go idle (i.e. ^Z the buildworld) and let
    the slave catch up, then you can still do an integrity check using the
    same procedure.

						-Matt