HAMMER update 23-Mar-08

[Matthew Dillon]

    Here's an update on the HAMMER work!

    Current status:

    * Passes all standard filesystem stress tests and buildworld will
      run with a HAMMER /usr/obj.  Large histories are able to accumulate
      without effecting performance.

    * Pruning and reblocking code is in and partially tested, but now needs
      more stringent testing.

    * Full historical access appears to be working but needs testing.
      Note that a sync is still needed to flush dirty cached data prior
      to acquiring a timestamp for the 'snapshot' to be set in stone.
      (dirty data cached in-memory has no historical tags and must be
      committed to physical disk before it can be accessed historically).


    Current bugs:

    * There is one known bug in the standard operations code paths
      that results in an assertion in HAMMER's I/O subsystem.

    * There are probably bugs in the reblocking and/or pruning code.  More
      likely in the reblocking code.


    There are two big-ticket and several little-ticket items left.  HAMMER
    will officially go Alpha when the big-ticket items are done, and beta
    when we get a few of the little-ticket items done.

    Big ticket items left:

    * UNDO (crash recovery) code.  Currently it writes out undo records but
      they are not yet sequenced, buffer writes are not yet ordered, and
      there is no mount-time recovery code yet.

      This is the last item needed before HAMMER can go operational.

    * Filesystem full handling.  Currently no space is reserved for dirty
      cached data so it is possible to create/write files and for HAMMER
      to not have sufficient space left on-disk to flush it.


    Little ticket items:

    * Automated reblocking (currently these functions are manually
      initialized via the hammer utility).

    * I/O clustering and preliminary BMAP op when writing out large files.

    * CRC checking (CRC fields are reserved but not entirely generated yet
      and not yet checked at all).

    * Disaster Recovery filesystem scan.

    * Boot support.

    I expect all of these items and more to be handled by the 2.0 release
    in July.

			    Additional HAMMER capabilities
				(no timeline yet)

    * Adding, removing, and resizing a HAMMER filesystem's backing store.

			Ultimate Goals and working towards them
				(no timeline yet)

    Our ultimate goal with HAMMER and DragonFly in general is to support
    fully cache coherent replication in a multi-machine environment.  This
    involves several steps and networking protocols.

    * Replication of synchronization streams based on the UNDO log.  If
      resynchronizing to a target which is too old a B-Tree scan will
      likely be required.

    * Cache coherency protocols for machine-machine coherency for both
      replicated and remote-HAMMER access.

    I have no time frame for these items yet.  It will depend on how quickly
    HAMMER moves to Alpha and Beta status.  I will say, however, now
    that HAMMER's on-disk format has solidified, that I have a very precise
    understanding of the protocols that will be needed to accomplish fully
    cache coherent remote access for both replicated and non-replicated
    (remote mount style) access.

    And, as you know, fully coherent filesystem access across machines is
    going to be the basis for DragonFly's clustering across said machines.

    In summary, things are progressing very well.

					-Matt
					Matthew Dillon 
					<dillon at backplane.com>