HAMMER2 progress report - 07-Aug-2012

[Matthew Dillon]

    Hammer2 continues to progress.  I've been working on the userland
    spanning tree protocol.

    * The socket/messaging system now connects, handshakes with public
      key encryption, and negotiates AES keys for the session data stream.

    * The low level transactional messaging subsystem is pretty solid now.

    * The initial spanning tree protocol implementation is propagating
      node information across the cluster and is handling connect/disconnect
      events properly.  So far I've only tested two hosts x 10 mounts,
      but pretty soon now I will start using vkernels to create much
      larger topologies for testing purposes.

      Essentially the topology is (ascii art):

                   Host #1                Any cluster (graph) topology
       ______________________________  ______________________________
      /                              \/                              \
      PFS mount ------\
      PFS mount ------\\             /---(TCP)-- Host #2 --(TCP)----\
      PFS mount ------ hammer2 service -----------(TCP)------------- Host #3
      PFS mount ------//             \---(TCP)-- Host #4 --(TCP)----/
      PFS mount ------/

      Full graph spanning tree protocol so there can be loops, multiple ways
      to get to the same target, and so on and so forth.  The SPANs propagate
      the best N (one or two) paths for each mount.

      Any given mount is just a HAMMER2 PFS, so there will be immense
      flexibility in what a 'mount' means.  i.e. is it a master node?  Is
      it a slave?  Is it a cache-only node?  Maybe its a diskless client-only
      node (no persistent storage at all), etc.

      Because each node is a PFS, and PFS's can be trivially created
      (any single physical HAMMER2 filesystem can contain any number
      of PFS's)... because of that there will be immense flexibility in
      how people construct their clusters.

    * The low level messaging subsystem is solid.  Message relaying is next
      on my TODO list (using the spanning tree to relay messages).  After
      that I'll have to get automatic-reconnection working properly.

    Once the low level messaging subsystem is solid I will be able to start
    working on the higher-level protocols, which is the fun part.  There is
    still a very long ways to go.

    Ultimately the feature set is going to be huge, which is one reason why
    there is so much work left to do.  For example, we want to be able to
    have millions of diskless or cache-only clients be able to connect into
    a cluster and have it actually work... which means that the topology
    would have to support 'satellite' hosts to aggregate the clients and
    implement a proxy protocol to the core of the topology without having
    to propagate millions of spanning tree nodes.  Ultimately the topology
    has to allow for proxy operation, otherwise the spanning tree overhead
    becomes uncontrolled.  This would also make it possible to have
    internet-facing hosts without compromising the cluster's core.

    Also note that dealing with multiple physical disks and failures will
    also be part of the equation.  The cluster mechanic described above is
    an abstraction for having multiple copies of the same filesystem in
    different places, with varying amounts of data and thus gaining
    redundancy.

    But we ALSO want to be able to have a SINGLE copy of the filesystem
    (homed at a particular machine) to use the SAME mechanism to glue
    together all of its physical storage into a single entity (plus with
    a copies mechanic for redundancy), and then allow that filesystem to
    take part in the multi-master cluster as one of the masters.

    All of these vastly different feature sets will use the same underlying
    transactional messaging protocol.

    x bazillion more features and that's my goal.

						-Matt