HAMMER2 updates in master - will be brought into the current release for Christmas

[Matthew Dillon]

Now that we've rolled the release, the master branch has received a
significant HAMMER2 update.  The intention is that it will be brought into
the current release this month sometime around Christmas after a bit more
testing and minor resource management work has been completed on it.  But
its in master now and so far its working great!

This update finally gives H2 filesystem meta-consistency protection for
snapshots and crashes.  This is on top of the lower-level radix tree
protections it already had.  The coherency protection handles
directory-entry-vs-inode dependencies which prior to this patch could
sometimes get broken by a crash, causing the nlinks count on the inode to
not match the number of directory entries referencing it, sometimes putting
directory entry flushes and the related inode flushes in different sync
groups, and often causing snapshot operations to 'cut' the filesystem while
it is in an inconsistent state, when made in the midst of modifying
activity.  Some people have noticed these issues recently, and while they
can be cleaned up with manual hammer2 directives after the fact, it was
really annoying to have to deal with.

The update now under test in master fixes all of these issues and is also
able to guarantee buffer cache consistency... hence also write() atomicy
boundaries on snapshot or crash.

As an added bonus, most of the front-end stalls that currently occur during
a filesystem sync have also been cleaned up.  The frontend.. that is, the
filesystem system calls made by programs, is now able to operate
concurrently with a filesystem sync.   Previously the front-end stalled on
any modifying operation made to any inode during a filesystem sync
(something I did to try to 'fix' the prior consistency issues, but which
didn't sufficiently fix the problems).  With the new work, the front-end
now only stalls on the specific inodes which are in the sync queue, and
will reorder any inodes it stalls on to the front of the sync queue in
order to retire their flushes as quickly as possible.  Once the inode is no
longer on the sync queue, the front-end can proceed with its modifying
operation concurrent with the continuing filesystem sync.  The result is
significantly better concurrency and far shorter-duration stalls for those
situations where a stall is mandatory.   The new code keeps track of and
handles any dependencies between inodes on the syncq and inodes not on the
syncq which arise during this operation.

The new patch also significantly improves snapshot operation, and adds a
new directive called 'snapshot-debug' (which I will probably rename) which
foregoes the standard sync-before-snapshot that the primary 'snapshot'
directive implements.  Both directives will snapshot a fully consistent
filesystem, the only difference is that snapshot-debug might not get
changes made just prior to issuing the command (it uses a recorded blockmap
from the most recent sync instead of forcing a new sync).

HAMMER2 still has the big-ticket clustering work on the TODO, but the work
going in today makes significant progress in providing the reliable
on-media topology that is a prerequisite to the continuing work.

-Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/users/attachments/20181209/c463ccfe/attachment-0002.htm>