Next DFly release will have an initial HAMMER2 implementation

[Matthew Dillon]

The next DragonFly release (probably in September some time) will have an
initial HAMMER2 implementation.  It WILL be considered experimental and
won't be an installer option yet.  This initial release will only have
single-image support operational plus basic features.  It will have live
dedup (for cp's), compression, fast recovery, snapshot, and boot support
out of the gate.

This first H2 release will not have clustering or multi-volume support, so
don't expect those features to work.  I may be able to get bulk dedup and
basic mirroring operational by release time, but it won't be very
efficient.  Also, right now, sync operations are fairly expensive and will
stall modifying operations to some degree during the flush, and there is no
reblocking (yet).  The allocator has a 16KB granularity (on HAMMER1 it was
2MB), so for testing purposes it will still work fairly well even without
reblocking.

The design is in a good place.  I'm quite happy with how the physical
layout turned out.  Allocations down to 1KB are supported.  The freemap has
a 16KB granularity with a linear counter (one counter per 512KB) for
packing smaller allocations.  INodes are 1KB and can directly embed 512
bytes of file data for files <= 512 bytes, or have four top-level blockrefs
for files > 512 bytes.  The freemap is also zoned by type for I/O locality.

The blockrefs are 'fat' at 128 bytes but enormously powerful.  That will
allow us to ultimately support up to a 512-bit crypto hash and blind dedup
using said hash.  Not on release, but that's the plan.

I came up with an excellent solution for directory entries.  The 1KB
allocation granularity was a bit high but I didn't want to reduce it.
However, because blockrefs are now 128 byte entities, and directory entries
are hashed just like in H1, I was able to code them such that a directory
entry is embedded in the blockref itself and does not require a separate
data reference or allocation beyond that.  Filenames up to 64 bytes long
can be accomodated in the blockref using the check-code area of the
blockref.  Longer filenames will use an additional data reference hanging
off the blockref to accomodate up to 255 char filenames.  Of course, a
minimum of 1KB will have to be allocated in that case, but filenames are <=
64 bytes in the vast majority of use cases so it just isn't an issue.

This gives directory entries optimal packing and indexing and is a huge win
in terms of performance since blockrefs are arrayed in 16KB and 64KB
blocks.  In addition, since inodes can embed up to four blockrefs, the
directory entries for 'small' directories with <= 4 entries ('.' and '..'
don't count) can actually be embedded in the directory inode itself.

So, generally speaking, the physical layout is in a very happy place.  The
basics are solid on my test boxes so it's now a matter of implementing as
many of the more sophisticated features as I can before release, and
continuing to work on the rest after the release.

-Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/users/attachments/20170818/b14d63d6/attachment-0002.htm>