Next DFly release will have an initial HAMMER2 implementation

[Tim Darby]

Exciting news, but I really can't live without multi-volume and mirroring,
so just hurry those along please. :)


Tim

On Fri, Aug 18, 2017 at 11:40 PM, Matthew Dillon <dillon at backplane.com>
wrote:

> The next DragonFly release (probably in September some time) will have an
> initial HAMMER2 implementation.  It WILL be considered experimental and
> won't be an installer option yet.  This initial release will only have
> single-image support operational plus basic features.  It will have live
> dedup (for cp's), compression, fast recovery, snapshot, and boot support
> out of the gate.
>
> This first H2 release will not have clustering or multi-volume support, so
> don't expect those features to work.  I may be able to get bulk dedup and
> basic mirroring operational by release time, but it won't be very
> efficient.  Also, right now, sync operations are fairly expensive and will
> stall modifying operations to some degree during the flush, and there is no
> reblocking (yet).  The allocator has a 16KB granularity (on HAMMER1 it was
> 2MB), so for testing purposes it will still work fairly well even without
> reblocking.
>
> The design is in a good place.  I'm quite happy with how the physical
> layout turned out.  Allocations down to 1KB are supported.  The freemap has
> a 16KB granularity with a linear counter (one counter per 512KB) for
> packing smaller allocations.  INodes are 1KB and can directly embed 512
> bytes of file data for files <= 512 bytes, or have four top-level blockrefs
> for files > 512 bytes.  The freemap is also zoned by type for I/O locality.
>
> The blockrefs are 'fat' at 128 bytes but enormously powerful.  That will
> allow us to ultimately support up to a 512-bit crypto hash and blind dedup
> using said hash.  Not on release, but that's the plan.
>
> I came up with an excellent solution for directory entries.  The 1KB
> allocation granularity was a bit high but I didn't want to reduce it.
> However, because blockrefs are now 128 byte entities, and directory entries
> are hashed just like in H1, I was able to code them such that a directory
> entry is embedded in the blockref itself and does not require a separate
> data reference or allocation beyond that.  Filenames up to 64 bytes long
> can be accomodated in the blockref using the check-code area of the
> blockref.  Longer filenames will use an additional data reference hanging
> off the blockref to accomodate up to 255 char filenames.  Of course, a
> minimum of 1KB will have to be allocated in that case, but filenames are <=
> 64 bytes in the vast majority of use cases so it just isn't an issue.
>
> This gives directory entries optimal packing and indexing and is a huge
> win in terms of performance since blockrefs are arrayed in 16KB and 64KB
> blocks.  In addition, since inodes can embed up to four blockrefs, the
> directory entries for 'small' directories with <= 4 entries ('.' and '..'
> don't count) can actually be embedded in the directory inode itself.
>
> So, generally speaking, the physical layout is in a very happy place.  The
> basics are solid on my test boxes so it's now a matter of implementing as
> many of the more sophisticated features as I can before release, and
> continuing to work on the rest after the release.
>
> -Matt
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/users/attachments/20170819/0105b573/attachment-0003.htm>