[GSOC] Introduction to HAMMER2 compression feature
fjwcash at gmail.com
Wed Jun 12 09:08:41 PDT 2013
On Wed, Jun 12, 2013 at 8:13 AM, Daniel Flores <daniel5555 at gmail.com> wrote:
> Hello everyone,
> My name is Daniel Flores and my proposal called “HAMMER2 compression
> feature” was accepted for this year’s Google Summer of Code. I already
> posted the draft of my proposal  in this kernel list, so I will not
> repeat much of it, but instead I want to focus on some design decisions
> that I’ve already made. Since I’m an inexperienced developer at this point,
> I’d be happy to receive your suggestions and criticism.
> The feature itself consists in that it attempts to compress a HAMMER2
> block, which is of 64KB in size. The result should be a block of 32KB,
> 16KB, 8KB, etc. in size.
> Currently I’m searching for the algorithms that are the most appropriate
> for a file system. Particularly I’m searching for algorithms that are very
> fast; don’t require a lot of memory and processing power and offer fairly
> good compression rate (but not necessarily the best compression rate out of
> all). Right now I have two candidates – DEFLATE  and LZO . Both of
> them have some available implementations which I intend to use, as Alex
> suggested in his review of my proposal.
> DEFLATE seems to be a good choice, because it works with small amounts of
> data and has a sliding window of 32KB – just nice for a 64KB block. It is
> based on another algorithm – LZ77, which is successfully used in
> compression feature for NTFS, so hopefully DEFLATE would be good as well.
> LZO seems to be a good choice, because, similarly, it works on small
> amounts of data, it is as fast as DEFLATE and was specifically designed to
> have a very fast decompression speed.
You might want to consider LZ4 . It was just recently added to ZFS
. It's multi-threaded-aware, extremely fast for both compression and
decompression, and gives very good compression ratios. It also includes a
short-circuit design to prevent wasting time trying to compress already
compressed blocks or highly incompressible blocks. It's also OSS and
several reference implementations are available in various languages.
fjwcash at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Kernel