git: hammer2 - Merge Daniel Flores's HAMMER2 GSOC project into the main tree

[Matthew Dillon]

commit 355d67fcd81e0a7b17007d691bb00bdd151f3d28
Author: Matthew Dillon <dillon at apollo.backplane.com>
Date:   Wed Sep 18 19:56:07 2013 -0700

    hammer2 - Merge Daniel Flores's HAMMER2 GSOC project into the main tree
    
    * This merge contains work primarily by Daniel Flores, and also by
      Matthew Dillon.  Daniel's work is focused around adding the compression
      infrastructure while my work was focused around changing HAMMER2's
      I/O infrastructure to work with the compression code.
    
      Daniel's work involved adding the compression functions and heuristics,
      modifying mainly vnops and vfsops to use them, adding the new buffer
      cache write thread, and adding the new hammer2 utility directives and
      related ioctls.
    
      My work involved changing the H2 I/O infrastructure to always double-buffer
      (i.e. logical buffers vs device buffers) because device buffers can now
      wind up being a different size than the related logical buffers.  I also
      had to make changes to the hammer2_mount and hammer2_pfsmount mechanics
      and other things to prevent deadlocks.
    
    			    Daniel's Work
    
    * Add the hammer2 setcomp directive which sets the compression mode on a
      directory or file.  If applie to a directory, the compression mode is
      inherited by any new files or directories created under the directory.
      Pre-existing subdirectories and files are not affected.
    
      The directive has a recursive option to recurse down and set the mode on
      everything underneath.
    
    * Add wthread_bioq and related fields to hammer2_mount to support a
      buffer cache buffer writing thread.  This thread is responsible for
      calculating compression sizes, allocating device buffer blocks, and
      compressing logical buffers into the device buffers.
    
    * Implement HAMMER2_COMP_AUTOZERO, HAMMER2_COMP_LZ4, and HAMMER2_COMP_ZLIB
      compression modes.  AUTOZERO is the zero-block detection code.  LZ4 will
      do zero-block-detection and LZ4 otherwise, and ZLIB will do
      zero-block-detection and gzip otherwise.
    
      This work entails a ton of new files imported from the LZ4 and ZLIB
      projects plus lots of wiring.
    
      The new files had to be cleaned up considerably, as well, since they
      were originally intended for userland.
    
    * Move synchronous device buffer handling out of hammer2_vop_write() and
      into the support thread.  Numerous procedures were moved out of
      hammer2_vnops.c and into hammer2_vfsops.c as well.
    
      This greatly simplifies hammer2_vop_write() as well as the truncate and
      extend code, and improves the critical-path performance for write()
      (at least until the buffer cache fills up or gets too far behind).
    
    * Implement semi-synchronous decompression callbacks for read I/O and
      read-ahead I/O.
    
    * Add HAMMER2IOC_INODE_COMP_REC_SET and HAMMER2IOC_INODE_COMP_REC_SET2
      ioctls to support the setcomp directive.
    
    			  Matthew's Work
    
    * The hammer2_inode copies additional fields from the inode data, allowing
      the inode data to be deallocated after use.
    
    * Due to the way the buffer cache now operates, multiple deletions of the
      same chain key can occur within the same transaction.  Adjust the RBTREE
      compare code to handle the case.
    
    * Track chain structure use on a pfsmount-by-pfsmount basis for current
      and future management of the kmalloc pools used by hammer2.
    
    * Rework the way inodes are locked to track chain modifications.
    
    * Rewrite hammer2_chain_delete_duplicate().
    
    * Rewrite hammer2_trans_init() and the flush code primarily to fix
      deadlocks in the flush synchronization mechanics.
    
    * Interlock very low level chain operations with a spin lock instead
      of the full-blown chain lock to deal with potential deadlocks and
      fix a few SMP races.
    
    * For the moment make all logical buffers 64KB.  Not efficient for small
      files and will be changed back at some point but necessary for efficient
      compression at the moment.
    
    * Implement an asynchronous buffer cache callback feature.
    
    * Use a localized in-hammer2_inode-structure size field for frontend
      operations, including extend and truncate, to remove confusion against
      backend flushes.  This way the inode data can be modified only during
      the flush and not before where it might cause confusion with previously
      staged flushes.

Summary of changes:
 sbin/hammer2/Makefile                              |    2 +-
 sbin/hammer2/cmd_setcomp.c                         |  219 ++++
 sbin/hammer2/cmd_stat.c                            |    3 +-
 sbin/hammer2/hammer2.h                             |    7 +
 sbin/hammer2/main.c                                |   20 +
 sbin/hammer2/print_inode.c                         |   81 ++
 sys/vfs/hammer2/Makefile                           |    7 +-
 sys/vfs/hammer2/hammer2.h                          |   40 +-
 sys/vfs/hammer2/hammer2_chain.c                    |  207 +++-
 sys/vfs/hammer2/hammer2_disk.h                     |    4 +-
 sys/vfs/hammer2/hammer2_flush.c                    |  106 +-
 sys/vfs/hammer2/hammer2_freemap.c                  |    8 +
 sys/vfs/hammer2/hammer2_inode.c                    |  100 +-
 sys/vfs/hammer2/hammer2_ioctl.c                    |   40 +
 sys/vfs/hammer2/hammer2_ioctl.h                    |    4 +
 sys/vfs/hammer2/hammer2_lz4.c                      |  526 +++++++++
 sys/vfs/hammer2/hammer2_lz4.h                      |   93 ++
 sys/vfs/hammer2/hammer2_lz4_encoder.h              |  467 ++++++++
 sys/vfs/hammer2/hammer2_subr.c                     |   70 +-
 sys/vfs/hammer2/hammer2_vfsops.c                   |  726 +++++++++++-
 sys/vfs/hammer2/hammer2_vnops.c                    |  956 ++++++---------
 sys/vfs/hammer2/zlib/hammer2_zlib.h                |  551 +++++++++
 sys/vfs/hammer2/zlib/hammer2_zlib_adler32.c        |  175 +++
 sys/vfs/hammer2/zlib/hammer2_zlib_deflate.c        | 1210 +++++++++++++++++++
 sys/vfs/hammer2/zlib/hammer2_zlib_deflate.h        |  337 ++++++
 sys/vfs/hammer2/zlib/hammer2_zlib_inffast.c        |  340 ++++++
 sys/vfs/hammer2/zlib/hammer2_zlib_inffast.h        |   11 +
 .../vfs/hammer2/zlib/hammer2_zlib_inffixed.h       |    0
 sys/vfs/hammer2/zlib/hammer2_zlib_inflate.c        | 1052 +++++++++++++++++
 sys/vfs/hammer2/zlib/hammer2_zlib_inflate.h        |  113 ++
 sys/vfs/hammer2/zlib/hammer2_zlib_inftrees.c       |  304 +++++
 sys/vfs/hammer2/zlib/hammer2_zlib_inftrees.h       |   62 +
 sys/vfs/hammer2/zlib/hammer2_zlib_trees.c          | 1232 ++++++++++++++++++++
 .../vfs/hammer2/zlib/hammer2_zlib_trees.h          |    0
 sys/vfs/hammer2/zlib/hammer2_zlib_zconf.h          |  289 +++++
 sys/vfs/hammer2/zlib/hammer2_zlib_zutil.c          |  178 +++
 sys/vfs/hammer2/zlib/hammer2_zlib_zutil.h          |  148 +++
 37 files changed, 8988 insertions(+), 700 deletions(-)
 create mode 100644 sbin/hammer2/cmd_setcomp.c
 create mode 100644 sbin/hammer2/print_inode.c
 create mode 100644 sys/vfs/hammer2/hammer2_lz4.c
 create mode 100644 sys/vfs/hammer2/hammer2_lz4.h
 create mode 100644 sys/vfs/hammer2/hammer2_lz4_encoder.h
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib.h
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_adler32.c
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_deflate.c
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_deflate.h
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_inffast.c
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_inffast.h
 copy contrib/zlib-1.2/inffixed.h => sys/vfs/hammer2/zlib/hammer2_zlib_inffixed.h (100%)
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_inflate.c
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_inflate.h
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_inftrees.c
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_inftrees.h
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_trees.c
 copy contrib/zlib-1.2/trees.h => sys/vfs/hammer2/zlib/hammer2_zlib_trees.h (100%)
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_zconf.h
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_zutil.c
 create mode 100644 sys/vfs/hammer2/zlib/hammer2_zlib_zutil.h

http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/355d67fcd81e0a7b17007d691bb00bdd151f3d28


-- 
DragonFly BSD source repository