Real World DragonFlyBSD Hammer DeDup figures from HiFX - Second Disk 20% ( 88 GB ) Disk Space reclaimed

[Siju George]

Hi,

I De-Duped the second disk on the backup Server today.

Before De-Dup

Filesystem                Size   Used  Avail Capacity  Mounted on
Backup2                   454G   335G   119G    74%    /Backup2

After De-Dup

Filesystem                Size   Used  Avail Capacity  Mounted on
Backup2                   454G   247G   207G    54%    /Backup2

So 20% or 88 GB reclaimed without deleting anything :-)

No. of PFSes    -       3 Slaves and 3 Masters

De-dup Simulate

====================================================================
dfly-bkpsrv# hammer dedup-simulate /Backup2
Dedup-simulate /Backup2: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 0
Dedup-simulate /Backup2 succeeded
Simulated dedup ratio = 1.26
dfly-bkpsrv# hammer dedup-simulate /Backup2/pfs/Data
Dedup-simulate /Backup2/pfs/Data: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 1
Dedup-simulate /Backup2/pfs/Data succeeded
Simulated dedup ratio = 1.37
You have new mail.
dfly-bkpsrv# hammer dedup-simulate /Backup2/pfs/VersionControl
Dedup-simulate /Backup2/pfs/VersionControl: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 2
Dedup-simulate /Backup2/pfs/VersionControl succeeded
Simulated dedup ratio = 2.79
dfly-bkpsrv# hammer dedup-simulate /Backup2/pfs/test
Dedup-simulate /Backup2/pfs/test: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 3
Dedup-simulate /Backup2/pfs/test succeeded
Simulated dedup ratio = 0.00
dfly-bkpsrv# hammer dedup-simulate /Backup2/www-daily-bak/www-hot
Dedup-simulate /Backup2/www-daily-bak/www-hot: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 5
Dedup-simulate /Backup2/www-daily-bak/www-hot succeeded
Simulated dedup ratio = 1.64
dfly-bkpsrv# hammer dedup-simulate /Backup2/mysql-daily-bak/mysql-hot
Dedup-simulate /Backup2/mysql-daily-bak/mysql-hot: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 6
Dedup-simulate /Backup2/mysql-daily-bak/mysql-hot succeeded
Simulated dedup ratio = 5.99
====================================================================

Real De-dup

=========================================================
dfly-bkpsrv# hammer dedup /Backup2
Dedup /Backup2: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 0
Dedup /Backup2 succeeded
Dedup ratio = 1.25
      580 MB referenced
      462 MB allocated
     2274 KB skipped
           0 CRC collisions
           0 SHA collisions
           0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup2/Data
Dedup /Backup2/Data: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 1
Dedup /Backup2/Data succeeded
Dedup ratio = 1.37
      266 GB referenced
      194 GB allocated
       31 MB skipped
        1960 CRC collisions
           0 SHA collisions
          20 bigblock underflows
You have new mail.

dfly-bkpsrv# hammer dedup /Backup2/VersionControl
Dedup /Backup2/VersionControl: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 2
Dedup /Backup2/VersionControl succeeded
Dedup ratio = 2.75
      160 MB referenced
       58 MB allocated
      861 KB skipped
           0 CRC collisions
           0 SHA collisions
           0 bigblock underflows
You have new mail.
dfly-bkpsrv# hammer dedup /Backup2/test
Dedup /Backup2/test: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 3
Dedup /Backup2/test succeeded
Dedup ratio = 0.00
         0 B referenced
         0 B allocated
         0 B skipped
           0 CRC collisions
           0 SHA collisions
           0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup2/www-daily-bak/www-hot
Dedup /Backup2/www-daily-bak/www-hot: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 5
Dedup /Backup2/www-daily-bak/www-hot succeeded
Dedup ratio = 1.64
     5599 MB referenced
     3420 MB allocated
       12 MB skipped
           4 CRC collisions
           0 SHA collisions
           0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup2/mysql-daily-bak/mysql-hot
Dedup /Backup2/mysql-daily-bak/mysql-hot: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 6
Dedup /Backup2/mysql-daily-bak/mysql-hot succeeded
Dedup ratio = 6.28
       49 GB referenced
     8079 MB allocated
         0 B skipped
           0 CRC collisions
           0 SHA collisions
           0 bigblock underflows
You have new mail.
============================================================