Hammer or ZFS based backup, encryption

[Freddie Cash]

On Sun, Feb 22, 2009 at 11:10 PM, Bill Hacker <wbh at conducive.org> wrote:
> Freddie Cash wrote:
>> On Sat, Feb 21, 2009 at 10:39 AM, Csaba Henk <csaba.henk at creo.hu> wrote:
>>> I need to setup a backup machine, and I intend to utilize today's
>>> snapshotty filesystems (which boils down to Dfly+Hammer or FBSD+ZFS --
>>> btrfs is not there yet, and I don't feel like dwelving into Solaris).
>>> Set up such an OS with such an fs, and backup by syncing to the
>>> snapshotty fs and create a snapshot.
>>>
>>> I wonder about the following things:
>>>
>>> 1) Any idea how does this approach scale related to more conventional
>>> solutions,
>>> like rdiff-backup or dump(8)? I see the the pros, but are there any
>>> cons? How effective is taking regular snapshots space-wise?
>>>
>>> 2) Is there any practical argument for choosing between Dfly+Hammer and
>>> FBSD+ZFS? (Feel free to give biased answers :) )
>>>
>>> 3) I'd like to encrypt stuff, either at device or fs level. For
>>> FreeBSD there is geli(8). I haven't found anything for DragonFly.
>>> Is there any way to get at it on DragonFly?
>>
>> We do this at work, using FreeBSD 7.1 and ZFS, for backing up over 80
>> remote Linux and FreeBSD servers, and 1 Windows station.  We have two
>> servers, one that does the backups every night, and another that
>> mirrors the backups during the day.
>
> *trimmed* (description of a quite decent ZFS approach)
>
>>
>> After the initial rsync of a server, which can take several days as it
>> can easily max out an ADSL link's upload bandwidth, the daily run
>> takes about 6 hours, most of which is waiting for rsync to generate
>> the file listing.
>>
>
> *snipped*
>
> But there we are, Startup 'seeding; is unavidable, but thereafter ... among
> other things, looking to reduce the reliance on rsync (and similar CVS'ish
> or git'ish techniques) having to 'inventory' stuff at a high per-file level
> that a 'hammer mirror-stream' (or GMIRROR to networked RAID) could do 'as
> you go along' at a lower level - closer to the actual blocks as they are
> being written.
>
> How, and how well, would ZFS handle redundant pools on separate sites?
>
> And can that be a streaming process - even if that means the redundancy
> target is r/o for 'the duration', as a hammer slave would be?

ZFS includes "snapshot send" and "snapshot recv" commands for
streaming snapshots across the network.  However, there is currently a
bottleneck in the recv code that prevents it from saturating even a
100 Mbit link.  This affects ZFS v6 which is in FreeBSD 7.x, and I'm
fairly certain that it hasn't been fixed in ZFS v13 which is in
FreeBSD 8.  Once that has been fixed, then it can be used for
near-real-time streaming of snapshots across the network.

Until then, you can rsync snapshots from the master to the slave
server(s), which is what we're doing.

-- 
Freddie Cash
fjwcash at gmail.com