the 'why' of pseudofs

[Bill Hacker]

Matthew Dillon wrote:
    There are several reasons for using PFSs.

    * Shared allocation space.  You don't have to worry about blowing
      out small filesystems and having to resize them.
    * Each PFS has its own inode space, allowing mirroring to be used
      to manage backups on a per-PFS basis.  Thus mirroring slaves can
      be conveniently created and destroyed, and masters can be used
      to differentiate what you do and do not want to back up.  e.g.
      I want to backup /home, I don't want to backup /usr/obj.
      In this respect there is actually a lot more to it... PFSs are
      the primary enabler for most of the future multi-master clustering
      work.  Even slaves are extremely inconvenient to do without PFSs
      to manage independant inode spaces.
    * Each PFS can have its own history/snapshot retention policy.
      For example you want to retain history on /home but who cares
      about /tmp or /usr/obj ?  You might want to retain only a few
      days worth of snapshots for /var but hundreds of days for /home.
    * Each PFS can be pruned / reblocked independantly of the others.
      For example /build on pkgbox is configured to spend a lot longer
      pruning and reblocking then /archive.
    With regards to softlinks vs null mounts, null mounts are preferred
    because softlinks are not always handled properly, or handled in
    the expected way, by utilities.
    An example of this would be, say, /usr/src.  If /usr/src is a softlink
    then the /usr/obj paths generated would be the expanded softlink.
    So instead of getting /usr/obj/usr/src/... you would instead get
    /usr/obj/pfs/@@0xffffffffffffffff:0007/src/...
    It can get messy very quickly when the filesystem space is glued 
    together with softlinks instead of mounts.

						-Matt

OK - let's say one of the goals here will be more comprehensive 
documentation - no only of what 'is now' (above) but w/r the perils, 
tribulations - or possibly the advantages, even if edge-case - of doing 
elsewise.

I'll first append Michaels' response - embedding a few words:

PFS is the smallest unit of mirroring, and [therefor also the smallest 
> |the] unit to which you can
apply specific retainment policies. For example while you do not want
to retain much history for /tmp, you might want to do so for /home.
When it comes to mirroring, you clearly do not want to mirror changes to
PFS /tmp, while you want to mirror changes to PFS /home.
Good concept. Bad choice of examples [1].

> If everything
would lie on a single huge filesystem "/", we could not decide what to
mirror and what not. That's the major design decision. 

. .. my /, /usr, /var, /home, /tmp are (traditionally) on separate 
partitions, if not slices.

I don't need hammer there. Logs aside, these could damn near be in ROM 
for as much as two years at a go.

I need HAMMER on the 500 GB single to several TB arrays where client 
applications, IMAP mailstore, web sites, and other *dynamic* data reside.

You might ask, why not simply specify which directories to mirror 
and which to leave out (without considering PFS)? The issue here is,
that, AFAIK, mirroring works on a very low level, where only inode
numbers are available and not full pathnames, so something like:
  
  tar -cvzf /tmp/backup.tgz --reject="/tmp;/var/tmp"

would not work, or would be slow.

Another issue is locality. Metadata from one PFS lies more close
together and as such is faster to iterate.
====================== Questions and counterpoints ================

Some GOSPEL we can agree on came at the end of Matt's reply.

Let's take these as givens:

>    With regards to softlinks vs null mounts, null mounts are preferred
>    *because softlinks are not always handled properly, or handled in
>    the expected way*, by utilities.
>
====
    *It can get messy very quickly when the filesystem space is glued 
    together with softlinks instead of mounts.*

====

To which I'll remind that neither (Matt's) excellent 'cpdup', nor rsync, 
nor a seasoned admin working manually can be *certain* that all 
softlinks will ever and always be handled as one wished they had been 
. ... mere seconds after the bullet holes appear in the feet.

They can be problem-solvers some of the time, but a potential 
maintenance hand-grenade ALL the time.

Further, a 'pseudo' mount is, IMNSHO, just another 'virtual' band-aid of 
a different flavor. It is NOT (yet) 'enough' more assured to be handled 
correctly. Not because the fs & utils cannot be made to do the right 
thing, but because the *sysadmin* and habit of long years may not.

New game. More education needed. New habits to be built, and old ones 
'unlearned'.

If/as/when a utility must actually *care* about the foundation level of 
inodes, AND THEN 'virtualize' a known-to-not-match set of these, we 
should be aware that hazards lurk, and have them 'boxed' ahead of time.

'virtual' means the next word is a lie, and if it hides, it bites.

So we need different utilities to avoid foot-shooting.
As in the 'version-display' capability in 'ls' already in work.
More will be needed...

I  *want* the benefits of a file system that is robust in new ways that 
address real needs. Enter hammerfs.

I DO NOT want to add-back fragility that negates those very benefits by 
moving the failure modes from device and OS and fs into the space 
between an admin's ears. Exit 'too much' dependence on softlinks and 
pseudo mounts.

Re-balancing seems to be in order.

I'll now motor off for a day or two, do a more research, 
experimentation, and preparation.

Back with specifics 'soon'.

Thanks,

Bill

NOTE: /, /usr/ var/, /home, /tmp are not even on the radar.

Properly reserved for system and sysadmins, these are easily cloned and 
kept synced, backed-up, even swapped in/out 'en bloc' - so long as one 
keeps non-admin users, webish-ness, data bases, mailstore, 
disaster-recovery image storage, etc. entirely OFF them.

Those needs should be mounted from different slices as a minimum, 
different devices (or arrays) *preferably*.

Think 20-200 GB system device, RAID1 - and UFS/FFS is good enough.

Elsewhere - it is on the 500 GB, 1 TB, 2 TB - and up - 'working storage' 
where HAMMER may rule.

- hammer mirror-stream over 100BT or Gig-E,

- the ability to up-rank the slave to master,*quickly*,

. .. and the incremental changes to even very large storage can be kept 
in sync faster, better, and with fewer resources than by any other method.

Providing one doesn't blow it away with fat-fingers....