hammer prune explanation

[Erik Wikström]

On 2008-05-10 22:59, Matthew Dillon wrote:
> :Yeah, I was thinking about wildcarding as well.
> :
> :But is it possible to implement it within cmd_prune.c, or do I have to
> :modify the ioctl kernel code? If done in cmd_prune.c, I somehow have to
> :iterate over all deleted files and call the prune command for it.
> :
> :I thought, it's easier to introduce a check in the kernel, whether the
> :file that should be pruned matches a given pattern. Doesn't sound very
> :hard to do, if it is easy to get the pathname for a given inode.
> :
> :Are you thinking about something like the archive flag?
> 
>     I think it is probably best to implement that level of sophistication
>     in the utility rather then in the kernel.  The pruning ioctl code
>     has no concept of files or directories... literally it has no concept.
>     All it understands, really, are object id's (aka inode numbers) and
>     records.
> 
>     The hammer utility on the other hand can actually scan the filesystem
>     hierarchy.
> 
>     Locating wholely deleted files and directories is not hard to do.
>     As-of queries can be used to access earlier versions of a directory.
> 
>     We might want to add some kernel support to make it more efficient,
>     for example to make it possible for the hammer utility to have
>     visibility into all deleted directory entries.  It could use that
>     visbility to do as-of accesses and through that mechanic would thus
>     have visibility into all deleted files and directories.
> 
>     Inode numbers are never reused, so the inode number (and hence object
>     id) of a deleted file will be just as unique as the inode number 
>     for one that is still visible.
> 
> : >     Right now any serious HAMMER user need to set up at least a daily
> : >     cron job to prune and reblock the filesystem.  I add a '-t timeout'
> : >     feature to the HAMMER utility to make allow the operations to be
> : >     set up in a cron job and keep the filesystem up to snuff over a long
> : >     period of time.  So, e.g. you would have a nightly cron job that
> : >     did this:
> : >
> : > 	# spend up to 5 minutes pruning the filesystem and another
> : > 	# 5 minutes reblocking it, then stop.
> : > 	hammer -t 300 prune /myfilesystem; hammer -t 300 reblock /myfilesystem
> :
> :Does this degrade filesystem seriously?
> :
> :Regards,
> :
> :   Michael
> 
>     For the time it is running it will be maxing out the filesystem, e.g.
>     similar to doing a 'find / ...'.  The idea is to limit the run time
>     (hence the -t) so your nightly cron job does a small chunk of the
>     filesystem every night, resulting in a clean well ordered filesystem
>     over a long period of time.  So, for example, spend 10 minutes a day
>     doing housekeeping.  Filesystems are rarely operating at 100% 24x7 and
>     there are other ways to spread out the overhead if it became necessary
>     to do so.  Usually picking a chunk of time during off-hours is sufficient.

That will probably work quite well for servers which are running 24x7,
but how about using HAMMER on desktops/laptops (which might not be
running except when in use, though the disk might not be used all the
time). Could some kind of low priority process be used instead? Perhaps
one that only runs for less than a minute but instead runs every 10
minutes or so, the idea being to spread out the pruning so that it wont
affect (severely) normal usage but still keep the FS in good shape.

-- 
Erik Wikström