Globbing (was Re: HAMMER update 10-Feb-2008)

[Bill Hacker]

0432c$0$854$415eb37d at crater_reader.dragonflybsd.org>
In-Reply-To: <47b0432c$0$854$415eb37d at crater_reader.dragonflybsd.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Lines: 55
Message-ID: <47b04f81$0$854$415eb37d at crater_reader.dragonflybsd.org>
NNTP-Posting-Host: 218.253.81.177
X-Trace: 1202737026 crater_reader.dragonflybsd.org 854 218.253.81.177
Xref: crater_reader.dragonflybsd.org dragonfly.kernel:12081

Oliver Fromme wrote:
> Bill Hacker wrote:
>  > But the larger issue is indeed expanded globbing support *somewhere*.
>  > 
>  > And not just for new file systems, but old ones that have long-since 
>  > outgrown the currently available resources.
>  > 
>  > Massive drive and name space isn't a lot of use if we are short matching 
>  > utils to manage it well.
> 
> Filename expansion ("globbing") has nothing to do with
> file systems.  It is purely a shell feature (and every
> shell implements its own).  Putting it into the kernel
> makes no sense.  Note that there are globbing support
> functions in libc (POSIX/SUS wants them, and some apps
> use them), but no shell uses them, even the base /bin/sh
> implements its own filename expansion.
> 
> The limit you're talking about ("too many arguments")
> is the execve() argument limit.  Historically all the
> arguments plus environment could not exceed 64 KB.
> A few years ago FreeBSD increased the limit to 256 KB.
> I don't know if DragonFly did the same, but it doesn't
> matter much -- The point is that things like "rm *"
> are unsafe if you don't know to what amount the glob
> expression will expand.  No matter whether the limit
> is 64 KB or 256 KB, there _is_ a limit.
> 
> That's why xargs(1) exists:  "echo * | xargs rm" is
> safe (echo is a shell-builtin, so it is not subject to
> the execve argument limit).
> 
> Best regards
>    Oliver
> 

Thanks. Point taken.

FWIW, I switched to:

find /path/to/dir/ -delete

- which solved my problem safely.

But I think Adrian's original observation - perhaps modified to apply to 
shells in general or execve() in particular, not kernelspace - is still 
valid when we must deal with today's rapidly expanding storage sets.

Short term, perhaps all that is really needed are a few added hints in 
man pages as to the fact that there *are* limits, and what options exist 
(as above) to comfortably work around them.

Regards,

Bill