copying pkgsrc tree around jails on beefy machine - fast on 2nd iteration, a bit slower than first on 3rd iteration

[David Rhodus]

On 1/5/06, Matthew Dillon <dillon at xxxxxxxxxxxxxxxxxxxx> wrote:
> :
> :If the machine has 3-4GB of memory why shouldn't the vnode and buffer
> :cache do some better auto tunning.  If the kernel has a VM system
> :shouldn't the kernel be aware of how much memory is in use for the
> :entire system and make all of the system caches larger ?
>
>     The kernel is fairly conservative with regards to the number of vnodes
>     it allows to be cached because the memory required to hold the vnodes
>     and related VM structures can exceed the kernel's own fairly limited
>     address space.  When doing copy operations on large numbers of small
>     files, the vnode limit will often be hit before the memory limit.
>     (When copying larger files the memory limit is hit before the vnode
>     limit).
>
>     You can adjust the number of vnodes the kernel caches with the
>     'kern.maxvnodes' sysctl.  So, e.g. /usr/pkgsrc has around 105000
>     files so if you want to support caching of a cpdup you would need
>     to be able to cache ~210000 vnodes.  If you make the number too
>     big, however, you run a serious risk of running the kernel out of
>     KVM.
>
>     Unless you actually *NEED* to cache whole file structures for
>     performance reasons in a production environment, I would *NOT*
>     recommend that the system default be changed.
>
>                                         -Matt

Shouldn't the vnode recycler keep the number of vnodes down so that
you don't exhaust the KVM space ?

I did a quick little test on my machine and it looks like you'll hit
this condition in vfs_mount.c if maxvnodes is set too high.
    635                         if (vnlru_nowhere % 100 == 0)
    636                                 printf("vnlru_proc: vnode
recycler stopped working!\n");

-DR