new lwbuf api and mpsafe sf_bufs

[Matthew Dillon]

:http://gitweb.dragonflybsd.org/~sjg/dragonfly.git/commitdiff/caac5766643042650328f83531dc8fd767df5f60
:
:The basic thought behind the new lwbuf api is that sf_buf's have been
:abused more and more as time has passed for purposes that don't
:necessarily match the original intent. Most uses of sf_bufs beyond
:sendfile are exceedingly short-lived, short enough that they would be
:better served by cpu-local caches of kvm space, even if that means a
:bit of overlap now and again. lwbuf attempts to match the semantics of
:xio, with one addition as suggested by Simon, a sanity check and tlb
:invalidation in lwbuf_kva if the lwbuf strays to some other cpu.
:Currently lwbuf allocates a page of kvm at a time on objcache object
:construction and stores the kvm with the object in objcache when it is
:returned to the cache.
:
:sf_bufs, which are still most applicable to sendfile, have been
:simplified on top of the lwbuf api, converted to objcache and the
:global hash has been protected with a spinlock.
:
:XIO and exec have been modified to use lwbuf's.
:
:Please comment/hate/review/test
:
:Thanks,
:Sam

    Hmm.  If I understand this correctly you are removing SFB_CPUPRIVATE
    from the SFBUF API and splitting the entire set of API procedures out
    for private-cpu use into a new LWBUF API?

    It looks like you moved the cpumask mmu syncing to lwbuf_kva(),
    but I think I see a bug in lwbuf_alloc().

    From my reading of the code lwbuf_alloc() is not able to reuse
    the KVA mappings for previously cached LWBUF's because it is 
    unconditionally changing the vm_page (lwb->m = m; pmap_kenter_quick...).
    Because of the pmap change, lwb->cpumask would have to be set to
    gd->gd_cpumask, NOT or'd with the previous mask.

    i.e. the bug is:

	lwb->cpumask |= gd->gd_cpumask;

    It needs to be:

	lwb->cpumask = gd->gd_cpumask;

    If you want to separate out private mappings like that I think the
    LWBUFs have to cached just like they are with SFBUFs.  I like the
    use of the objcache but I think the LWBUF code needs to retain 
    a sysctl-programmable number of lwb elements in a RBTREE and try to
    reuse the mappings.  i.e. it needs to be a bit more sophisticated
    then your current design.

					-Matt
					Matthew Dillon 
					<dillon at backplane.com>