new lwbuf api and mpsafe sf_bufs

[Aggelos Economopoulos]

Samuel Greear wrote:
> http://gitweb.dragonflybsd.org/~sjg/dragonfly.git/commitdiff/caac5766643042650328f83531dc8fd767df5f60
> 
> The basic thought behind the new lwbuf api is that sf_buf's have been
> abused more and more as time has passed for purposes that don't
> necessarily match the original intent. Most uses of sf_bufs beyond
> sendfile are exceedingly short-lived, short enough that they would be
> better served by cpu-local caches of kvm space, even if that means a
> bit of overlap now and again. lwbuf attempts to match the semantics of
> xio, with one addition as suggested by Simon, a sanity check and tlb
> invalidation in lwbuf_kva if the lwbuf strays to some other cpu.
> Currently lwbuf allocates a page of kvm at a time on objcache object
> construction and stores the kvm with the object in objcache when it is
> returned to the cache.
> 
> sf_bufs, which are still most applicable to sendfile, have been
> simplified on top of the lwbuf api, converted to objcache and the
> global hash has been protected with a spinlock.
> 
> XIO and exec have been modified to use lwbuf's.
> 
> Please comment/hate/review/test

No time to comment atm, but a few questions to get a discussion started:
a) does this make any measurable difference in microbenchmarks?
b) have you tested it on mp?
c) is the extra complexity worth it? (just glanced at the patch, looks
simple enough to me)
d) do people think the extra kva usage is going to be an issue?

Hopefully I'll have more to ask tomorrow.

Aggelos