new lwbuf api and mpsafe sf_bufs
dillon at apollo.backplane.com
Tue Mar 10 17:31:11 PDT 2009
:The basic thought behind the new lwbuf api is that sf_buf's have been
:abused more and more as time has passed for purposes that don't
:necessarily match the original intent. Most uses of sf_bufs beyond
:sendfile are exceedingly short-lived, short enough that they would be
:better served by cpu-local caches of kvm space, even if that means a
:bit of overlap now and again. lwbuf attempts to match the semantics of
:xio, with one addition as suggested by Simon, a sanity check and tlb
:invalidation in lwbuf_kva if the lwbuf strays to some other cpu.
:Currently lwbuf allocates a page of kvm at a time on objcache object
:construction and stores the kvm with the object in objcache when it is
:returned to the cache.
:sf_bufs, which are still most applicable to sendfile, have been
:simplified on top of the lwbuf api, converted to objcache and the
:global hash has been protected with a spinlock.
:XIO and exec have been modified to use lwbuf's.
Hmm. If I understand this correctly you are removing SFB_CPUPRIVATE
from the SFBUF API and splitting the entire set of API procedures out
for private-cpu use into a new LWBUF API?
It looks like you moved the cpumask mmu syncing to lwbuf_kva(),
but I think I see a bug in lwbuf_alloc().
From my reading of the code lwbuf_alloc() is not able to reuse
the KVA mappings for previously cached LWBUF's because it is
unconditionally changing the vm_page (lwb->m = m; pmap_kenter_quick...).
Because of the pmap change, lwb->cpumask would have to be set to
gd->gd_cpumask, NOT or'd with the previous mask.
i.e. the bug is:
lwb->cpumask |= gd->gd_cpumask;
It needs to be:
lwb->cpumask = gd->gd_cpumask;
If you want to separate out private mappings like that I think the
LWBUFs have to cached just like they are with SFBUFs. I like the
use of the objcache but I think the LWBUF code needs to retain
a sysctl-programmable number of lwb elements in a RBTREE and try to
reuse the mappings. i.e. it needs to be a bit more sophisticated
then your current design.
<dillon at backplane.com>
More information about the Submit