new lwbuf api and mpsafe sf_bufs

[Samuel Greear]

On Mon, Mar 9, 2009 at 12:15 PM, Aggelos Economopoulos
<aoiko at cc.ece.ntua.gr> wrote:

> No time to comment atm, but a few questions to get a discussion started:
> a) does this make any measurable difference in microbenchmarks?

This isn't the easiest thing to microbenchmark, but see the bottom of
this mail for some timings against the exec path.

> b) have you tested it on mp?

I have now.

> c) is the extra complexity worth it? (just glanced at the patch, looks
> simple enough to me)

IMO this simplifies the code. sf_buf_alloc() in particular was
somewhat confusing internally, keeping used items on the freelist,
etc. I think ideally, assuming this goes in, all consumers of sf_*
would switch to being lwbuf consumers, and sf_* could move back into
uipc_syscalls.c (or the whole sendfile mess could be moved into
uipc_sendfile.c).

In terms of complexity, I had thought about using a per-cpu hashtable
for lwbuf, but upon consideration I didn't think it was worth the
added complexity or memory consumption. A middle ground might be a
simple open hashing scheme against small tables, but each lwbuf would
still have to track the cpu that owned it and ipi a deallocation if it
wandered to another cpu. This would likely be effective on
microbenchmarks but I am not so sure about real-world. You would be
trading the costs of pmap_kenter_quick() for a hash lookup, and while
the hash lookup is going to be faster that may be outweighed by the
extra overhead elsewhere (free) and in most cases (not
microbenchmarks) what you seek isn't going to be there anyway.
Currently, sf_bufs when used in contexts apart from sendfile(2) are
very short-lived. They are typically allocated, used and then free'd
immediately. If one wanted to do something more complicated I would
suggest never freeing the lwbufs except under memory pressure or in a
LRU fashion on allocation. Sharing the sf_bufs makes a lot of sense
for sendfile(2) itself, because webservers, ftp servers, etc. that
make most effective use of it typically have a fairly small common
working dataset.

> d) do people think the extra kva usage is going to be an issue?

I would suggest that in most cases this will use less kva. Not
changing the defaults you have 6656 pages dedicated to sf_bufs w/ 2GB
of memory. Typically I would expect you would see no more than a
couple of hundred pages in use unless you are using sendfile(2).

>
> Hopefully I'll have more to ask tomorrow.
>
> Aggelos
>

#!/bin/sh

if [ "$1" = "sub" ]
then
   for i in `seq 3000`
   do
       sh &
   done
   wait
   exit
fi

for i in `seq 10`
do
   time $0 sub
done


SMP on T5450 (1.66GHz Core2Duo)

Unpatched
      12.12 real        21.14 user         1.59 sys
      12.14 real        21.42 user         1.48 sys
      12.12 real        21.34 user         1.52 sys
      12.13 real        21.06 user         1.71 sys
      12.16 real        21.31 user         1.61 sys
      12.14 real        21.14 user         1.57 sys
      12.16 real        21.37 user         1.72 sys
      12.17 real        21.29 user         1.68 sys
      12.15 real        21.31 user         1.65 sys
      12.19 real        21.07 user         1.61 sys

Patched
      12.09 real        21.37 user         1.45 sys
      12.07 real        21.27 user         1.51 sys
      12.08 real        21.17 user         1.56 sys
      12.11 real        21.31 user         1.51 sys
      12.11 real        21.44 user         1.54 sys
      12.10 real        21.33 user         1.49 sys
      12.10 real        21.18 user         1.70 sys
      12.12 real        21.45 user         1.53 sys
      12.12 real        21.37 user         1.64 sys
      12.14 real        21.32 user         1.64 sys

UP on T5450

Unpatched
      22.33 real        20.79 user         0.81 sys
      22.36 real        20.87 user         0.82 sys
      22.35 real        20.57 user         0.87 sys
      22.38 real        20.71 user         0.88 sys
      22.39 real        20.82 user         0.84 sys
      22.37 real        20.81 user         0.89 sys
      22.38 real        20.84 user         0.80 sys
      22.40 real        20.78 user         0.87 sys
      22.40 real        20.71 user         0.90 sys
      22.40 real        20.82 user         0.81 sys

Patched
      22.34 real        20.75 user         0.82 sys
      22.34 real        20.78 user         0.78 sys
      22.38 real        20.75 user         0.89 sys
      22.37 real        20.82 user         0.75 sys
      22.38 real        20.91 user         0.83 sys
      22.38 real        20.85 user         0.76 sys
      22.40 real        20.82 user         0.95 sys
      22.39 real        20.85 user         0.86 sys
      22.40 real        20.85 user         0.82 sys
      22.39 real        20.90 user         0.85 sys