SMP performance on drgonfly

[Kris Kennaway]

Hi guys,

For a while now I have been interested in the alternative ideas about
SMP serialization and concurrency that are being pursued by the
dragonfly project, so I thought it would be interesting to see how
they have played out over the past 4 years of development.

One problem I ran into is that dragonfly is not able to boot on many
recent (particularly AMD) systems that I tried, because they require
ACPI for PCI bus enumeration and the older FreeBSD 4.x-derived code
base does not do this.

Finally I did find an Intel 8-core system that is able to run
Dragonfly.  The 1.9 kernel I first tried hung during boot, but 1.8
boots successfully.  I recompiled the kernel for SMP and disabled
INVARIANTS and the I[345]86_CPU options.  /etc/malloc.conf was set to
aj.  These seemed to be the only obvious tuning options I could find,
but perhaps I missed something.

The benchmark I ran was the sysbench SQL database benchmark in
read-only mode, which seems to be a good SMP test case.  I ran it
against mysql 5.0.37 with some config file tuning (I can go into
details if required).  The system has 4GB of RAM (actually 8GB, but
the rest is ignored on i386 without PAE), so the database fits in
memory and no disks are accessed after the initial warmup run.  I
compared Dragonfly 1.8 with both the libc_r and libthread_xu thread
libraries to FreeBSD 7.0 on the same hardware.

I was somewhat surprised by the results:

  http://obsecurity.dyndns.org/dfly.png

The libc_r result is to be expected: a userland threading library will
not give any scaling improvements as the client workload increases.
The slight peaks at higher loads are probably just random variation.

However I was surprised at the libthread_xu results.  There is a very
small performance increase up to 3 clients, but the peak performance
is only about 30% higher than the baseline.  I confirmed that SMP is
active and processes are being scheduled on all 8 CPUs, but the system
is basically never performing more than ~1.3 CPU's of work.

Can anyone suggest what might be wrong?

Kris