Our SMP implementation scalability

[Bill Huey (hui)]

On Thu, Jan 18, 2007 at 09:26:32AM +1100, Dmitri Nikulin wrote:
> To memory-quote Matt, it should scale really well in theory, because
> most things are naturally lockless and so aren't co-dependent. FreeBSD
> scales badly (though improving) because there are complex locks
> everywhere, and even in the best case there's still a lot of data
> structures that end up making each CPU take turns. There's no way to
> make that scale well - the only clean solution is with replication and
> messaging like DragonFly is gaining. Linux has similar problems in
> theory, but has had so much more manpower applied to optimization that

Linux and dfBSD lock philosophies are the same but expressed through
different techiques as I understand it from my discussion with Matt a
while back. I know aspects of Linux locking to a certain degree. Linux
emphasized per CPU data locality and shallow locking chains.

> it performs very well anyway. FreeBSD is in a difficult position of
> not having an architecture that's easy to develop and optimize by few
> people (like DragonFly), and not having the manpower to fix
> consequences of a complex architecture (like Linux). That's not to say
> it's a bad system for low end SMP, but I doubt we'll ever see it
> utilize 1024-core servers very well.

FreeBSD's shared/exclusive lock is useless since it's still effectively
bounded to serializing atomic operation if you have short reader paths.
It's effectively a spinlock at that moment. Linux uses RCU to deal with
that problem in a lockless fashion yet still maintaining a highly
concurrent read-mostly behavior. This ignores also their preemptible
kernel abilities that basically make implementing RCU into their system
difficult because of quiescence calculations.

Linux gets it's solid MP performance by bruteforce refactoring of
problematic sections which is relatively easy give that it's a bit
under designed. dfBSD parallels Linux in that it's attacking the
problems through a combination of refactoring and locking as I
understand it from discussion about the dfBSD effort.

> I'm interested to know how NetBSD's SMP is going to evolve, because I
> haven't heard much about it at all. What they have now seems to be the
> same giant lock that FreeBSD 4 had, which means the kernel won't
> scale. However, in my benchmarks on a dual core, the userland M:N
> threading 'scales' as well as it possibly could for two cores (at
> least 2x throughput, compared to the same user threads on one kernel
> thread).

If you don't have to hit the kernel, userspace threading should scale
well for systems that don't need to call into kernel space often. You're
screwed if you have to though.

bill