SPL vs. Critical Section vs. Mutexing for device synchronisation

[Thomas E. Spanjaard]

Matthew Dillon wrote:
    This only works because we hold the BGL.  Without the BGL critical
    sections cannot be used to protect against interrupts.  Therefore,
    while the interrupt subsystem is able to depend on critical sections
    now, IT WON'T BE ABLE TO IN THE FUTURE.  A new MP-SAFE API is needed.
As I understand it, Joerg's major point is that until we have such an 
MP-safe API, we shouldn't change the running system yet, but instead do 
it at once once the new API has been tested.

    Frankly I don't see how we can possibly avoid the use of a 
    locked bus cycle instruction for either case.  The only way to truely
    avoid locked bus cycles is through cpu localization (aka using IPIs
    to execute all related operations on the same cpu).
I am in favour of this. From the performance point of view, servicing 
interrupts on another machine is very degrading, and I assume that DFly 
isn't supposed to be run on just SMP machines sharing APICs, but also on 
clusters of machines with small to broad interconnects. Also, any 
related activity, i.e. pushing packets through the network stack, should 
be done on the CPU which has the shortest path to the device in question.

    The use of cpu localization is one of DragonFly's major goals.  I
    *WANT* to use a cpu localization mechanism whenever possible, and indeed
    we are using such a mechanism in our networking code with incredible
    results.  But that doesn't mean that cpu localization works in every
    case.
Oh wait. I should have read this first :). Can you give me an example of 
a case in which CPU localization doesn't work?

    In anycase, these are all very complex issues.  Even if we come up with
    a clean solution, we can't go from step A to step Z in a single step. 
Even though -Development is already here, isn't it an option to run a 
separate tree just for this major change, so we can compare them once 
this is 'done'?

Cheers,
		-- Thomas E. Spanjaard
		   tgen at xxxxxxxxxxxxx