SPL vs. Critical Section vs. Mutexing for device synchronisation

[Joerg Sonnenberger]

On Fri, Jun 03, 2005 at 10:43:31AM -0700, Matthew Dillon wrote:
> :(A) Defered interrupt processing via critical sections
> :The primary advantage of critical sections is the simplicity. They
> :are very easy to use and as long as there is no actual interrupt
> :they are very cheap too. As a result they work best when used over
> :short periods of time. Interrupt processing can be implemented either
> :via a FIFO model via interrupt masks.
> 
>     This only works because we hold the BGL.  Without the BGL critical
>     sections cannot be used to protect against interrupts.  Therefore,
>     while the interrupt subsystem is able to depend on critical sections
>     now, IT WON'T BE ABLE TO IN THE FUTURE.  A new MP-SAFE API is needed.

Yes, that's why the current splXXX->crit_enter / splx->crit_exit swap
is partly pointless. It doesn't bring us nearer to our goal, at least
not farther than simply replacing e.g. splimp with a inline function
calling crit_enter would have. It also makes the code harder to read
because we can't now distinguish against what exactly we try to protect
for.

>     At the moment I have created a mutex-like (locked bus cycle) 
>     serialization API that is MP safe.  The abstraction is general enough
>     that we should be able to replace the internals with something 
>     better (aka lockless) in the future.  But right now it's the only
>     thing we have which is inter-cpu safe.

(a) It currently doesn't allow recursion. That's not a big deal from the
implementation point, but makes the use much more difficult. It would be
nice to change this. IMO it makes the usage unncessary complicated.

(b) We still haven't solved the interface issues at all, e.g.
between network stack and interface queues. This is also a point where
we have to be *very* careful to avoid dead lock conditions.

> 
> :The down-side of critical sections is the coarse granularity making
> :it unsuitable for any thing not taking a short period of term. Similiar
> :issues to the mutex concept below apply. It also means we have to be
> :on the same CPU as the interrupt.
> 
>     But this isn't necessarily true.  What really matters here is 
> 
>     * What percentage of the time is a cpu holding a critical section, and
> 
>     * What percentage of interrupts are being delayed due to code being in
>       a critical section.

As I said before, as long as a critical section is short, this doesn't
matter. But device drivers do hold the protection over longer periods
of time. It's not such a big problem for us as it would be e.g. for
FreeBSD, since we have a different hard clock implementation. It's still
something which can seriously hurt.

>     When a device driver is interacting between its upper and lower layers
>     you have to remember that the upper layers can be called from any 
>     process and thus any cpu.  If that layer must interact with a lower
>     layer the only way to do it is to either use a locked bus cycle or
>     to use an IPI message to forward the operation to the same cpu that
>     the interrupt is bound to.

I'm very well aware of that. Depending on the type of driver this is
or is not an issue. From the point of a network driver, the only
interactions with other CPUs are (a) queueing of packets (--> non-blocking
list?) (b) ioctl commands (--> not time-critical, IPI) (c) start processing
(--> should not happen often? IPI).

I'm not sure for the other drivers.

>     I like the idea of per-device interrupt deferal but you have to 
>     realize that in an SMP environment this almost certainly requires
>     the use of locked bus cycle instructions.

I think we have three different types of needs:
(a) critical section (per-cpu)
This just changes / checks a counter and is one conditional + handling
in the interrupt path.
(b) interrupt mask (per-cpu)
Similiar to the critical section. This should be the prefered modus operandi.
Fall-back to (c) for shared interrupts has to be supported.
(c) device mutex (global)
This is the expensive case. We can't avoid bus locked instructions here,
but since we have a fixed target CPU for the interrupt, the local
interrupt mask is a special case of this. We have to check for interrupts
pending when we want to unlock the mutex and process them.

Joerg