Question regarding IPIQs
dillon at apollo.backplane.com
Wed Aug 28 14:35:12 PDT 2013
:I've been studying dragonfly for a while now and this is something I
:could never figure out, I'm pretty sure I'm missing something and I'd
:appreciate if someone could give me some pointers.
:Suppose the following cenario.
:Cenario A [A]
:cpu0: on whichever context
:cpu0: spin_lock(&a) (now in a critical section)
:cpu0: wakeup(ident1) (ident1 has a sleeper on cpu1)
:cpu0: lwkt_send_ipiq3(cpu1, wakeup(ident1))
Illegal. You should not call wakeup() with a spin lock held.
You most definitely should not be calling any IPIQ functions with
a spin lock held.
You have two solutions here. First, use a regular lock or a token
instead of a spin lock. Second, flag the condition and issue the
wakeup() or IPIQ function after releasing the spin lock.
In DragonFly, spin locks are only intended to be used for very
short sequences of self-contained or mostly self-contained code.
:But ipiq for cpu1 is full, so I enable interrupts, although I'm still on
:a critical section, process my own ipiq and spin until cpu1 queue is not
:full, but at this point I'm still holding spin_lock a.
Critical sections do not disable interrupts, they simply prevent normal
preemptive processing (preemption by an interrupt thread or IPI)
from occuring. Interrupts will still occur, but they will simply flag
the pending condition in mycpu->gd_reqflags, EOI the interrupt, and
immediately return. The interrupt is then processed normally when the
critical section is exited.
However, if you issue an IPIQ operation inside a critical section which
would otherwise be forced to spin-wait due to a full IPIQ queue, it
will synchronously execute any incoming IPIs while it is spin-waiting
to avoid deadlocking the system.
:Both realize the other is full and process their own queues, but they break the
:atomicity required by the two calls to crit_enter() earlier
:What am I missing ? (Sorry for the noise).
Programs which enter critical sections can always assume that there will
be no asynchronous atomicy breakage, but there are no guarantees regarding
synchronous atomicy. For example, you can tsleep() just fine while
holding a critical section. Nominal IPIQ operations fall under the same
rules. Incoming IPIs will not be executed asynchronously but they can
still be executed synchronously if your outgoing IPI operation would
otherwise be forced to spin-wait.
<dillon at backplane.com>
More information about the Kernel