Question regarding IPIQs

[Matthew Dillon]

:Hi there, 
:
:I've been studying dragonfly for a while now and this is something I
:could never figure out, I'm pretty sure I'm missing something and I'd
:appreciate if someone could give me some pointers. 
:
:Suppose the following cenario.
:
:Cenario A [A]
:cpu0: on whichever context
:cpu0: spin_lock(&a) (now in a critical section)
:cpu0: wakeup(ident1) (ident1 has a sleeper on cpu1)
:cpu0: lwkt_send_ipiq3(cpu1, wakeup(ident1))

    Illegal.  You should not call wakeup() with a spin lock held.
    You most definitely should not be calling any IPIQ functions with
    a spin lock held.

    You have two solutions here.  First, use a regular lock or a token
    instead of a spin lock.  Second, flag the condition and issue the
    wakeup() or IPIQ function after releasing the spin lock.

    In DragonFly, spin locks are only intended to be used for very
    short sequences of self-contained or mostly self-contained code.

:But ipiq for cpu1 is full, so I enable interrupts, although I'm still on
:a critical section, process my own ipiq and spin until cpu1 queue is not
:full, but at this point I'm still holding spin_lock a. 

    Critical sections do not disable interrupts, they simply prevent normal
    preemptive processing (preemption by an interrupt thread or IPI)
    from occuring.  Interrupts will still occur, but they will simply flag
    the pending condition in mycpu->gd_reqflags, EOI the interrupt, and
    immediately return.  The interrupt is then processed normally when the
    critical section is exited.

    However, if you issue an IPIQ operation inside a critical section which
    would otherwise be forced to spin-wait due to a full IPIQ queue, it
    will synchronously execute any incoming IPIs while it is spin-waiting
    to avoid deadlocking the system.

:Both realize the other is full and process their own queues, but they break the
:atomicity required by the two calls to crit_enter() earlier
:
:What am I missing ? (Sorry for the noise).
:
:Thanks

    Programs which enter critical sections can always assume that there will
    be no asynchronous atomicy breakage, but there are no guarantees regarding
    synchronous atomicy.  For example, you can tsleep() just fine while
    holding a critical section.  Nominal IPIQ operations fall under the same
    rules.  Incoming IPIs will not be executed asynchronously but they can
    still be executed synchronously if your outgoing IPI operation would
    otherwise be forced to spin-wait.

					-Matt
					Matthew Dillon 
					<dillon at backplane.com>