Question regarding IPIQs

[Christiano F. Haesbaert]

Hi there, 

I've been studying dragonfly for a while now and this is something I
could never figure out, I'm pretty sure I'm missing something and I'd
appreciate if someone could give me some pointers. 

Suppose the following cenario.

Cenario A [A]
cpu0: on whichever context
cpu0: spin_lock(&a) (now in a critical section)
cpu0: wakeup(ident1) (ident1 has a sleeper on cpu1)
cpu0: lwkt_send_ipiq3(cpu1, wakeup(ident1))

But ipiq for cpu1 is full, so I enable interrupts, although I'm still on
a critical section, process my own ipiq and spin until cpu1 queue is not
full, but at this point I'm still holding spin_lock a. 

This is the code I'm referring to, lwkt_ipiq:221-229

	while (ip->ip_windex - ip->ip_rindex > MAXCPUFIFO / 4) {
	    if (atomic_poll_acquire_int(&target->gd_npoll)) {
		logipiq(cpu_send, func, arg1, arg2, gd, target);
		cpu_send_ipiq(target->gd_cpuid);
	    }
	    KKASSERT(ip->ip_windex - ip->ip_rindex != MAXCPUFIFO - 1);
	    lwkt_process_ipiq();
	    cpu_pause();
	}

cpu1: ipiq is full and is running an ithread, so it's in a critical section.
cpu1: spin_lock(&a);
cpu1: ??deadlocked??

cpu1 will never process its own ipiq, therefore cpu0 will never make progress,
since cpu0 holds spin_lock a, cpu1 will never make progress as well.

Would this imply that a code that may generate an ipi down the stack, may never
hold a spinlock ? I understand it is very likely that there is no current path
that does a wakeup holding a spinlock.

Furthermore, in this cenario you end up processing your own ipiq even if you
were already on a critical section, can't the ipiq callbacks actually race
against the code that lead to wakeup()->lwkt_sent_ipiq3()->lwkt_process_ipiq() ?

Cenario [B]
Still regarding the paragraph above, what if both cpu0 and cpu1 have their ipiqs
full _and_ come from a code in a critical section, as in:

cpu0: cpu0 own ipiq is full
cpu0: crit_enter()
cpu0: crit_enter()
cpu0: wakeup(ident1) (ident1 has a sleeper on cpu1)
...

cpu1: cpu1 own ipiq is full
cpu1: crit_enter()
cpu1: crit_enter()
cpu1: wakeup(ident0) (ident0 has a sleeper on cpu0)
...

Both realize the other is full and process their own queues, but they break the
atomicity required by the two calls to crit_enter() earlier

What am I missing ? (Sorry for the noise).

Thanks

-- 
Christiano Farina HAESBAERT
Do NOT send me html mail.