[DragonFlyBSD - Bug #3476] (In Progress) x86_64: restart_cpus() doesn't wait for the cpus and can spin forever

tuxillo bugtracker-admin at leaf.dragonflybsd.org
Sun Nov 22 04:38:49 PST 2026


Issue #3476 has been updated by tuxillo.

Status changed from New to In Progress

Test results, from a 4-vCPU QEMU guest with the attached restartwait.c running 5000 rounds of stop_cpus() and restart_cpus(stopped_cpus) on an idle system:

<pre>
before: restartwait: 5000 rounds: restart_cpus() returned with cpus still stopped in 5000 rounds (15000 cpus); restart_cpus() max 0 us; last cpu left Xcpustop max 8 us after it returned
after:  restartwait: 5000 rounds: restart_cpus() returned with cpus still stopped in 0 rounds (0 cpus); restart_cpus() max 152 us; last cpu left Xcpustop max 0 us after it returned
</pre>

The guest has no MONITOR, so a stopped cpu waits in cpu_halt(), which never returns. Both kernels had the fix from #3477 for that.

I saw the spin forever once, in an 8-vCPU guest running 5000 rounds of stop_cpus(), smp_invltlb() and restart_cpus() under TLB shootdown load: the cpu running the test sat in restart_cpus() and the other cpus then hung waiting on it. That kernel also had the stopped-cpu Xinvltlb change discussed in #3475; without it the same load hangs on that problem first. With this patch added, the same test passed 5000 rounds. That's one run each, so it's an example, not a rate.

I haven't run it on hardware yet.

----------------------------------------
Bug #3476: x86_64: restart_cpus() doesn't wait for the cpus and can spin forever
http://bugs.dragonflybsd.org/issues/3476#change-14973

* Author: tuxillo
* Status: In Progress
* Priority: Normal
* Target version: 6.6
* Start date: 2026-11-22
----------------------------------------
As of master e2619a112f, restart_cpus() on x86_64 returns before the cpus it restarts have left Xcpustop, and if one of them is quick it never returns at all. It is called by ddb's continue and by dump_reactivate_cpus(), which the dm_crypt dump path uses.

restart_cpus() sets started_cpus and then waits with @while (CPUMASK_CMPMASKNEQ(stopped_cpus, map))@. Both callers pass stopped_cpus itself as map, so the loop only checks whether stopped_cpus has changed since the call. If no cpu has cleared its bit yet, the loop ends at once. If a cpu clears its bit before the first test, stopped_cpus never equals map again, since bits are only cleared from then on, and the caller spins in cpu_pause() for good. The window is a few instructions, wider when the caller takes an interrupt there or, in a VM, when its vCPU is descheduled.

The loop came in with c07315c47f (2014), the cpumask_t conversion. Before that it was @while ((stopped_cpus & map) != 0)@.

The attached patch restores that test, on the cpus that were signalled: @while (CPUMASK_TESTMASK(stopped_cpus, mask))@.

---Files--------------------------------
restartwait.c (3.86 KB)
0001-x86_64-make-restart_cpus-wait-for-the-cpus-to-restar.patch (1.5 KB)


-- 
You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: http://bugs.dragonflybsd.org/my/account


More information about the Bugs mailing list