[DragonFlyBSD - Bug #3475] x86_64: a crash dump can stall in smp_invltlb() and never finish
tuxillo
bugtracker-admin at leaf.dragonflybsd.org
Sun Nov 22 06:07:37 PST 2026
Issue #3475 has been updated by tuxillo.
The attached cpustop-xinvltlb.diff replaces cpustop-complete.diff. A stopped cpu keeps taking Xinvltlb: Xcpustop clears gd_xinvaltlb and the cpu's smp_smurf_mask bit when it parks, and puts the flag back on restart. Under LOOPRECOVER, a cpu that has waited a second on a stopped originator sends it the Xinvltlb again.
On a 12-thread Coffee Lake desktop, on master 64a78aecd3 with this patch, #3476 and #3477, I panicked it 5 times under the attached tlbchurn.c load. I left out the dump change proposed above, so the dump still calls smp_invltlb() for every chunk. 5 of 5 minidumps completed, with no smp_invltlb() or smp_inval_intr() warnings. kgdb on three of the dumps showed cpus stopped inside smp_inval_intr() in two of them (cpu 4 in one, cpus 3 and 9 in the other, cpu 9 in its own shootdown), and they still answered. With the dump change added as well (the set as it would be committed), 5 of 5 completed again, and 3 of those 5 caught cpus inside smp_inval_intr(). In each of those three, one stopped cpu printed a single "smp_inval_intr N, WARNING blocked >1 sec" while waiting on an originator frozen in the middle of its own command (for example "pm_inval_intr: WARNING, taking too long cpus=1->9 done=00000002 mask=00000000 mode=0"). Nothing waits on it there, so the dump finished; the message buffer was saved about a second into the dump, so I can't tell whether the re-send to the stopped originator cleared it later.
A test module that runs stop_cpus(), smp_invltlb() and restart_cpus() in a loop passed 1000 and 5000 rounds under the same load there (7 rounds caught a cpu inside smp_inval_intr(), smp_invltlb() max 39 us), and 5000 rounds in an 8-vCPU QEMU guest, where ddb's continue also came back 20 of 20 times.
Still open: after the 5000-round run the desktop lost its network and keyboard and kept its cpus busy until I reset it, and in the guest a run without the shootdowns once panicked with "Bad link elm" in _tsleep_remove(). I suspect the IPI queue messages a stopped cpu runs in lwkt_smp_stopped(), which can catch per-cpu state half changed, but I haven't shown it. Normal use does nowhere near that many stop/restart cycles.
----------------------------------------
Bug #3475: x86_64: a crash dump can stall in smp_invltlb() and never finish
http://bugs.dragonflybsd.org/issues/3475#change-14974
* Author: tuxillo
* Status: New
* Priority: Normal
* Target version: 6.6
* Start date: 2026-11-21
----------------------------------------
As of master e2619a112f, a crash dump on x86_64 can stop making progress and never finish. The minidump (blk_write() in minidump_machdep.c) and the full dump (cb_dumpdata() in dump_machdep.c) both call smp_invltlb() after mapping each chunk into crashdumpmap. The other cpus are stopped for the panic, and when one of them doesn't answer the IPI, smp_invltlb() waits invlpg_timeout seconds (10, or 60 in a VM), prints "smp_invltlb: giving up", and the next chunk waits again. In a QEMU guest that was a minute per chunk, so a 570 MB dump would have taken more than a day.
The cross-cpu flush isn't needed: pmap_kenter_quick() already does invlpg on the current cpu, and only the dumping cpu reads crashdumpmap. The calls came in with fb8345e665 (2011) to replace pmap_kenter()'s IPI-queue invalidation. FreeBSD's amd64 dump code doesn't flush there at all.
The attached patch flushes only the dumping cpu (cpu_invltlb()) in both places, and fixes the pmap_kenter_temporary() comment that told callers to use smp_invltlb().
To test it I panicked with sysctl debug.panic=1 while other processes kept the cpus busy with TLB shootdowns. In a 4-vCPU QEMU guest, 2 of 6 minidumps stalled before the patch and 10 of 10 completed after it. On a 12-thread machine with 32 GB, with the attached tlbchurn.c as the load, 5 of 5 minidumps and 3 of 3 full dumps completed with the patch; I didn't run an unpatched kernel there. In two of those minidumps another cpu spent the whole dump in smp_inval_intr() waiting for a cpu that never answered, which is what the old code would have waited on.
>From reading the code, the cpu that never answers probably took the stop IPI while inside smp_inval_intr() and parked there. That's a separate problem, and this patch doesn't touch it.
---Files--------------------------------
tlbchurn.c (3.64 KB)
cpustop-complete.diff (1.77 KB)
cpustop-xinvltlb.diff (3.68 KB)
--
You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: http://bugs.dragonflybsd.org/my/account
More information about the Bugs
mailing list