[DragonFlyBSD - Bug #3421] Thread stuck in vmiopg randomly.
arcade@b1t.name
bugtracker-admin at leaf.dragonflybsd.org
Fri Sep 18 02:33:18 PDT 2026
Задача #3421 була оновлена arcade at b1t.name.
Файл photo_8_2026-09-18_12-31-58.jpg доданий
Файл photo_7_2026-09-18_12-31-58.jpg доданий
Файл photo_6_2026-09-18_12-31-58.jpg доданий
Файл photo_5_2026-09-18_12-31-58.jpg доданий
Файл photo_4_2026-09-18_12-31-58.jpg доданий
Файл photo_3_2026-09-18_12-31-58.jpg доданий
Файл photo_2_2026-09-18_12-31-58.jpg доданий
Файл photo_1_2026-09-18_12-31-58.jpg доданий
0k, update.
So I tried updating everything from recent source just to be sure what am I testing. This generally means I got a lot of issues (each resulting in me forcing reboot due to stuck stuff). I also paired wit LLM (Muse Spark, Qwen 2.8) to get more debug data:
--- a/sys/kern/vfs_bio.c
+++ b/sys/kern/vfs_bio.c
@@ -59,6 +59,7 @@
#include <sys/buf2.h>
#include <sys/spinlock2.h>
#include <vm/vm_page2.h>
+#include <sys/namecache.h>
#include "opt_ddb.h"
---
static void
vfs_vmio_release(struct buf *bp)
{
int i;
vm_page_t m;
for (i = 0; i < bp->b_xio.xio_npages; i++) {
m = bp->b_xio.xio_pages[i];
bp->b_xio.xio_pages[i] = NULL;
/*
* We need to own the page in order to safely unwire it.
*
* Debug bug 3421: the busy print below fires only on real
* hangs (no spam in practice). Dump waiter stack, sleeper
* census and file identity right here, no manual db needed.
*/
if (m->busy_count & PBUSY_LOCKED) {
struct proc *p = curthread->td_proc;
kprintf("vmiopg-wait: m=%p busy=%08x obj=%p type=%d "
"pindex=%jx phys=%jx waiter=%d (%s)\n",
(void *)m, m->busy_count,
(void *)m->object,
m->object ? (int)m->object->type : -1,
(uintmax_t)m->pindex,
(uintmax_t)m->phys_addr,
p ? p->p_pid : -1,
p ? p->p_comm : "-");
if (m->object && m->object->type == OBJT_VNODE &&
m->object->handle) {
struct vnode *vp = m->object->handle;
struct namecache *ncp =
TAILQ_FIRST(&vp->v_namecache);
kprintf("vmiopg-file: mount=%s name=%s vtype=%d size=%jx\n",
vp->v_mount ? vp->v_mount->mnt_stat.f_mntonname : "?",
ncp && ncp->nc_name ? ncp->nc_name : "?",
vp->v_type,
(uintmax_t)m->object->size)
);
kprintf("vmiopg-file2: parent=%s vtag=%d usecount=%d\n",
pdp && pdp->nc_name ? pdp->nc_name : "?",
vp->v_tag, vp->v_usecount);
}
#ifdef VM_PAGE_DEBUG
kprintf("holder=%s:%d\n",
m->busy_func ? m->busy_func : "?",
m->busy_line);
#endif print_backtrace(12);
allproc_scan(&vmiopg_dump_sleepers, NULL, 0);
}
vm_page_busy_wait(m, FALSE, "vmiopg");
--- one new function
static int
vmiopg_dump_sleepers(struct proc *p, void *arg __unused)
{
struct lwp *lp;
static const char *skip[] = {
"wait", "idle", "pause", "ttyin", "piperd", "kqread",
"nanslp", "udevw", "autofscv", "umtxsl", NULL
};
RB_FOREACH(lp, lwp_rb_tree, &p->p_lwp_tree) {
struct thread *td = lp->lwp_thread;
int i;
if (td == NULL || td->td_wchan == NULL)
continue;
for (i = 0; skip[i]; ++i) {
if (td->td_wmesg && !strcmp(td->td_wmesg, skip[i]))
break;
}
if (skip[i])
continue;
kprintf("vmiopg-sleeper: pid=%d (%s) wmesg=%s wchan=%p\n",
p->p_pid, p->p_comm,
td->td_wmesg ? td->td_wmesg : "-",
td->td_wchan);
}
return 0;
}
^^^^ this is final code, but I yet never got complete dump from it. Some lags and output added.
Sep 16 16:53:16 slipper kernel: vmiopg-wait: m=0xfffff8007c1fb980 busy=80000000 obj=0xfffff80687ad0800 type=2 pindex=6c4 phys=781f53000 waiter=-1 (-)
Sep 16 16:58:03 slipper kernel: vmiopg-wait: m=0xfffff80040730c00 busy=80000000 obj=0xfffff80687ac6a80 type=2 pindex=6c4 phys=c5f8000 waiter=-1 (-)
Sep 16 17:08:34 slipper kernel: vmiopg-wait: m=0xfffff80040872d80 busy=80000000 obj=0xfffff80687ac8ec0 type=2 pindex=6c4 phys=ee3b000 waiter=13278 (cc1)
Sep 16 17:18:54 slipper kernel: vmiopg-wait: m=0xfffff8004086a180 busy=80000000 obj=0xfffff80687ab0bc0 type=2 pindex=6c4 phys=ed23000 waiter=-1 (-)
Sep 16 17:26:29 slipper kernel: vmiopg-wait: m=0xfffff800408f6b00 busy=80000000 obj=0xfffff80687aa6a80 type=2 pindex=6c4 phys=feb6000 waiter=-1 (-)
Sep 16 17:33:46 slipper kernel: vmiopg-wait: m=0xfffff8007d1bdd00 busy=80000000 obj=0xfffff80687ad0bc0 type=2 pindex=6c4 phys=7a179a000 waiter=-1 (-)
Sep 16 17:42:34 slipper kernel: vmiopg-wait: m=0xfffff8004078ae80 busy=80000000 obj=0xfffff80687ad0080 type=2 pindex=6c4 phys=d13d000 waiter=-1 (-)
Sep 16 17:54:58 slipper kernel: vmiopg-wait: m=0xfffff80040737680 busy=80000000 obj=0xfffff80687ad06c0 type=2 pindex=6c4 phys=c6cd000 waiter=4899 (make)
Sep 16 17:58:41 slipper kernel: vmiopg-wait: m=0xfffff8004072f300 busy=80000000 obj=0xfffff80687ac6d00 type=2 pindex=6c4 phys=c5c6000 waiter=-1 (-)
The weird addition is bug was easy to trigger once, but I recall it was triggered twice one time only through all of reboots.
Also when I succeeded to bump kernel over libthread stuff from tomorrow… everything went normal. I built a world a few times on last patch version, built ~300 ports - bug is not manifesting at all. Still need to update initrd to latest version though.
I'll try to unpatch and retest, maybe patch itself already triggers something…
----------------------------------------
Bug #3421: Thread stuck in vmiopg randomly.
http://bugs.dragonflybsd.org/issues/3421#change-14699
* Автор: arcade at b1t.name
* Статус: New
* Пріоритет: Normal
* Версія: 6.6
* Початок: 2026-09-15
----------------------------------------
Might be related to #3392.
I got some weird process hangs for a long time, with last crypto upgrade they became hard, and easily repeatable. Previously it was mostly Telegram doing parallel disk IO. Currently maul -j16 buildworld is a way to trigger, though sometimes any ocassional process can borg. When we have a tainted process everything else continue to work, only one process is stuck. Sometimes another process can stuck too.
I can try taking testing further, probably rebuilding kernel with KTR if that helps…
AI slop incoming (muse spark 1.3):
---
Subject: Unkillable processes stuck in vmiopg/vmpfw under parallel build on HAMMER2: leaked busy page, no I/O in flight
Environment: DragonFly 6.5-DEVELOPMENT, reproduced on 9b3eb087a6 (Apr 2026) and 4bb22ffcf2 (Sep 2026, current master). Notebook, 16 CPUs, 32 GB RAM, NVMe. Root on dm-crypt AES-XTS (AESNI) + HAMMER2, swap also encrypted. No KDB/DDB custom config at first; debugger present for later captures.
Reproducer: make -j16 buildworld-class load (buildincludes, libc). Single leaf process (observed: make, as, cc1) wedges in uninterruptible sleep; parents sit in wait; rest of the system stays alive with gigabytes free and zero swap use. kill -9 does not kill it. Sometimes recovers on its own, usually needs reboot. Lower -j reduces frequency.
DDB traces (two separate hangs):
1. sys_read → hammer2_read_file → cluster_readx → getblk → getnewbuf → vfs_vmio_release → vm_page_busy_wait → tsleep(vmiopg), faulting file read (e.g. .depend).
2. sys_write → hammer2_write_file → breadnx → getblk → getnewbuf → vfs_vmio_release → vm_page_busy_wait, writer (as writing .o).
Third hang: cc1 in vmpfw while a h2xop-ROOT worker thread sits in vmiopg permanently.
Ruled out: disk/dm-crypto layers are healthy — dd through /dev/mapper/root gives 321 MB/s during the hang, iostat shows zero activity on both rootd0 and nvme0, both mpipe_dm_crypt threads idle in wait, all h2xop threads idle, dmesg has no dm/HAMMER2 errors, show tokens shows no token deadlock. Read-flood through the same dm device does not unstick the waiter. No thread in the system holds the page: waiter waits on a PG_BUSY page with no I/O in flight and no owner — looks like a leaked busy bit in the buffer-cache page recycling path (getnewbuf reusing a buffer whose backing page never gets unbusied).
Side issue: entering DDB and running ps hard-rebooted the machine once (no dump captured). trace <thread_addr> and show tokens worked fine.
Happy to re-capture with KTR or test patches; KDB-capable kernel available.
---
---Файли--------------------------------
photo_2026-09-15_13-26-54.jpg (322 КБ)
photo_2026-09-15_13-26-49.jpg (342 КБ)
photo_8_2026-09-18_12-31-58.jpg (234 КБ)
photo_7_2026-09-18_12-31-58.jpg (283 КБ)
photo_6_2026-09-18_12-31-58.jpg (281 КБ)
photo_5_2026-09-18_12-31-58.jpg (366 КБ)
photo_4_2026-09-18_12-31-58.jpg (319 КБ)
photo_3_2026-09-18_12-31-58.jpg (249 КБ)
photo_2_2026-09-18_12-31-58.jpg (294 КБ)
photo_1_2026-09-18_12-31-58.jpg (673 КБ)
--
You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: http://bugs.dragonflybsd.org/my/account
More information about the Bugs
mailing list