[issue1597] panic: assertion: parent != NULL in hammer_cursor_removed_node (v2.5.1.187.gc1543-DEV, X86_64)

Sat Nov 7 08:18:33 PST 2009

Saifi Khan <saifi.khan at datasynergy.org> added the comment:

Observed couple of times.

Here is some more thread related information.

Script started on Sun Nov  8 03:10:03 2009

# kgdb ker	nel.1 vmcore.1
GNU gdb (GDB) 7.0
This GDB was configured as "x86_64-dragonfly".

Reading symbols from /var/crash/kernel.1...done.

Unread portion of the kernel message buffer:
panic: assertion: parent != NULL in hammer_cursor_removed_node
mp_lock = 00000000; cpuid = 0
Trace beginning at frame 0xffffffffc53ed198
Ã°Hâ€¹]Ã Lâ€¹eÃ¨Lâ€¹mÃ°Lâ€¹uÃ¸Ã‰ÃƒUHâ€°Ã¥ATSIâ€°Ã¼Ã¨â€š[() at Ã°Hâ€¹]Ã Lâ€¹eÃ¨Lâ€¹mÃ°Lâ€¹uÃ¸Ã‰ÃƒUHâ€°Ã¥ATSIâ€°Ã¼Ã¨â€š[+0x1f7
Ã°Hâ€¹]Ã Lâ€¹eÃ¨Lâ€¹mÃ°Lâ€¹uÃ¸Ã‰ÃƒUHâ€°Ã¥ATSIâ€°Ã¼Ã¨â€š[() at Ã°Hâ€¹]Ã Lâ€¹eÃ¨Lâ€¹mÃ°Lâ€¹uÃ¸Ã‰ÃƒUHâ€°Ã¥ATSIâ€°Ã¼Ã¨â€š[+0x1f7
Âº() at Âº+0x38
Hâ€°Ã–Iâ€°Ã‹Iâ€¹ÂÃˆ() at Hâ€°Ã–Iâ€°Ã‹Iâ€¹ÂÃˆ+0x165
Hâ€°Ã–Iâ€°Ã‹Iâ€¹ÂÃˆ() at Hâ€°Ã–Iâ€°Ã‹Iâ€¹ÂÃˆ+0x14b
() at +0x203
Â·Ã€Ã«	HÃÂ·() at Â·Ã€Ã«	HÃÂ·+0x3ff
â€°Ã¾Iâ€°ÃµIâ€°Ã¼HÂÂ¿H() at â€°Ã¾Iâ€°ÃµIâ€°Ã¼HÂÂ¿H+0x3b6
Ã‰ÃƒUHâ€°Ã¥â€¹â€¡Â¤() at Ã‰ÃƒUHâ€°Ã¥â€¹â€¡Â¤+0x1fd
Ãš;uÃ«$;t Hâ€¹@hHâ€¦Ã€uÃ³Ã‡Eâ‚¬() at Ãš;uÃ«$;t Hâ€¹@hHâ€¦Ã€uÃ³Ã‡Eâ‚¬+0x2d
â€¹>Ã¨Â¦Ã°() at â€¹>Ã¨Â¦Ã°+0x3e
@Lâ€°wHâ€¹E@â€°â€¡Â() at @Lâ€°wHâ€¹E@â€°â€¡Â+0x110
() at +0x486
f%() at f%+0x1f
Ã†Â±+_â‚¬HÃ‡Ã‡@Ã³Æ’â‚¬Ã¨Â®>=() at Ã†Â±+_â‚¬HÃ‡Ã‡@Ã³Æ’â‚¬Ã¨Â®>=+0x372
Â¾â€6() at Â¾â€6+0xb3
Debugger("panic")

CPU0 stopping CPUs: 0x00000002
 stopped
panic: from debugger
mp_lock = 00000000; cpuid = 0
boot() called on cpu#0
Uptime: 1h44m18s

dumping to dev ad4s1b, blockno 4456960

Reading symbols from /boot/modules/acpi.ko...done.
Loaded symbols for /boot/modules/acpi.ko
Reading symbols from /boot/modules/null.ko...done.
Loaded symbols for /boot/modules/null.ko
_get_mycpu () at ./machine/thread.h:73
73	    __asm ("movq %%gs:globaldata,%0" : "=r" (gd) : "m"(__mycpu__dummy));
(kgdb) thread
[Current thread is 27 (pid 94642/0, hammer)]
(kgdb) thread apply all bt

Thread 84 (kernel idle_0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff805bd1cd in cpu_idle () at
/usr/src/sys/platform/pc64/x86_64/machdep.c:932
#3  0x0000000000000000 in ?? ()

Thread 83 (pid 0/0, swapper):
#0  0x0000000000000000 in ?? ()
Cannot access memory at address 0x0

Thread 82 (kernel ithread emerg):
#0  0x0000000000000000 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 81 (kernel ithread 0):
#0  0x0000000000000000 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 80 (kernel softclock 0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80383d90 in softclock_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_timeout.c:313
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 79 (kernel usched 0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80372737 in sched_thread (dummy=<value optimized out>) at
/usr/src/sys/kern/usched_bsd4.c:1169
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 78 (kernel dsched 0):
#0  0x0000000000000000 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 77 (kernel devfs_msg_core):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---    at
/usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808d53a0,
flags=-2138221664)
    at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff8050c985 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  devfs_msg_core (arg=<value optimized out>) at
/usr/src/sys/vfs/devfs/devfs_core.c:1038
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 76 (kernel netisr_cpu 0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff80891dd0,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 75 (kernel disk_msg_core):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff8087b920,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff8038c8f6 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  disk_msg_core (arg=<value optimized out>) at /usr/src/sys/kern/subr_disk.c:363
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 74 (kernel ifnet 0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff8088e090,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---
Thread 73 (kernel taskq_cpu 0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff9334bac8, flags=1024,
wmesg=0xffffffff80642947 "tqthr", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8037fcb9 in ssleep (ident=0xffffffff9334bac8,
spin=0xffffffff9334bb00, flags=0, 
    wmesg=0xffffffff80642947 "tqthr", timo=0) at /usr/src/sys/kern/kern_synch.c:707
#4  0xffffffff803960b1 in TQ_SLEEP (arg=<value optimized out>) at
/usr/src/sys/kern/subr_taskqueue.c:98
#5  taskqueue_thread_loop (arg=<value optimized out>) at
/usr/src/sys/kern/subr_taskqueue.c:374
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 72 (kernel crypto):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff808f1070, flags=1024,
wmesg=0xffffffff806823a0 "crypto_wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff80380572 in lksleep (ident=0xffffffff808f1070,
lock=0xffffffff808f10a0, flags=0, 
    wmesg=0xffffffff806823a0 "crypto_wait", timo=0) at
/usr/src/sys/kern/kern_synch.c:722
#4  0xffffffff80591609 in crypto_proc () at /usr/src/sys/opencrypto/crypto.c:1367
#5  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 71 (kernel crypto returns):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff808f10d0, flags=1024,
wmesg=0xffffffff8068224e "crypto_ret_wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff80380572 in lksleep (ident=0xffffffff808f10d0,
lock=0xffffffff808f1100, flags=0, 
    wmesg=0xffffffff8068224e "crypto_ret_wait", timo=0) at
/usr/src/sys/kern/kern_synch.c:722
#4  0xffffffff8059028d in crypto_ret_proc () at
/usr/src/sys/opencrypto/crypto.c:1430
#5  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 70 (kernel ithread 69):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---Backtrace stopped:
previous frame inner to this frame (corrupt stack?)

Thread 69 (kernel xpt_thrd):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff8083c4a0, flags=1024,
wmesg=0xffffffff805dea77 "ccb_scanq", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8016daeb in xpt_scanner_thread (dummy=<value optimized out>) at
/usr/src/sys/bus/cam/cam_xpt.c:1421
#4  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 68 (kernel ithread 67):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 67 (kernel acpi_task):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff83651620,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff80ad8a6a in lwkt_waitport (arg=<value optimized out>) at
@/sys/msgport2.h:85
#5  acpi_task_thread (arg=<value optimized out>) at
/usr/src/sys/dev/acpica5/Osd/OsdSchedule.c:100
#6  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 66 (kernel ithread 9):
#0  0x0000000000000000 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 65 (kernel ithread 1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---Thread 64 (kernel ithread 12):
#0  0x0000000000000000 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 63 (kernel ithread 64):
#0  0x0000000000000000 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 62 (kernel ithread 4):
#0  0x0000000000000000 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 61 (kernel ithread 7):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 60 (kernel usb0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff9d052570, flags=-1661944816,
wmesg=0xffffffff8067584f "usbevt", timo=6000)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8055c407 in usb_event_thread (arg=<value optimized out>) at
/usr/src/sys/bus/usb/usb.c:459
#4  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 59 (kernel usbtask-hc):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff808ec900, flags=0,
wmesg=0xffffffff80675848 "usbtsk", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8055c22c in usb_task_thread (arg=0xffffffff808ec900) at
/usr/src/sys/bus/usb/usb.c:488
#4  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 58 (kernel usbtask-dr):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#2  0xffffffff8037f5c0 in
tsleep (ident=0xffffffff808ec928, flags=0, wmesg=0xffffffff80675848 "usbtsk",
timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8055c22c in usb_task_thread (arg=0xffffffff808ec928) at
/usr/src/sys/bus/usb/usb.c:488
#4  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 57 (kernel ithread 11):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 56 (kernel ithread 14):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 55 (kernel ithread 15):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 54 (kernel random):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff83652070, flags=-1658381296,
wmesg=0xffffffff8063cf65 "rwait", timo=12)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff80365c40 in rand_thread_loop (dummy=<value optimized out>) at
/usr/src/sys/kern/kern_nrandom.c:568
#4  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 53 (kernel udp_thread 0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---    at
/usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808bdb10,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 52 (kernel tcp_thread 0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808bbf90,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff804375e3 in lwkt_waitport (dummy=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  tcpmsg_service_loop (dummy=<value optimized out>) at
/usr/src/sys/netinet/tcp_subr.c:423
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 51 (kernel rtable_cpu 0):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff83652260,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff803fcb3d in lwkt_waitport (dummy=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  rtable_service_loop (dummy=<value optimized out>) at
/usr/src/sys/net/route.c:199
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 50 (kernel hammer-M):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff9f5c3128, flags=0,
wmesg=0xffffffff8066d783 "hmrwwa", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8051a5dd in hammer_flusher_master_thread (arg=<value optimized out>)
    at /usr/src/sys/vfs/hammer/hammer_flusher.c:252
#4  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 49 (kernel hammer-S0):
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#0  _crit_exit_quick () at
/usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff9cc125f8, flags=0,
wmesg=0xffffffff8066d758 "hmrssw", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff80519ec2 in hammer_flusher_slave_thread (arg=<value optimized out>)
    at /usr/src/sys/vfs/hammer/hammer_flusher.c:444
#4  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 48 (kernel hammer-S1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff9cc12a78, flags=0,
wmesg=0xffffffff8066d758 "hmrssw", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff80519ec2 in hammer_flusher_slave_thread (arg=<value optimized out>)
    at /usr/src/sys/vfs/hammer/hammer_flusher.c:444
#4  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 47 (kernel hammer-S2):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff9cc12ef8, flags=0,
wmesg=0xffffffff8066d758 "hmrssw", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff80519ec2 in hammer_flusher_slave_thread (arg=<value optimized out>)
    at /usr/src/sys/vfs/hammer/hammer_flusher.c:444
#4  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 46 (kernel hammer-S3):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff9cc13378, flags=0,
wmesg=0xffffffff8066d758 "hmrssw", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff80519ec2 in hammer_flusher_slave_thread (arg=<value optimized out>)
    at /usr/src/sys/vfs/hammer/hammer_flusher.c:444
#4  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 45 (kernel pagedaemon):
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#0  _crit_exit_quick () at
/usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff808ec668, flags=-1621075152,
wmesg=0xffffffff80646682 "psleep", timo=500)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8054b2f8 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1530
#4  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 44 (kernel vmdaemon):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff808ec6c4, flags=0,
wmesg=0xffffffff80646682 "psleep", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8054a4ae in vm_daemon () at /usr/src/sys/vm/vm_pageout.c:1630
#4  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 43 (kernel bufdaemon):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff8087cbd0, flags=-1621050416,
wmesg=0xffffffff80646682 "psleep", timo=100)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8037fcb9 in ssleep (ident=0xffffffff8087cbd0,
spin=0xffffffff8088cbe4, flags=0, 
    wmesg=0xffffffff80646682 "psleep", timo=100) at
/usr/src/sys/kern/kern_synch.c:707
#4  0xffffffff803be63b in buf_daemon () at /usr/src/sys/kern/vfs_bio.c:2356
#5  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 42 (kernel bufdaemon_hw):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff8087cbd4, flags=-1621038128,
wmesg=0xffffffff80646682 "psleep", timo=100)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8037fcb9 in ssleep (ident=0xffffffff8087cbd4,
spin=0xffffffff8088cbe4, flags=0, 
    wmesg=0xffffffff80646682 "psleep", timo=100) at
/usr/src/sys/kern/kern_synch.c:707
#4  0xffffffff803be2e5 in buf_daemon_hw () at /usr/src/sys/kern/vfs_bio.c:2416
#5  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 41 (kernel syncer):
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#0  _crit_exit_quick () at
/usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff808747ec, flags=0,
wmesg=0xffffffff806475d4 "syncer", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff803d1b64 in sched_sync () at /usr/src/sys/kern/vfs_sync.c:278
#4  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 40 (kernel vnlru):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff83653730, flags=-1621013504,
wmesg=0xffffffff80649244 "vlruwt", timo=100)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff803d13de in vnlru_proc () at /usr/src/sys/kern/vfs_mount.c:701
#4  0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 39 (pid 672/0, cron):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff80879960, flags=-1074800184,
wmesg=0xffffffff8064059b "nanslp", timo=6000)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff8038292f in nanosleep1 (uap=0xffffffffbfefdb48) at
/usr/src/sys/kern/kern_time.c:315
#4  sys_nanosleep (uap=0xffffffffbfefdb48) at /usr/src/sys/kern/kern_time. c:349
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffbfefdbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x00007ffffffffa70 in ?? ()
#8  0x00007ffffffffa60 in ?? ()
#9  0x0000000000000008 in ?? ()
#10 0x0000000000000004 in ?? ()
#11 0x0000000800a6d7e0 in ?? ()
#12 0x0000000800a6d858 in ?? ()
#13 0x00000000000000f0 in ?? ()
#14 0x000000000000003c in ?? ()
#15 0x000000000000003c in ?? ()
#16 0x0000000000000000 in ?? ()

Thread 38 (pid 724/0, getty):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#2  0xffffffff8037f595 in
tsleep (ident=0xffffffff8362bc60, flags=256, wmesg=0xffffffff80643b5b "ttyin",
timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362bc50, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
    at /usr/src/sys/kern/tty.c:2583
#4  0xffffffff803a3f7b in ttread (tp=0xffffffff8362bc50, uio=0xffffffffc006bad8,
flag=<value optimized out>)
    at /usr/src/sys/kern/tty.c:1737
#5  0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6  0xffffffff805ac1c8 in scread (ap=0xffffffffc006b9a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7  0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde730, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8  0xffffffff8050f17f in devfs_specf_read (fp=0xffffffff9ce69eb8,
uio=0xffffffffc006bad8, cred=<value optimized out>, 
    flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc006bad8, flags=0,
res=0xffffffffc006bb48)
    at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc006bad8, flags=0, res=0xffffffffc006bb48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc006bbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 37 (pid 726/0, getty):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8362c140, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362c130, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
    at /usr/src/sys/kern/tty.c:2583
#4  0xffffffff803a3f7b in ttread (tp=0xffffffff8362c130, uio=0xffffffffc0073ad8,
flag=<value optimized out>)
    at /usr/src/sys/kern/tty.c:1737
#5  0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6  0xffffffff805ac1c8 in scread (ap=0xffffffffc00739a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7  0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde910, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8  0xffffffff8050f17f in devfs_specf_read (fp=0xffffffff9ce69f88,
uio=0xffffffffc0073ad8, cred=<value optimized out>, 
    flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc0073ad8, flags=0,
res=0xffffffffc0073b48)
    at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc0073ad8, flags=0, res=0xffffffffc0073b48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc0073bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---
Thread 36 (pid 727/0, getty):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8362d4c0, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362d4b0, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
    at /usr/src/sys/kern/tty.c:2583
#4  0xffffffff803a3f7b in ttread (tp=0xffffffff8362d4b0, uio=0xffffffffc0078ad8,
flag=<value optimized out>)
    at /usr/src/sys/kern/tty.c:1737
#5  0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6  0xffffffff805ac1c8 in scread (ap=0xffffffffc00789a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7  0xffffffff8034d05c in dev_dread (dev=0xffffffff9cddea00, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8  0xffffffff8050f17f in devfs_specf_read (fp=0xffffffff9ce695c8,
uio=0xffffffffc0078ad8, cred=<value optimized out>, 
    flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc0078ad8, flags=0,
res=0xffffffffc0078b48)
    at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc0078ad8, flags=0, res=0xffffffffc0078b48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc0078bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 35 (pid 729/0, getty):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8362d9a0, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362d990, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
    at /usr/src/sys/kern/tty.c:2583
#4  0xffffffff803a3f7b in ttread (tp=0xffffffff8362d990, uio=0xffffffffc00a2ad8,
flag=<value optimized out>)
    at /usr/src/sys/kern/tty.c:1737
#5  0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6  0xffffffff805ac1c8 in scread (ap=0xffffffffc00a29a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7  0xffffffff8034d05c in dev_dread (dev=0xffffffff9cddebe0, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8  0xffffffff8050f17f in devfs_specf_read (fp=0xffffffff9ce69b10,
uio=0xffffffffc00a2ad8, cred=<value optimized out>, 
    flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc00a2ad8, flags=0,
res=0xffffffffc00a2b48)
    at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc00a2ad8, flags=0, res=0xffffffffc00a2b48)
at /usr/src/sys/kern/sys_generic.c:238
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#11 0xffffffff803968c1 in
sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc00a2bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 34 (pid 643/0, sshd):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8087bc30, flags=256,
wmesg=0xffffffff8069038e "select", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80397901 in doselect (nd=<value optimized out>, in=0x8005801b8,
ou=0x0, ex=0x0, tv=0x0, res=0xffffffffbfa93b48)
    at /usr/src/sys/kern/sys_generic.c:965
#4  0xffffffff80397c1f in sys_select (uap=0xffffffffbfa93b48) at
/usr/src/sys/kern/sys_generic.c:780
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffbfa93bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x0000000000000006 in ?? ()
#8  0x00000008005801b8 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 33 (pid 828/0, su):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffffbfcc3758, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=829, status=0xffffffffbfa9fa64,
options=2, rusage=0x0, res=0xffffffffbfa9fb48)
    at /usr/src/sys/kern/kern_exit.c:869
#4  0xffffffff803578be in sys_wait4 (uap=0xffffffffbfa9fb48) at
/usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffbfa9fbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x000000000000033d in ?? ()
#8  0x00007ffffffff74c in ?? ()
#9  0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 32 (pid 829/0, sh):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff9cc17258, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc0282a64,
options=2, rusage=0x0, res=0xffffffffc0282b48)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---    at
/usr/src/sys/kern/kern_exit.c:869
#4  0xffffffff803578be in sys_wait4 (uap=0xffffffffc0282b48) at
/usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffc0282bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x00000000ffffffff in ?? ()
#8  0x00007ffffffff764 in ?? ()
#9  0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 31 (pid 94579/0, vi):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffffc02db1d8, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803a0e22 in ttysleep (tp=0xffffffffc02db1c8, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
    at /usr/src/sys/kern/tty.c:2583
#4  0xffffffff803a3f7b in ttread (tp=0xffffffffc02db1c8, uio=0xffffffffc0302ad8,
flag=<value optimized out>)
    at /usr/src/sys/kern/tty.c:1737
#5  0xffffffff803a81d4 in ptsread (ap=0xffffffffc03029a8) at
/usr/src/sys/kern/tty_pty.c:396
#6  0xffffffff8034d05c in dev_dread (dev=0xffffffff9f5a9f10, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#7  0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb27000,
uio=0xffffffffc0302ad8, cred=<value optimized out>, 
    flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#8  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc0302ad8, flags=0,
res=0xffffffffc0302b48)
    at /usr/src/sys/sys/file2.h:57
#9  kern_preadv (fd=0, auio=0xffffffffc0302ad8, flags=0, res=0xffffffffc0302b48)
at /usr/src/sys/kern/sys_generic.c:238
#10 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#11 0xffffffff805be85e in syscall2 (frame=0xffffffffc0302bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#12 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#13 0x0000000000000000 in ?? ()

Thread 30 (pid 94595/0, sh):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff9cc188d8, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc3b2da64,
options=2, rusage=0x0, res=0xffffffffc3b2db48)
    at /usr/src/sys/kern/kern_exit.c:869
#4  0xffffffff803578be in sys_wait4 (uap=0xffffffffc3b2db48) at
/usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffc3b2dbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#7  0x00000000ffffffff in
?? ()
#8  0x00007ffffffff634 in ?? ()
#9  0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 29 (pid 94608/0, mail):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffffc4020f00, flags=1280,
wmesg=0xffffffff80642c0b "piperd", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80398d6b in pipe_read (fp=0xffffffffbfb27340,
uio=0xffffffffc63a9ad8, cred=<value optimized out>, 
    fflags=<value optimized out>) at /usr/src/sys/kern/sys_pipe.c:642
#4  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc63a9ad8, flags=0,
res=0xffffffffc63a9b48)
    at /usr/src/sys/sys/file2.h:57
#5  kern_preadv (fd=0, auio=0xffffffffc63a9ad8, flags=0, res=0xffffffffc63a9b48)
at /usr/src/sys/kern/sys_generic.c:238
#6  0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#7  0xffffffff805be85e in syscall2 (frame=0xffffffffc63a9bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#8  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#9  0x0000000000000000 in ?? ()

Thread 28 (pid 94639/0, sh):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffffbfcc4958, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc406da64,
options=2, rusage=0x0, res=0xffffffffc406db48)
    at /usr/src/sys/kern/kern_exit.c:869
#4  0xffffffff803578be in sys_wait4 (uap=0xffffffffc406db48) at
/usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffc406dbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
#7  0x00000000ffffffff in ?? ()
#8  0x00007ffffffff744 in ?? ()
#9  0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 27 (pid 94642/0, hammer):
#0  _get_mycpu () at ./machine/thread.h:73
#1  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:657
#2  0xffffffff8036ba66 in boot (howto=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:378
#3  0xffffffff8036be0a in panic (fmt=0xffffffff805e9d54 "from debugger") at
/usr/src/sys/kern/kern_shutdown.c:813
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#4  0xffffffff80188325 in
db_panic (addr=<value optimized out>, have_addr=0, count=0, modif=0x0)
    at /usr/src/sys/ddb/db_command.c:448
#5  0xffffffff801889db in db_command () at /usr/src/sys/ddb/db_command.c:344
#6  db_command_loop () at /usr/src/sys/ddb/db_command.c:470
#7  0xffffffff8018b751 in db_trap (type=<value optimized out>, code=<value
optimized out>) at /usr/src/sys/ddb/db_trap.c:71
#8  0xffffffff805b9ad9 in kdb_trap (type=3, code=0, regs=0xffffffffc53ed0c8)
    at /usr/src/sys/platform/pc64/x86_64/db_interface.c:176
#9  0xffffffff805be1b5 in trap (frame=0xffffffffc53ed0c8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:686
#10 0xffffffff805b7f5e in calltrap () at
/usr/src/sys/platform/pc64/x86_64/exception.S:179
#11 0xffffffff808f6658 in main_console.9668 ()
#12 0x0000000000000780 in ?? ()
#13 0xffffffffbfd5c6b0 in ?? ()
#14 0x0000000000000000 in ?? ()

Thread 26 (kernel idle_1):
#0  0xffffffff80acd36f in acpi_cpu_idle () at
/usr/src/sys/dev/acpica5/acpi_cpu_cstate.c:865
#1  0xffffffff805bd20f in cpu_idle () at
/usr/src/sys/platform/pc64/x86_64/machdep.c:944
#2  0x0000000000000000 in ?? ()

Thread 25 (kernel softclock 1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80383d90 in softclock_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_timeout.c:313
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 24 (kernel usched 1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff80372737 in sched_thread (dummy=<value optimized out>) at
/usr/src/sys/kern/usched_bsd4.c:1169
#3  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 23 (kernel dsched 1):
#0  0x0000000000000000 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 22 (kernel netisr_cpu 1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#2  0xffffffff8037e2ee in
lwkt_sleep (wmesg=0xffffffff8063face "waitport", flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff80891f88,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 21 (kernel ifnet 1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff8088e248,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 20 (kernel taskq_cpu 1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f5c0 in tsleep (ident=0xffffffff9334bb18, flags=1024,
wmesg=0xffffffff80642947 "tqthr", timo=0)
    at /usr/src/sys/kern/kern_synch.c:626
#3  0xffffffff8037fcb9 in ssleep (ident=0xffffffff9334bb18,
spin=0xffffffff9334bb50, flags=0, 
    wmesg=0xffffffff80642947 "tqthr", timo=0) at /usr/src/sys/kern/kern_synch.c:707
#4  0xffffffff803960b1 in TQ_SLEEP (arg=<value optimized out>) at
/usr/src/sys/kern/subr_taskqueue.c:98
#5  taskqueue_thread_loop (arg=<value optimized out>) at
/usr/src/sys/kern/subr_taskqueue.c:374
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 19 (kernel udp_thread 1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808bdcc8,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---Backtrace stopped:
previous frame inner to this frame (corrupt stack?)

Thread 18 (kernel tcp_thread 1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808bc148,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff804375e3 in lwkt_waitport (dummy=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  tcpmsg_service_loop (dummy=<value optimized out>) at
/usr/src/sys/netinet/tcp_subr.c:423
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 17 (kernel rtable_cpu 1):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
    at /usr/src/sys/kern/kern_synch.c:790
#3  0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff83652420,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4  0xffffffff803fcb3d in lwkt_waitport (dummy=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5  rtable_service_loop (dummy=<value optimized out>) at
/usr/src/sys/net/route.c:199
#6  0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 16 (pid 212/0, dhclient):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8087bc30, flags=256,
wmesg=0xffffffff8064e511 "poll", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff8039801c in sys_poll (uap=0xffffffffbfef8b48) at
/usr/src/sys/kern/sys_generic.c:1096
#4  0xffffffff805be85e in syscall2 (frame=0xffffffffbfef8bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#5  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#6  0x00007ffffffffa90 in ?? ()
#7  0x0000000000000001 in ?? ()
#8  0x00000000ffffffff in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 15 (pid 226/0, dhclient):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#2  0xffffffff8037f595 in
tsleep (ident=0xffffffff8087bc30, flags=-1079445272, wmesg=0xffffffff8064e511
"poll", timo=8640000)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff8039801c in sys_poll (uap=0xffffffffbfa8fb48) at
/usr/src/sys/kern/sys_generic.c:1096
#4  0xffffffff805be85e in syscall2 (frame=0xffffffffbfa8fbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#5  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#6  0x00007ffffffffa90 in ?? ()
#7  0x0000000000000002 in ?? ()
#8  0x000000007ffffd78 in ?? ()
#9  0x00000008005a00c0 in ?? ()
#10 0x0000000000000005 in ?? ()
#11 0x0000000100000016 in ?? ()
#12 0x00000000000000d1 in ?? ()
#13 0x00000008006c00c0 in ?? ()
#14 0x000000007ffffd78 in ?? ()
#15 0x0000000000000001 in ?? ()
#16 0x00000008006d0000 in ?? ()
#17 0x0000000000000002 in ?? ()
#18 0x000000000020c49b in ?? ()
#19 0x00000000005c6a78 in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 14 (pid 722/0, getty):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8362b780, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362b770, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
    at /usr/src/sys/kern/tty.c:2583
#4  0xffffffff803a3f7b in ttread (tp=0xffffffff8362b770, uio=0xffffffffbfa17ad8,
flag=<value optimized out>)
    at /usr/src/sys/kern/tty.c:1737
#5  0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6  0xffffffff805ac1c8 in scread (ap=0xffffffffbfa179a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7  0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde550, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8  0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb26b88,
uio=0xffffffffbfa17ad8, cred=<value optimized out>, 
    flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffbfa17ad8, flags=0,
res=0xffffffffbfa17b48)
    at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffbfa17ad8, flags=0, res=0xffffffffbfa17b48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa17bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#13 0xffffffff805b8193 in
Xfast_syscall () at /usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 13 (pid 725/0, getty):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8362bfa0, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362bf90, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
    at /usr/src/sys/kern/tty.c:2583
#4  0xffffffff803a3f7b in ttread (tp=0xffffffff8362bf90, uio=0xffffffffc006fad8,
flag=<value optimized out>)
    at /usr/src/sys/kern/tty.c:1737
#5  0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6  0xffffffff805ac1c8 in scread (ap=0xffffffffc006f9a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7  0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde820, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8  0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb26d28,
uio=0xffffffffc006fad8, cred=<value optimized out>, 
    flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc006fad8, flags=0,
res=0xffffffffc006fb48)
    at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc006fad8, flags=0, res=0xffffffffc006fb48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc006fbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 12 (pid 723/0, getty):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8362be00, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362bdf0, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
    at /usr/src/sys/kern/tty.c:2583
#4  0xffffffff803a3f7b in ttread (tp=0xffffffff8362bdf0, uio=0xffffffffbfa9bad8,
flag=<value optimized out>)
    at /usr/src/sys/kern/tty.c:1737
#5  0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6  0xffffffff805ac1c8 in scread (ap=0xffffffffbfa9b9a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7  0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde640, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8  0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb26df8,
uio=0xffffffffbfa9bad8, cred=<value optimized out>, 
    flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffbfa9bad8, flags=0,
res=0xffffffffbfa9bb48)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---    at
/usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffbfa9bad8, flags=0, res=0xffffffffbfa9bb48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa9bbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 11 (pid 728/0, getty):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8362d800, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362d7f0, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
    at /usr/src/sys/kern/tty.c:2583
#4  0xffffffff803a3f7b in ttread (tp=0xffffffff8362d7f0, uio=0xffffffffc007cad8,
flag=<value optimized out>)
    at /usr/src/sys/kern/tty.c:1737
#5  0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6  0xffffffff805ac1c8 in scread (ap=0xffffffffc007c9a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7  0xffffffff8034d05c in dev_dread (dev=0xffffffff9cddeaf0, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8  0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb26f30,
uio=0xffffffffc007cad8, cred=<value optimized out>, 
    flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9  0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc007cad8, flags=0,
res=0xffffffffc007cb48)
    at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc007cad8, flags=0, res=0xffffffffc007cb48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc007cbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 10 (pid 823/0, sshd):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffffbfd13ed0, flags=256,
wmesg=0xffffffff80644f6c "sbwait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff803b3445 in ssb_wait (ssb=0x0) at /usr/src/sys/kern/uipc_socket2.c:85
#4  0xffffffff803b0e9b in soreceive (so=0xffffffffbfd13e50, psa=0x0,
uio=0xffffffffbfa1aad8, sio=0x0, controlp=0x0, 
    flagsp=0xffffffffbfa1aa0c) at /usr/src/sys/kern/uipc_socket.c:910
#5  0xffffffff8039d608 in so_pru_soreceive (fp=0xffffffffbfb27138,
uio=0xffffffffbfa1aad8, cred=<value optimized out>, 
    fflags=0) at /usr/src/sys/sys/socketops.h:72
#6  soo_read (fp=0xffffffffbfb27138, uio=0xffffffffbfa1aad8, cred=<value
optimized out>, fflags=0)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---    at
/usr/src/sys/kern/sys_socket.c:90
#7  0xffffffff8039671e in dofileread (fd=5, auio=0xffffffffbfa1aad8, flags=0,
res=0xffffffffbfa1ab48)
    at /usr/src/sys/sys/file2.h:57
#8  kern_preadv (fd=5, auio=0xffffffffbfa1aad8, flags=0, res=0xffffffffbfa1ab48)
at /usr/src/sys/kern/sys_generic.c:238
#9  0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#10 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa1abf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#11 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#12 0x0000000000000005 in ?? ()
#13 0x00007ffffffff210 in ?? ()
#14 0x0000000000000004 in ?? ()
#15 0x0000000000000004 in ?? ()
#16 0x0000000000001000 in ?? ()
#17 0x00007fffffffe93f in ?? ()
#18 0x0000000000000003 in ?? ()
#19 0x0000000000000000 in ?? ()

Thread 9 (pid 826/0, sh):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff9cc16dd8, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc0233a64,
options=2, rusage=0x0, res=0xffffffffc0233b48)
    at /usr/src/sys/kern/kern_exit.c:869
#4  0xffffffff803578be in sys_wait4 (uap=0xffffffffc0233b48) at
/usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffc0233bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x00000000ffffffff in ?? ()
#8  0x00007ffffffff574 in ?? ()
#9  0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 8 (pid 1/0, init):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff9cc11858, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=-1, status=0x0, options=0, rusage=0x0,
res=0xffffffff9cc42b48)
    at /usr/src/sys/kern/kern_exit.c:869
#4  0xffffffff803578be in sys_wait4 (uap=0xffffffff9cc42b48) at
/usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffff9cc42bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#6  0xffffffff805b8193 in
Xfast_syscall () at /usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x00000000ffffffff in ?? ()
#8  0x0000000000000000 in ?? ()

Thread 7 (pid 825/0, sshd):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8087bc30, flags=256,
wmesg=0xffffffff8069038e "select", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80397901 in doselect (nd=<value optimized out>, in=0x800580210,
ou=0x800580208, ex=0x0, tv=0x0, 
    res=0xffffffffc023cb48) at /usr/src/sys/kern/sys_generic.c:965
#4  0xffffffff80397c1f in sys_select (uap=0xffffffffc023cb48) at
/usr/src/sys/kern/sys_generic.c:780
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffc023cbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x000000000000000a in ?? ()
#8  0x0000000800580210 in ?? ()
#9  0x0000000800580208 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 6 (pid 94593/0, cron):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffffbfbe0f00, flags=1280,
wmesg=0xffffffff80642c0b "piperd", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80398d6b in pipe_read (fp=0xffffffff9ce6a128,
uio=0xffffffffc02adad8, cred=<value optimized out>, 
    fflags=<value optimized out>) at /usr/src/sys/kern/sys_pipe.c:642
#4  0xffffffff8039671e in dofileread (fd=5, auio=0xffffffffc02adad8, flags=0,
res=0xffffffffc02adb48)
    at /usr/src/sys/sys/file2.h:57
#5  kern_preadv (fd=5, auio=0xffffffffc02adad8, flags=0, res=0xffffffffc02adb48)
at /usr/src/sys/kern/sys_generic.c:238
#6  0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#7  0xffffffff805be85e in syscall2 (frame=0xffffffffc02adbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#8  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#9  0x0000000000000005 in ?? ()
#10 0x0000000800550000 in ?? ()
#11 0x0000000000004000 in ?? ()
#12 0x0000000800550000 in ?? ()
#13 0x00000000000000c0 in ?? ()
#14 0x00000000ffffffff in ?? ()
#15 0x0000000000000003 in ?? ()
#16 0x0000000800a80220 in ?? ()
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#17 0x00000008005900e8 in
?? ()
#18 0x0000000000000000 in ?? ()

Thread 5 (pid 94597/0, sh):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff9cc17fd8, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc0297a64,
options=2, rusage=0x0, res=0xffffffffc0297b48)
    at /usr/src/sys/kern/kern_exit.c:869
#4  0xffffffff803578be in sys_wait4 (uap=0xffffffffc0297b48) at
/usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffc0297bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x00000000ffffffff in ?? ()
#8  0x00007ffffffff654 in ?? ()
#9  0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 4 (pid 94604/0, sh):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffffbfcc4058, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc0293a64,
options=2, rusage=0x0, res=0xffffffffc0293b48)
    at /usr/src/sys/kern/kern_exit.c:869
#4  0xffffffff803578be in sys_wait4 (uap=0xffffffffc0293b48) at
/usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffc0293bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x00000000ffffffff in ?? ()
#8  0x00007ffffffff1e4 in ?? ()
#9  0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 3 (pid 94603/0, sh):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffffbfcc3bd8, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc02bea64,
options=2, rusage=0x0, res=0xffffffffc02beb48)
    at /usr/src/sys/kern/kern_exit.c:869
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#4  0xffffffff803578be in
sys_wait4 (uap=0xffffffffc02beb48) at /usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffc02bebf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x00000000ffffffff in ?? ()
#8  0x00007fffffffef34 in ?? ()
#9  0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 2 (pid 349/0, syslogd):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff8087bc30, flags=256,
wmesg=0xffffffff8069038e "select", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80397901 in doselect (nd=<value optimized out>, in=0x8005900c8,
ou=0x0, ex=0x0, tv=0x0, res=0xffffffffbfa97b48)
    at /usr/src/sys/kern/sys_generic.c:965
#4  0xffffffff80397c1f in sys_select (uap=0xffffffffbfa97b48) at
/usr/src/sys/kern/sys_generic.c:780
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffbfa97bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x0000000000000007 in ?? ()
#8  0x00000008005900c8 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 1 (pid 94640/0, hammer):
#0  _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1  lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2  0xffffffff8037f595 in tsleep (ident=0xffffffff9cc18d58, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
    at /usr/src/sys/kern/kern_synch.c:615
#3  0xffffffff80357813 in kern_wait (pid=94642, status=0xffffffffc030ea64,
options=0, rusage=0x0, res=0xffffffffc030eb48)
    at /usr/src/sys/kern/kern_exit.c:869
#4  0xffffffff803578be in sys_wait4 (uap=0xffffffffc030eb48) at
/usr/src/sys/kern/kern_exit.c:671
#5  0xffffffff805be85e in syscall2 (frame=0xffffffffc030ebf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6  0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7  0x00000000000171b2 in ?? ()
#8  0x00007fffffffe684 in ?? ()
#9  0x0000000000000000 in ?? ()
(kgdb) 
(kgdb) q
# exit
Script done on Sun Nov  8 03:10:45 2009

----------
priority:  -> bug
status: unread -> chatting

_____________________________________________________
DragonFly issue tracker <bugs at lists.dragonflybsd.org>
<http://bugs.dragonflybsd.org/issue1597>
_____________________________________________________