[issue2169] panic - option #NETATALK in x86_64 kernel

John Marino (via DragonFly issue tracker) sinknull at leaf.dragonflybsd.org
Sun Oct 30 07:39:26 PDT 2011


New submission from John Marino <dragon at marino.st>:

For testing purposes I compiled NETATALK into a 64-bit kernel and was rewarded 
with the attached panic as soon as the atalkd started.

I have the full crash dump if required.

----------
assignedto: dillon
files: core5.txt
messages: 10249
nosy: dillon, marino
priority: bug
status: unread
title: panic - option #NETATALK in x86_64 kernel

_____________________________________________________
DragonFly issue tracker <bugs at lists.dragonflybsd.org>
<http://bugs.dragonflybsd.org/issue2169>
_____________________________________________________banshee.synsport.com dumped core - see /var/crash/vmcore.5

Sun Oct 30 15:20:36 CET 2011

Version String: DragonFly v2.13.0.151.gdc8442-DEVELOPMENT #9: Sun Oct 30 15:10:23 CET 2011 root at banshee.synsport.com:/usr/obj/usr/src/sys/X86_64_BANSHEE_SMP

panic: assertion "so->so_pcb == NULL && (so->so_state & SS_NOFDREF)" failed in sofree at /usr/src/sys/kern/uipc_socket.c:347

GNU gdb (GDB) 7.3
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-dragonfly".
For bug reporting instructions, please see:
<http://bugs.dragonflybsd.org/>...
Reading symbols from /boot/kernel/kernel...done.

Unread portion of the kernel message buffer:
<118>Starting atalkd.
panic: assertion "so->so_pcb == NULL && (so->so_state & SS_NOFDREF)" failed in sofree at /usr/src/sys/kern/uipc_socket.c:347
cpuid = 0
Trace beginning at frame 0xffffffe0378d2ba0
????u/H??????????PPH?}?A??????() at ????u/H??????????PPH?}?A??????+0x1ed
????u/H??????????PPH?}?A??????() at ????u/H??????????PPH?}?A??????+0x1ed
() at +0xd9
S??@?() at S??@?+0x23
() at +0x23
Cm?t}??() at Cm?t}??+0x7f
Debugger("panic")

CPU0 stopping CPUs: 0x0000000e
 stopped
Physical memory: 3810 MB
Dumping 802 MB: 787 771 755 739 723 707 691 675 659 643 627 611 595 579 563 547 531 515 499 483 467 451 435 419 403 387 371 355 339 323 307 291 275 259 243 227 211 195 179 163 147 131 115 99 83 67 51 35 19 3

[New pid 810/0, atalkd]
[New pid 804/0, sh]
[New pid 796/0, postgres]
[New pid 795/0, postgres]
[New pid 794/0, postgres]
[New pid 761/0, usbd]
[New pid 596/0, nfsd]
[New pid 576/0, nfsd]
[New pid 575/0, nfsd]
[New pid 421/0, syslogd]
[New pid 198/0, wpa_supplicant]
[New pid 2/0, sh]
[New kernel vnlru]
[New kernel pagezero]
[New kernel vmdaemon]
[New kernel hammer-S2]
[New kernel rtable_cpu 3]
[New kernel ithread 3]
[New kernel ithread 7]
[New kernel ithread 15]
[New kernel ral0 taskq]
[New kernel ithread 23]
[New kernel usb0]
[New kernel ithread 195]
[New kernel crypto 3]
[New kernel firmware taskq]
[New kernel taskq_cpu 3]
[New kernel ifnet 3]
[New kernel netisr_cpu 3]
[New kernel dsched 3]
[New kernel usched 3]
[New kernel ithread emerg]
[New kernel softclock 3]
[New kernel idle_3]
[New pid 594/0, nfsd]
[New pid 593/0, nfsd]
[New pid 595/0, nfsd]
[New pid 585/0, nfsd]
[New pid 592/0, nfsd]
[New pid 588/0, nfsd]
[New pid 589/0, nfsd]
[New pid 584/0, nfsd]
[New pid 586/0, nfsd]
[New pid 580/0, nfsd]
[New pid 569/0, mountd]
[New kernel bufdaemon]
[New kernel swapcached]
[New kernel hammer-S1]
[New kernel rtable_cpu 2]
[New kernel ithread 14]
[New kernel usb1]
[New kernel ithread 22]
[New kernel acpi_task]
[New kernel crypto 2]
[New kernel taskq_cpu 2]
[New kernel ifnet 2]
[New kernel devfs_msg_core]
[New kernel netisr_cpu 2]
[New kernel dsched 2]
[New kernel usched 2]
[New kernel softclock 2]
[New kernel idle_2]
[New pid 721/0, powerd]
[New pid 669/0, dntpd]
[New pid 573/0, nfsd]
[New kernel consttyd]
[New kernel bufdaemon_hw]
[New kernel pagedaemon]
[New pid 1/0, init]
[New kernel hammer-S0]
[New kernel rtable_cpu 1]
[New kernel ithread 1]
[New kernel ithread 17]
[New kernel ithread 21]
[New kernel usbtask-dr]
[New kernel xpt_thrd]
[New kernel ithread 197]
[New kernel crypto 1]
[New kernel taskq_cpu 1]
[New kernel ifnet 1]
[New kernel disk_msg_core]
[New kernel netisr_cpu 1]
[New kernel dsched 1]
[New kernel usched 1]
[New kernel softclock 1]
[New kernel idle_1]
[New pid 793/0, postgres]
[New pid 791/0, postgres]
[New pid 234/0, dhclient]
[New pid 597/0, nfsd]
[New pid 591/0, nfsd]
[New pid 587/0, nfsd]
[New pid 590/0, nfsd]
[New pid 583/0, nfsd]
[New pid 581/0, nfsd]
[New pid 577/0, nfsd]
[New pid 452/0, rpcbind]
[New pid 220/0, dhclient]
[New pid 106/0, adjkerntz]
[New kernel syncer]
[New kernel hammer-S3]
[New kernel hammer-M]
[New kernel rtable_cpu 0]
[New kernel random]
[New pid 0/0, swapper]
[New kernel ithread 4]
[New kernel ithread 192]
[New kernel ithread 16]
[New kernel usbtask-hc]
[New kernel ithread 9]
[New kernel crypto returns]
[New kernel crypto 0]
[New kernel taskq_cpu 0]
[New kernel ifnet 0]
[New kernel netisr_cpu 0]
[New kernel dsched 0]
[New kernel usched 0]
[New kernel ithread 196]
[New kernel softclock 0]
[New kernel idle_0]
Reading symbols from /boot/kernel/acpi.ko...done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/ahci.ko...done.
Loaded symbols for /boot/kernel/ahci.ko
Reading symbols from /boot/kernel/ehci.ko...done.
Loaded symbols for /boot/kernel/ehci.ko
_get_mycpu () at ./machine/thread.h:69
69	    __asm ("movq %%gs:globaldata,%0" : "=r" (gd) : "m"(__mycpu__dummy));
(kgdb) #0  _get_mycpu () at ./machine/thread.h:69
#1  md_dumpsys (di=<optimized out>) at /usr/src/sys/platform/pc64/x86_64/dump_machdep.c:263
#2  0xffffffff80403755 in dumpsys () at /usr/src/sys/kern/kern_shutdown.c:925
#3  0xffffffff80294e1c in db_fncall (dummy1=<optimized out>, dummy2=<optimized out>, dummy3=<optimized out>, dummy4=<optimized out>) at /usr/src/sys/ddb/db_command.c:542
#4  0xffffffff802952ef in db_command (aux_cmd_tablep_end=0xffffffff80775b40, aux_cmd_tablep=0xffffffff80775b08, cmd_table=<optimized out>, last_cmdp=<optimized out>) at /usr/src/sys/ddb/db_command.c:344
#5  db_command_loop () at /usr/src/sys/ddb/db_command.c:470
#6  0xffffffff802981a1 in db_trap (type=<optimized out>, code=<optimized out>) at /usr/src/sys/ddb/db_trap.c:71
#7  0xffffffff806798c8 in kdb_trap (type=<optimized out>, code=<optimized out>, regs=<optimized out>) at /usr/src/sys/platform/pc64/x86_64/db_interface.c:176
#8  0xffffffff8067f4c7 in trap (frame=0xffffffe0378d2ad8) at /usr/src/sys/platform/pc64/x86_64/trap.c:718
#9  0xffffffff80677e1e in calltrap () at /usr/src/sys/platform/pc64/x86_64/exception.S:187
#10 0xffffffff8067975f in breakpoint () at ./cpu/cpufunc.h:73
#11 Debugger (msg=<optimized out>) at /usr/src/sys/platform/pc64/x86_64/db_interface.c:361
#12 0xffffffff80404044 in panic (fmt=0xffffffff806c0840 "assertion \"%s\" failed in %s at %s:%u") at /usr/src/sys/kern/kern_shutdown.c:822
#13 0xffffffff8044ccf3 in sofree (so=0xffffffe05d4382c8) at /usr/src/sys/kern/uipc_socket.c:347
#14 0xffffffff804e8235 in at_pcbdetach (so=0xffffffe05d4382c8, ddp=0xffffffe00fb97db0) at /usr/src/sys/netproto/atalk/ddp_usrreq.c:503
#15 0xffffffff804e82f7 in ddp_detach (msg=0xffffffe05e105978) at /usr/src/sys/netproto/atalk/ddp_usrreq.c:80
#16 0xffffffff804a7ddd in netmsg_service_loop (arg=<optimized out>) at /usr/src/sys/net/netisr.c:308
#17 0xffffffff8040e8bf in lwkt_deschedule_self (td=0x0) at /usr/src/sys/kern/lwkt_thread.c:355
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(kgdb) (kgdb) 
Token           collisions owner
pmap_token               0 not held
dev_token                0 not held
vm_token                 0 not held
vmspace_token            0 not held
kvm_token                0 not held
proc_token              30 not held
tty_token                2 not held
vnode_token              0 not held
vmobj_token              2 not held
(kgdb) 
cpu  pid    thread    flags comm               wchan    wmesg
  0    -1 80cb2978 00000000 idle_0             00000000
  0    -1 80b2a8a8 00010402 softclock 0        00000000
  0    -1 80a7e568 00010400 ithread 196        00000000
  0    -1 80a930a0 00000400 usched 0           00000000
  0    -1 80aa32a8 00000400 dsched 0           00000000
  0    -1 80b65700 00000403 netisr_cpu 0       00000000
  0    -1 80b526c0 00040400 ifnet 0            00000000 waitport
  0    -1 319b3270 00020680 taskq_cpu 0        315abad0 tqthr
  0    -1 319b48f0 04020e80 crypto 0           80ca6360 crypto_wait
  0    -1 319b5af0 00020e80 crypto returns     80ca7b00 crypto_ret_wait
  0    -1 80a4c4d0 00010400 ithread 9          00000000
  0    -1 319b6cf0 00020e80 usbtask-hc         80ca1a40 usbtsk
  0    -1 80a4e2c8 00010400 ithread 16         00000000
  0    -1 80a7d448 00010400 ithread 192        00000000
  0    -1 80a4af68 00010400 ithread 4          00000000
  0     0 80a481c0 00020080 swapper            80bdbb0c nowork
  0    -1 319b8370 00020680 random             319b8370 rwait
  0    -1 319b90f0 00040600 rtable_cpu 0       00000000 waitport
  0    -1 319babf0 00020680 hammer-M           39f68160 hmrwwa
  0    -1 319bbdf0 00020680 hammer-S3          37daabc0 hmrssw
  0    -1 319be1f0 00020e80 syncer             80b12f90 syncer
  0   106 5d152df0 00820680 adjkerntz          5d17ad68 pause
  0   220 5d1548f0 00020680 dhclient           5d187b90 kqread
  0   452 5d1551f0 00820680 rpcbind            5d717390 kqread
  0   577 5d157ef0 00020680 nfsd               0fd366a8 nfsd
  0   581 5d080070 00020680 nfsd               0fd36888 nfsd
  0   583 5d080970 00020680 nfsd               0fd36a68 nfsd
  0   590 5d084870 00020680 nfsd               0fd36c48 nfsd
  0   587 5d082d70 00020680 nfsd               0fd36e28 nfsd
  0   591 5d085170 00020680 nfsd               0fd37008 nfsd
  0   597 5d088770 00020680 nfsd               0fd371e8 nfsd
  0   234 5d1536f0 00020680 dhclient           5d188990 kqread
  0   791 5d15c6f0 00820680 postgres           5d701b90 kqread
  0   793 5d15bdf0 00820680 postgres           5e235390 kqread
  1    -1 80cbc978 00000001 idle_1             00000000
  1    -1 80b2acd8 00010402 softclock 1        00000000
  1    -1 80a934b8 00000400 usched 1           00000000
  1    -1 80aa36c0 00000400 dsched 1           00000000
  1    -1 80b65b08 00040400 netisr_cpu 1       00000000 waitport
  1    -1 319b2970 00040600 disk_msg_core      00000000 waitport
  1    -1 80b52ac8 00040400 ifnet 1            00000000 waitport
  1    -1 319b36f0 00020680 taskq_cpu 1        315abb20 tqthr
  1    -1 319b4d70 04020e80 crypto 1           80ca63c0 crypto_wait
  1    -1 80a7e9b0 00010400 ithread 197        00000000
  1    -1 319b5f70 00020e80 xpt_thrd           80a24060 ccb_scanq
  1    -1 319b7170 00020e80 usbtask-dr         80ca1a68 usbtsk
  1    -1 80a4f830 00010400 ithread 21         00000000
  1    -1 80a4e710 00010400 ithread 17         00000000
  1    -1 80a4a290 00010402 ithread 1          00000000 tty_token
  1    -1 319b9570 00040600 rtable_cpu 1       00000000 waitport
  1    -1 319bb070 00020680 hammer-S0          37daaa88 hmrssw
  1     1 319b24f0 00020680 init               319b2070 wait
  1    -1 319bc270 00020f80 pagedaemon         80c99db0 psleep
  1    -1 319bd470 00020f80 bufdaemon_hw       80b3ef6c psleep
  1    -1 319be670 00020f80 consttyd           319be670 waiting
  1   573 5d155af0 00020680 nfsd               5d718390 kqread
  1   669 5d1590f0 00820680 dntpd              80b2a840 nanslp
  1   721 5d151bf0 00820680 powerd             80b2a840 nanslp
  2    -1 80cc6978 00000001 idle_2             00000000
  2    -1 80b2b108 00010402 softclock 2        00000000
  2    -1 80a938d0 00000400 usched 2           00000000
  2    -1 80aa3ad8 00000400 dsched 2           00000000
  2    -1 80b65f10 00040400 netisr_cpu 2       00000000 waitport
  2    -1 319b2df0 00040600 devfs_msg_core     00000000 waitport
  2    -1 80b52ed0 00040400 ifnet 2            00000000 waitport
  2    -1 319b3b70 00020680 taskq_cpu 2        315abb70 tqthr
  2    -1 319b51f0 04020e80 crypto 2           80ca6420 crypto_wait
  2    -1 319b63f0 00040e00 acpi_task          00000000 waitport
  2    -1 80a4fc78 00010402 ithread 22         00000000
  2    -1 319b75f0 00020e80 usb1               37e67488 usbevt
  2    -1 80a4da38 00010400 ithread 14         00000000
  2    -1 319b99f0 00040600 rtable_cpu 2       00000000 waitport
  2    -1 319bb4f0 00020680 hammer-S1          37daaaf0 hmrssw
  2    -1 319bc6f0 00020f80 swapcached         80ca1700 csleep
  2    -1 319bd8f0 00020f80 bufdaemon          80b3ef68 psleep
  2   569 5d1563f0 00020680 mountd             5d719190 kqread
  2   580 5d1587f0 00020680 nfsd               5d680628 nfsd
  2   586 5d082470 00020680 nfsd               5d681708 nfsd
  2   584 5d081270 00020680 nfsd               5d6818e8 nfsd
  2   589 5d083f70 00020680 nfsd               5d681ac8 nfsd
  2   588 5d083670 00020680 nfsd               5d681ca8 nfsd
  2   592 5d085a70 00020680 nfsd               5d681e88 nfsd
  2   585 5d081b70 00020680 nfsd               5d682068 nfsd
  2   595 5d087570 00020680 nfsd               5d682248 nfsd
  2   593 5d086370 00020680 nfsd               5d682428 nfsd
  2   594 5d086c70 00020680 nfsd               5d682608 nfsd
  3    -1 80cd0978 00000001 idle_3             00000000
  3    -1 80b2b538 00010402 softclock 3        00000000
  3    -1 80a8e6c0 00010400 ithread emerg      00000000
  3    -1 80a93ce8 00000400 usched 3           00000000
  3    -1 80aa3ef0 00000400 dsched 3           00000000
  3    -1 80b66318 00040400 netisr_cpu 3       00000000 waitport
  3    -1 80b532d8 00040400 ifnet 3            00000000 waitport
  3    -1 319b3ff0 00020680 taskq_cpu 3        315abbc0 tqthr
  3    -1 319b4470 00020680 firmware taskq     315abc10 tqthr
  3    -1 319b5670 04020e80 crypto 3           80ca6480 crypto_wait
  3    -1 80a7e120 00010400 ithread 195        00000000
  3    -1 319b6870 00020e80 usb0               37e44c88 usbevt
  3    -1 80a500c0 00010400 ithread 23         00000000
  3    -1 319b7ef0 00020680 ral0 taskq         315aedc0 tqthr
  3    -1 80a4de80 00010400 ithread 15         00000000
  3    -1 80a4bc40 00010400 ithread 7          00000000
  3    -1 80a4ab20 00010400 ithread 3          00000000
  3    -1 319b9e70 00040600 rtable_cpu 3       00000000 waitport
  3    -1 319bb970 00020680 hammer-S2          37daab58 hmrssw
  3    -1 319bcb70 00020e80 vmdaemon           80c99e0c psleep
  3    -1 319bcff0 00020e80 pagezero           80ca19e4 pgzero
  3    -1 319bdd70 00020e80 vnlru              319bdd70 vlruwt
  3     2 5d09eaf0 00820680 sh                 5d09e670 wait
  3   198 5d153ff0 00820680 wpa_supplicant     5d184b90 kqread
  3   421 5d1524f0 00820680 syslogd            5d714390 kqread
  3   575 5d156cf0 00020680 nfsd               5d4373c8 nfsd
  3   576 5d1575f0 00020680 nfsd               5d4375a8 nfsd
  3   596 5d087e70 00020680 nfsd               5d437788 nfsd
  3   761 5d15a2f0 00820680 usbd               5d71c590 kqread
  3   794 5d15cff0 00820680 postgres           5e236d90 kqread
  3   795 5d15d8f0 00820680 postgres           5e236f90 kqread
  3   796 5d15e1f0 00820680 postgres           5e237390 kqread
  3   804 5d09f3f0 00820680 sh                 5d1512f0 wait
  3   810 5d1599f0 00840600 atalkd             00000000 waitmsg
(kgdb) 
cpu    curthread    wchan
0    netisr_cpu 0    00000000
1        idle_1    00000000
2        idle_2    00000000
3        idle_3    00000000
(kgdb) 
Filesystem      Refs    Flags
ufs             1       0x00000000
hammer          1       0x00000000
msdos           0       0x00000000
tmpfs           0       0x00000000
mfs             0       0x00000000
cd9660          0       0x00040000
procfs          1       0x00080000
nfs             0       0x00020000
null            7       0x00100000
devfs           2       0x00080000
(kgdb) Filesystem: ufs, Refs: 1, Flags: 0x00000000
VFS ops: 
$1 = {
  vfs_mount = 0xffffffff8058b250 <ffs_mount>, 
  vfs_start = 0xffffffff80464e5a <vfs_stdstart>, 
  vfs_unmount = 0xffffffff8058a975 <ffs_unmount>, 
  vfs_root = 0xffffffff80590a51 <ufs_root>, 
  vfs_quotactl = 0xffffffff80590a00 <ufs_quotactl>, 
  vfs_statfs = 0xffffffff80589ef4 <ffs_statfs>, 
  vfs_sync = 0xffffffff8058a76e <ffs_sync>, 
  vfs_vget = 0xffffffff8058a3d1 <ffs_vget>, 
  vfs_fhtovp = 0xffffffff8058a392 <ffs_fhtovp>, 
  vfs_checkexp = 0xffffffff80590a0b <ufs_check_export>, 
  vfs_vptofh = 0xffffffff80589e9b <ffs_vptofh>, 
  vfs_init = 0xffffffff80589ec3 <ffs_init>, 
  vfs_uninit = 0xffffffff8058d974 <ufs_uninit>, 
  vfs_extattrctl = 0xffffffff80464ebd <vfs_stdextattrctl>, 
  vfs_statvfs = 0xffffffff80464f41 <vfs_stdstatvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
Filesystem: hammer, Refs: 1, Flags: 0x00000000
VFS ops: 
$2 = {
  vfs_mount = 0xffffffff805c66c5 <hammer_vfs_mount>, 
  vfs_start = 0xffffffff80464e5a <vfs_stdstart>, 
  vfs_unmount = 0xffffffff805c664c <hammer_vfs_unmount>, 
  vfs_root = 0xffffffff805c62dc <hammer_vfs_root>, 
  vfs_quotactl = 0xffffffff80464e65 <vfs_stdquotactl>, 
  vfs_statfs = 0xffffffff805c5de8 <hammer_vfs_statfs>, 
  vfs_sync = 0xffffffff805c62f4 <hammer_vfs_sync>, 
  vfs_vget = 0xffffffff805c61fc <hammer_vfs_vget>, 
  vfs_fhtovp = 0xffffffff805c60db <hammer_vfs_fhtovp>, 
  vfs_checkexp = 0xffffffff805c6050 <hammer_vfs_checkexp>, 
  vfs_vptofh = 0xffffffff805c5fe7 <hammer_vfs_vptofh>, 
  vfs_init = 0xffffffff805c5ef5 <hammer_vfs_init>, 
  vfs_uninit = 0xffffffff80464eb2 <vfs_stduninit>, 
  vfs_extattrctl = 0xffffffff80464ebd <vfs_stdextattrctl>, 
  vfs_statvfs = 0xffffffff805c5cf0 <hammer_vfs_statvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
Filesystem: msdos, Refs: 0, Flags: 0x00000000
VFS ops: 
$3 = {
  vfs_mount = 0xffffffff80489196 <msdosfs_mount>, 
  vfs_start = 0xffffffff80464e5a <vfs_stdstart>, 
  vfs_unmount = 0xffffffff80489063 <msdosfs_unmount>, 
  vfs_root = 0xffffffff80488e45 <msdosfs_root>, 
  vfs_quotactl = 0xffffffff80464e65 <vfs_stdquotactl>, 
  vfs_statfs = 0xffffffff80488fb8 <msdosfs_statfs>, 
  vfs_sync = 0xffffffff80488ee0 <msdosfs_sync>, 
  vfs_vget = 0xffffffff80464e86 <vfs_stdvget>, 
  vfs_fhtovp = 0xffffffff80488dff <msdosfs_fhtovp>, 
  vfs_checkexp = 0xffffffff80488db7 <msdosfs_checkexp>, 
  vfs_vptofh = 0xffffffff80488d90 <msdosfs_vptofh>, 
  vfs_init = 0xffffffff80485f14 <msdosfs_init>, 
  vfs_uninit = 0xffffffff80484f50 <msdosfs_uninit>, 
  vfs_extattrctl = 0xffffffff80464ebd <vfs_stdextattrctl>, 
  vfs_statvfs = 0xffffffff80464f41 <vfs_stdstatvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
Filesystem: tmpfs, Refs: 0, Flags: 0x00000000
VFS ops: 
$4 = {
  vfs_mount = 0xffffffff805d1225 <tmpfs_mount>, 
  vfs_start = 0xffffffff80464e5a <vfs_stdstart>, 
  vfs_unmount = 0xffffffff805d0e71 <tmpfs_unmount>, 
  vfs_root = 0xffffffff805d0cea <tmpfs_root>, 
  vfs_quotactl = 0xffffffff80464e65 <vfs_stdquotactl>, 
  vfs_statfs = 0xffffffff805d119a <tmpfs_statfs>, 
  vfs_sync = 0xffffffff80464e70 <vfs_stdsync>, 
  vfs_vget = 0xffffffff80464e86 <vfs_stdvget>, 
  vfs_fhtovp = 0xffffffff805d169b <tmpfs_fhtovp>, 
  vfs_checkexp = 0xffffffff80464e9c <vfs_stdcheckexp>, 
  vfs_vptofh = 0xffffffff805d0c47 <tmpfs_vptofh>, 
  vfs_init = 0xffffffff80464ea7 <vfs_stdinit>, 
  vfs_uninit = 0xffffffff80464eb2 <vfs_stduninit>, 
  vfs_extattrctl = 0xffffffff80464ebd <vfs_stdextattrctl>, 
  vfs_statvfs = 0xffffffff80464f41 <vfs_stdstatvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
Filesystem: mfs, Refs: 0, Flags: 0x00000000
VFS ops: 
$5 = {
  vfs_mount = 0xffffffff8058cb37 <mfs_mount>, 
  vfs_start = 0xffffffff8058c7e2 <mfs_start>, 
  vfs_unmount = 0xffffffff8058a975 <ffs_unmount>, 
  vfs_root = 0xffffffff80590a51 <ufs_root>, 
  vfs_quotactl = 0xffffffff80590a00 <ufs_quotactl>, 
  vfs_statfs = 0xffffffff8058c542 <mfs_statfs>, 
  vfs_sync = 0xffffffff8058a76e <ffs_sync>, 
  vfs_vget = 0xffffffff8058a3d1 <ffs_vget>, 
  vfs_fhtovp = 0xffffffff8058a392 <ffs_fhtovp>, 
  vfs_checkexp = 0xffffffff80590a0b <ufs_check_export>, 
  vfs_vptofh = 0xffffffff80589e9b <ffs_vptofh>, 
  vfs_init = 0xffffffff8058c537 <mfs_init>, 
  vfs_uninit = 0xffffffff80464eb2 <vfs_stduninit>, 
  vfs_extattrctl = 0xffffffff80464ebd <vfs_stdextattrctl>, 
  vfs_statvfs = 0xffffffff80464f41 <vfs_stdstatvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
Filesystem: cd9660, Refs: 0, Flags: 0x00040000
VFS ops: 
$6 = {
  vfs_mount = 0xffffffff803d1808 <cd9660_mount>, 
  vfs_start = 0xffffffff80464e5a <vfs_stdstart>, 
  vfs_unmount = 0xffffffff803d0dd0 <cd9660_unmount>, 
  vfs_root = 0xffffffff803d211f <cd9660_root>, 
  vfs_quotactl = 0xffffffff80464e65 <vfs_stdquotactl>, 
  vfs_statfs = 0xffffffff803d0d5c <cd9660_statfs>, 
  vfs_sync = 0xffffffff80464e70 <vfs_stdsync>, 
  vfs_vget = 0xffffffff803d2103 <cd9660_vget>, 
  vfs_fhtovp = 0xffffffff803d0cff <cd9660_fhtovp>, 
  vfs_checkexp = 0xffffffff803d0cb9 <cd9660_checkexp>, 
  vfs_vptofh = 0xffffffff803d0c90 <cd9660_vptofh>, 
  vfs_init = 0xffffffff803cfa38 <cd9660_init>, 
  vfs_uninit = 0xffffffff803cf6e9 <cd9660_uninit>, 
  vfs_extattrctl = 0xffffffff80464ebd <vfs_stdextattrctl>, 
  vfs_statvfs = 0xffffffff80464f41 <vfs_stdstatvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
Filesystem: procfs, Refs: 1, Flags: 0x00080000
VFS ops: 
$7 = {
  vfs_mount = 0xffffffff804823c4 <procfs_mount>, 
  vfs_start = 0xffffffff80464e5a <vfs_stdstart>, 
  vfs_unmount = 0xffffffff8048236c <procfs_unmount>, 
  vfs_root = 0xffffffff804824a7 <procfs_root>, 
  vfs_quotactl = 0xffffffff80464e65 <vfs_stdquotactl>, 
  vfs_statfs = 0xffffffff804822cc <procfs_statfs>, 
  vfs_sync = 0xffffffff80464e70 <vfs_stdsync>, 
  vfs_vget = 0xffffffff80464e86 <vfs_stdvget>, 
  vfs_fhtovp = 0xffffffff80464e91 <vfs_stdfhtovp>, 
  vfs_checkexp = 0xffffffff80464e9c <vfs_stdcheckexp>, 
  vfs_vptofh = 0xffffffff80464e4f <vfs_stdvptofh>, 
  vfs_init = 0xffffffff80464ea7 <vfs_stdinit>, 
  vfs_uninit = 0xffffffff80464eb2 <vfs_stduninit>, 
  vfs_extattrctl = 0xffffffff80464ebd <vfs_stdextattrctl>, 
  vfs_statvfs = 0xffffffff80464f41 <vfs_stdstatvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
Filesystem: nfs, Refs: 0, Flags: 0x00020000
VFS ops: 
$8 = {
  vfs_mount = 0xffffffff80541ce0 <nfs_mount>, 
  vfs_start = 0xffffffff80464e5a <vfs_stdstart>, 
  vfs_unmount = 0xffffffff80540ed3 <nfs_unmount>, 
  vfs_root = 0xffffffff8054098a <nfs_root>, 
  vfs_quotactl = 0xffffffff80464e65 <vfs_stdquotactl>, 
  vfs_statfs = 0xffffffff8054063b <nfs_statfs>, 
  vfs_sync = 0xffffffff80540525 <nfs_sync>, 
  vfs_vget = 0xffffffff80464e86 <vfs_stdvget>, 
  vfs_fhtovp = 0xffffffff80464e91 <vfs_stdfhtovp>, 
  vfs_checkexp = 0xffffffff80464e9c <vfs_stdcheckexp>, 
  vfs_vptofh = 0xffffffff80464e4f <vfs_stdvptofh>, 
  vfs_init = 0xffffffff8053d9ee <nfs_init>, 
  vfs_uninit = 0xffffffff8053d9b3 <nfs_uninit>, 
  vfs_extattrctl = 0xffffffff80464ebd <vfs_stdextattrctl>, 
  vfs_statvfs = 0xffffffff8054019c <nfs_statvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
Filesystem: null, Refs: 7, Flags: 0x00100000
VFS ops: 
$9 = {
  vfs_mount = 0xffffffff8047fcb4 <nullfs_mount>, 
  vfs_start = 0xffffffff80464e5a <vfs_stdstart>, 
  vfs_unmount = 0xffffffff8047fc48 <nullfs_unmount>, 
  vfs_root = 0xffffffff8047fc09 <nullfs_root>, 
  vfs_quotactl = 0xffffffff8047fbf4 <nullfs_quotactl>, 
  vfs_statfs = 0xffffffff8047faeb <nullfs_statfs>, 
  vfs_sync = 0xffffffff80464e70 <vfs_stdsync>, 
  vfs_vget = 0xffffffff80464e86 <vfs_stdvget>, 
  vfs_fhtovp = 0xffffffff8047fad2 <nullfs_fhtovp>, 
  vfs_checkexp = 0xffffffff8047fa8c <nullfs_checkexp>, 
  vfs_vptofh = 0xffffffff8047fa81 <nullfs_vptofh>, 
  vfs_init = 0xffffffff80464ea7 <vfs_stdinit>, 
  vfs_uninit = 0xffffffff80464eb2 <vfs_stduninit>, 
  vfs_extattrctl = 0xffffffff8047fa6c <nullfs_extattrctl>, 
  vfs_statvfs = 0xffffffff80464f41 <vfs_stdstatvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
Filesystem: devfs, Refs: 2, Flags: 0x00080000
VFS ops: 
$10 = {
  vfs_mount = 0xffffffff8059b1a7 <devfs_vfs_mount>, 
  vfs_start = 0xffffffff80464e5a <vfs_stdstart>, 
  vfs_unmount = 0xffffffff8059b0fd <devfs_vfs_unmount>, 
  vfs_root = 0xffffffff8059b3d9 <devfs_vfs_root>, 
  vfs_quotactl = 0xffffffff80464e65 <vfs_stdquotactl>, 
  vfs_statfs = 0xffffffff8059b03f <devfs_vfs_statfs>, 
  vfs_sync = 0xffffffff80464e70 <vfs_stdsync>, 
  vfs_vget = 0xffffffff8059b012 <devfs_vfs_vget>, 
  vfs_fhtovp = 0xffffffff8059afd3 <devfs_vfs_fhtovp>, 
  vfs_checkexp = 0xffffffff80464e9c <vfs_stdcheckexp>, 
  vfs_vptofh = 0xffffffff8059afa0 <devfs_vfs_vptofh>, 
  vfs_init = 0xffffffff80464ea7 <vfs_stdinit>, 
  vfs_uninit = 0xffffffff80464eb2 <vfs_stduninit>, 
  vfs_extattrctl = 0xffffffff80464ebd <vfs_stdextattrctl>, 
  vfs_statvfs = 0xffffffff80464f41 <vfs_stdstatvfs>, 
  vfs_account = 0xffffffff80464ec8 <vfs_stdaccount>
}
(kgdb) $11 = {mnt_list = {tqe_next = 0xffffffe05cfc9fb8, tqe_prev = 0xffffffff809d0f20}, mnt_op = 0xffffffff80a006c0, mnt_vfc = 0xffffffff80a00680, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe037c9ae00, mnt_nvnodelist = {tqh_first = 0xffffffe037c9ac00, tqh_last = 0xffffffe037c9ae78}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 6, lk_lockholder = 0
xffffffffffffffff}, mnt_flag = 20480, mnt_kern_flag = 138346496, mnt_maxsymlinklen = 255, mnt_stat = {f_spare2 = 0, f_bsize = 16384, f_iosize = 16384, f_blocks = 45185024, f_bfree = 41130121, f_bavail = 41130121, f_files = 1233728, f_ffree = 0, f_fsid = {val = {1343492506, 41653}}, f_owner = 0, f_type = 2, f_flags = 20480, f_syncwrites = 17, f_asyncwrites = 120, f_fstypename = "hammer\000\000\000\000\000\000\000\000\000", f_mntonname = "/", '\000' <repeats 78 times>, f_syncreads = 0, f!
 _asyncreads = 1217, f_spares1 = 0, f_mntfromname = "ROOT\000/5VP7RQNY.s1d", '\000' <repeats 61 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 16384, f_frsize = 16384, f_blocks = 45185024, f_bfree = 41130130, f_bavail = 41130130, f_files = 1233728, f_ffree = 0, f_favail = 0, f_fsid = 1889473200, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 2588525574, time_mid = 22253, time
_hi_and_version = 4576, clock_seq_hi_and_reserved = 147 '\223', clock_seq_low = 132 '\204', node = "mbm\311%\331"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe039f68000, mnt_time = 0, mnt_iosize_max = 131072, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe039ea26b0}, mnt_nvnodelistsize = 872, mnt_vn_use_ops = 0xffffffe00fd6d1c0, mnt_vn!
 _coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe00fd6d1c0, mnt_vn_spec_ops = 0xffffffe00fd6d400, mnt_vn_fifo_ops = 0xffffffe00fd6d640, mnt_ncmountpt = {ncp = 0xffffffe05cf90ed8, mount = 0xffffffe039ea2438}, mnt_ncmounton = {ncp = 0x0, mount = 0x0}, mnt_refs = 112, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe039ea2740}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioo
ps = 0x0}
$12 = {mnt_list = {tqe_next = 0xffffffe05d39e4b8, tqe_prev = 0xffffffe039ea2438}, mnt_op = 0xffffffff809fe500, mnt_vfc = 0xffffffff809fe4c0, mnt_namecache_gen = 699, mnt_syncer = 0xffffffe037af6c00, mnt_nvnodelist = {tqh_first = 0xffffffe037af7200, tqh_last = 0xffffffe037af6c78}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffff
ffffffffffff}, mnt_flag = 4096, mnt_kern_flag = 272564224, mnt_maxsymlinklen = 0, mnt_stat = {f_spare2 = 0, f_bsize = 512, f_iosize = 512, f_blocks = 2, f_bfree = 0, f_bavail = 0, f_files = 698, f_ffree = 0, f_fsid = {val = {167837441, 10}}, f_owner = 0, f_type = 10, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 0, f_fstypename = "devfs\000\000\000\000\000\000\000\000\000\000", f_mntonname = "/dev", '\000' <repeats 75 times>, f_syncreads = 0, f_asyncreads = 0, f_spares1 = 0, f_mntf!
 romname = "devfs", '\000' <repeats 74 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 512, f_frsize = 512, f_blocks = 2, f_bfree = 0, f_bavail = 0, f_files = 698, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\00
0\000"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe037b24d70, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05cfca230}, mnt_nvnodelistsize = 101, mnt_vn_use_ops = 0xffffffe037ad6f40, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe037ad6f40, mnt_vn_spec_ops = 0xff!
 ffffe037ad7180, mnt_vn_fifo_ops = 0x0, mnt_ncmountpt = {ncp = 0xffffffe05cf91088, mount = 0xffffffe05cfc9fb8}, mnt_ncmounton = {ncp = 0xffffffe05cf90f68, mount = 0xffffffe039ea2438}, mnt_refs = 19, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05cfca2c0}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
$13 = {mnt_list = {tqe_next = 0xffffffe05d39e7f8, tqe_prev = 0xffffffe05cfc9fb8}, mnt_op = 0xffffffff809fae40, mnt_vfc = 0xffffffff809fae00, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe05d19be00, mnt_nvnodelist = {tqh_first = 0xffffffe05d180600, tqh_last = 0xffffffe05d19be78}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffffff
ffffffffff}, mnt_flag = 4096, mnt_kern_flag = 134217728, mnt_maxsymlinklen = 60, mnt_stat = {f_spare2 = 0, f_bsize = 2048, f_iosize = 16384, f_blocks = 387047, f_bfree = 65686, f_bavail = 34723, f_files = 97278, f_ffree = 95818, f_fsid = {val = {1301764766, 1794447206}}, f_owner = 0, f_type = 1, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 2, f_fstypename = "ufs", '\000' <repeats 12 times>, f_mntonname = "/boot", '\000' <repeats 74 times>, f_syncreads = 0, f_asyncreads = 7, f_spar!
 es1 = 0, f_mntfromname = "/dev/serno/5VP7RQNY.s1a", '\000' <repeats 56 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\00
0', node = "\000\000\000\000\000"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe05d3ae1f8, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39e730}, mnt_nvnodelistsize = 5, mnt_vn_use_ops = 0xffffffe05d3ffd00, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe05d3ffd0!
 0, mnt_vn_spec_ops = 0xffffffe05d3e0080, mnt_vn_fifo_ops = 0xffffffe05d3e02c0, mnt_ncmountpt = {ncp = 0xffffffe039fad4e8, mount = 0xffffffe05d39e4b8}, mnt_ncmounton = {ncp = 0xffffffe039facf48, mount = 0xffffffe039ea2438}, mnt_refs = 1, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39e7c0}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
$14 = {mnt_list = {tqe_next = 0xffffffe05d39eb38, tqe_prev = 0xffffffe05d39e4b8}, mnt_op = 0xffffffff809d2960, mnt_vfc = 0xffffffff809d2920, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe05d19c800, mnt_nvnodelist = {tqh_first = 0xffffffe05d19c800, tqh_last = 0xffffffe05d19c878}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffffff
ffffffffff}, mnt_flag = 4096, mnt_kern_flag = 12517376, mnt_maxsymlinklen = 0, mnt_stat = {f_spare2 = 0, f_bsize = 16384, f_iosize = 16384, f_blocks = 45185024, f_bfree = 41130131, f_bavail = 41130131, f_files = 1233728, f_ffree = 0, f_fsid = {val = {1343492506, -1532862911}}, f_owner = 0, f_type = 2, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 0, f_fstypename = "null", '\000' <repeats 11 times>, f_mntonname = "/var", '\000' <repeats 75 times>, f_syncreads = 0, f_asyncreads = 0, !
 f_spares1 = 0, f_mntfromname = "/pfs/@@-1:00001", '\000' <repeats 64 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000'
, node = "\000\000\000\000\000"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe05d3e0500, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39ea70}, mnt_nvnodelistsize = 1, mnt_vn_use_ops = 0xffffffe05d3e0740, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe05d3e0740,!
  mnt_vn_spec_ops = 0x0, mnt_vn_fifo_ops = 0x0, mnt_ncmountpt = {ncp = 0xffffffe039fad7b8, mount = 0xffffffe05d39e7f8}, mnt_ncmounton = {ncp = 0xffffffe039facb58, mount = 0xffffffe039ea2438}, mnt_refs = 20, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39eb00}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
$15 = {mnt_list = {tqe_next = 0xffffffe05d39ee78, tqe_prev = 0xffffffe05d39e7f8}, mnt_op = 0xffffffff809d2960, mnt_vfc = 0xffffffff809d2920, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe05d19ce00, mnt_nvnodelist = {tqh_first = 0xffffffe05d19ce00, tqh_last = 0xffffffe05d19ce78}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffffff
ffffffffff}, mnt_flag = 4096, mnt_kern_flag = 12517376, mnt_maxsymlinklen = 0, mnt_stat = {f_spare2 = 0, f_bsize = 16384, f_iosize = 16384, f_blocks = 45185024, f_bfree = 41130131, f_bavail = 41130131, f_files = 1233728, f_ffree = 0, f_fsid = {val = {1343492506, 1311015715}}, f_owner = 0, f_type = 2, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 0, f_fstypename = "null", '\000' <repeats 11 times>, f_mntonname = "/tmp", '\000' <repeats 75 times>, f_syncreads = 0, f_asyncreads = 0, f!
 _spares1 = 0, f_mntfromname = "/pfs/@@-1:00002", '\000' <repeats 64 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000',
 node = "\000\000\000\000\000"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe05d3e0980, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39edb0}, mnt_nvnodelistsize = 1, mnt_vn_use_ops = 0xffffffe05d3e0bc0, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe05d3e0bc0, !
 mnt_vn_spec_ops = 0x0, mnt_vn_fifo_ops = 0x0, mnt_ncmountpt = {ncp = 0xffffffe039fad8d8, mount = 0xffffffe05d39eb38}, mnt_ncmounton = {ncp = 0xffffffe039faceb8, mount = 0xffffffe039ea2438}, mnt_refs = 1, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39ee40}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
$16 = {mnt_list = {tqe_next = 0xffffffe05d39f1b8, tqe_prev = 0xffffffe05d39eb38}, mnt_op = 0xffffffff809d2960, mnt_vfc = 0xffffffff809d2920, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe05d19d400, mnt_nvnodelist = {tqh_first = 0xffffffe05d19d400, tqh_last = 0xffffffe05d19d478}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffffff
ffffffffff}, mnt_flag = 4352, mnt_kern_flag = 12517376, mnt_maxsymlinklen = 0, mnt_stat = {f_spare2 = 0, f_bsize = 16384, f_iosize = 16384, f_blocks = 45185024, f_bfree = 41130121, f_bavail = 41130121, f_files = 1233728, f_ffree = 0, f_fsid = {val = {1343492506, -1586088510}}, f_owner = 0, f_type = 2, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 0, f_fstypename = "null", '\000' <repeats 11 times>, f_mntonname = "/usr", '\000' <repeats 75 times>, f_syncreads = 0, f_asyncreads = 0, !
 f_spares1 = 0, f_mntfromname = "/pfs/@@-1:00003", '\000' <repeats 64 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000'
, node = "\000\000\000\000\000"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe05d3e0e00, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39f0f0}, mnt_nvnodelistsize = 1, mnt_vn_use_ops = 0xffffffe05d3e1040, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe05d3e1040,!
  mnt_vn_spec_ops = 0x0, mnt_vn_fifo_ops = 0x0, mnt_ncmountpt = {ncp = 0xffffffe039fadba8, mount = 0xffffffe05d39ee78}, mnt_ncmounton = {ncp = 0xffffffe039faa3f8, mount = 0xffffffe039ea2438}, mnt_refs = 22, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39f180}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
$17 = {mnt_list = {tqe_next = 0xffffffe05d39f4f8, tqe_prev = 0xffffffe05d39ee78}, mnt_op = 0xffffffff809d2960, mnt_vfc = 0xffffffff809d2920, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe05d19da00, mnt_nvnodelist = {tqh_first = 0xffffffe05d19da00, tqh_last = 0xffffffe05d19da78}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffffff
ffffffffff}, mnt_flag = 4096, mnt_kern_flag = 12517376, mnt_maxsymlinklen = 0, mnt_stat = {f_spare2 = 0, f_bsize = 16384, f_iosize = 16384, f_blocks = 45185024, f_bfree = 41130131, f_bavail = 41130131, f_files = 1233728, f_ffree = 0, f_fsid = {val = {1343492506, 1079525286}}, f_owner = 0, f_type = 2, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 0, f_fstypename = "null", '\000' <repeats 11 times>, f_mntonname = "/home", '\000' <repeats 74 times>, f_syncreads = 0, f_asyncreads = 0, !
 f_spares1 = 0, f_mntfromname = "/pfs/@@-1:00004", '\000' <repeats 64 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000'
, node = "\000\000\000\000\000"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe05d3e1280, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39f430}, mnt_nvnodelistsize = 1, mnt_vn_use_ops = 0xffffffe05d3e14c0, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe05d3e14c0,!
  mnt_vn_spec_ops = 0x0, mnt_vn_fifo_ops = 0x0, mnt_ncmountpt = {ncp = 0xffffffe039fadd58, mount = 0xffffffe05d39f1b8}, mnt_ncmounton = {ncp = 0xffffffe039face28, mount = 0xffffffe039ea2438}, mnt_refs = 1, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39f4c0}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
$18 = {mnt_list = {tqe_next = 0xffffffe05d39f838, tqe_prev = 0xffffffe05d39f1b8}, mnt_op = 0xffffffff809d2960, mnt_vfc = 0xffffffff809d2920, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe05d19e200, mnt_nvnodelist = {tqh_first = 0xffffffe05d19e200, tqh_last = 0xffffffe05d19e278}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffffff
ffffffffff}, mnt_flag = 4096, mnt_kern_flag = 12517376, mnt_maxsymlinklen = 0, mnt_stat = {f_spare2 = 0, f_bsize = 16384, f_iosize = 16384, f_blocks = 45185024, f_bfree = 41130131, f_bavail = 41130131, f_files = 1233728, f_ffree = 0, f_fsid = {val = {1343492506, -1358263993}}, f_owner = 0, f_type = 2, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 0, f_fstypename = "null", '\000' <repeats 11 times>, f_mntonname = "/usr/obj", '\000' <repeats 71 times>, f_syncreads = 0, f_asyncreads =!
  0, f_spares1 = 0, f_mntfromname = "/pfs/@@-1:00005", '\000' <repeats 64 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\
000', node = "\000\000\000\000\000"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe05d3e1700, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39f770}, mnt_nvnodelistsize = 1, mnt_vn_use_ops = 0xffffffe05d3e1940, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe05d3e1!
 940, mnt_vn_spec_ops = 0x0, mnt_vn_fifo_ops = 0x0, mnt_ncmountpt = {ncp = 0xffffffe039fae268, mount = 0xffffffe05d39f4f8}, mnt_ncmounton = {ncp = 0xffffffe039fae028, mount = 0xffffffe05d39ee78}, mnt_refs = 1, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39f800}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
$19 = {mnt_list = {tqe_next = 0xffffffe05d39fb78, tqe_prev = 0xffffffe05d39f4f8}, mnt_op = 0xffffffff809d2960, mnt_vfc = 0xffffffff809d2920, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe05d19ea00, mnt_nvnodelist = {tqh_first = 0xffffffe05d19ea00, tqh_last = 0xffffffe05d19ea78}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffffff
ffffffffff}, mnt_flag = 4096, mnt_kern_flag = 12517376, mnt_maxsymlinklen = 0, mnt_stat = {f_spare2 = 0, f_bsize = 16384, f_iosize = 16384, f_blocks = 45185024, f_bfree = 41130131, f_bavail = 41130131, f_files = 1233728, f_ffree = 0, f_fsid = {val = {1343492506, 1166824485}}, f_owner = 0, f_type = 2, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 0, f_fstypename = "null", '\000' <repeats 11 times>, f_mntonname = "/var/crash", '\000' <repeats 69 times>, f_syncreads = 0, f_asyncreads !
 = 0, f_spares1 = 0, f_mntfromname = "/pfs/@@-1:00006", '\000' <repeats 64 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '
\000', node = "\000\000\000\000\000"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe05d3e1b80, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39fab0}, mnt_nvnodelistsize = 1, mnt_vn_use_ops = 0xffffffe05d3e1dc0, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe05d3e!
 1dc0, mnt_vn_spec_ops = 0x0, mnt_vn_fifo_ops = 0x0, mnt_ncmountpt = {ncp = 0xffffffe039fae6e8, mount = 0xffffffe05d39f838}, mnt_ncmounton = {ncp = 0xffffffe039fae538, mount = 0xffffffe05d39e7f8}, mnt_refs = 1, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39fb40}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
$20 = {mnt_list = {tqe_next = 0xffffffe05d380078, tqe_prev = 0xffffffe05d39f838}, mnt_op = 0xffffffff809d2960, mnt_vfc = 0xffffffff809d2920, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe05d19f200, mnt_nvnodelist = {tqh_first = 0xffffffe05d19f200, tqh_last = 0xffffffe05d19f278}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffffff
ffffffffff}, mnt_flag = 4096, mnt_kern_flag = 12517376, mnt_maxsymlinklen = 0, mnt_stat = {f_spare2 = 0, f_bsize = 16384, f_iosize = 16384, f_blocks = 45185024, f_bfree = 41130131, f_bavail = 41130131, f_files = 1233728, f_ffree = 0, f_fsid = {val = {1343492506, -1428233532}}, f_owner = 0, f_type = 2, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 0, f_fstypename = "null", '\000' <repeats 11 times>, f_mntonname = "/var/tmp", '\000' <repeats 71 times>, f_syncreads = 0, f_asyncreads =!
  0, f_spares1 = 0, f_mntfromname = "/pfs/@@-1:00007", '\000' <repeats 64 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\
000', node = "\000\000\000\000\000"}, f_uid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0xffffffe05d3e2000, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39fdf0}, mnt_nvnodelistsize = 1, mnt_vn_use_ops = 0xffffffe05d3e2240, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe05d3e2!
 240, mnt_vn_spec_ops = 0x0, mnt_vn_fifo_ops = 0x0, mnt_ncmountpt = {ncp = 0xffffffe039faead8, mount = 0xffffffe05d39fb78}, mnt_ncmounton = {ncp = 0xffffffe039fae928, mount = 0xffffffe05d39e7f8}, mnt_refs = 1, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d39fe80}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
$21 = {mnt_list = {tqe_next = 0x0, tqe_prev = 0xffffffe05d39fb78}, mnt_op = 0xffffffff809d2f40, mnt_vfc = 0xffffffff809d2f00, mnt_namecache_gen = 0, mnt_syncer = 0xffffffe05d19fa00, mnt_nvnodelist = {tqh_first = 0xffffffe05d718a00, tqh_last = 0xffffffe05d19fa78}, mnt_lock = {lk_spinlock = {counta = 0, countb = 0}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_unused1 = 0, lk_wmesg = 0xffffffff80719839 "vfslock", lk_timo = 0, lk_lockholder = 0xffffffffffffffff}, mn
t_flag = 4096, mnt_kern_flag = 268435456, mnt_maxsymlinklen = 0, mnt_stat = {f_spare2 = 0, f_bsize = 4096, f_iosize = 4096, f_blocks = 1, f_bfree = 0, f_bavail = 0, f_files = 6164, f_ffree = 6158, f_fsid = {val = {117505794, 7}}, f_owner = 0, f_type = 7, f_flags = 4096, f_syncwrites = 0, f_asyncwrites = 0, f_fstypename = "procfs\000\000\000\000\000\000\000\000\000", f_mntonname = "/proc", '\000' <repeats 74 times>, f_syncreads = 0, f_asyncreads = 0, f_spares1 = 0, f_mntfromname = "proc!
 fs", '\000' <repeats 73 times>, f_spares2 = 0, f_spare = {0, 0}}, mnt_vstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, f_owner = 0, f_type = 0, f_syncreads = 0, f_syncwrites = 0, f_asyncreads = 0, f_asyncwrites = 0, f_fsid_uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}, f_uid_uuid 
= {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq_hi_and_reserved = 0 '\000', clock_seq_low = 0 '\000', node = "\000\000\000\000\000"}}, mnt_data = 0x0, mnt_time = 0, mnt_iosize_max = 65536, mnt_reservedvnlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d3802f0}, mnt_nvnodelistsize = 2, mnt_vn_use_ops = 0xffffffe05d3e2480, mnt_vn_coherency_ops = 0x0, mnt_vn_journal_ops = 0x0, mnt_vn_norm_ops = 0xffffffe05d3e2480, mnt_vn_spec_ops = 0x0, mnt_vn_fifo_ops = 0x0, mnt_ncmountpt!
  = {ncp = 0xffffffe039faf198, mount = 0xffffffe05d380078}, mnt_ncmounton = {ncp = 0xffffffe039facac8, mount = 0xffffffe039ea2438}, mnt_refs = 1, mnt_token = {t_ref = 0x0, t_collisions = 0, t_collmask = 0, t_desc = 0xffffffff80719841 "permnt"}, mnt_jlist = {tqh_first = 0x0, tqh_last = 0xffffffe05d380380}, mnt_jbitmap = 0x0, mnt_streamid = 0, mnt_bioops = 0x0}
(kgdb) 
Id Refs Address    Size     Name
 1    7 0x80200000 e64500   kernel
 2    1 0x81065000 620368   acpi.ko
 3    1 0x81686000 bac88    ahci.ko
 4    1 0x81741000 40658    ehci.ko
(kgdb) 

------------------------------------------------------------------------
ps -axl

Segmentation fault (core dumped)

------------------------------------------------------------------------
vmstat -s

   124531 cpu context switches
   234233 device interrupts
      334 software interrupts
        0 traps
  5068912 system calls
        1 kernel threads created
      440  fork() calls
        0 vfork() calls
        0 rfork() calls
      293 exec() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
      175 vnode pager pageins
      175 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
        0 pages reactivated
     8900 copy-on-write faults
        0 copy-on-write optimized faults
   253467 zero fill pages zeroed
    80474 zero fill pages prezeroed
       16 intransit blocking page faults
   496968 total VM faults taken
        0 pages affected by kernel thread creation
    28166 pages affected by  fork()
        0 pages affected by vfork()
        0 pages affected by rfork()
   866612 pages freed
        0 pages freed by daemon
    35288 pages freed by exiting processes
    84183 pages active
   344809 pages inactive
        0 pages in VM cache
   168115 pages wired down
   341638 pages free
     4096 bytes per page
   173351 global smp invltlbs
     2607 total path lookups
     6037 total component lookups
          cache hits (83% pos + 6% neg)
          deletions 0%, falsehits 0%

------------------------------------------------------------------------
vmstat -m


Memory statistics by type                          Type  Kern
              Type   InUse  MemUse HighUse   Limit  Requests  Limit Limit
      HAMMER-inodes    876    767K      0K134203388K       898    0     0
      HAMMER-others   1273    242K      0K  374784K      4623    0     0
            acpidev     87      4K      0K  374784K        87    0     0
           pci_link     36      3K      0K  374784K        36    0     0
            acpisem     16      1K      0K  374784K        16    0     0
           acpitask      0      0K      0K  374784K         2    0     0
             acpica    309     15K      0K  374784K     76353    0     0
            subproc     42    131K      0K  374784K       813    0     0
           atkbddev      2      1K      0K  374784K         2    0     0
         acd_driver      1      3K      0K  374784K         1    0     0
            ata_cam      6      4K      0K  374784K        24    0     0
           objcache     64     13K      0K  374784K        64    0     0
  objcache magazine   1487   1000K      0K  374784K      1487    0     0
           nexusdev      3      1K      0K  374784K         3    0     0
              lockf      3      1K      0K  374784K        18    0     0
            syscons      6     23K      0K  374784K         6    0     0
             crypto      1      1K      0K  374784K         1    0     0
           propstng   1378     54K      0K  374784K      1378    0     0
        prop string   1371     11K      0K  374784K      1482    0     0
           propnmbr   1629     90K      0K  374784K      1629    0     0
            pdict16     20      2K      0K  374784K        20    0     0
           propdict    675    127K      0K  374784K       675    0     0
    prop dictionary    671    168K      0K  374784K       724    0     0
             kbdmux      6      8K      0K  374784K         6    0     0
             isadev     25      2K      0K  374784K        25    0     0
                USB     32      4K      0K  374784K        36    0     0
             USBdev      3      1K      0K  374784K         9    0     0
               ZONE      1      1K      0K  374784K         1    0     0
             atexit      1      1K      0K  374784K         1    0     0
            vmspace     44     28K      0K  374784K        44    0     0
          proc-args     39      2K      0K  374784K       544    0     0
          exec-args      5   1300K      0K  374784K         5    0     0
       HAMMER-mount      1     68K      0K  374784K         1    0     0
              devfs   3728    589K      0K  374784K      6615    0     0
             kqueue     35      5K      0K  374784K       109    0     0
        UFS dirhash      7      5K      0K  374784K         7    0     0
          UFS mount      3      5K      0K  374784K         3    0     0
          UFS ihash      1   4096K      0K  374784K         1    0     0
           FFS node      4      2K      0K134203388K         4    0     0
            pagedep      1    512K      0K  374784K         1    0     0
           inodedep      1   4096K      0K  374784K         1    0     0
             newblk      1      1K      0K  374784K         1    0     0
           p1003.1b      1      1K      0K  374784K         1    0     0
               kenv     20      5K      0K  374784K        20    0     0
          file desc     42     32K      0K  374784K       821    0     0
               file     79      9K      0K  374784K      5303    0     0
              sigio      1      1K      0K  374784K         1    0     0
                kld     78      6K      0K  374784K       110    0     0
          ar_driver      0      0K      0K  374784K        10    0     0
             module    259     24K      0K  374784K       260    0     0
             varsym    222      8K      0K  374784K       226    0     0
         NFS daemon     21     10K      0K  374784K        21    0     0
           NFS hash      1   4096K      0K  374784K         1    0     0
        NFS srvsock      2      2K      0K  374784K         2    0     0
       ip6_moptions      1      1K      0K  374784K         1    0     0
           syncache      4     48K      0K  374784K         4    0     0
            tcptemp     25      2K      0K  374784K        25    0     0
                ipq     50      3K      0K  374784K        50    0     0
           in_multi     13      1K      0K  374784K        13    0     0
            ata_dma      6      2K      0K  374784K         6    0     0
               CCMS   1017     88K      0K  374784K      1017    0     0
          80211scan      3      4K      0K  374784K         4    0     0
       80211ratectl      2      1K      0K  374784K         2    0     0
          80211node      1     11K      0K  374784K         3    0     0
        80211nodeie      3      1K      0K  374784K         6    0     0
           80211com      1      7K      0K  374784K         1    0     0
        80211crypto      2      1K      0K  374784K         2    0     0
           80211vap      1      3K      0K  374784K         1    0     0
           routetbl    180     30K      0K  374784K       269    0     0
              faith      1      2K      0K  374784K         1    0     0
              clone      3     12K      0K  374784K         3    0     0
             ifaddr    330     39K      0K  374784K       330    0     0
        ether_multi     47      3K      0K  374784K        67    0     0
              ifnet      2      3K      0K  374784K         2    0     0
                BPF     17     74K      0K  374784K        24    0     0
      MSDOSFS mount      1   4096K      0K  374784K         1    0     0
       NULLFS mount      7      4K      0K  374784K         7    0     0
             vnodes   1017    509K      0K134203388K      1017    0     0
        Export Host      2      1K      0K  374784K         2    0     0
           vnodeops     18     11K      0K  374784K        18    0     0
          nameibufs      7      7K      0K  374784K         7    0     0
              mount     12     10K      0K  374784K        15    0     0
           vfscache   2600   4294K      0K  374784K      2610    0     0
         BIO buffer      0      0K      0K  374784K        15    0     0
              unpcb      6      2K      0K  374784K        52    0     0
          ad_driver      2      1K      0K  374784K         2    0     0
             socket     34     16K      0K  374784K       156    0     0
             soname      4      1K      0K  374784K       281    0     0
                pcb     66     54K      0K  374784K       159    0     0
                tag      0      0K      0K  374784K         8    0     0
               mbuf    392    196K      0K  374784K       392    0     0
             mbufcl     82    164K      0K  374784K        82    0     0
            mclmeta     82      2K      0K  374784K        82    0     0
        ata_generic     97     26K      0K  374784K        97    0     0
               ptys    256    128K      0K  374784K       256    0     0
               ttys    134     18K      0K  374784K       134    0     0
                shm      7    121K      0K  374784K         9    0     0
            CAM SIM      7      2K      0K  374784K         7    0     0
        ISOFS mount      1   4096K      0K  374784K         1    0     0
                sem      3      6K      0K  374784K         3    0     0
         CAM periph      5      1K      0K  374784K        12    0     0
            scsi_cd      0      0K      0K  374784K         1    0     0
      CAM dev queue      7      1K      0K  374784K         7    0     0
           ppbusdev      3      1K      0K  374784K         3    0     0
                msg      4     27K      0K  374784K         4    0     0
            CAM XPT     62     17K      0K  374784K       211    0     0
               rman    123     13K      0K  374784K       547    0     0
               pipe     23      5K      0K  374784K       435    0     0
           ioctlops      0      0K      0K  374784K        30    0     0
          taskqueue     14      1K      0K  374784K        14    0     0
               sbuf      0      0K      0K  374784K         2    0     0
               SWAP      2   8741K      0K  374784K         2    0     0
          CAM queue     23      1K      0K  374784K       169    0     0
               kobj    193    435K      0K  374784K       193    0     0
       eventhandler     29      2K      0K  374784K        29    0     0
               disk      6      1K      0K  374784K         6    0     0
            MD disk      2      1K      0K  374784K         2    0     0
                bus   1054     80K      0K  374784K      6988    0     0
            callout      4   8192K      0K  374784K         4    0     0
             sysctl      0      0K      0K  374784K       466    0     0
          sysctloid    686     24K      0K  374784K       686    0     0
            tslpque      3    192K      0K  374784K         3    0     0
             dsched    947    313K      0K  374784K       947    0     0
       lwkt message      9      3K      0K  374784K       645    0     0
             thread     82     93K      0K  374784K        82    0     0
            memdesc      1      3K      0K  374784K         1    0     0
        MPipe Array      2      1K      0K  374784K         2    0     0
              cache   2697    246K      0K  374784K      2713    0     0
             devbuf    772    135K      0K  374784K      1228    0     0
               temp    259     15K      0K  374784K    124832    0     0
             ip6ndp     10      1K      0K  374784K        10    0     0
            uidinfo      4      5K      0K  374784K         5    0     0
               cred      3      1K      0K  374784K         9    0     0
               pgrp     17      3K      0K  374784K        19    0     0
            session     16      1K      0K  374784K        18    0     0
               proc     42     77K      0K  374784K       812    0     0
                lwp     40     20K      0K  374784K       810    0     0

Memory Totals:  In Use    Free    Requests
                50276K      0K      253723

------------------------------------------------------------------------
vmstat -z


ITEM            SIZE     LIMIT    USED    FREE  REQUESTS

SWAPMETA:        192,  3050402,      1,     20,        1
PV ENTRY:         80,  2543456, 113412, 1197295,   558991
MAP ENTRY:       104,        0,    518,   1530,     1162
MAP:             272,        0,      0,     10,        0
VM OBJECT:       176,        0,    496,     59,     5120


------------------------------------------------------------------------
vmstat -i

interrupt                    total       rate
atkbd0                           6          0
sio1                             0          0
sio0                             0          0
re0                              0          0
ohci0/atapci1/atapci2        29603         72
ral0                          3548          8
ata0                             0          0
ata1                             0          0
swi_siopoll                      0          0
swi_cambio                      71          0
swi_vm                           0          0
swi_mp_taskq/swi_taskq       59174        145
Total                        92402        226

------------------------------------------------------------------------
pstat -T

 79/98624 files
0M/32767M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity  Type
/dev/#C19:0x20051   67108608        0 67108608     0%    Interleaved

------------------------------------------------------------------------
ipcs -a

Message Queues:
T     ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP CBYTES  QNUM QBYTES LSPID LRPID   STIME    RTIME    CTIME

Shared Memory:
T     ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP NATTCH         SEGSZ   CPID   LPID    ATIME    DTIME    CTIME
m  65536    5432001 --rw-------    pgsql    pgsql    pgsql    pgsql      4     29638656    791    791 15:15:00 15:15:00 15:15:00

Semaphores:
T     ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP NSEMS   OTIME    CTIME
s  65536    5432001 --rw-------    pgsql    pgsql    pgsql    pgsql     17no-entry 15:15:00
s  65537    5432002 --rw-------    pgsql    pgsql    pgsql    pgsql     17no-entry 15:15:00
s  65538    5432003 --rw-------    pgsql    pgsql    pgsql    pgsql     17no-entry 15:15:00


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:  16384	(max characters in a message)
	msgmni:     40	(# of message queues)
	msgmnb:   2048	(max characters in a message queue)
	msgtql:     40	(max # of messages in system)
	msgssz:      8	(size of a message segment)
	msgseg:   2048	(# of message segments in system)

shminfo:
	shmmax:   2558943232	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          512	(max number of shared memory identifiers)
	shmseg:         1024	(max shared memory segments per process)
	shmall:       624742	(max amount of shared memory in pages)

seminfo:
	semmap:     30	(# of entries in semaphore map)
	semmni:     10	(# of semaphore identifiers)
	semmns:     60	(# of semaphores in system)
	semmnu:     30	(# of undo structures in system)
	semmsl:     60	(max # of semaphores per id)
	semopm:    100	(max # of operations per semop call)
	semume:     10	(max # of undo entries per process)
	semusz:    104	(size in bytes of undo structure)
	semvmx:  32767	(semaphore maximum value)
	semaem:  16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit    GLease    Vacate     Evict
        0         0         0         0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses
        0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit    GLease    Vacate     Evict
        0         0         0         0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Lease Stats:
   Leases     PeakL   GLeases
        0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
	0 packets sent
		0 data packets (0 bytes)
		0 data packets (0 bytes) retransmitted
		0 Fast Retransmits (0 early)
		0 packets sent by Limited Transmit
		0 spurious RTO retransmits
		0 spurious Fast Retransmits (0 early)
		0 Eifel-detected spurious retransmits
		0 RTT-detected spurious retransmits
		0 resends initiated by MTU discovery
		0 ack-only packets (0 delayed)
		0 URG only packets
		0 window probe packets
		0 window update packets
		0 control packets
	0 packets received
		0 acks (for 0 bytes)
		0 acks for unsent data
		0 packets (0 bytes) received in-sequence
		0 completely duplicate packets (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		0 window update packets
		0 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
	0 connection requests
	0 connection accepts
	0 bad connection attempts
	0 listen queue overflows
	0 connections established (including accepts)
	0 connections closed (including 0 drops)
		0 connections updated cached RTT on close
		0 connections updated cached RTT variance on close
		0 connections updated cached ssthresh on close
	0 embryonic connections dropped
	0 segments updated rtt (of 0 attempts)
	0 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	0 correct ACK header predictions
	0 correct data packet header predictions
	0 syncache entries added
		0 retransmitted
		0 dupsyn
		0 dropped
		0 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	0 cookies sent
	0 cookies received
udp:
	11 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	0 with no checksum
	0 dropped due to no socket
	11 broadcast/multicast datagrams dropped due to no socket
	0 dropped due to full socket buffers
	0 not for hashed pcb
	0 delivered
	0 datagrams output
ip:
	29 total packets received
	18 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	11 packets for this host
	0 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	0 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	0 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
icmp:
	0 calls to icmp_error
	0 errors not generated 'cuz old message was icmp
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	0 message responses generated
	0 invalid return addresses
	0 no return routes
	ICMP address mask responses are disabled
igmp:
	0 messages received
	0 messages received with too few bytes
	0 messages received with bad checksum
	0 membership queries received
	0 membership queries received with invalid field(s)
	0 membership reports received
	0 membership reports received with invalid field(s)
	0 membership reports received for groups to which we belong
	0 membership reports sent
ip6:
	1 total packet received
	0 with size smaller than minimum
	0 with data size < data length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 fragments that exceeded limit
	0 packets reassembled ok
	0 packets for this host
	0 packets forwarded
	1 packet not forwardable
	0 redirects sent
	6 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 packets that violated scope rules
	1 multicast packet which we don't join
	Input histogram:
		UDP: 1
	Mbuf statistics:
		0 one mbuf
		1 one ext mbuf
		0 two or more ext mbuf
	0 packets whose headers are not continuous
	0 tunneling packets that can't find gif
	0 packets discarded due to too may headers
	0 failures of source address selection
	0 forward cache hit
	0 forward cache miss
icmp6:
	0 calls to icmp_error
	0 errors not generated because old message was icmp error or so
	0 errors not generated because rate limitation
	Output histogram:
		multicast listener report: 6
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	Histogram of error messages to be generated:
		0 no route
		0 administratively prohibited
		0 beyond scope
		0 address unreachable
		0 port unreachable
		0 packet too big
		0 time exceed transit
		0 time exceed reassembly
		0 erroneous header field
		0 unrecognized next header
		0 unrecognized option
		0 redirect
		0 unknown
	0 message responses generated
	0 messages with too many ND options
	0 messages with bad ND options
	0 bad neighbor solicitation messages
	0 bad neighbor advertisement messages
	0 bad router solicitation messages
	0 bad router advertisement messages
	0 bad redirect messages
	0 path MTU changes
ddp:
	0 packets with short headers 
	0 packets with long headers 
	0 packets with no checksum 
	0 packets too short 
	0 packets with bad checksum 
	0 packets with not enough data 
	0 packets forwarded 
	0 packets encapsulated 
	0 packets rcvd for unreachable dest 
	0 packets dropped due to no socket space 

------------------------------------------------------------------------
netstat -m

1/111936 mbufs in use (current/max):
64/6656 mbuf clusters in use (current/max)
	65 mbufs and mbuf clusters allocated to data
128 Kbytes allocated to network (0% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

------------------------------------------------------------------------
netstat -id





More information about the Bugs mailing list