Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)

Mon Aug 8 12:02:33 PDT 2016

* Matthew Dillon on Mon, Aug 08, 2016 at 11:08:40AM -0700:
> And (2) What is the device attachment for the swap drive (in the dmesg
> output).

The swap drive is vbd1s0b, with the following attachment:
| virtio_pci2: <VirtIO PCI Block adapter> port 0xc040-0xc07f mem 0xfebf1000-0xfebf1fff irq 10 at device 6.0 on pci0
| vtblk1: <VirtIO Block Adapter> on virtio_pci2
| virtio_pci2: host features: 0x710006d4 <EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
| virtio_pci2: negotiated features: 0x10000254 <RingIndirect,WriteCache,BlockSize,DiskGeometry,MaxNumSegs>

I have attached the full dmesg for reference.

> Indefinite wait buffer typically means the disk driver locked up while the
> system was trying to swap to/from it.  Two questions:  (1) Is the swap
> encrypted ?

It is indeed encrypted.  We don’t want to trust the data center
operator, hence everything with the exception of /boot is encrypted.

>From fstab:
| /dev/part-by-uuid/3c563c78-c204-11e5-aff0-010000000000 none swap sw,crypt 0 0

    Stefan

> On Mon, Aug 8, 2016 at 5:18 AM, Stefan Unterweger <232.20711 at chiffre.aleturo.com> wrote:
> > Another crash with 09b0c4f76, but this one looks different.  The console
> > was scrolling endless streams of lines like this one:

> > | swap_pager: indefinite wait buffer:  bp 0xffffffe00e57bea8 offset: 4133535744, size: 4096

-- 
▪ Die Internetbleibe.  Schick, magisch, leistungsstark.  https://internetbleibe.de/
▪ medoly media UG (haftungsbeschränkt) | Hausburgstraße 13, 10249 Berlin
▪ info at medolymedia.de | https://medolymedia.de/ | Tel. 030 609 826‒560 | Fax …‒569
▪ Geschäftsführer: Matthias Nothhaft | HRB 131198 (Amtsgericht Berlin-Charlottenburg), Sitz: Berlin, USt-ID: DE275221203
-------------- next part --------------
Copyright (c) 2003-2016 The DragonFly Project.
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
DragonFly 4.6-RELEASE #6: Sun Aug  7 01:22:04 CEST 2016
    root at rhaal:/usr/obj/usr/src/sys/X86_64_GENERIC
i8254 clock: 1193168 Hz
TSC clock: 2600099450 Hz
CPU: AMD Opteron 62xx class CPU (2600.10-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x600f12  Stepping = 2
  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0x96982203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,AVX,VMM>
  AMD Features=0x24500800<SYSCALL,NX,MMX+,Page1GB,LM>
  AMD Features2=0x10be7<LAHF,CMP,SVM,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4>
  MONITOR/MWAIT Features=0x2<INTBRK>
real memory  = 4293504000 (4094 MB)
avail memory = 3953410048 (3770 MB)
lapic: divisor index 0, frequency 500003421 Hz
SMI Frequency (worst case): 37037 Hz (27 us)
Initialize MI interrupts
wdog: In-kernel automatic watchdog reset enabled
kbd1 at kbdmux0
md0: Preloaded image <initrd.img> 15728640 bytes at 0xffffffff8270d580
md1: Malloc disk
ACPI: RSDP 0x00000000000FC920 000014 (v00 BOCHS )
ACPI: RSDT 0x00000000DFFFBCA0 000040 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
ACPI: FACP 0x00000000DFFFFF80 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
ACPI: DSDT 0x00000000DFFFBCE0 00151D (v01 BXPC   BXDSDT   00000001 INTL 20100528)
ACPI: FACS 0x00000000DFFFFF40 000040
ACPI: APIC 0x00000000DFFFFC60 000270 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
ACPI: HPET 0x00000000DFFFFC20 000038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
ACPI: SRAT 0x00000000DFFFF770 0004A8 (v01 BOCHS  BXPCSRAT 00000001 BXPC 00000001)
ACPI: SSDT 0x00000000DFFFD8E0 001E8E (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
ACPI: SSDT 0x00000000DFFFD870 00003D (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
ACPI: SSDT 0x00000000DFFFD200 00066E (v01 BXPC   BXSSDTPC 00000001 INTL 20100528)
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
rdrand0: No RdRand support.
acpi0: <BOCHS BXPCRSDT> on motherboard
ACPI: 4 ACPI AML tables successfully acquired and loaded
ACPI FADT: SCI testing interrupt mode ...
ACPI FADT: SCI select level/low
objcache_reclaimlist
objcache_reclaimlist
objcache_reclaimlist
objcache_reclaimlist
acpi0: Power Button (fixed)
acpi_timer0 on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
acpi_hpet0: frequency 100000000
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci_link4: Unable to route IRQs: AE_NOT_FOUND
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0xc160-0xc16f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 1.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
acd0: DVDROM <QEMU DVD-ROM/1.0> at ata1-master WDMA2
uhci0: <Intel 82371SB (PIIX3) USB controller> port 0xc100-0xc11f irq 11 at device 1.2 on pci0
usbus0: controller did not stop
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
pci0: <bridge> (vendor 0x8086, dev 0x7113) at device 1.3 irq 9
vgapci0: <VGA-compatible display> mem 0xfd000000-0xfdffffff at device 2.0 on pci0
vgapci0: Boot video device
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
virtio_pci0: <VirtIO PCI Balloon adapter> port 0xc120-0xc13f irq 11 at device 3.0 on pci0
virtio_pci1: <VirtIO PCI Block adapter> port 0xc000-0xc03f mem 0xfebf0000-0xfebf0fff irq 10 at device 5.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci1
virtio_pci1: host features: 0x710006d4 <EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci1: negotiated features: 0x10000254 <RingIndirect,WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci2: <VirtIO PCI Block adapter> port 0xc040-0xc07f mem 0xfebf1000-0xfebf1fff irq 10 at device 6.0 on pci0
vtblk1: <VirtIO Block Adapter> on virtio_pci2
virtio_pci2: host features: 0x710006d4 <EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci2: negotiated features: 0x10000254 <RingIndirect,WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci3: <VirtIO PCI Network adapter> port 0xc140-0xc15f mem 0xfebf2000-0xfebf2fff irq 11 at device 7.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci3
virtio_pci3: host features: 0x711f8060 <EventIdx,RingIndirect,NotifyOnEmpty,RxModeExtra,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,TxAllGSO,MacAddress>
virtio_pci3: negotiated features: 0x110f8020 <RingIndirect,NotifyOnEmpty,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,MacAddress>
vtnet0: MAC address: 02:01:06:f6:1b:63
add dynamic link state
virtio_pci4: <VirtIO PCI Block adapter> port 0xc080-0xc0bf mem 0xfebf3000-0xfebf3fff irq 11 at device 8.0 on pci0
vtblk2: <VirtIO Block Adapter> on virtio_pci4
virtio_pci4: host features: 0x710006d4 <EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci4: negotiated features: 0x10000254 <RingIndirect,WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci5: <VirtIO PCI Block adapter> port 0xc0c0-0xc0ff mem 0xfebf4000-0xfebf4fff irq 10 at device 10.0 on pci0
vtblk3: <VirtIO Block Adapter> on virtio_pci5
virtio_pci5: host features: 0x710006d4 <EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci5: negotiated features: 0x10000254 <RingIndirect,WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model IntelliMouse Explorer, device ID 4
cpu0: <ACPI CPU> on acpi0
cpu_cst0: <ACPI CPU C-State> on cpu0
cpu1: <ACPI CPU> on acpi0
cpu_cst1: <ACPI CPU C-State> on cpu1
ACPI: Enabled 16 GPEs in block 00 to 0F
orm0: <ISA Option ROM> at iomem 0xe7800-0xeffff on isa0
pmtimer0 on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x500 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1: can't drain, serial port might not exist, disabling
uhub0: 2 ports with 2 removable, self powered
hpt27xx: no controller detected.
CAM: Configuring 2 busses
CAM: finished configuring all busses
cd0 at ata1 bus 0 target 0 lun 0
cd0: <QEMU QEMU DVD-ROM 1.0> Removable CD-ROM SCSI-0 device 
cd0: 16.000MB/s transfers
cd0: cd present [329728 x 2048 byte records]
ugen0.2: <QEMU> at usbus0
uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 1.00/0.00, addr 2> on usbus0
no B_DEVMAGIC (bootdev=0)
Device Mapper version 4.16.0 loaded
dm_target_zero: Successfully initialized
dm_target_crypt: Successfully initialized
dm_target_error: Successfully initialized
Mounting root from ufs:md0s0
DMA space used: 11060k, remaining available: 131072k
Mounting devfs
dm_target_crypt: Setting min/max mpipe buffers: 2/39
dm_target_crypt: Setting min/max mpipe buffers: 2/39
HAMMER(Rhaal) recovery check seqno=08be4d76
HAMMER(Rhaal) recovery range 300000001dba3a60-300000001dba3a60
HAMMER(Rhaal) recovery nexto 300000001dba3a60 endseqno=08be4d77
HAMMER(Rhaal) mounted clean, no recovery needed
chroot_kernel: set new rootnch/rootvnode to /new_root
dm_target_crypt: Setting min/max mpipe buffers: 2/39
dm_target_crypt: Setting min/max mpipe buffers: 2/39
dm_target_crypt: Setting min/max mpipe buffers: 2/39
dm_target_crypt: Setting min/max mpipe buffers: 2/39
dm_target_crypt: Setting min/max mpipe buffers: 2/39
HAMMER: read-only -> read-write
HAMMER(Rhaal-Daten) recovery check seqno=65c8ff52
HAMMER(Rhaal-Daten) recovery range 300000000b81f6b0-300000000b920bc0
HAMMER(Rhaal-Daten) recovery nexto 300000000b82bb50 endseqno=65c907c1
HAMMER(Rhaal-Daten) recovery undo  300000000b81f6b0-300000000b920bc0 (1053968 bytes)(RW)
HAMMER(Rhaal-Daten) Found REDO_SYNC 300000000b82bb50
HAMMER(Rhaal-Daten) recovery complete
HAMMER(Rhaal-Daten) recovery redo  300000000b81f6b0-300000000b920bc0 (1053968 bytes)(RW)
HAMMER(Rhaal-Daten) Embedded extended redo 300000000b82bb50, -50336 extbytes
HAMMER: unable to find objid 0000000189ab8f52:00060000
HAMMER: unable to find objid 0000000189ab8f52:00060000
HAMMER: unable to find objid 0000000189ab8f52:00060000
HAMMER: unable to find objid 0000000189ab8f52:00060000
HAMMER: unable to find objid 0000000189ac0f51:00060000
HAMMER: unable to find objid 0000000189ac0f51:00060000
HAMMER: unable to find objid 0000000189ac0f51:00060000
HAMMER: unable to find objid 0000000189ac0f51:00060000
HAMMER(Rhaal-Daten) End redo recovery
HAMMER(Wornkel-2) recovery check seqno=029316e9
HAMMER(Wornkel-2) recovery range 300000000d3b1f00-300000000d3b1f00
HAMMER(Wornkel-2) recovery nexto 300000000d3b1f00 endseqno=029316ea
HAMMER(Wornkel-2) mounted clean, no recovery needed
dm_target_crypt: Setting min/max mpipe buffers: 2/39
swap low/high-water marks set to 83874/125811