Re: Random server crashes every few weeks (smp_invltlb: endless loop […] retrysmp_invltlb: ipi sent)
Michael Neumann
mneumann at ntecs.de
Sat May 28 04:17:49 PDT 2016
I am running DragonFly on linux-kvm with virtio-blk for years, and I've
only seen rarely issues (maybe 1-2 times a year there is a crash). I
will give it a try once I update the machine.
Regards,
Michael
On 05/28/16 02:23, Matthew Dillon wrote:
> Ok, Zachary noted that ivadasz had a patch. Imre and I went over it on
> IRC and he committed the patch to master. I also committed some
> additional changes so it would be great if anyone using master + virtio
> in a virtual-hosted environment can [re]test the changes.
>
> There are likely going to be numerous other issues with virtual hosting
> not yet addressed.
>
> -Matt
>
> On Fri, May 27, 2016 at 10:08 AM, Matthew Dillon <dillon at backplane.com
> <mailto:dillon at backplane.com>> wrote:
>
> Virtio (for block storage devices) could be the cause. There are
> known bugs in the DragonFly driver for virtio which haven't been
> tracked down yet (not enough of the devs are using virtual hosting
> to be able to reproduce the problem in a debugable way).
>
> -Matt
>
> On Fri, May 27, 2016 at 7:39 AM, Steve Petrie, P.Eng.
> <apetrie at aspetrie.net <mailto:apetrie at aspetrie.net>> wrote:
>
> Greetings To DragonFlyBSD List,
>
> The subject of random server crashes with DragonFly running
> running on a virtualized host machine, is of great interest (and
> concern) to me. Caveat: I am an (almost) complete DragonFly newbie.
>
> Please see my commens inline below.
>
> Steve
>
> ----- Original Message ----- From: "Stefan Unterweger"
> <232.20711 at chiffre.aleturo.com
> <mailto:232.20711 at chiffre.aleturo.com>>
> To: "Matthew Dillon" <dillon at backplane.com
> <mailto:dillon at backplane.com>>
> Cc: <users at dragonflybsd.org <mailto:users at dragonflybsd.org>>
> Sent: Friday, May 27, 2016 3:38 AM
> Subject: Re: Random server crashes every few weeks (smp_invltlb:
> endless loop […] retrysmp_invltlb: ipi sent)
>
>
>
> * Matthew Dillon on Thu, May 26, 2016 at 11:00:18AM -0700:
>
> It's really hard to say from something which is
> virtually hosted. It kinda
> sounds like the virtual host isn't assigning enough of
> its own cpus to the
> virtual host. The fact that DragonFly is complaining
> about smp_invltlb()
> implies that the host's virtualized cpu threads are not
> getting scheduled
> properly.
>
> One thing to note is that we do not do any instruction
> escapes to hint to
> virtual hosts when a cpu is in a tight loop waiting for
> synchronization.
> It would be nice if we had some support for that, it
> would probably make
> DFly play better on virtualized systems.
>
>
> This is an interesting suggestion, which at least would
> explain at least
> some of the cases where I’ve experienced the crashes (the
> daily HAMMER
> cronjob, heavy paging under stress, I/O bursts and so on).
>
> So in effect, could it be that the crashes are more likely
> as either my
> own server comes under load or some -other- server who
> happens to run in
> the same hypervisor?
>
> Would this warrant opening a ticket with Profitbricks, or is
> it just as
> likely that I’m wasting my time and will only get a response
> along the
> lines of ‘Use Linux; Dragonfly BSD most certainly is not
> supported’?
>
> I suggest setting the number of cores to 1. That will
> get rid of all SMP
> interplay and hopefully remove the issues the virtual
> host is choking on.
>
>
> Interestingly enough, I have seen the opposite so far. At
> first, I have
> run the server on only one core, to save money and because
> it doesn’t
> really yet need any more. When on one core, it still
> freezes, along
> approximately the same pattern, but I never got a trace there.
>
> My guess then was that perhaps there would have been some
> odd race
> condition between paging, HAMMER and dm_crypt—adding another
> core
> temporarily seemed more stable and then regressed back to
> the mean.
>
> I will try to set up another VM to see whether I can
> reliably reproduce
> such a crash.
>
>
> After a great deal of research, I chose DragonFly as the OS for
> a new website (not yet online). Three main attributes drew me to
> DragonFly: 1. reputation for reliability and speed, 2. hammer
> file system, 3. responsiveness of DragonFly open source community.
>
> However, my business plan for this new website (not yet online),
> requires starting out hosting it on a VM under QEMU / KVM
> virtualization, because I cannot justify the much higher cost of
> dedicated server hosting hardware. And I like the brutally
> competitive quasi-commoditized hosting services market for QEMU
> / KVM virtualization offerings.
>
> I do have an experimental working DragonFly installation on a
> QEMU / KVM VM hosted at Elastic Hosts www.elastichosts.com
> <http://www.elastichosts.com> I access this VM through TightVNC
> and I get a DragonFly console using PuTTY.
>
> But I had to suspend work on testing this DragonFly VM
> installation, due to other business priorities. I hope to get
> back to it later in 2016.
>
> However, I can highly recommend Elastic Hosts for their solid
> cloud infrastructure and their strong customer support.
>
> So if Stefan wants to expand his testing of the DragonFly crash
> to a VM with a (probably) different underlying architecture than
> at ProfitBricks, I would recommend giving an Elastic Hosts QEMU
> / KVM VM a try.
>
> Alternatively, if Stafan has (or develops) some simple limited
> self-contained testing setup, for reproducing the DragonFly
> crash he is experiencing with the ProfitBricks VM he's presently
> using, I would be interested to try to set up the same testing
> scenario on my current DragonFly (and later on an upgraded
> version of DragonFly) on the Elastic Hosts VM where I presently
> have (an outdated version of) DragonFly operational.
>
> Steve
>
>
>
> Thanks for your answer,
> Stefan
>
>
>
> PS: Just in case, as I’ve forgotten it previously: here’s
> the dmesg from
> the server in question.
>
> | Copyright (c) 2003-2015 The DragonFly Project.
> | Copyright (c) 1992-2003 The FreeBSD Project.
> | Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991,
> 1992, 1993, 1994
> | The Regents of the University of California. All rights
> reserved.
> | DragonFly v4.4.1-RELEASE #2: Sun Dec 6 19:10:59 EST 2015
> |
> root at www.shiningsilence.com:/usr/obj/home/justin/release/4_4/sys/X86_64_GENERIC
> | TSC clock: 2600054420 Hz, i8254 clock: 1193169 Hz
> | CPU: AMD Opteron 62xx class CPU (2600.11-MHz K8-class CPU)
> | Origin = "AuthenticAMD" Id = 0x600f12 Stepping = 2
> |
> Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
> |
> Features2=0x96982203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,AVX,VMM>
> | AMD Features=0x24500800<SYSCALL,NX,MMX+,Page1GB,LM>
> | AMD
> Features2=0x10be7<LAHF,CMP,SVM,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4>
> | MONITOR/MWAIT Features=0x2<INTBRK>
> | real memory = 3219762176 (3070 MB)
> | avail memory = 2990727168 (2852 MB)
> | lapic: divisor index 0, frequency 500005713 Hz
> | SMI Frequency (worst case): 28571 Hz (35 us)
> | Initialize MI interrupts
> | wdog: In-kernel automatic watchdog reset enabled
> | kbd1 at kbdmux0
> | md0: Preloaded image <initrd.img> 15728640 bytes at
> 0xffffffff82739ac0
> | md1: Malloc disk
> | ACPI: RSDP 0x00000000000FC980 000014 (v00 BOCHS )
> | ACPI: RSDT 0x00000000BFFFBCA0 000040 (v01 BOCHS BXPCRSDT
> 00000001 BXPC 00000001)
> | ACPI: FACP 0x00000000BFFFFF80 000074 (v01 BOCHS BXPCFACP
> 00000001 BXPC 00000001)
> | ACPI: DSDT 0x00000000BFFFBCE0 00151D (v01 BXPC BXDSDT
> 00000001 INTL 20100528)
> | ACPI: FACS 0x00000000BFFFFF40 000040
> | ACPI: APIC 0x00000000BFFFFC60 000270 (v01 BOCHS BXPCAPIC
> 00000001 BXPC 00000001)
> | ACPI: HPET 0x00000000BFFFFC20 000038 (v01 BOCHS BXPCHPET
> 00000001 BXPC 00000001)
> | ACPI: SRAT 0x00000000BFFFF770 0004A8 (v01 BOCHS BXPCSRAT
> 00000001 BXPC 00000001)
> | ACPI: SSDT 0x00000000BFFFD8E0 001E8E (v01 BOCHS BXPCSSDT
> 00000001 BXPC 00000001)
> | ACPI: SSDT 0x00000000BFFFD870 00003D (v01 BOCHS BXPCSSDT
> 00000001 BXPC 00000001)
> | ACPI: SSDT 0x00000000BFFFD200 00066E (v01 BXPC BXSSDTPC
> 00000001 INTL 20100528)
> | cryptosoft0: <software crypto> on motherboard
> | aesni0: <AES-CBC,AES-XTS> on motherboard
> | padlock0: No ACE support.
> | rdrand0: No RdRand support.
> | acpi0: <BOCHS BXPCRSDT> on motherboard
> | ACPI: 4 ACPI AML tables successfully acquired and loaded
> | ACPI FADT: SCI testing interrupt mode ...
> | ACPI FADT: SCI select level/low
> | objcache_reclaimlist
> | objcache_reclaimlist
> | objcache_reclaimlist
> | objcache_reclaimlist
> | acpi0: Power Button (fixed)
> | acpi_timer0 on acpi0
> | acpi_hpet0: <High Precision Event Timer> iomem
> 0xfed00000-0xfed003ff on acpi0
> | acpi_hpet0: frequency 100000000
> | pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> | pci0: <ACPI PCI bus> on pcib0
> | pci_link4: Unable to route IRQs: AE_NOT_FOUND
> | isab0: <PCI-ISA bridge> at device 1.0 on pci0
> | isa0: <ISA bus> on isab0
> | atapci0: <Intel PIIX3 WDMA2 controller> port
> 0xc120-0xc12f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device
> 1.1 on pci0
> | ata0: <ATA channel 0> on atapci0
> | ata1: <ATA channel 1> on atapci0
> | acd0: DVDROM <QEMU DVD-ROM/1.0> at ata1-master WDMA2
> | uhci0: <Intel 82371SB (PIIX3) USB controller> port
> 0xc0c0-0xc0df irq 11 at device 1.2 on pci0
> | usbus0: controller did not stop
> | usbus0 on uhci0
> | pci0: <bridge> (vendor 0x8086, dev 0x7113) at device 1.3 irq 9
> | vgapci0: <VGA-compatible display> mem
> 0xfd000000-0xfdffffff at device 2.0 on pci0
> | vgapci0: Boot video device
> | virtio_pci0: <VirtIO PCI Balloon adapter> port
> 0xc0e0-0xc0ff irq 11 at device 3.0 on pci0
> | virtio_pci1: <VirtIO PCI Block adapter> port 0xc000-0xc03f
> mem 0xfebf0000-0xfebf0fff irq 10 at device 5.0 on pci0
> | vtblk0: <VirtIO Block Adapter> on virtio_pci1
> | virtio_pci1: host features: 0x710006d4
> <EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
> | virtio_pci1: negotiated features: 0x254
> <WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
> | virtio_pci2: <VirtIO PCI Block adapter> port 0xc040-0xc07f
> mem 0xfebf1000-0xfebf1fff irq 10 at device 6.0 on pci0
> | vtblk1: <VirtIO Block Adapter> on virtio_pci2
> | virtio_pci2: host features: 0x710006d4
> <EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
> | virtio_pci2: negotiated features: 0x254
> <WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
> | virtio_pci3: <VirtIO PCI Network adapter> port
> 0xc100-0xc11f mem 0xfebf2000-0xfebf2fff irq 11 at device 7.0
> on pci0
> | vtnet0: <VirtIO Networking Adapter> on virtio_pci3
> | virtio_pci3: host features: 0x711f8060
> <EventIdx,RingIndirect,NotifyOnEmpty,RxModeExtra,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,TxAllGSO,MacAddress>
> | virtio_pci3: negotiated features: 0x110f8020
> <RingIndirect,NotifyOnEmpty,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,MacAddress>
> | usbus0: 12Mbps Full Speed USB v1.0
> | vtnet0: MAC address: 02:01:06:f6:1b:63
> | add dynamic link state
> | virtio_pci4: <VirtIO PCI Block adapter> port 0xc080-0xc0bf
> mem 0xfebf3000-0xfebf3fff irq 11 at device 8.0 on pci0
> | vtblk2: <VirtIO Block Adapter> on virtio_pci4
> | virtio_pci4: host features: 0x710006d4
> <EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
> | virtio_pci4: negotiated features: 0x254
> <WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
> | atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq
> 1 on acpi0
> | atkbd0: <AT Keyboard> irq 1 on atkbdc0
> | kbd0 at atkbd0
> | ugen0.1: <Intel> at usbus0
> | uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00,
> addr 1> on usbus0
> | psm0: <PS/2 Mouse> irq 12 on atkbdc0
> | psm0: model IntelliMouse Explorer, device ID 4
> | cpu0: <ACPI CPU> on acpi0
> | cpu_cst0: <ACPI CPU C-State> on cpu0
> | cpu1: <ACPI CPU> on acpi0
> | cpu_cst1: <ACPI CPU C-State> on cpu1
> | ACPI: Enabled 16 GPEs in block 00 to 0F
> | orm0: <ISA Option ROM> at iomem 0xe9800-0xeffff on isa0
> | pmtimer0 on isa0
> | vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem
> 0xa0000-0xbffff on isa0
> | sc0: <System console> at flags 0x100 on isa0
> | sc0: VGA <16 virtual consoles, flags=0x300>
> | sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
> | sio0: type 16550A
> | sio1: can't drain, serial port might not exist, disabling
> | hpt27xx: no controller detected.
> | CAM: Configuring 2 busses
> | CAM: finished configuring all busses
> | cd0 at ata1 bus 0 target 0 lun 0
> | cd0: <QEMU QEMU DVD-ROM 1.0> Removable CD-ROM SCSI-0 device
> | cd0: 16.000MB/s transfers
> | cd0: cd present [329728 x 2048 byte records]
> | uhub0: 2 ports with 2 removable, self powered
> | ugen0.2: <QEMU> at usbus0
> | uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 1.00/0.00,
> addr 2> on usbus0
> | no B_DEVMAGIC (bootdev=0)
> | Device Mapper version 4.16.0 loaded
> | dm_target_zero: Successfully initialized
> | dm_target_crypt: Successfully initialized
> | dm_target_error: Successfully initialized
> | Mounting root from ufs:md0s0
> | DMA space used: 1236k, remaining available: 131072k
> | Mounting devfs
> | dm_target_crypt: Setting min/max mpipe buffers: 2/30
> | dm_target_crypt: Setting min/max mpipe buffers: 2/30
> | HAMMER(Rhaal) recovery check seqno=055a4f51
> | HAMMER(Rhaal) recovery range 300000000cc2da60-300000000cc2da60
> | HAMMER(Rhaal) recovery nexto 300000000cc2da60
> endseqno=055a4f52
> | HAMMER(Rhaal) mounted clean, no recovery needed
> | chroot_kernel: set new rootnch/rootvnode to /new_root
> | dm_target_crypt: Setting min/max mpipe buffers: 2/30
> | dm_target_crypt: Setting min/max mpipe buffers: 2/30
> | dm_target_crypt: Setting min/max mpipe buffers: 2/30
> | HAMMER: read-only -> read-write
> | HAMMER(Rhaal-Daten) recovery check seqno=352f5bc4
> | HAMMER(Rhaal-Daten) recovery range
> 3000000000d5c108-3000000000d77bc8
> | HAMMER(Rhaal-Daten) recovery nexto 3000000000d77bc8
> endseqno=352f5cc1
> | HAMMER(Rhaal-Daten) recovery undo
> 3000000000d5c108-3000000000d77bc8 (113344 bytes)(RW)
> | HAMMER(Rhaal-Daten) Found REDO_SYNC 3000000000cb87a0
> | HAMMER(Rhaal-Daten) recovery complete
> | HAMMER(Rhaal-Daten) recovery redo
> 3000000000d5c108-3000000000d77bc8 (113344 bytes)(RW)
> | HAMMER(Rhaal-Daten) Find extended redo 3000000000cb87a0,
> 670056 extbytes
> | HAMMER(Rhaal-Daten) End redo recovery
> | dm_target_crypt: Setting min/max mpipe buffers: 2/30
> | swap low/high-water marks set to 83874/125811
>
>
>
>
More information about the Users
mailing list