dual port EM nic wedging under load

Thu Nov 23 11:11:00 PST 2006

I have been trying to benchmark the various BSDs for firewall and 
packet forwarding performance and wanted to give DragonFly BSD a try. 
However, when using netrate from FreeBSD to blast across a dual port 
EM nic on DragonFly BSD, the em nics wedge and the only way to 
recover is to ifconfig down/up them.  Try with and without polling as 
well.  Sometimes it takes 1 second, sometimes upto 30 seconds.

dmesg below and stats below.  I was trying with both UP and SMP from 
the 1.6 RELEASE CD and even tried a downloaded kernel from HEAD, but 
the same results.  I by start blasting UDP packets across the router 
at full speed, and the interfaces will wedge.  The network setup can 
be seen at http://www.tancsa.com/blast.html

Copyright (c) 2003, 2004, 2005, 2006 The DragonFly Project.
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
DragonFly 1.6.2-RELEASE #7: Wed Nov 22 21:35:54 EST 2006
    mdtancsa at r2-dragonfly.sentex.ca:/usr/obj/usr/src/sys/router
TSC clock: 2015130374 Hz, i8254 clock: 1193259 Hz
CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2015.01-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20fb1  Stepping = 1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Hyperthreading: 2 logical CPUs
  AMD Features=0xe0500000<<b20>,AMIE,<b29>,DSP,3DNow!>
real memory  = 2147418112 (2097088K bytes)
avail memory = 2077835264 (2029136K bytes)
DragonFly/MP: Multiprocessor motherboard
 cpu0 (BSP): apic id:  0, version: 0x00040010, at 0xfee00000
 cpu1 (AP):  apic id:  1, version: 0x00040010, at 0xfee00000
 Warning: APIC I/O disabled
Preloaded elf kernel "/kernel.old" at 0xc048e000.
Preloaded elf module "/modules.old/acpi.ko" at 0xc048e214.
sched_ithd: stray interrupt 7
Pentium Pro MTRR support enabled
md0: Malloc disk
pcibios: BIOS version 3.00
Using $PIR table, 11 entries at 0xc00fddf0
npx0: <math processor> on motherboard
npx0: INT 16 interface
Using XMM optimized bcopy/copyin/copyout
acpi0: <Nvidia AWRDACPI> on motherboard
acpi_bus_number: can't get _ADR
acpi_bus_number: can't get _ADR
acpi0: Power Button (fixed)
Warning: ACPI is disabling APM's device.  You can't run both
acpi_bus_number: can't get _ADR
acpi_bus_number: can't get _ADR
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_button0: <Power Button> on acpi0
compare 0
compare 0
compare 0
legacypci0 on motherboard
pcib0: <Host to PCI bridge> on legacypci0
pci0: <PCI bus> on pcib0
pci0: <unknown card> (vendor=0x10de, dev=0x005e) at 0.0
isab0: <PCI to ISA bridge (vendor=10de device=0050)> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <unknown card> (vendor=0x10de, dev=0x0052) at 1.1 irq 10
atapci0: <Generic PCI ATA controller> port 0xf000-0xf00f at device 6.0 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
atapci1: <Generic PCI ATA controller> port 
0xd800-0xd80f,0xb70-0xb73,0x970-0x977,0xbf0-0xbf3,0x9f0-0x9f7 mem 
0xdb002000-0xdb002fff irq 11 at device 7.0 on pci0
ata2: at 0x9f0 on atapci1
ata3: at 0x970 on atapci1
atapci2: <Generic PCI ATA controller> port 
0xc400-0xc40f,0xb60-0xb63,0x960-0x967,0xbe0-0xbe3,0x9e0-0x9e7 mem 
0xdb001000-0xdb001fff irq 5 at device 8.0 on pci0
ata4: at 0x9e0 on atapci2
ata5: at 0x960 on atapci2
pcib1: <PCI to PCI bridge (vendor=10de device=005c)> at device 9.0 on pci0
pci5: <PCI bus> on pcib1
pci5: <ATI Mach64-GM graphics accelerator> at 8.0 irq 10
pci0: <PCI to Other bridge (vendor=10de device=0057)> at 10.0 irq 5
pcib2: <PCI to PCI bridge (vendor=10de device=005d)> at device 11.0 on pci0
pci4: <PCI bus> on pcib2
bge0: <Broadcom BCM5751 Gigabit Ethernet, ASIC rev. 0x4001> mem 
0xd7000000-0xd700ffff irq 5 at device 0.0 on pci4
miibus0: <MII bus> on bge0
ukphy0: <Generic IEEE 802.3u media interface> on miibus0
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
bge0: MAC address: 00:10:18:14:15:12
pcib3: <PCI to PCI bridge (vendor=10de device=005d)> at device 12.0 on pci0
pci3: <PCI bus> on pcib3
bge1: <Broadcom BCM5751 Gigabit Ethernet, ASIC rev. 0x4001> mem 
0xd5000000-0xd500ffff irq 3 at device 0.0 on pci3
miibus1: <MII bus> on bge1
ukphy1: <Generic IEEE 802.3u media interface> on miibus1
ukphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
bge1: MAC address: 00:10:18:14:27:d5
pcib4: <PCI to PCI bridge (vendor=10de device=005d)> at device 13.0 on pci0
pci2: <PCI bus> on pcib4
bge2: <Broadcom BCM5751 Gigabit Ethernet, ASIC rev. 0x4001> mem 
0xd3000000-0xd300ffff irq 11 at device 0.0 on pci2
miibus2: <MII bus> on bge2
ukphy2: <Generic IEEE 802.3u media interface> on miibus2
ukphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
bge2: MAC address: 00:10:18:14:38:d2
pcib5: <PCI to PCI bridge (vendor=10de device=005d)> at device 14.0 on pci0
pci1: <PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection, Version - 3.2.15> port 
0x9000-0x901f mem 0xd1000000-0xd101ffff,0xd1020000-0xd103ffff irq 10 
at device 0.0 on pci1
em0: MAC address: 00:15:17:0b:70:98
em0: Speed: N/A, Duplex:N/A
em1: <Intel(R) PRO/1000 Network Connection, Version - 3.2.15> port 
0x9400-0x941f mem 0xd1060000-0xd107ffff,0xd1040000-0xd105ffff irq 11 
at device 0.1 on pci1
em1: MAC address: 00:15:17:0b:70:99
em1: Speed: N/A, Duplex:N/A
orm0: <Option ROMs> at iomem 
0xc0000-0xc97ff,0xcc000-0xcdfff,0xce000-0xcffff,0xd0000-0xd1fff on isa0
pmtimer0 on isa0
fdc0: ready for input in output
fdc0: cmd 3 failed at out byte 1 of 3
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model Generic PS/2 mouse, device ID 0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1: can't drain, serial port might not exist, disabling
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
SMP: AP CPU #1 Launched!
vlan0: MAC address: 00:00:00:00:00:00
Warning: ACPI idle hook not yet supported for SMP
ad0: 114473MB <ST3120026A> [232581/16/63] at ata0-master BIOSDMA
ad2: 38166MB <ST340014A> [77545/16/63] at ata1-master BIOSDMA
Mounting root from ufs:/dev/ad2s1a
em0: Link is up 1000 Mbps Full Duplex
em1: Link is up 1000 Mbps Full Duplex
em0: Link is Down
em0: Link is up 1000 Mbps Full Duplex

[r2-dragonfly]# pciconf -lv
none0 at pci0:0:0: class=0x058000 card=0x815a1043 chip=0x005e10de 
rev=0xa3 em0 at pci1:0:0:   class=0x020000 card=0x115e8086 
chip=0x105e8086 rev=0x06 hdr=0x00
    vendor   = 'Intel Corporation'
    device   = 'PRO/1000 PT'
    class    = network
    subclass = ethernet
em1 at pci1:0:1:   class=0x020000 card=0x115e8086 chip=0x105e8086 
rev=0x06 hdr=0x00
    vendor   = 'Intel Corporation'
    device   = 'PRO/1000 PT'
    class    = network
    subclass = ethernet

em stats posrt wedge

em0: Adapter: 0xda491ec0
em0: Excessive collisions = 0
em0: Symbol errors = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 7202325
em0: Receive No Buffers = 1472
em0: Receive length errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Carrier extension errors = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 5
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 7202331
em0: Good Packets Rcvd = 6560
em0: Good Packets Xmtd = 16
em1: Adapter: 0xda492540
em1: Excessive collisions = 0
em1: Symbol errors = 0
em1: Sequence errors = 0
em1: Defer count = 0
em1: Missed Packets = 0
em1: Receive No Buffers = 0
em1: Receive length errors = 0
em1: Receive errors = 0
em1: Crc errors = 0
em1: Alignment errors = 0
em1: Carrier extension errors = 0
em1: XON Rcvd = 0
em1: XON Xmtd = 0
em1: XOFF Rcvd = 0
em1: XOFF Xmtd = 0
em1: Good Packets Rcvd = 5
em1: Good Packets Xmtd = 5909
as seen leading up to and during the wedge
[r2-dragonfly]# netstat -ni 1
            input        (Total)           output
   packets  errs      bytes    packets  errs      bytes colls
         0     0          0          0     0          0     0
         1     0         60          0     0          0     0
         0     0          0          0     0          0     0
         1     0         60          0     0          0     0
         0     0          0          0     0          0     0
      5889 279836     419388       5888     0     376832     0
         0 380706          0          0     0          0     0
         0 379626          0          0     0          0     0
         1 382278         60          0     0          0     0

--------------------------------------------------------------------
Mike Tancsa,                                      tel +1 519 651 3400
Sentex Communications,                            mike at sentex.net
Providing Internet since 1994                    www.sentex.net
Cambridge, Ontario Canada                         www.sentex.net/mike