dual port EM nic wedging under load

Mike Tancsa mike at sentex.net
Sun Nov 26 15:00:35 PST 2006


At 11:42 AM 11/26/2006, Matthew Dillon wrote:

:Using polling and fastfwd on, I am able to get about 300Kpps in a
:unidirectional blast and still see that rate even with 10 poorly
:written ipfw rules !?!
    Are you sure the ipfw ruleset is being executed?  :-) ... whenever
    something looks to be too good to be true....
It does look odd, but this is what I was doing.  Here is without ipfw
[r2-dragonfly]# kldunload ipfw
IP firewall unloaded
[r2-dragonfly]# ifstat -b
       bge0                em0                 em1
 Kbps in  Kbps out   Kbps in  Kbps out   Kbps in  Kbps out
    0.00      0.00      0.00      0.00      0.00      0.00
    2.87      0.00      0.00      0.00      0.00      0.00
    0.00      0.00      0.00      0.00      0.00      0.00
    0.46      0.00      0.00      0.00      0.00      0.00
    0.00      0.00  78943.98      0.46      0.46  78872.37
    0.46      0.00  140623.2      0.00      0.00  140623.2
    0.00      0.00  140622.8      0.00      0.00  140622.8
    0.46      0.00  140610.4      0.00      0.00  140610.4
    0.00      0.00  140607.5      0.00      0.00  140607.5
    2.91      0.00  140501.6  47328.70  47337.99  140501.6
    0.00      0.00  140607.2  140607.2  140607.2  140607.2
    0.46      0.00  140607.9  140607.9  140607.9  140607.9
    0.00      0.00  140607.0  140607.0  140607.0  140607.0
    0.46      0.00  140599.7  140599.7  140599.7  140599.7
    0.00      0.00  140615.4  140615.4  140615.4  140615.4
    0.46      0.00  140606.8  140606.8  140606.8  140606.8
    0.00      0.00  140606.2  140606.2  140606.2  140606.2
    0.46      0.00  140539.4  140539.4  140609.0  140609.0
    0.00      0.00  140596.2  140596.2  140526.5  140526.5
    0.00      0.00  140624.2  99093.16  99093.16  140624.2
    0.46      0.00  140615.0      0.00      0.00  140615.0
First one stream, then a second... Rate is about the same.

Then I load the kld, and add some poorly written rules

[r2-dragonfly]# ipfw show
00010 0   0 deny log ip from 10.0.0.1 to any
00020 0   0 deny log ip from 172.0.0.1 to any
00030 0   0 deny log tcp from 172.0.1.1 to any
00040 0   0 deny log udp from 172.2.1.1 to any
00050 0   0 deny log tcp from any to 10.90.1.1 135-139
00060 0   0 unreach port tcp from 192.168.43.98 to me 25
00070 0   0 deny log tcp from any to 10.66.2.2 445
00080 0   0 deny log tcp from any to 10.61.2.2 138
00090 0   0 deny log ip from any to 219.0.0.0/16
00100 0   0 deny log esp from 218.0.0.0/8 to any
01000 1 242 allow ip from any to any
65535 0   0 deny ip from any to any
[r2-dragonfly]#
Same throughput

[r2-dragonfly]# ifstat -b
       bge0                em0                 em1
 Kbps in  Kbps out   Kbps in  Kbps out   Kbps in  Kbps out
    0.46      0.00      0.00      0.00      0.00      0.00
    0.00      0.00      0.00      0.00      0.00      0.00
    0.93      0.00  97387.37      0.00      0.00  97378.09
    0.00      0.00  140628.8      0.00      0.00  140628.8
    0.87      0.00  140607.9      0.00      0.00  140607.9
    0.93      0.00  140541.2      0.00      0.00  140541.2
    0.00      0.00  140647.7  27050.50  27055.88  140647.7
    0.46      0.00  140607.2  140607.2  140607.2  140607.2
    0.00      0.00  140606.5  140606.5  140606.5  140606.5
    0.46      0.00  140607.2  140607.2  140607.2  140607.2
    0.46      0.00  140603.3  140603.3  140603.3  140603.3
    0.46      0.00  140611.8  140611.8  140611.8  140611.8
    0.00      0.00  140606.9  140606.9  140606.9  140606.9
    0.46      0.00  140525.3  140525.3  140525.3  140525.3
    0.00      0.00  140598.5  140598.5  140598.5  140598.5
    0.46      0.00  140617.8  140617.8  140617.8  140617.8
    0.00      0.00  140606.9  140606.9  140606.9  140606.9
Whats odd however, is

[r2-dragonfly]# ipfw show
00010   0     0 deny log ip from 10.0.0.1 to any
00020   0     0 deny log ip from 172.0.0.1 to any
00030   0     0 deny log tcp from 172.0.1.1 to any
00040   0     0 deny log udp from 172.2.1.1 to any
00050   0     0 deny log tcp from any to 10.90.1.1 135-139
00060   0     0 unreach port tcp from 192.168.43.98 to me 25
00070   0     0 deny log tcp from any to 10.66.2.2 445
00080   0     0 deny log tcp from any to 10.61.2.2 138
00090   0     0 deny log ip from any to 219.0.0.0/16
00100   0     0 deny log esp from 218.0.0.0/8 to any
01000 482 19162 allow ip from any to any
65535   0     0 deny ip from any to any
[r2-dragonfly]#
Rule 1000 should have WAY more hits

And sure enough

[r2-dragonfly]# ipfw show
00005   0     0 deny ip from any to 192.168.44.1
00010   0     0 deny log ip from 10.0.0.1 to any
00020   0     0 deny log ip from 172.0.0.1 to any
00030   0     0 deny log tcp from 172.0.1.1 to any
00040   0     0 deny log udp from 172.2.1.1 to any
00050   0     0 deny log tcp from any to 10.90.1.1 135-139
00060   0     0 unreach port tcp from 192.168.43.98 to me 25
00070   0     0 deny log tcp from any to 10.66.2.2 445
00080   0     0 deny log tcp from any to 10.61.2.2 138
00090   0     0 deny log ip from any to 219.0.0.0/16
00100   0     0 deny log esp from 218.0.0.0/8 to any
00200   0     0 count ip from any to 192.168.44.1
00210   0     0 deny ip from any to 192.168.44.1
01000 489 20536 allow ip from any to any
01010   0     0 count ip from any to 192.168.44.1
65535   0     0 deny ip from any to any
[r2-dragonfly]#
[r2-dragonfly]# ipfw add 210 deny ip from any to 192.168.44.1
00210 deny ip from any to 192.168.44.1
[r2-dragonfly]# ipfw show
00010   0     0 deny log ip from 10.0.0.1 to any
00020   0     0 deny log ip from 172.0.0.1 to any
00030   0     0 deny log tcp from 172.0.1.1 to any
00040   0     0 deny log udp from 172.2.1.1 to any
00050   0     0 deny log tcp from any to 10.90.1.1 135-139
00060   0     0 unreach port tcp from 192.168.43.98 to me 25
00070   0     0 deny log tcp from any to 10.66.2.2 445
00080   0     0 deny log tcp from any to 10.61.2.2 138
00090   0     0 deny log ip from any to 219.0.0.0/16
00100   0     0 deny log esp from 218.0.0.0/8 to any
00200   0     0 count ip from any to 192.168.44.1
00210   0     0 deny ip from any to 192.168.44.1
01000 488 20294 allow ip from any to any
01010   0     0 count ip from any to 192.168.44.1
65535   0     0 deny ip from any to any
[r2-dragonfly]# ipfw add 5 deny ip from any to 192.168.44.1
00005 deny ip from any to 192.168.44.1
[r2-dragonfly]# ipfw show
00005   0     0 deny ip from any to 192.168.44.1
00010   0     0 deny log ip from 10.0.0.1 to any
00020   0     0 deny log ip from 172.0.0.1 to any
00030   0     0 deny log tcp from 172.0.1.1 to any
00040   0     0 deny log udp from 172.2.1.1 to any
00050   0     0 deny log tcp from any to 10.90.1.1 135-139
00060   0     0 unreach port tcp from 192.168.43.98 to me 25
00070   0     0 deny log tcp from any to 10.66.2.2 445
00080   0     0 deny log tcp from any to 10.61.2.2 138
00090   0     0 deny log ip from any to 219.0.0.0/16
00100   0     0 deny log esp from 218.0.0.0/8 to any
00200   0     0 count ip from any to 192.168.44.1
00210   0     0 deny ip from any to 192.168.44.1
01000 489 20536 allow ip from any to any
01010   0     0 count ip from any to 192.168.44.1
65535   0     0 deny ip from any to any
But when I use all, it seems to work

[r2-dragonfly]# ipfw add 4 deny all from any to 192.168.44.1
00004 deny ip from any to 192.168.44.1
[r2-dragonfly]# ipfw show
00004 238982 9081316 deny ip from any to 192.168.44.1
00005      0       0 deny ip from any to 192.168.44.1
00010      0       0 deny log ip from 10.0.0.1 to any
00020      0       0 deny log ip from 172.0.0.1 to any
00030      0       0 deny log tcp from 172.0.1.1 to any
00040      0       0 deny log udp from 172.2.1.1 to any
00050      0       0 deny log tcp from any to 10.90.1.1 135-139
00060      0       0 unreach port tcp from 192.168.43.98 to me 25
00070      0       0 deny log tcp from any to 10.66.2.2 445
00080      0       0 deny log tcp from any to 10.61.2.2 138
00090      0       0 deny log ip from any to 219.0.0.0/16
00100      0       0 deny log esp from 218.0.0.0/8 to any
00200      0       0 count ip from any to 192.168.44.1
00210      0       0 deny ip from any to 192.168.44.1
01000    793   32521 allow ip from any to any
01010      0       0 count ip from any to 192.168.44.1
65535      0       0 deny ip from any to any
[r2-dragonfly]#
Something seems busted.


:Also it handles the load quite smoothly for the bi-directional test
:
:Here is the output of ifstat -b as seen from the box acting as router
:
:You can see the first stream starting up, and then the second on the
:opposite stream.  Rates remain constant throughout, which is quite
:different from FreeBSD. ipfw on Dragonfly has no ill effect for some reason.
:...
    I'm not sure if you are overloading the system intentionally or not.
    There seem to be an excessive number of ENOBUF issues during packet
    processing, assuming the count start at 0 when the test began.
:em0: Missed Packets =    221709682     <<<<<<
:em0: Receive No Buffers = 53994896     <<<<<< This seems excessive to me
:em0: Receive length errors = 0
:em0: Receive errors = 0
:em0: Crc errors = 0
:em0: Alignment errors = 0
:em0: Carrier extension errors = 0
:em0: RX overruns = 55192               <<<<<<
:...
:em0: Good Packets Rcvd = 187109273
:em0: Good Packets Xmtd =  30553884
:em1: Missed Packets =     17508539     <<<<<<
:em1: Receive No Buffers = 14148594     <<<<<<
:em1: Receive length errors = 0
:em1: Receive errors = 0
:em1: Crc errors = 0
:em1: Alignment errors = 0
:em1: Carrier extension errors = 0
:em1: RX overruns = 0                   <<<<<<
:em1: Good Packets Rcvd =  30553871
:em1: Good Packets Xmtd = 187031079
    One really nice thing about DragonFly's NETIF polling code is that you
    can control the polling frequency (kern.polling.pollhz) and other
    parameters on the fly with sysctl's.  You can actually change the polling
    frequency on a live system!


Yes, if I set the polling hz to 5000, I am getting 596,231 on the 
single stream, and when I bring online the stream in the opposite 
direction, I get 335,006 in one direction and 160,262 in the other.

kern.polling.each_burst: 5
HZ    1 blast      blast A and blast B
3000   450,072       349,696   349,814
4000   588,514       391,615   391,753
5000   592,655       344,472   360,468
6000   593,966       353,047   352,177
Changing
kern.polling.each_burst: 20,40
doesnt seem to have much impact statistically.

Even at 5000Hz, throughput is pretty steady... what comes in, goes out

[r2-dragonfly]# ifstat -b
       bge0                em0                 em1
 Kbps in  Kbps out   Kbps in  Kbps out   Kbps in  Kbps out
    0.47      0.00  158176.3  158176.3  158219.2  158219.2
    0.00      0.00  167004.1  167004.1  166937.3  166937.3
    0.46      0.00  167158.7  167158.7  167158.7  167158.7
    0.00      0.00  167161.0  167161.0  167194.9  167194.9
    0.46      0.00  167110.8  167110.8  167076.9  167076.9
    0.00      0.00  155460.8  155460.8  155460.8  155460.8
    0.46      0.00  167033.7  167033.7  167033.7  167033.7
^C
[r2-dragonfly]#
Top shows

load 
averages:  0.18,  0.05,  0.01 
                                 up 0+05:17:52  15:58:38
29 processes:  1 running, 28 sleeping
CPU0 states:  0.0% user,  0.0% nice, 88.7% system,  0.0% interrupt, 11.3% idle
CPU1 states:  0.0% user,  0.0% nice,  0.0% system,  1.6% interrupt, 98.4% idle
Mem: 9432K Active, 10M Inact, 58M Wired, 9216K Buf, 1929M Free

[r2-dragonfly]# sysctl -a kern.polling
kern.polling.burst: 119
kern.polling.each_burst: 30
kern.polling.burst_max: 150
kern.polling.user_frac: 50
kern.polling.reg_frac: 20
kern.polling.short_ticks: 39675403
kern.polling.lost_polls: 679638
kern.polling.pending_polls: 1
kern.polling.residual_burst: 0
kern.polling.handlers: 2
kern.polling.enable: 1
kern.polling.phase: 2
kern.polling.suspect: 12710
kern.polling.stalled: 2
kern.polling.pollhz: 4000
[r2-dragonfly]#
One thing I found on FreeBSD that helps with polling is setting 
kern.polling.idle_poll=1

By comparison, FreeBSD 7 with polling (no firewall) and NP (non 
polling, normal HZ)
[r2-current]# sysctl -a kern.clockrate
kern.clockrate: { hz = 4000, tick = 250, profhz = 2666, stathz = 533 }
[r2-current]#
4000    582,652      288,532   290,964
NP      590,068      312,334   320,538

        ---Mike 






More information about the Users mailing list