dual port EM nic wedging under load
Mike Tancsa
mike at sentex.net
Sun Nov 26 15:00:35 PST 2006
At 11:42 AM 11/26/2006, Matthew Dillon wrote:
:Using polling and fastfwd on, I am able to get about 300Kpps in a
:unidirectional blast and still see that rate even with 10 poorly
:written ipfw rules !?!
Are you sure the ipfw ruleset is being executed? :-) ... whenever
something looks to be too good to be true....
It does look odd, but this is what I was doing. Here is without ipfw
[r2-dragonfly]# kldunload ipfw
IP firewall unloaded
[r2-dragonfly]# ifstat -b
bge0 em0 em1
Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out
0.00 0.00 0.00 0.00 0.00 0.00
2.87 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
0.46 0.00 0.00 0.00 0.00 0.00
0.00 0.00 78943.98 0.46 0.46 78872.37
0.46 0.00 140623.2 0.00 0.00 140623.2
0.00 0.00 140622.8 0.00 0.00 140622.8
0.46 0.00 140610.4 0.00 0.00 140610.4
0.00 0.00 140607.5 0.00 0.00 140607.5
2.91 0.00 140501.6 47328.70 47337.99 140501.6
0.00 0.00 140607.2 140607.2 140607.2 140607.2
0.46 0.00 140607.9 140607.9 140607.9 140607.9
0.00 0.00 140607.0 140607.0 140607.0 140607.0
0.46 0.00 140599.7 140599.7 140599.7 140599.7
0.00 0.00 140615.4 140615.4 140615.4 140615.4
0.46 0.00 140606.8 140606.8 140606.8 140606.8
0.00 0.00 140606.2 140606.2 140606.2 140606.2
0.46 0.00 140539.4 140539.4 140609.0 140609.0
0.00 0.00 140596.2 140596.2 140526.5 140526.5
0.00 0.00 140624.2 99093.16 99093.16 140624.2
0.46 0.00 140615.0 0.00 0.00 140615.0
First one stream, then a second... Rate is about the same.
Then I load the kld, and add some poorly written rules
[r2-dragonfly]# ipfw show
00010 0 0 deny log ip from 10.0.0.1 to any
00020 0 0 deny log ip from 172.0.0.1 to any
00030 0 0 deny log tcp from 172.0.1.1 to any
00040 0 0 deny log udp from 172.2.1.1 to any
00050 0 0 deny log tcp from any to 10.90.1.1 135-139
00060 0 0 unreach port tcp from 192.168.43.98 to me 25
00070 0 0 deny log tcp from any to 10.66.2.2 445
00080 0 0 deny log tcp from any to 10.61.2.2 138
00090 0 0 deny log ip from any to 219.0.0.0/16
00100 0 0 deny log esp from 218.0.0.0/8 to any
01000 1 242 allow ip from any to any
65535 0 0 deny ip from any to any
[r2-dragonfly]#
Same throughput
[r2-dragonfly]# ifstat -b
bge0 em0 em1
Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out
0.46 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
0.93 0.00 97387.37 0.00 0.00 97378.09
0.00 0.00 140628.8 0.00 0.00 140628.8
0.87 0.00 140607.9 0.00 0.00 140607.9
0.93 0.00 140541.2 0.00 0.00 140541.2
0.00 0.00 140647.7 27050.50 27055.88 140647.7
0.46 0.00 140607.2 140607.2 140607.2 140607.2
0.00 0.00 140606.5 140606.5 140606.5 140606.5
0.46 0.00 140607.2 140607.2 140607.2 140607.2
0.46 0.00 140603.3 140603.3 140603.3 140603.3
0.46 0.00 140611.8 140611.8 140611.8 140611.8
0.00 0.00 140606.9 140606.9 140606.9 140606.9
0.46 0.00 140525.3 140525.3 140525.3 140525.3
0.00 0.00 140598.5 140598.5 140598.5 140598.5
0.46 0.00 140617.8 140617.8 140617.8 140617.8
0.00 0.00 140606.9 140606.9 140606.9 140606.9
Whats odd however, is
[r2-dragonfly]# ipfw show
00010 0 0 deny log ip from 10.0.0.1 to any
00020 0 0 deny log ip from 172.0.0.1 to any
00030 0 0 deny log tcp from 172.0.1.1 to any
00040 0 0 deny log udp from 172.2.1.1 to any
00050 0 0 deny log tcp from any to 10.90.1.1 135-139
00060 0 0 unreach port tcp from 192.168.43.98 to me 25
00070 0 0 deny log tcp from any to 10.66.2.2 445
00080 0 0 deny log tcp from any to 10.61.2.2 138
00090 0 0 deny log ip from any to 219.0.0.0/16
00100 0 0 deny log esp from 218.0.0.0/8 to any
01000 482 19162 allow ip from any to any
65535 0 0 deny ip from any to any
[r2-dragonfly]#
Rule 1000 should have WAY more hits
And sure enough
[r2-dragonfly]# ipfw show
00005 0 0 deny ip from any to 192.168.44.1
00010 0 0 deny log ip from 10.0.0.1 to any
00020 0 0 deny log ip from 172.0.0.1 to any
00030 0 0 deny log tcp from 172.0.1.1 to any
00040 0 0 deny log udp from 172.2.1.1 to any
00050 0 0 deny log tcp from any to 10.90.1.1 135-139
00060 0 0 unreach port tcp from 192.168.43.98 to me 25
00070 0 0 deny log tcp from any to 10.66.2.2 445
00080 0 0 deny log tcp from any to 10.61.2.2 138
00090 0 0 deny log ip from any to 219.0.0.0/16
00100 0 0 deny log esp from 218.0.0.0/8 to any
00200 0 0 count ip from any to 192.168.44.1
00210 0 0 deny ip from any to 192.168.44.1
01000 489 20536 allow ip from any to any
01010 0 0 count ip from any to 192.168.44.1
65535 0 0 deny ip from any to any
[r2-dragonfly]#
[r2-dragonfly]# ipfw add 210 deny ip from any to 192.168.44.1
00210 deny ip from any to 192.168.44.1
[r2-dragonfly]# ipfw show
00010 0 0 deny log ip from 10.0.0.1 to any
00020 0 0 deny log ip from 172.0.0.1 to any
00030 0 0 deny log tcp from 172.0.1.1 to any
00040 0 0 deny log udp from 172.2.1.1 to any
00050 0 0 deny log tcp from any to 10.90.1.1 135-139
00060 0 0 unreach port tcp from 192.168.43.98 to me 25
00070 0 0 deny log tcp from any to 10.66.2.2 445
00080 0 0 deny log tcp from any to 10.61.2.2 138
00090 0 0 deny log ip from any to 219.0.0.0/16
00100 0 0 deny log esp from 218.0.0.0/8 to any
00200 0 0 count ip from any to 192.168.44.1
00210 0 0 deny ip from any to 192.168.44.1
01000 488 20294 allow ip from any to any
01010 0 0 count ip from any to 192.168.44.1
65535 0 0 deny ip from any to any
[r2-dragonfly]# ipfw add 5 deny ip from any to 192.168.44.1
00005 deny ip from any to 192.168.44.1
[r2-dragonfly]# ipfw show
00005 0 0 deny ip from any to 192.168.44.1
00010 0 0 deny log ip from 10.0.0.1 to any
00020 0 0 deny log ip from 172.0.0.1 to any
00030 0 0 deny log tcp from 172.0.1.1 to any
00040 0 0 deny log udp from 172.2.1.1 to any
00050 0 0 deny log tcp from any to 10.90.1.1 135-139
00060 0 0 unreach port tcp from 192.168.43.98 to me 25
00070 0 0 deny log tcp from any to 10.66.2.2 445
00080 0 0 deny log tcp from any to 10.61.2.2 138
00090 0 0 deny log ip from any to 219.0.0.0/16
00100 0 0 deny log esp from 218.0.0.0/8 to any
00200 0 0 count ip from any to 192.168.44.1
00210 0 0 deny ip from any to 192.168.44.1
01000 489 20536 allow ip from any to any
01010 0 0 count ip from any to 192.168.44.1
65535 0 0 deny ip from any to any
But when I use all, it seems to work
[r2-dragonfly]# ipfw add 4 deny all from any to 192.168.44.1
00004 deny ip from any to 192.168.44.1
[r2-dragonfly]# ipfw show
00004 238982 9081316 deny ip from any to 192.168.44.1
00005 0 0 deny ip from any to 192.168.44.1
00010 0 0 deny log ip from 10.0.0.1 to any
00020 0 0 deny log ip from 172.0.0.1 to any
00030 0 0 deny log tcp from 172.0.1.1 to any
00040 0 0 deny log udp from 172.2.1.1 to any
00050 0 0 deny log tcp from any to 10.90.1.1 135-139
00060 0 0 unreach port tcp from 192.168.43.98 to me 25
00070 0 0 deny log tcp from any to 10.66.2.2 445
00080 0 0 deny log tcp from any to 10.61.2.2 138
00090 0 0 deny log ip from any to 219.0.0.0/16
00100 0 0 deny log esp from 218.0.0.0/8 to any
00200 0 0 count ip from any to 192.168.44.1
00210 0 0 deny ip from any to 192.168.44.1
01000 793 32521 allow ip from any to any
01010 0 0 count ip from any to 192.168.44.1
65535 0 0 deny ip from any to any
[r2-dragonfly]#
Something seems busted.
:Also it handles the load quite smoothly for the bi-directional test
:
:Here is the output of ifstat -b as seen from the box acting as router
:
:You can see the first stream starting up, and then the second on the
:opposite stream. Rates remain constant throughout, which is quite
:different from FreeBSD. ipfw on Dragonfly has no ill effect for some reason.
:...
I'm not sure if you are overloading the system intentionally or not.
There seem to be an excessive number of ENOBUF issues during packet
processing, assuming the count start at 0 when the test began.
:em0: Missed Packets = 221709682 <<<<<<
:em0: Receive No Buffers = 53994896 <<<<<< This seems excessive to me
:em0: Receive length errors = 0
:em0: Receive errors = 0
:em0: Crc errors = 0
:em0: Alignment errors = 0
:em0: Carrier extension errors = 0
:em0: RX overruns = 55192 <<<<<<
:...
:em0: Good Packets Rcvd = 187109273
:em0: Good Packets Xmtd = 30553884
:em1: Missed Packets = 17508539 <<<<<<
:em1: Receive No Buffers = 14148594 <<<<<<
:em1: Receive length errors = 0
:em1: Receive errors = 0
:em1: Crc errors = 0
:em1: Alignment errors = 0
:em1: Carrier extension errors = 0
:em1: RX overruns = 0 <<<<<<
:em1: Good Packets Rcvd = 30553871
:em1: Good Packets Xmtd = 187031079
One really nice thing about DragonFly's NETIF polling code is that you
can control the polling frequency (kern.polling.pollhz) and other
parameters on the fly with sysctl's. You can actually change the polling
frequency on a live system!
Yes, if I set the polling hz to 5000, I am getting 596,231 on the
single stream, and when I bring online the stream in the opposite
direction, I get 335,006 in one direction and 160,262 in the other.
kern.polling.each_burst: 5
HZ 1 blast blast A and blast B
3000 450,072 349,696 349,814
4000 588,514 391,615 391,753
5000 592,655 344,472 360,468
6000 593,966 353,047 352,177
Changing
kern.polling.each_burst: 20,40
doesnt seem to have much impact statistically.
Even at 5000Hz, throughput is pretty steady... what comes in, goes out
[r2-dragonfly]# ifstat -b
bge0 em0 em1
Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out
0.47 0.00 158176.3 158176.3 158219.2 158219.2
0.00 0.00 167004.1 167004.1 166937.3 166937.3
0.46 0.00 167158.7 167158.7 167158.7 167158.7
0.00 0.00 167161.0 167161.0 167194.9 167194.9
0.46 0.00 167110.8 167110.8 167076.9 167076.9
0.00 0.00 155460.8 155460.8 155460.8 155460.8
0.46 0.00 167033.7 167033.7 167033.7 167033.7
^C
[r2-dragonfly]#
Top shows
load
averages: 0.18, 0.05, 0.01
up 0+05:17:52 15:58:38
29 processes: 1 running, 28 sleeping
CPU0 states: 0.0% user, 0.0% nice, 88.7% system, 0.0% interrupt, 11.3% idle
CPU1 states: 0.0% user, 0.0% nice, 0.0% system, 1.6% interrupt, 98.4% idle
Mem: 9432K Active, 10M Inact, 58M Wired, 9216K Buf, 1929M Free
[r2-dragonfly]# sysctl -a kern.polling
kern.polling.burst: 119
kern.polling.each_burst: 30
kern.polling.burst_max: 150
kern.polling.user_frac: 50
kern.polling.reg_frac: 20
kern.polling.short_ticks: 39675403
kern.polling.lost_polls: 679638
kern.polling.pending_polls: 1
kern.polling.residual_burst: 0
kern.polling.handlers: 2
kern.polling.enable: 1
kern.polling.phase: 2
kern.polling.suspect: 12710
kern.polling.stalled: 2
kern.polling.pollhz: 4000
[r2-dragonfly]#
One thing I found on FreeBSD that helps with polling is setting
kern.polling.idle_poll=1
By comparison, FreeBSD 7 with polling (no firewall) and NP (non
polling, normal HZ)
[r2-current]# sysctl -a kern.clockrate
kern.clockrate: { hz = 4000, tick = 250, profhz = 2666, stathz = 533 }
[r2-current]#
4000 582,652 288,532 290,964
NP 590,068 312,334 320,538
---Mike
More information about the Users
mailing list