Turn on adaptive MPSAFE for network threads, MPSAFE for IP and ARP

[Sepherosa Ziehau]

On Tue, Mar 10, 2009 at 12:28 AM, Matthias Schmidt
<matthias at dragonflybsd.org> wrote:
> Hi Sephe,
>
> * Sepherosa Ziehau wrote:
>> Hi all,
>>
>> I plan to set following tunables by default:
>> net.netisr.mpsafe_thread="1"
>> net.inet.udp.mpsafe_thread="1"
>> net.inet.tcp.mpsafe_thread="1"
>> net.link.ether.inet.arp_mpsafe="1"
>> net.inet.ip.mpsafe="1"
>>
>> The first three enables adaptive MPSAFE for network threads.
>> The last two enable IP and ARP MPSAFE.
>>
>> This gives us BGL free ip forwarding path and BGL free bridge
>> forwarding path.  pf is still under BGL though.
>
> Do you have any statistics regarding network performance?  Is there a
> "visible" difference?

On my Phenom9550 (2GB memory) w/ dual port 82571EB, one direction
forwarding, packets even spreaded to each core.  INVARIANTS is turned
on in the kernel config (I don't think it makes much sense to run a
system without INVARIANTS).

WIthout ipfw, BGL, ip forwarding:
~370Kpps - ~380Kpps

Without ipfw, BGL free, ip forwarding:
~600Kpps

With ipfw default pass rule, BGL free, ip/bridge forwarding:
~570Kpps

With ipfw 10 no matching rules + default pass rule, BGL free,
ip/bridge forwarding:
~530Kpps

Without ipfw, BGL free, ip fast forwarding (ipflow):
~650Kpps - ~670Kpps

Without ipfw, BGL free, bridging forwarding:
~650Kpps

For pure forwarding (i.e. no firewalling), the major bottle neck is on
transmit path (as I have measured, if you used spinlock on ifq, the
whole forwarding could be choked).  Hopefully ongoing multi queue work
could make the situation much better.

Best Regards,
sephe

-- 
Live Free or Die