network forwarding patch

[Sepherosa Ziehau]

Hi,

Please review/test following patch:
http://leaf.dragonflybsd.org/~sephe/ifq_if_start.diff3

- ifnet.if_snd is protected by adaptive serializer
- ifnet's serializer holding is pushed down into each if_output implementation
- ifq_dispatch() is called by ether_output_frame().  ifq_dispatch()
will first try holding serializer, if the holding failed, which
indicates serializer contention, it will schedule ifnet.if_start on
ifnet's CPU
- Calling of ifnet.if_start is interlocked by ifnet.if_snd.altq_started
- Ethernet device drivers do not directly call ifp->if_start, instead
they call if_devstart()
- ifq_poll -> dev_encap success -> ifq_dequeue, is no longer safe,
instead ifq_dequeue -> dev_encap fail -> ifq_prepend is used
- five sysctl variables are added, which could help further testing
without code changing

With the above patch, under MP safe and non-fastforwarding case, I got
additional 250Kpps performance boost if streams are evenly distributed
on each CPUs.

I have tested this patch with most of the devices I have.  The
untested devices are listed at:
http://leaf.dragonflybsd.org/~sephe/netif
If you have one of the device listed above and could help testing, I
would be quite appreciated.

Since the patch became relative big, I plan to commit the above patch
before next weekend.

TODO:
- 802.11 device drivers need special version of if_devstart() (due to mgtq)
- rtw needs own if_devstart() (due to device driver's internal beaconq)

Best Regards,
sephe

-- 
Live Free or Die