protocol threads mpsafe mode

[Sepherosa Ziehau]

Hi all,

Following patch allows us to experiments network threads in three mode:
1) BGL (default)
2) Adaptive BGL.  A msg flag is added (MSGF_MPSAFE).  The protocol
threads ({netmsg,tcpmsg}_service_loop_mpsafe) run without BGL by
default, if they get msg that does not have MSGF_MPSAFE, then they
hold BGL
3) No BGL.  Protocol threads are started with TDF_MPSAFE, old protocol
functions ({netmsg,tcpmsg}_service_loop) are used.

NETISR register needs additional flag parameter (only
NETISR_FLAG_MPSAFE is defined), so netisr_run (used on ethernet input
path) could hold BGL based on the registered netisr flag.

PR_MPSAFE (protosw.pr_flags) is added, so transport_processing_oncpu()
could hold BGL conditionally (I didn't feel well about how to set this
flag in the current patch (in ip_input.c), any suggestions?)

Following tunables are added:
net.inet.tcp.mpsafe_thread (0: BGL, 1: adaptive BGL, 2: no BGL)
net.inet.udp.mpsafe_thread (0: BGL, 1: adaptive BGL, 2: no BGL)
netisr.mpsafe_thread (0: BGL, 1: adaptive BGL, 2: no BGL)
net.inet.tcp.mpsafe_proto (0: will not set PR_MPSAFE, 1: set PR_MPSAFE)
net.inet.udp.mpsafe_proto (0: will not set PR_MPSAFE, 1: set PR_MPSAFE)

Patch:
http://leaf.dragonflybsd.org/~sephe/proto.diff1

Please review it.

Best Regards,
sephe

-- 
Live Free or Die