protocol threads mpsafe mode

[Matthew Dillon]

    I think we can go with something similar, allowing you to start
    committing the work without interfering with stability on HEAD.
    How about a few cleanups though.  For testing purposes we don't
    really care about another few nanoseconds so:

    * Always start the protocol threads TDF_MPSAFE.  LWKT is MP-SAFE so
      there is no need to conditionalize it.  The msgport code is also
      MP-SAFE, so we really only have to worry about the function dispatch.

    * You might as well not bother with two different netmsg_service_loop
      threads.  Just have one, acquire and release the mp lock based on 
      NETISR_FLAG_MPSAFE and netisr_mpsafe_thread.


    Here's what I'm thinking.  This is untested but note that the critical
    path is handled nicely.

    ...
    netmsg_service_loop(...)
    {
	struct netmsg *msg;
	int mplocked;

	/*
	 * thread was started TDF_MPSAFE 
	 */
	mplocked = 0;	

	/*
	 * Loop on netmsgs, adjust the mplock dynamically.
	 */
	while ((msg = lwkt_waitport(&curthread->td_msgport, 0))) {
	    /*
	     * 0: BGL, 1: adaptive, 2: no BGL.
	     */
	    switch(netisr_mpsafe_thread) {
	    case 0:
		if (mplocked == 0) {
			get_mplock();
			mplocked = 1;
		}
		msg->nm_dispatch(msg);
		/* leave mplocked */
		break;
	    case 1:
		if (msg->nm_lmsg.ms_flags & MSGF_MPSAFE) {
			if (mplocked) {
			    rel_mplock();
			    mplocked = 0;
			}
			msg->nm_dispatch(msg);
			/* leave mpunlocked */
		} else {
			if (mplocked == 0) {
			    get_mplock();
			    /* mplocked = 1; not needed */
			}
			msg->nm_dispatch(msg);
			rel_mplock();
			mplocked = 0;
			/* leave mpunlocked, next msg might be mpsafe */
		}
		break;
	    case 2:
		if (mplocked) {
			rel_mplock();
			mplocked = 0;
		}
		msg->nm_dispatch(msg);
		/* leave mpunlocked */
		break;
	    }
	}
    }

    It concentrates the messy debugging code all in one place, which is
    also good.

    --

    netisr_run():   I'm not sure you need any MP tests at all in this
    code, won't it already be properly held or not held due already having
    been resolved by the original dispatch?  This routine is only called
    on re-dispatch.

    netisr_queue(): Would check the netisr flag and set or clear TDF_MPSAFE
    appropriately.

    schednetisr(): Would check the netisr flag and set or clear TDF_MPSAFE
    appropriately.

					-Matt
					Matthew Dillon 
					<dillon at backplane.com>