network01.patch - MPSAFE network interrupt

[Matthew Dillon]

    Ok, here is my first go at making the network interrupt MP SAFE.  The
    patch can be fetched here:

	fetch http://apollo.backplane.com/DFlyMisc/network01.patch

    In order to test the patch you need to:

	(1) Apply the patch
	(2) ON SMP systems sysctl kern.intr_mpsafe=1 (or assign it in
	    /boot/loader.conf).  Note: you can boot with intr_mpsafe's
	    0 default and set it to 1 at any time.
	(3) ON SMP systems Make sure there are no other drivers sharing
	    the network interrupt.  vmstat -i will tell you.  For testing
	    purposes, remove any other non-network drivers that share the
	    interrupt.
	(4) Make sure that both the INVARIANTS and INVARIANT_SUPPORT options
	    are set.
	(5) Do a full kernel recompile.

    On SMP systems, to verify that the network interrupt is running MPSAFE,
    ping or exercise the network a little, then do a 'ps axl' and grep out
    the ithread for the network interrupt irq number.  The thread should
    be marked 'M' (which means 'MP'... not holding the big giant lock). 
    It is quite typical for firewire to share the same irq.  If this is
    the case, you will have to compile a kernel without firewire to test
    the network.

    IF YOU DONT HAVE A SMP BOX, the patch also needs serious testing on UP
    boxes.  The changes to the interrupt interlock infrastructure applies 
    to both UP and SMP.  On UP systems we are now relying on the serializer
    instead of a critical section to protect the networking hardware.

    This patch needs serious testing.  I intend to commit it some time next
    week.  Possibly as early as monday but it will depend heavily on what
    kind of feedback I get.

    --

    Making the network interface MP safe involves protecting portions of
    the IFNET structure, in particular the flags and altq.  Since the 
    code paths are fairly long I could not use the spinlock implementation,
    but it turned out to be a good thing because using the LWKT serialization
    subsystem (which is just as efficient for the non-blocking case) allowed
    me to integrate the network interrupt serialization with the IFNET 
    serialization but using the same serialization structure for both.

    This means that only a single serialization lock is acquired for the
    entire interrupt path.

    The rest of the work involved protecting all entry points into the
    network device driver, mainly if_start, if_init, if_ioctl, and so forth.
    It turned out to be fairly minimally invasive to the actual network
    drivers themselves which means I was able to make ALL of them MPSAFE
    rather then just a few test cases.

    --

    After we release (middle of December) I am going to make a big push to
    get Jeff's parallel routing implementation in and to make the TCP and
    UDP network protocol threads MP safe.  The current patch was designed
    to dovetail into those efforts.
  
						-Matt