Severe packet loss on fxp interfaces with the new bridging code...

[Chris Csanady]

On an SMP machine without any of the new mpsafe sysctls active, bridging
over fxp interfaces barely works.  Maximum bandwidth is somewhere around
100K/s, and I am seeing up to 80% packet loss.  (as measured with a
simultaneous scp and ping)  After a bit of detective work, I have found
the location of these mysteriously vanishing packets, but I do not
understand the true nature of the problem.
At if_fxp.c, line 1335, fxp_add_rfabuf() often fails to allocate a
replacement mbuf for use in the receive ring.  It then silently proceeds
to reuse the packet.   Any ideas on why the
	m_getcl(MB_DONTWAIT, MT_DATA, M_PKTHDR)

would fail here?  In any case, it might be nice to add an
"else ifp->if_ierrors++;" clause to the relevant if block...
While tracking this down, I also discovered that one of the BPF_MTAP()
calls is misplaced.  If you tcpdump on a bridge interface, you will miss
any traffic destined for the local machine itself.  The below diff  
corrects
this, as well as acounts for packets sent out via the bridge interface.

Chris

--- if_bridge.c.orig    2005-12-23 06:47:31.000000000 -0600
+++ if_bridge.c 2005-12-23 07:40:07.000000000 -0600
@@ -1329,6 +1329,7 @@
                if (m == 0)
                        break;
                BPF_MTAP(ifp, m);
+               ifp->if_opackets++;
                eh = mtod(m, struct ether_header *);
                dst_if = NULL;
@@ -1520,6 +1521,12 @@
        m->m_flags &= ~M_PROTO1; /* XXX Hack - loop prevention */
+       /*
+        * Tap all packets arriving on the bridge, no matter if
+        * they are local destinations or not.  In is in.
+        */
+       BPF_MTAP(bifp, m);
+
#define IFP2AC(ifp) ((struct arpcom *)(ifp))
#define IFP2ENADDR(ifp) (IFP2AC(ifp)->ac_enaddr)
        if (memcmp(eh->ether_dhost, IFP2ENADDR(bifp),
@@ -1537,12 +1544,6 @@
                goto out;
        }
-       /*
-        * Tap all packets arriving on the bridge, no matter if
-        * they are local destinations or not.  In is in.
-        */
-       BPF_MTAP(bifp, m);
-
        if (ETHER_IS_MULTICAST(eh->ether_dhost)) {
                /* Tap off 802.1D packets; they do not get  
forwarded. */
                if (memcmp(eh->ether_dhost, bstp_etheraddr,