Network Slowdowns?

[Matthew Dillon]

    Packet overhead for bulk data transfers is only around 3.4%.  The
    ethernet MAC is 12 bytes.  The TCP/IP header is 40 bytes (20 bytes
    for UDP/IP), plus additional bytes for options, and the data payload
    is usually around 1448 bytes.

    Ethernet framing overhead is harder to calculate.  10BaseT is easy,
    it's just manchester encoded at 20MHz -- so 8 bits per byte at 10MHz,
    basically.  I think there is a short preamble and a CRC tacked on.

    100BaseT encodes 4 bits into 5 over the wire (10 bits per byte), but
    the line frequency is actually 125MHz, not 100MHz, so one byte of
    data takes 10 bits @ 8ns/bit = 80ns == 12.5 MBytes/sec.  If you
    subtract the 3.4% packet overhead you get around 12 MBytes/sec.  I
    think there is also a premable and some other junk there that
    reduces the effective bandwidth to 11 MBytes/sec with 1500 MTU packets.

    GiGE also uses a line frequency of 125MHz, encoding 2 (or is it 3?)
    bits into each 8ns period.  But (and now I am looking at the Wikipedia
    entry for GigE) it transmits in both directions on all four cable pairs.
    So you get 125 x 2 x 4 = 1 GBit.  Each 8 bit byte is converted to 10 bits.
    It's all very confusing, actually.  GigE requires a lot of signal 
    processing and uses echo cancellation to work.

    CPU overhead is a different beast entirely.   SSH has a lot of encryption
    and decryption overhead... my transfers over localhost top out at
    21.8 MBytes/sec on my test box.  At 100BaseT speeds the packet rate
    is only around 6000-8000 packets per second and cpu overhead for packet
    processing is usually in the noise.  At GiGE rates, however, the packet
    rate is over 66,000 packets per second and cpu overhead can be
    significant even if you discount interrupt and device access overheads.

						-Matt