Initial messaging / message-port infrastructure is in

[David Leimbach]

Ah... I need coffee. :)  Light Weight Kernel Threading perhaps? :)

On Sunday, July 20, 2003, at 6:48AM, David Leimbach wrote:

Looks interesting... I need to get my FBSD box running again so I can  
play with this
I suppose?  [I clobbered it a few days ago :)]

Anyway.. what does LWKT mean?  I somehow have missed it... I assume  
its an
acronym?

Dave
On Saturday, July 19, 2003, at 8:56PM, Matthew Dillon wrote:
    I've committed the initial messaging/message-port infrastructure.  
 Here
    are the cvsweb links:

http://www.dragonflybsd.org/cgi-bin/cvsweb.cgi/src/sys/kern/ 
lwkt_msgport.c
http://www.dragonflybsd.org/cgi-bin/cvsweb.cgi/src/sys/sys/msgport.h
http://www.dragonflybsd.org/cgi-bin/cvsweb.cgi/src/sys/sys/thread.h

http://www.dragonflybsd.org/cgi-bin/cvsweb.cgi/src/sys/kern/ 
lwkt_msgport.c?rev=HEAD&content-type=text/x-cvsweb-markup
http://www.dragonflybsd.org/cgi-bin/cvsweb.cgi/src/sys/sys/msgport.h? 
rev=HEAD&content-type=text/x-cvsweb-markup
http://www.dragonflybsd.org/cgi-bin/cvsweb.cgi/src/sys/sys/thread.h? 
rev=HEAD&content-type=text/x-cvsweb-markup

    None of it is tested yet since nothing uses it yet (though DEV is  
about to
    start using it in a degenerate form), but I think people will be
    interested in seeing all of those comments I was making about  
messaging
    and IPIs realized by this commit.

    This also shows off one of the big advantages of asynchronous IPI
    messaging.  The messaging and port functions do not need to obtain
    any mutexes (even if there were no MP lock) even when they wind up
    queueing or dequeueing something.  But that isn't the only  
reason...
    because the messaging operations are combined with scheduling ops  
and
    one IPI will cover both the queueing and the scheduling aspects of
    a message.

    So right now we save at least 8 mutex equivalents with a single  
IPI
    message:

    Traditional mutex Design			LWKT/IPI design

    cpu1: get_queue_mtx				cpu1: send IPIQ to target cpu
    cpu1: queue_message
    cpu1: rel_queue_mtx
    cpu1: get_sched_mtx
    cpu1: schedule_target_thread
    cpu1: rel_sched_mtx
    cpu1: (IPI the target cpu anyway
	  in case it is idle??)
    cpu2: (receive wakeup IPI??)
    cpu2: get_sched_mtx				cpu2: receive IPIQ
    cpu2: locate_scheduled_thread		cpu2: queue message
    cpu2: rel_sched_mtx				cpu2: schedule target thread
    cpu2: get_queue_mtx				cpu2: dequeue message
    cpu2: dequeue_message			cpu2: execute message function
    cpu2: rel_queue_mtx
    cpu2: execute message function		
    Even if I were generous and ignored the fact that even in a  
mutexed
    system cpu1 might want to wake up cpu2 with an IPI, the fact  
remains
    that in a mutexed system *8* mutex operations are executed before
    the message can be acted upon, and in the LWKT system *NO* mutex
    operations are executed before the message can be acted upon.

    The question then becomes: is the IPI latency (which costs  
neither cpu
    any actual cycles but can take a 'long' time (e.g. 1uS)) and the  
IPI
    interrupt overhead (on cpu2, which does eat cycles on cpu2) worth  
the
    8 *CONTESTED* mutexes we just saved?  I would say:  probably.

    And when you add in the other efficiencies.. for example, the  
fact that
    lwkt_getport() and lwkt_waitport() (basically ALL message  
processing
    involving the thread owning the message port in question)  
requires only
    a critical section to manipulate.  No mutexes, no IPIs, no  
nothing.

    Now is it worth it?  I think so!

						-Matt