serializing token

[Matthew Dillon]

:Is it true that functions that can block can cause 
:a release of a token?  Is that actually a "must
:cause the release of a token"?  If so then 
:being able to print out source code and highlight
:critical sections seems like it might be more 
:difficult for the new coder than say a mutex based
:locking system.

    Well, I'd would say that a new coder would be more likely to create
    a deadlock situation using mutexes.  Also, keep in mind that you cannot
    safely block holding a mutex in FreeBSD-5 (and its a bad idea to do so
    in any system), which means that there is serious code pollution between
    layers (an upper layer must know absolutely for sure whether a lower layer
    might block in order to determine whether a mutex can be safely held
    through the call to the lower layer).  In Dragonfly an upper layer need
    only allow for the fact that a lower layer might block in order to recheck
    (if necessary) the atomicy of data being protected by a token, and there
    is no code pollution whatsoever becaue the upper layer can simply assume
    that a lower layer might block if the upper layer is not sure, or if you
    want to program defensively.

    So in fact being able to block while holding a token is a feature of
    DragonFly, as long as the programmer is aware of the side effects.

:If the interrupt can get the token then it's even harder
:to draw a critical section in code where data is 
:consistent.  Also shouldn't some interrupts be treated with
:a higher priority than others?
:
:How can I write code with this API and know without a doubt
:that my data is how I left it.
:
:Dave

    I'm sure what you mean by harder to draw a critical section.  A 
    critical section will prevent an interrupt from occuring on the same
    cpu, but critical sections are only required in certain circumstances
    where a preemption might interfere with a piece of code.  For example,
    the slab allocator enters a critical section while manipulating the
    per-cpu globaldata cache in order to avoid conflicting with itself if
    it were called from a preemptive interrupt.

    Writing code the API is fairly simple, but it does depend on what you
    do 'inside' the area being protected by the token.  The vast majority
    of code that uses tokens, just as with mutexes, does not do anything fancy
    within the protected area and doesn't really have to worry about any of
    these issues.  For the remaining code you either have to be aware of the
    exact side effects (with mutexes), or you have to program defensively
    if you are not sure what a lower layer will do (with tokens).

    There is also a huge class of code where no rechecking is needed.  Take
    this for example:

    if (someglobal) {
	lwkt_gettoken(..)
	while (someglobal) {
	    someglobal &= ~someotherprocedurethatmightbloc();
	    call procedure that might block()
	}
	lwkt_reltoken(..)
    }

    This code snippit above is using a token to protect a global variable.
    It is able to optimize itself by not bothering to get the token unless
    it thinks it has work to do, and then integrating the recheck case in
    the while().  The token is protecting against preemptive modifications
    to the global variable rather then protecting against changes made to
    the global variable by the called procedures or as a side effect if 
    either procedure blocks.

    The really complex cases are relatively few... cases involving structural
    pointers such as unreferenced vnode pointers, vm_page's, and so forth.
    There are plenty of ways to protect such structures.  The vnode case is
    probably the most complex (and it is just as complex or even more complex
    when implemented with mutexes), most of the rest of the structures can be
    prevented from moving around with a ref count.

					-Matt
					Matthew Dillon 
					<dillon at xxxxxxxxxxxxx>