phk malloc, was (Re: ptmalloc2)

[Jonathan Dama]

I couldn't disagree more.  What's happened is that address
space allocation and physical resource allocation have been
split in two.  There are plenty of reasons for this, not the
least of which is that there is no reason to pessimize every
process.  (Overcommit is a form of lazy-initialization)
Moreover, it permits time-sharing of the available physical 
resources.  I might liken this to circuit-switching versus
packet-switching.  

All your program has to do to be safe is to touch each page
of the allocated address space--the problem goes away.  In
your example of allocating at startup--this is a very fair
solution.* 

Overcommit has ill-deserved reputation.  One which results
from peoples' tendany to conflate the disparate concepts of
address space allocation and physical resource allocation.

The changes that I've mentioned to phkmalloc do not in any
way make it less safe than before.  phkmalloc with
internally allocated structures that do not make use of
overcommit will not be any safer than those that do make use
of overcommit.  Thus, once an OS is overcommit enabled, it is 
to the advantage of the malloc routines to make use of this
property.  This is because I would use overcommit in respect
to some of the caching that is used.  This caching is not
related to memory that has been allocated via malloc is
merely a technique to minimize the system-call overhead.  

But perhaps you were complaining that the VM does overcommit
at all...

having a knob to disable overcommit on a system is at best a
compatibility feature for programs that do not take the
appropriate steps to reserve the physical resources they
assume will be available.

-Jon

*and use madvise to call not me to avoid being the victim of
the OS hunting for pages.

**Disks are cheap, why not just impose some limits on the
number of running processes and give yourself swap-space to
the hilt.

***I agree it would be nice if there was a MEMORY_CONSTRAINED
signal that the OS could use to encourage processes to return 
pages to the system.


****Maybe Matt can elaborate on some of the issues
surrounding not implementing overcommit in the VM.