phk malloc, was (Re: ptmalloc2)

[Dan Melomedman]

Jonathan Dama wrote:
> All your program has to do to be safe is to touch each page
> of the allocated address space--the problem goes away.  In
> your example of allocating at startup--this is a very fair
> solution.* 

I think a possible problem here is that many programs are not
written this way, and sysadmins aren't aware either. I think
MessageWall fits this description for example.

> But perhaps you were complaining that the VM does overcommit
> at all...

It's a criticism (hopefully constructive), not complaint.
> 
> having a knob to disable overcommit on a system is at best a
> compatibility feature for programs that do not take the
> appropriate steps to reserve the physical resources they
> assume will be available.

IMHO the problem here is that many coders and sysadmins are not
aware of overcommit related reliability issues in the scenarios I've
mentioned. Should we really blame them if the documentation doesn't
really explain that malloc is actually lying to their code? Why not have
such a knob, let's say as an environment variable? Then I just run all such
overcommit-unaware software and it works as the coder expected it to
work.

> **Disks are cheap, why not just impose some limits on the
> number of running processes and give yourself swap-space to
> the hilt.

This isn't a scarce physical memory issue. This is a reliability issue.
Sometimes you want to guarantee a process won't crash if it thinks it
preallocated memory. Sometimes you want several of such processes running
under different UID/GIDs for security and fault isolation in addition to
using rlimit() resource limits. Sometimes you must guarantee
preallocation of physical memory simply due to performance reasons. What
about people with near real-time requirements? Paging is too slow. Also
swapping is sometimes highly undesirable due to security concerns.
In many cases you don't have or care for such requirements, but in some do,
and I'd think these are not such rare cases considering that people are
increasingly gaining appreciation for more reliable, secure and
fault-tolerant software and operating systems. Many people are just
unaware.

> 
> ***I agree it would be nice if there was a MEMORY_CONSTRAINED
> signal that the OS could use to encourage processes to return 
> pages to the system.

It would be nice to tell malloc to allocate and lock physical memory
through an environment variable, as well IMO.