phk malloc, was (Re: ptmalloc2)

[Gary Thorpe]

com>
In-Reply-To: <20050223060503.GA18114 at xxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Lines: 58
Message-ID: <421eace4$0$719$415eb37d at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
NNTP-Posting-Host: 149.99.113.127
X-Trace: 1109306597 crater_reader.dragonflybsd.org 719 149.99.113.127
Xref: crater_reader.dragonflybsd.org dragonfly.kernel:7835

Dan Melomedman wrote:

> Bill Hacker wrote:
> 
>>I still thing fixing the application is preferable.
> 
> 
> What's there to fix if it's not the application's fault?
> Application is asking the OS exactly what it wants by design to make certain
> guarantees: physical memory, the OS on the other hand only promises to,
> but doesn't guarantee to give it to the application. When I go to a bank
> to cash a check, I expect hard, cold cash, not an IOU. Should the bank
> blame me for asking money because they don't have it?

I don't think most understand your point. They should read this 
paragraph until they do.

> 
> 
>>Documenting an 'overcommit switch' is all well and good, but theory 
>>aside, how stable can you expect MessageWall to be *in the real world* 
>>on a potentially resource-challenged LinBox that is running other things 
>>as well?
> 
> 
> In theory it should be much more stable. If not, then there's something
> wrong with the Linux kernel, and I'll report a bug. Robert Love wrote the
> overcomit accounting patch sometime in 2002 to make the out of memory
> situation impossible on Linux with careful tuning. It moves all memory
> failures to allocation routines. You either get physical memory or the
> allocation routines return failure, you retry later. This is much better
> than random SIGKILLs sent to your preallocating services for which the
> OS can't find physical memory when the box is stressed.

It is probably more performance optimal than that: you can probably just 
update the accounting for physical pages even though none are actually 
assigned yet.

> In theory this should be possible:
> 
> 1) Turn off swap.
> 2) Set rlimits on _all_ services within a reasonable amount to not exceed
> the available physical memory within a certain percentage - give a
> reasonable amount of slack, lets say 20 percent to the system when all
> said and done.
> 3) Shut off the overcommit and let 'er rip.
> 
> The above should guarantee that 'preallocating' service will never crash
> due to overcommit. The other processes are allowed to grow to their
> rlimit, and will be killed by the kernel when they reach it.
> 
> And we'll see how it does. This is on a test box of course. Am I missing
> anything?

No, that's pretty much how you would get the same effect on *BSD. I 
think the kernel not allowing overcomit is actually more flexible, but 
who knows.