<div dir="ltr"><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 14, 2015 at 1:47 PM, Matthew Dillon <span dir="ltr"><<a href="mailto:dillon@backplane.com" target="_blank">dillon@backplane.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>DragonFly only uses large pages in-kernel and only for a few things. They do not extend to user-land. The main reason is that to do it properly and make large pages deterministic we would have to completely rewrite the memory allocator, otherwise the benefit is not well realized. FreeBSD didn't and their large page allocations are relatively non-deterministic, meaning it is hard to count on them happening all the time under normal operating conditions.<br><br></div><div>DragonFly does have a shared page table optimization for common memory maps which is useful for things like database servers (postgres, which forks instead of threads). This feature can be turned on with the machdep.pmap_mmu_optimize sysctl. It is considered experimental. Basically what it does is recognize when two or more distinct processes mmap() the same shared file or VM object (so it also works for any fork-shared memory areas) and will simply have the processes MMU maps access the same physical MMU directory page. This has a big positive effect on reducing VM page faults on heavily shared forked situations.<br><br></div><div>Big pages reduce TLB overheads but can wind up being pretty memory inefficient. Memory efficiency tends to be more important these days... that is, servers are generally limited by memory and not limited by cpu.<br></div></div></blockquote><div><br></div><div>Yes, in the general case you are right. I am looking at a special case of applications which allocate all available memory and execute in isolation on servers.</div><div><br></div><div>I do know with Linux, use of large pages first requires the sysadmin to tell the kernel to reserve some quantity of memory specifically for mapping using large pages by applications using mmap. Is this an extension of a memory management layer that avoids reworking the entire subsystem (like you suggest) to incorporate consideration of multiple page sizes throughout the system?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div><br>--<br><br></div><div>If I recall, FreeBSD mostly removed page coloring from their VM page allocation subsystem. DragonFly kept it and integrated it into the fine-grained-locked VM page allocator. There's no advantage to manipulating the parameters for two reasons.<br><br></div><div>First, all page coloring really does is try to avoid degenerate situations in the cpu caches. The cpu caches are already 4-way or 8-way set-associative. The page coloring improves this but frankly even the set associativeness in the base cpu caches gets us most of the way there. So adjusting the page coloring algorithms will not yield any improvements.<br></div></div></blockquote><div><br></div><div>That makes perfect sense, and I believe this was one reason Linus Torvalds has been against inclusion of coloring mechanism in Linux itself.</div><div><br></div><div>I am not interested in using coloring to manage multiple applications, but rather to impact the performance for an individually executing application with various memory regions of different properties.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Secondly, the L1 cache is a physical memory cache but it is also virtually indexed. This is a cpu hardware optimization that allows the cache lookup to be initiated concurrent with the TLB lookup. Because of this, physical set associatively does not actually solve all the problems which can occur with a virtually indexed cache.<br><br></div><div>So the userland memory allocator implements an offsetting feature for allocations which attempts to address the virtually indexed cache issues. This feature is just as important as the physical page coloring feature for performance purposes.<br></div></div></blockquote><div><br></div><div>This is interesting, thank you. My main target is the LLC as it is significantly larger on server processors.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div><br></div>-Matt<br><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 14, 2015 at 10:10 AM, Alex Merritt <span dir="ltr"><<a href="mailto:merritt.alex@gmail.com" target="_blank">merritt.alex@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello!<div><br></div><div>I am interested in learning whether Dragonfly supports large pages (2M and 1G), and secondly, what mechanisms exist for applications to have influence over the colors used to assign the physical pages backing their memory, specifically for private anonymous mmap'd regions. Regarding coloring, I'd like to be able to evaluate applications with a small number of colors (restricting their access to the last-level cache) and compare their performance to more/all colors available. I am initially looking to work in hacks to achieve this to perform some preliminary experiments, perhaps by way of a kernel module or something.</div><div><br></div><div>A cursory search of the code showed no hints at support for large pages, but I did find there are more internal functions governing the allocation of pages based on colors, compared to FreeBSD (10.1). In FreeBSD it seems colors are only considered for regions which are added that are backed by a file, but I am not 100% certain.</div><div><br></div><div>I appreciate any help!</div><div><br></div><div>Thanks,</div><div>Alex</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>