Large pages; page coloring

[Alex Merritt]

I learned this recently, having gained access to newer Intel processors:
these CPUs (Sandybridge, Haswell) use a form of indexing into the LLC which
is no longer direct (i.e. taking specific bits from a physical address to
determine which set the cache line in the LLC it goes into), but rather
what they call "complex indexing"[1]. Presumably this is some proprietary
hashing.

I wanted to ask -- does page coloring, using direct indexing logic by the
kernel, have an advantage if such hashing is used, also if we are unaware
of the specific algorithm used to index the LLC? If we are unable to
determine which pages will conflict in the cache without careful study, and
assuming this algorithm may change between microarchitectures, it seems
there may be less benefit to applying the technique.

[1]  Intel Manual Vol.2A Table 3-17, cpuid command 04H

-Alex

On Tue, Apr 14, 2015 at 10:47 AM, Matthew Dillon <dillon at backplane.com>
wrote:
>
> --
>
> If I recall, FreeBSD mostly removed page coloring from their VM page
> allocation subsystem.  DragonFly kept it and integrated it into the
> fine-grained-locked VM page allocator.  There's no advantage to
> manipulating the parameters for two reasons.
>
> First, all page coloring really does is try to avoid degenerate situations
> in the cpu caches.  The cpu caches are already 4-way or 8-way
> set-associative.  The page coloring improves this but frankly even the set
> associativeness in the base cpu caches gets us most of the way there.  So
> adjusting the page coloring algorithms will not yield any improvements.
>
> Secondly, the L1 cache is a physical memory cache but it is also virtually
> indexed.  This is a cpu hardware optimization that allows the cache lookup
> to be initiated concurrent with the TLB lookup.  Because of this, physical
> set associatively does not actually solve all the problems which can occur
> with a virtually indexed cache.
>
> So the userland memory allocator implements an offsetting feature for
> allocations which attempts to address the virtually indexed cache issues.
> This feature is just as important as the physical page coloring feature for
> performance purposes.
>
> -Matt
>
>
> On Tue, Apr 14, 2015 at 10:10 AM, Alex Merritt <merritt.alex at gmail.com>
> wrote:
>
>> Hello!
>>
>> I am interested in learning whether Dragonfly supports large pages (2M
>> and 1G), and secondly, what mechanisms exist for applications to have
>> influence over the colors used to assign the physical pages backing their
>> memory, specifically for private anonymous mmap'd regions. Regarding
>> coloring, I'd like to be able to evaluate applications with a small number
>> of colors (restricting their access to the last-level cache) and compare
>> their performance to more/all colors available. I am initially looking to
>> work in hacks to achieve this to perform some preliminary experiments,
>> perhaps by way of a kernel module or something.
>>
>> A cursory search of the code showed no hints at support for large pages,
>> but I did find there are more internal functions governing the allocation
>> of pages based on colors, compared to FreeBSD (10.1). In FreeBSD it seems
>> colors are only considered for regions which are added that are backed by a
>> file, but I am not 100% certain.
>>
>> I appreciate any help!
>>
>> Thanks,
>> Alex
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/kernel/attachments/20150529/c14b9f8f/attachment-0002.htm>