IPI Benchmark Data?

[Matthew Dillon]

The main improvement was to the MMU page-invalidation routines.  In 4.4
these routines used the kernel's generic IPI messaging API.  But page
invalidations require all cpus to synchronize their changes and since the
generic IPI messaging API honors critical sections that could cause most of
the cpu's to stall for longer than intended if just one of the cpus
happened to be in a long critical section.  In 4.6 the page-invalidation
code runs through a dedicated IPI vector which ignores critical sections
and does not have this stall issue.

A second optimization as also implemented, but could not be tested well
enough to enable for the release.  This optimization can be turned on with
a sysctl in the 4.6 release (sysctl machdep.optimized_invltlb=1).  This
optimization is meant to avoid sending IPIs to CPUs which are idle and thus
might be in a low-power mode.  Such cpus will respond much more slowly to
the IPI vector and not only increase latency for the cpus running at full
power, but also have a tendancy to kick the cpus in low-power mode out of
low-power mode.  This second optimization is most useful on systems with a
lot of cores (multi-socket systems and systems with > 4 cores).  This one
will eventually be turned on by default once sufficient testing of the
feature has been done.

There were numerous other optimizations to reduce the amount of IPI
signalling needed.  Probably the biggest one is that the buffer cache no
longer synchronizes the MMU when throwing away a buffer cache buffer.  It
only synchronizes the MMU when allocating one.  This cut out roughly half
of the system-wide IPIs in a nominally loaded system.  We have an
additional optimization, also disabled by default, which nearly eliminates
buffer-cache-related IPIs in situations where filesystem disk bandwidth is
very high (sysctl vfs.repurpose_enable=1).  It's only really applicable
under heavy loads when consolidated disk I/O bandwidth exceeds 200
MBytes/sec.

-Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/kernel/attachments/20160802/21aa9d27/attachment-0002.html>