Unexpected benefit with Ryzen - reducing power for home server

[Matthew Dillon]

I received an unexpected benefit today when I checked for BIOS updates and
found that my ASRock AB350M Pro4 Ryzen motherboard now supports ECC.  I
decided that it is finally time to upgrade my home server from the
venerable E3-1270 (Xeon, 3.4GHz, 4-core/8-thread) to the 2700X (Ryzen,
~4.1GHz, 8-core/16-thread).

In doing so, I am using ECC memory and the only EUDIMMs I have are 2133.
At those memory frequencies, overclocking the cpu itself doesn't really add
much benefit so I decided to test just how LOW a power envelope I could set
and still retain full multi-threaded performance for compile jobs.  That
is, the system is limited by relatively low memory bandwidth so there's no
need to overclock the cpu itself.  It turns out... really low!  Ryzen CPUs
are very sensitive to memory bandwidth.

To make these changes I tell the BIOS to use the AMD CBS settings (usually
in the OC Tweaker menu), then go into the CBS settings at
Advanced->CBS->NBIO->XFR 2.0 Configuration.  I Enable XFR2, then set the
PPT, TDC, and EDC limits.  Set TDC and EDC high enough so they don't get in
the way, then limit power consumption by adjusting PPT.  By using the PPT
limit instead of manually setting the CPU frequency, the motherboard gets
the best of both worlds... it will idle just as low as it did before, it
will still run one or two cores at full speed (~4.1 GHz), and it will
ratchet down the frequency when all cores are loaded.  Using a PPT limit
with XFR2 is far, far superior to using manual OC frequency settings for
the CPU.

With standard XFR enabled in auto mode the 2700X will pull around 180W at
the wall at full load.  This might be useful if I had DDR4 3000 memory in
it, but I don't, so there's no real need to pull that much power at full
load.  I was able to reduce this all the way down to 85W at full load
without really impacting a concurrent -j 32 nativekernel NO_MODULES=TRUE
test compile.  Now, obviously, more compute-bound workloads will suffer a
lot more, but I mostly do compiles and with 16 cpu threads compiles tend to
be limited by memory bandwidth and not CPU frequency.

NOTE!  Some BIOSes these are in mW, others they are in W, double check!
Incorrect settings can crowbar your PSU and/or blow up the CPU!  The below
tests are with XFR2 enabled with a PPT power envelope limit set.  XFR2
controlls the CPU frequency (rather than using manual OC settings for the
CPU frequency).

time make -j 32 nativekernel NO_MODULES=TRUE >& /tmp/bk.out

PPT 100000 (160W) - 1:06 (3.6 GHz at full load, 1-core 4.2 GHz)
PPT 75000 (135W)   - 1:07 (3.4 GHz at full load, 1-core 4.2 GHz)
PPT 65000 (115W)   - 1:08 (3.2 GHz at full load, 1-core 4.1 GHz)
PPT 55000 (90W)     - 1:09 (3.0 GHz at full load, 1-core 4.1 GHz)
PPT 50000 (85W)     - 1:10 (2.8 GHz at full load, 1-core 4.0 GHz approx)

(TDC and EDC set to 300000 to get them out of the way)

I settled on running the server with a PPT of 65000.  That seems to be a
reasonable trade-off and its a huge benefit to power costs.   I've noticed
similar behavior on the Threadripper 2990WX (the 32-core/64-thread
beast)... there's just no need to run the TR2990WX at 330W.  225W or 250W
is just fine for multi-threaded workloads.

--

This is *very* impressive efficiency.  Whod a thought that one would be
able to run an 8-core/16-thread CPU at full load at only 85W and still reap
most of the benefit in a memory-heavy workload!

Of course, in the server space, we've known for a long time that maximum
efficiency occurs with a high number of cores running at lower frequencies,
and that efficiency trumps performance on machines with high core counts.
But I never considered that the consumer Ryzen CPUs could also benefit from
the same thing until now.  Now that consumer CPUs are starting to get up
there in core-count, it does make sense.  AMD has a real winner with their
XFR2 feature not only for overclocking, but also for reducing the power
envelope.  Not to mention supporting ECC on all of their CPUs, consumer and
server (which many Ryzen consumer mobos now support as demand has increased
for the feature).

-Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/users/attachments/20180902/865c5489/attachment-0002.htm>