Parallel compression not using all available CPU

[PeerCorps Trust Fund]

Hi,

It turns out that it was a combination of two things - turning off hyperthreading in BIOS and using a faster disk.

I found a post from the author of lbzip2 which seems to describe what might be happening in this case, but reference was made to a user using an i5 mobile CPU:

########################################################################

"bzip2 author here. I strongly suspect that you see what you see because your Intel core i5 is probably only dual core PLUS hyper-threaded, not real quad-core. Meaning, you have two instances of the L2 per-core cache, not four, and each two hyperthreads share an L2 cache.

Since the bzip2 compression/decompression is very cache sensitive (see "man bzip2"), the scaling factor will be determined mostly by how many OS-threads can dispose over a dedicated cache each. In your case this number is probably 2.

Since you run two threads per core, those contend for the shared L2 cache, basically each messing with the other (flushing / invalidating the shared cache for the other). This contention shows up as double CPU time, because "waiting for cache" (or "waiting for main memory") is accounted for as CPU time.

Hyperthreading is not useful but detrimental for lbzip2; so you should export LBZIP2="-n 2". You should not run more worker threads per core than: core-dedicated-cache-size divided by 8MB."

########################################################################

Running the compression again on the same file from an SSD with hyperthreading turned off, I was able to fully saturate all of the cores using lbzip2. None of this seemed obvious at first, but it rectified the situation. The biggest difference came from turning off hyperthreading (idle CPU - 20% vs the previous 90%) and then running from an SSD with hyperthreading turned off (idle CPU = 0%).

Previously, the compression was run from a single HDD, not an SSD. Concerning the compression test using the same HDD under FreeBSD, well I don't know why it was able to saturate the CPU. Perhaps it has something to do with ZFS's aggressive caching. Turning that off and re-running the test would likely answer the question. Pixz performed similarly when the above two modifications were made.



On 12/11/2016 02:00 AM, Jasse Jansson wrote:
> Have you tried to disable hypertreads in the BIOS ???
> It's a long shot, I know, but it might help.
>
> On 2016-12-10 22:14, PeerCorps Trust Fund wrote:
>> Hi,
>>
>> On both systems HAMMER was used. One small correction concerning the 2c/2t machine, both compression programs did effectively utilize that CPU which had an idle % of 0.0. It is the bigger machine, 16c/32t where the CPU isn't effectively maxed out. I'll continue to try and investigate why and report back if I find anything.
>>
>>
>> On 12/10/2016 10:26 PM, Justin Sherrill wrote:
>>> On the two DragonFly systems, was it Hammer or UFS?  I would be
>>> surprised if that made a difference, but it might?
>>>
>>> On Sat, Dec 10, 2016 at 6:19 AM, PeerCorps Trust Fund
>>> <ipc at peercorpstrust.org> wrote:
>>>> Hi,
>>>>
>>>> I've observed that parallel compression tools such as pixz and lbzip2 do not
>>>> make use of all of the available CPU under Dragonfly. On other OSes, it
>>>> does.
>>>>
>>>> When testing on a 50 gb file, using top I've observed that CPU idle
>>>> percentages consistently hover around the 90% range for pixz and ~70% for
>>>> lbzip2. These values under FreeBSD and Linux are typically ~0.0% idle until
>>>> compression is complete. Correspondingly, compression takes significantly
>>>> longer under Dragonfly, so the CPU is really being under utilized in this
>>>> case as opposed to erroneous reporting by top.
>>>>
>>>> This was tested on two systems, one 16c/32t and a 2c/2t system on a recent
>>>> master DragonFly v4.7.0.973.g8d7da-DEVELOPMENT #2: Wed Dec  7 11:44:04 EET
>>>> 2016.
>>>>
>>>> Has anyone else possibly observed this?
>>>>
>>>> --
>>>> Mike
>>>>
>>>
>>
>
>