FlopMark

The figures produced by the benchmark show two curves per chart:

1. The dashed line shows ideal scaling: twice the workers, twice as fast.
2. The solid line shows measured scaling: it starts close to ideal scaling, then bends away.

This is not a flaw of the benchmark. It happens everywhere from phones and laptops to supercomputers and clusters on the cloud (where the picture becomes more complex, of course). Below, we briefly explain three reasons. Two of them are directly related to communication: between the workers, and with the memory.

Many modern devices mix two kinds of cores: fast performance cores, and slower efficiency cores that save energy. The first workers usually get the fast ones. Once all performance cores are taken, the slower efficiency cores join the game and the curve bends (even more).

Thermal throttling plays a role too. A warm phone slows itself down, and the system can move workers from one core to another during the run.

Some steps are sequential by nature: in that case, one worker completes the operation while the others wait for the result. Here are some examples:

• In the LU solve, choosing the next pivot is performed by only one worker.
• In the Poisson solve, the workers must stop at every step to sum up their results.

These algorithmic restrictions set a natural limit. In the picture below, if one worker finishes the task in a certain amount of time, then with 8 workers the same task is only 4.7× faster, not 8×. (Illustrative: assuming a tenth of the work is purely sequential.)

Every core reads and writes its numbers in the same memory, through the same path. Alone, a core has this path to itself. With eight cores, they share it, and each one spends more of its time waiting for numbers.

On many devices the power table shows it: from one core to all of them, arithmetic often grows much more than memory speed. The Poisson solve is more memory-intensive than arithmetic-intensive, so this test shows the memory limit best.

Researchers redesign algorithms so that workers talk less often: fewer meetings, and bigger chunks of work between them. A recent research direction is to exploit randomness to reduce communication between cores by, for example, letting LU skip pivoting (in a clever way).

Another way is to change the question. Instead of solving the same problem faster, the extra cores solve a bigger problem in the same time. This is called weak scaling, and it is often how supercomputers are used.