The term “multithreading” gets thrown around a lot, and it can refer to a number of different technologies. It’s often not entirely clear what those technologies are, sometimes even to software developers. We’re going to talk about hardware multithreading, which is completely different from software multithreading. So let’s clarify the difference:
Hardware multithreading is when a single core is capable of executing multiple instruction streams at once. Non-multithreaded cores can only handle one stream at a time. For example, Intel’s “E-cores” on its desktop CPUs are single-thread cores, while its “P-cores” are dual-threaded cores. The main purpose of hardware multithreading is to make sure a CPU stays busy. In most applications, a CPU spends a lot of time sitting around and waiting for data, so being able to handle two applications at once increases the chances that the CPU will have data to work on.
Software multithreading is when a single program manages simultaneous execution streams. A program with only one execution stream is “serial.” For example, some file compression utilities will launch a number of threads to compress files in parallel. However, multithreading is not the only way to parallel exection. Many large, parallel HPC codes actually operate by launching hundreds or even thousands of serial programs that communicate via an external piece of software called daemon.
x86 Hyperthreading
Most x86 CPUs have a hardware multithreading technology called hyperthreading (HT). Via Wikipedia, this image illustrates very nicely how HT in x86 CPUs works. The different colors represent different instruction streams. The CPU is currently handling red and yellow streams. White indicates “empty” spots – registers and execution units and such that have nothing to do at the moment. Essentially, you have a single pipeline, but instructions from two different streams can be interleaved.
One significant issue with this architecture, which is currently on most x86 CPUs, is that it’s not very secure. With instructions from two different programs piled throughout the core, it’s possible to use some clever coding to grab the other program’s data. The best way to secure an x86 machine is to simply disable hyperthreading, as discussed in this article.
Power SMT
IBM Power’s SMT architecture is very different from x86 HT. Rather than a single pipeline into which instructions are interleaved, a Power10 SMT-8 CPU core (which can be split into two SMT-4 CPU cores) has two bundles of four individual “execution slices,” each bundle of four governed by its own instruction decoder–this does mean I need at least two software threads or programs to use the entire core. It’s almost like having eight small cores inside of one big one. This slide, which IBM showed at Hot Chips 32, shows the slice-based architecture nicely.
The physical separation between the slices means that IBM’s SMT doesn’t have the security vulnerabilities that x86-style HT does. In general, IBM’s Power and Telum CPUs are built with careful physical separations to ensure many users can occupy a machine, even the same core, without any chance of “cross-talk” or other vulnerabilities.
What about HPC?
Most HPC users disable HT on their x86 CPUs. The purpose of hardware multithreading in both x86 and Power is to hide latency due to cache misses, and most linear algebra codes don’t have enough latency for it to matter. The more you jump around in memory, the more latency you have. If you work on linear algebra codes, you might think you have a lot of latency, but compared to a database, you’re living in paradise. In fact, a pretty good rule of thumb is that the more your operation looks like database lookup and modification, the more benefit you’ll get from SMT.
We can see this in a linear algebra benchmark I did with HYPRE on a 15-core Power10 CPU running the BiCGStab algorithm. Going from 1 thread to 2 gets a large performance boost, since half my slice bundles are are inactive. But as I add more MPI process, my performance actually degrades. There’s just no latency to cure, at least not any that isn’t overwhelmed by the additional process overhead.
However, there are lots of pieces of HPC codes that do have a lot of latency. Basically any time you are managing data rather than computing, latency is your worst enemy. Meshing codes, for example, frequently have tons and tons of cache misses. Setting up an algebraic multigrid (AMG) preconditioner is another operation that looks a lot more like database creation than doing math.
That’s pretty impressive. Going from 1 thread to 8 got me a 5x speedup. Thus we have a handy rule of thumb. For math-heavy workloads, one Power10 SMT-8 core performs about well as two x86 cores. But for data-heavy workloads, it can be more like one to five. For many HPC applications, IBM Power’s SMT-8 architecture can provide significant acceleration.