Part of the Computer Science curriculum.
Computer Architecture explained why chipmakers stopped making single cores faster and started shipping more of them; this discipline picks up exactly there, asking a different question — given many cores (or many machines), how does a programmer actually get a program to use them?
Shared memory (UMA or NUMA) gives every processor a single address space it can all read and write directly; distributed memory gives each processor its own private memory and forces communication through explicit messages — the single hardware distinction that determines which programming model a parallel program can even use.
Two different axes along which a program can be split: task parallelism runs different operations concurrently (each thread does something different), while data parallelism runs the same operation concurrently over different pieces of data — the axis CS2013 names explicitly as the two core decomposition strategies.
Domain decomposition splits the data into chunks and gives each processor the same code to run on its own chunk (a direct data-parallel realization); functional decomposition splits the problem into distinct stages or roles, each running different code — the same partitioning question divide-and-conquer already asked, now applied across processors instead of recursive calls.
Splitting work is the easy part; a real parallel program also has to move data between the pieces (communication), keep operations that depend on each other's results in the right order (synchronization), and identify exactly which operations have those dependencies in the first place — three separate, unavoidable costs every decomposition has to pay.
Granularity is the ratio of computation to communication in each parallel chunk — too fine and communication overhead dominates, too coarse and processors sit idle waiting for a few slow chunks; load balancing is the companion problem of making sure every processor actually gets a fair, similarly-sized share of the work.
Speedup = T(1) / T(p), the ratio of sequential to parallel running time on p processors; efficiency = Speedup / p, the fraction of each processor's time actually spent on useful work — the two numbers every parallel performance claim ultimately has to be measured against.
If a fraction f of a program is inherently sequential, no number of processors can push speedup past 1/f — a hard ceiling that turns "just add more cores" into a question that always has to start with "how much of this can even run in parallel?"
Amdahl's Law assumes the problem size is fixed while processors grow; Gustafson's reframing assumes the opposite — in practice, more processors are used to solve a bigger problem in the same time, not the same problem faster — and under that assumption, speedup scales almost linearly instead of hitting a hard ceiling.
Strong scaling holds the problem size fixed and adds processors — the exact regime Amdahl's Law bounds; weak scaling grows the problem size in proportion to the processors — the exact regime Gustafson's Law describes. Real HPC performance reports name which one they mean, because the two questions have very different honest answers.
A single master thread runs sequentially until it hits a `#pragma omp parallel` region, at which point it forks a team of worker threads that all execute that region together, then joins back into one thread at the region's end — the fork-join model every OpenMP program is built from.
`#pragma omp for` splits loop iterations across the threads already forked by an enclosing parallel region, and a `reduction` clause gives every thread its own private accumulator that OpenMP combines safely at the end — the standard, directive-only way to data-parallelize an ordinary loop without touching its body's logic.
When a reduction clause isn't expressive enough, `critical` and `atomic` give threads mutually exclusive access to a shared update — `atomic` compiling to a single fast hardware instruction where possible, `critical` covering arbitrary blocks of code at a higher cost — the two directive-level tools that keep a shared-memory program correct without hand-writing a lock.
MPI has no shared address space to fork threads into — instead, a fixed number of independent processes are launched together (SPMD: the same program, running on every process), each one identified by a unique integer rank within a communicator, and every bit of coordination between them has to happen through explicit messages.
`MPI_Send`/`MPI_Recv` block the calling process until the message transfer is safe to consider complete — simple, but a matching pair with the wrong order can deadlock the whole program; `MPI_Isend`/`MPI_Irecv` return immediately and let the process do other work while the message is still in flight, at the cost of needing an explicit `MPI_Wait` before touching the buffer again.
Most real MPI programs barely use raw send/recv pairs directly — `MPI_Bcast` sends one process's data to every process, `MPI_Reduce` combines every process's data back into one with an operator like sum or max, and `MPI_Scatter`/`MPI_Gather` split and reassemble an array across all ranks, each one a communication pattern general enough to be implemented far more efficiently than a hand-written loop of point-to-point calls.
One machine with several cores and a real memory bottleneck usually calls for OpenMP; a cluster of many machines with no shared memory calls for MPI; a problem with thousands of independent, simple, identical operations calls for the GPU's SIMT model already covered in Computer Architecture — a decision guide that ties memory architecture, decomposition, Amdahl/Gustafson, and the two programming models covered here into one real choice, made honestly (many real HPC codes use more than one of these together).