Process pools in Python

Spreading CPU-bound work across real cores.

from concurrent.futures import ProcessPoolExecutor


def crunch(n):
    return sum(i * i for i in range(n))


def main():
    workloads = [2_000_000, 3_000_000, 4_000_000]
    with ProcessPoolExecutor() as pool:
        results = list(pool.map(crunch, workloads))
    return results  # CPU-bound work spread over real cores


if __name__ == "__main__":
    totals = main()

How it works

1. ProcessPoolExecutor sidesteps the GIL with processes.
2. pool.map distributes the heavy crunch calls.
3. The __main__ guard is required for spawning.

Keywords and builtins used here

• as
• crunch
• def
• for
• if
• list
• main
• range
• return
• sum
• with

The run, in numbers

Lines

16

Characters to type

344

Tokens

75

Three-star pace

110 tpm

At the three-star pace of 110 tokens a minute, this run takes about 41 seconds.

Type this snippet

Step 3 of 4 in Threads & processes, step 30 of 53 in Pythonic Python.

← Previous Next →