Thread pools in Python

Running IO-bound work across a pool of threads.

from concurrent.futures import ThreadPoolExecutor
import urllib.request

urls = [
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/c",
]


def fetch_length(url):
    with urllib.request.urlopen(url, timeout=5) as res:
        return url, len(res.read())


with ThreadPoolExecutor(max_workers=3) as pool:
    sizes = dict(pool.map(fetch_length, urls))

How it works

1. ThreadPoolExecutor manages a fixed set of threads.
2. pool.map fans the function over every URL.
3. Threads shine for waiting on the network.

Keywords and builtins used here

• as
• def
• dict
• fetch_length
• len
• return
• with

The run, in numbers

Lines

17

Characters to type

356

Tokens

82

Three-star pace

105 tpm

At the three-star pace of 105 tokens a minute, this run takes about 47 seconds.

Type this snippet

Step 1 of 4 in Threads & processes, step 28 of 53 in Pythonic Python.

← Previous Next →