Limiting concurrency in Python

A semaphore caps how many coroutines are in the expensive part at once.

import asyncio


async def fetch(name, semaphore, live):
    async with semaphore:
        live.append(name)
        peak = len(live)
        await asyncio.sleep(0.01)
        live.remove(name)
        return name, peak


async def main():
    semaphore = asyncio.Semaphore(2)
    live = []
    tasks = [fetch(f"job{i}", semaphore, live) for i in range(6)]

    highest = 0
    for coro in asyncio.as_completed(tasks):
        name, peak = await coro
        highest = max(highest, peak)
    print("never more than", highest, "at once")


asyncio.run(main())

How it works

1. async with semaphore acquires and releases around the work.
2. The limit is what keeps a crawler polite.
3. as_completed yields results in finishing order.

Keywords and builtins used here

• async
• await
• def
• fetch
• for
• len
• main
• max
• print
• range
• return
• with

The run, in numbers

Lines

25

Characters to type

474

Tokens

136

Three-star pace

110 tpm

At the three-star pace of 110 tokens a minute, this run takes about 74 seconds.

Type this snippet

Step 3 of 4 in Async in depth, step 50 of 53 in Pythonic Python.

← Previous Next →

Limiting concurrency in other languages

• Go
• JavaScript