async_crawler.py in Python

Fetch many URLs concurrently and report sizes and timing.

"""Fetch several URLs concurrently and report sizes and timing."""
import asyncio
import time
import urllib.request

URLS = [
    "https://example.com",
    "https://example.org",
    "https://example.net",
]


def fetch(url):
    with urllib.request.urlopen(url, timeout=10) as res:
        return len(res.read())


async def fetch_all(urls):
    # blocking urllib calls hop onto threads; asyncio herds them
    tasks = [asyncio.to_thread(fetch, url) for url in urls]
    sizes = await asyncio.gather(*tasks, return_exceptions=True)
    return dict(zip(urls, sizes))


def main():
    start = time.perf_counter()
    results = asyncio.run(fetch_all(URLS))
    elapsed = time.perf_counter() - start
    for url, size in results.items():
        if isinstance(size, Exception):
            print(f"{url}: failed ({size})")
        else:
            print(f"{url}: {size:,} bytes")
    print(f"fetched {len(results)} urls in {elapsed:.2f}s")


if __name__ == "__main__":
    main()

How it works

1. asyncio.to_thread runs blocking urllib on threads.
2. gather awaits them all with error capture.
3. Timing shows the concurrency win.

Keywords and builtins used here

• as
• async
• await
• def
• dict
• else
• fetch
• fetch_all
• for
• if
• isinstance
• len
• main
• print
• return
• with
• zip

The run, in numbers

Lines

38

Characters to type

875

Tokens

217

Three-star pace

110 tpm

At the three-star pace of 110 tokens a minute, this run takes about 118 seconds.

Type this snippet

Step 1 of 2 in Encore, step 52 of 53 in Pythonic Python.

← Previous Next →