Streaming downloads in Python

Downloading a large file without loading it all in memory.

import requests

with requests.get("https://example.com/big.csv",
                  stream=True, timeout=30) as response:
    response.raise_for_status()
    total = 0
    with open("big.csv", "wb") as out:
        for chunk in response.iter_content(chunk_size=8192):
            out.write(chunk)
            total += len(chunk)

kind = response.headers.get("Content-Type")

How it works

1. stream=True defers reading the body.
2. iter_content yields fixed-size chunks.
3. Each chunk is written straight to disk.

Keywords and builtins used here

• as
• for
• len
• open
• with

The run, in numbers

Lines

12

Characters to type

311

Tokens

81

Three-star pace

105 tpm

At the three-star pace of 105 tokens a minute, this run takes about 46 seconds.

Type this snippet

Step 5 of 5 in requests, step 5 of 9 in Web scraping.

← Previous Next →