parallel_hash.py in Python

Hash every file under a directory using a process pool.

"""Hash every file under a directory using a pool of processes."""
import hashlib
import sys
from concurrent.futures import ProcessPoolExecutor
from pathlib import Path


def sha256_of(path):
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        while chunk := f.read(65536):
            digest.update(chunk)
    return str(path), digest.hexdigest()


def main():
    root = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
    files = [p for p in root.rglob("*") if p.is_file()]
    if not files:
        print("no files found")
        return
    with ProcessPoolExecutor() as pool:
        for name, digest in pool.map(sha256_of, files):
            print(f"{digest[:16]}  {name}")
    print(f"hashed {len(files)} files")


if __name__ == "__main__":
    main()

How it works

1. rglob collects the files to hash.
2. ProcessPoolExecutor spreads hashing over cores.
3. Each file is read in chunks and digested.

Keywords and builtins used here

• as
• def
• else
• for
• if
• len
• main
• open
• print
• return
• sha256_of
• str
• while
• with

The run, in numbers

Lines

29

Characters to type

687

Tokens

190

Three-star pace

110 tpm

At the three-star pace of 110 tokens a minute, this run takes about 104 seconds.

Type this snippet

Step 2 of 2 in Encore, step 53 of 53 in Pythonic Python.

← Previous