Streams in JavaScript

Reading a file in chunks, and the pipeline that wires stages together.

import { createReadStream, createWriteStream } from "node:fs";
import { mkdtemp, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { pipeline } from "node:stream/promises";
import { Transform } from "node:stream";

const dir = await mkdtemp(join(tmpdir(), "stream-"));
const source = join(dir, "in.txt");
await writeFile(source, "one\ntwo\nthree\n");

let bytes = 0;
for await (const chunk of createReadStream(source, { highWaterMark: 4 })) {
  bytes += chunk.length;
}
console.log("read", bytes, "bytes in chunks");

const upper = new Transform({
  transform(chunk, _encoding, done) {
    done(null, chunk.toString().toUpperCase());
  },
});
await pipeline(createReadStream(source), upper,
               createWriteStream(join(dir, "out.txt")));
console.log("piped");

How it works

1. A read stream emits chunks; the async iterator consumes them.
2. pipeline propagates errors and closes everything.
3. Streams are why Node can process a file bigger than memory.

Keywords and builtins used here

• await
• const
• for
• from
• import
• let
• of

The run, in numbers

Lines

25

Characters to type

806

Tokens

185

Three-star pace

105 tpm

At the three-star pace of 105 tokens a minute, this run takes about 106 seconds.

Type this snippet

Step 1 of 2 in Streams & lines, step 5 of 18 in Node.js.

← Previous Next →