Node.js Streams Tutorial: From Basics to Backpressure

Streams let Node process datasets larger than RAM. Used correctly, they keep memory flat regardless of input size — a 10GB log compresses in a few dozen megabytes of resident memory. Used incorrectly, they leak file descriptors and swallow errors. The difference is a handful of rules.

The Four Stream Types

- Readable — a source: fs.createReadStream, an HTTP response on the client, an HTTP request body on the server - Writable — a sink: fs.createWriteStream, an HTTP response on the server, process.stdout - Duplex — both, independently: a TCP socket - Transform — a Duplex where output derives from input: zlib.createGzip(), crypto ciphers, your own parsers

Everything is chunked Buffers (or strings) by default. Setting objectMode: true makes chunks arbitrary JavaScript values — the basis of record-by-record ETL pipelines.

Backpressure: The Whole Point

Every stream has an internal buffer bounded by highWaterMark — 16KB by default for byte streams (64KB for fs.createReadStream), 16 objects in object mode. When a writable's buffer is full, write() returns false. That return value is a request: stop pushing until the drain event. A producer that ignores it does not crash — it silently buffers everything in memory, which is how "the streaming rewrite" ends up using more RAM than readFileSync.

Manual handling looks like:

- function pump(src, dst) { - src.on('data', chunk => { if (!dst.write(chunk)) src.pause(); }); - dst.on('drain', () => src.resume()); - }

You should almost never write this yourself — but you must recognize when code you depend on fails to.

pipeline Over pipe

The classic .pipe() has two production-grade flaws: an error in any stage does not destroy the other stages (leaking descriptors and hanging requests), and errors do not propagate to one place. pipeline from node:stream/promises fixes both:

- import { pipeline } from 'node:stream/promises'; - await pipeline(createReadStream('huge.log'), createGzip(), createWriteStream('huge.log.gz'));

One awaitable expression, full backpressure, and guaranteed teardown of every stage on any failure — including premature close, the notoriously missed case where the destination dies first. pipeline also accepts async generators as middle stages, which is the nicest way to write a custom transform:

- await pipeline(source, async function* (src) { for await (const chunk of src) yield transform(chunk); }, dest);

Async Iteration

Readables are async iterables, so modern consumption is a loop, not an event soup:

- for await (const chunk of createReadStream('input.txt')) { process(chunk); }

This respects backpressure automatically (the source pauses while your awaits run) and converts stream errors into normal thrown exceptions your try/catch sees. Caveats: chunks are arbitrary byte slices, not lines — compose with readline.createInterface or a splitting transform for line-oriented work — and breaking out of the loop destroys the stream, which is usually what you want but surprises code that expected to resume.

Implementing Your Own

For a custom Transform, implement _transform(chunk, encoding, callback) and always call the callback exactly once — forgetting it stalls the pipeline forever, the classic homemade-stream bug. Respect the contract in custom Readables too: stop pushing when push() returns false and wait for _read to be called again.

Common Mistakes

- Attaching a data handler *and* using pipe/pipeline — the data listener switches the stream to flowing mode and defeats backpressure - No error handler on a raw stream — an unhandled error event crashes the process; pipeline centralizes this, raw .pipe() chains need a handler per stage - Concatenating all chunks into one buffer "temporarily" — you have rebuilt readFile with extra steps; if you truly need it all, readFile is simpler and faster - Ignoring the write() return value in hot loops - Forgetting that objectMode counts objects, not bytes, against highWaterMark — 16 huge objects can be gigabytes

Web Streams Interop

Node also ships the WHATWG Web Streams (ReadableStream, WritableStream, TransformStream) used by fetch, and the two worlds interconvert: Readable.toWeb(nodeStream) and Readable.fromWeb(webStream). Write library code against Web Streams when you target browsers, Deno, and edge runtimes; convert at the Node boundary. Semantics differ in details — Web Streams use pull-based desiredSize accounting rather than drain events — but pipeline and pipeTo play the same backpressure role.

Tuning

highWaterMark is the knob: larger buffers mean fewer, bigger I/O operations and more memory per stream. The defaults suit typical file work; for high-throughput proxies, measure with 256KB or 1MB. And avoid writing thousands of tiny chunks — batch small records before write(), because per-chunk overhead (callbacks, buffering bookkeeping) dominates below a few kilobytes.

One more operational note: a stream pipeline holds kernel resources — file descriptors, socket buffers — for its whole lifetime, so concurrency limits matter as much as memory. Processing 10,000 files means bounding how many pipelines run at once, exactly like bounding parallel fetches.

Streams reward the discipline: pipeline everywhere, async iteration for consumption, backpressure respected, errors handled once. Follow those four and constant-memory processing stops being a claim and becomes a measurable property of your service.