Web Streams API: Processing Data in Chunks Like a Pro
Master the Streams API for efficient data processing — read, transform, and write data incrementally without loading everything into memory.
Why Streams?
Traditional data processing loads everything into memory before acting on it: fetch the entire response, parse the whole JSON, process the complete file. Streams let you process data chunk by chunk as it arrives, using constant memory regardless of data size. This is essential for large files, real-time data, and memory-constrained environments.
ReadableStream Basics
``javascript
// Create a readable stream
const stream = new ReadableStream({
start(controller) {
controller.enqueue('Hello, ');
controller.enqueue('Streams ');
controller.enqueue('World!');
controller.close();
}
});
// Read from it
const reader = stream.getReader();
while (true) {
const { done, value } = await reader.read();
if (done) break;
console.log(value); // "Hello, ", "Streams ", "World!"
}
`
Streaming Fetch Responses
The most practical use — process large responses without loading them entirely.
`javascript
async function streamJSON(url) {
const response = await fetch(url);
const reader = response.body
.pipeThrough(new TextDecoderStream())
.getReader();
let buffer = ''; const results = [];
while (true) { const { done, value } = await reader.read(); if (done) break;
buffer += value; // Process complete JSON objects from the buffer const lines = buffer.split('\n'); buffer = lines.pop(); // Keep incomplete line
for (const line of lines) { if (line.trim()) { results.push(JSON.parse(line)); } } }
return results;
}
`
TransformStreams: The Power of Piping
TransformStream lets you create reusable data processors that plug into stream pipelines.
`javascript
// A transform that converts text to uppercase
function createUpperCaseStream() {
return new TransformStream({
transform(chunk, controller) {
controller.enqueue(chunk.toUpperCase());
}
});
}
// A transform that filters lines function createLineFilter(predicate) { let buffer = ''; return new TransformStream({ transform(chunk, controller) { buffer += chunk; const lines = buffer.split('\n'); buffer = lines.pop(); for (const line of lines) { if (predicate(line)) { controller.enqueue(line + '\n'); } } }, flush(controller) { if (buffer && predicate(buffer)) { controller.enqueue(buffer); } } }); }
// Chain transforms with pipeThrough
const response = await fetch('/api/logs');
const errorLogs = response.body
.pipeThrough(new TextDecoderStream())
.pipeThrough(createLineFilter(line => line.includes('ERROR')))
.pipeThrough(createUpperCaseStream());
`
Progress Tracking with Streams
`javascript
function trackProgress(totalBytes, onProgress) {
let loaded = 0;
return new TransformStream({
transform(chunk, controller) {
loaded += chunk.length;
onProgress(loaded / totalBytes);
controller.enqueue(chunk);
}
});
}
const response = await fetch(url); const total = +response.headers.get('content-length');
const stream = response.body.pipeThrough(
trackProgress(total, (pct) => {
console.log(\\${(pct * 100).toFixed(1)}% downloaded\);
})
);
``
Streams are the foundation of efficient data processing on the web. They power Service Workers, file uploads, real-time communication, and any scenario where you need to handle data incrementally.