Node.js: Code Outside the Browser / Streams and Buffers

Types of streams: Readable, Writable, Duplex, Transform

📚 The 4 stream types

Type

What it does

Example in Node

📖 Readable

You read from it

Incoming HTTP request body, fs.createReadStream()

✍️ Writable

You write to it

HTTP response you send back, fs.createWriteStream()

🔄 Duplex

Both readable and writable

A web socket (net module)

🔀 Transform

Duplex, but modifies data as it passes through

zlib (compression)

📖 Readable and ✍️ writable are the two you'll use constantly. Duplex and transform are more advanced/less common.

💡 Key fact: streams are EventEmitters

Every stream inherits from EventEmitter — the same class from the events lecture. So you already know the pattern: .on('eventName', callback).

📖 Readable Streams

⭐ Key events: data (a chunk is ready), end (no more data)
⭐ Key methods: pipe() (connect to another stream), read()

const fs = require('fs');

const readable = fs.createReadStream('bigfile.txt', {
  encoding: 'utf8',
  highWaterMark: 64 * 1024 // chunk size: 64kb
});

readable.on('data', (chunk) => {
  console.log(`📦 Received ${chunk.length} characters`);
});

readable.on('end', () => {
  console.log('🏁 No more data left to read');
});

readable.on('error', (err) => {
  console.error('❌ Read error:', err.message);
});

🖥️ Example output (for a ~150kb file):

📦 Received 65536 characters
📦 Received 65536 characters
📦 Received 19339 characters
🏁 No more data left to read

(Number of data events depends on file size ÷ chunk size — a 150kb file with a 64kb chunk size produces 3 chunks, the last one smaller.)

✍️ Writable Streams

⭐ Key events: drain, finish (all data written)
⭐ Key methods: write(), end()

const http = require('http');
const fs = require('fs');

http.createServer((req, res) => {
  const readable = fs.createReadStream('video.mp4');

  readable.on('error', (err) => {
    res.statusCode = 500;
    res.end('Server error');
  });

  readable.pipe(res);
}).listen(3000);

🖥️ Example output (server-side console is silent unless you log; here's what the client sees when hitting the server):

$ curl <http://localhost:3000> --output video.mp4
  % Total    % Received % Xferd  Average Speed   Time
100 24.3M  100 24.3M    0     0  18.2M      0  0:00:01

The file downloads in chunks rather than the server loading the whole 24MB video into memory first.

🔄 Duplex Streams

Readable and writable at once — data flows both directions.

const net = require('net');

// server side
const server = net.createServer((socket) => {
  socket.on('data', (data) => {
    console.log('Server received:', data.toString());
  });
  socket.write('Hello client!');
}).listen(3000);

// client side
const socket = net.connect(3000, () => {
  socket.write('Hello server!');
});

socket.on('data', (data) => {
  console.log('Client received:', data.toString());
});

🖥️ Example output:

Server received: Hello server!
Client received: Hello client!

(Both logs come from the same process here since client + server run in one file — order between the two lines can vary slightly since they're async.)

🔀 Transform Streams

A duplex stream that also changes the data as it passes through.

const fs = require('fs');
const zlib = require('zlib');

const readable = fs.createReadStream('bigfile.txt');
const writable = fs.createWriteStream('bigfile.txt.gz');
const transform = zlib.createGzip();

readable
  .pipe(transform)
  .pipe(writable)
  .on('finish', () => console.log('✅ Compression complete'))
  .on('error', (err) => console.error('❌ Error:', err.message));

🖥️ Example output:

✅ Compression complete

And in your file system, a new file appears: bigfile.txt.gz — noticeably smaller than bigfile.txt.

⚠️ Worth knowing

Everything above is about consuming streams Node already built for you (HTTP, fs, zlib, etc.). You can build custom streams too — that's a more advanced topic for later. For most apps, consuming existing streams is what matters day to day.

✅ Takeaway

  • 🌊 Streams process data in chunks → saves memory + time

  • 📚 4 types: 📖 readable, ✍️ writable, 🔄 duplex, 🔀 transform — readable/writable are the ones you'll use most

  • 👂 All streams are EventEmitters, so .on('data', ...) / .on('end', ...) work exactly like before

  • 🔗 pipe() is the go-to tool for wiring streams together with minimal code