Node.js: Code Outside the Browser / Streams and Buffers

Working with Buffers

✂️ Slicing & Combining Buffers

const buf = Buffer.from('hello world');

const slice = buf.subarray(0, 5);
console.log(slice.toString()); 
// 'hello'

const a = Buffer.from('foo');
const b = Buffer.from('bar');
const combined = Buffer.concat([a, b]);
console.log(combined.toString()); 
// 'foobar'

  • .subarray(start, end) pulls out a portion of a buffer — the modern replacement for the older .slice() (still works, but subarray is the current recommended name).

  • Buffer.concat([...]) joins multiple buffers into one — useful when data arrives in chunks and needs to be reassembled.

✍️ Writing Into a Buffer

const buf = Buffer.alloc(5);
buf.write('hi');
console.log(buf);           
// <Buffer 68 69 00 00 00>
console.log(buf.toString()); 
// 'hi'

  • .write(string) writes text into an existing buffer, starting at index 0 by default.

  • Any bytes not written to stay at whatever they were set to — here, zero, since the buffer was created with alloc.

🔍 Comparing & Searching

const a = Buffer.from('abc');
const b = Buffer.from('abc');

console.log(a.equals(b)); 
// true — same content

const buf = Buffer.from('hello world');
console.log(buf.indexOf('world')); // 6
console.log(buf.includes('world')); // true

  • .equals() checks whether two buffers contain the exact same bytes — regular === won't work here, since it compares object references, not content.

  • .indexOf() and .includes() work like their string/array equivalents, searching for a value inside the buffer.

💡 Why Buffers Matter

  • 📁 Files — fs.readFile() returns a buffer unless you specify an encoding

  • 🌐 Network — incoming HTTP request bodies and TCP data arrive as buffers, before any decoding

  • 🖼️ Binary formats — images, audio, PDFs, and other non-text files are handled as raw bytes

  • 🔐 Crypto — hashing and encryption operate on bytes, not text

const fs = require('fs');

const data = fs.readFileSync('note.txt'); 
// no encoding → returns a Buffer
console.log(data);           
// <Buffer 48 65 6c 6c 6f>
console.log(data.toString()); 
// 'Hello'

🌍 Real-Life Examples

1️⃣ Saving a Base64 Image (from an API or form upload)

Frontends often send images as base64 text — a buffer converts that back into real image bytes to save as a file.

const fs = require('fs');

const base64Image = 'iVBORw0KGgoAAAANSUhEUgAAAAUA...'; 
// shortened
const imgBuffer = Buffer.from(base64Image, 'base64');

fs.writeFileSync('photo.png', imgBuffer);
console.log('🖼️ Image saved!');

  • The base64 string is decoded straight into binary image data — no intermediate steps needed.

  • This is exactly what happens behind the scenes when an app lets users upload a profile picture via a JSON API.

2️⃣ Checking a File's Real Type by Its Magic Bytes

File extensions can lie — the first few bytes of a file (its "magic number") reveal what it actually is. This is how tools verify uploads instead of trusting the filename.

const fs = require('fs');

const buf = fs.readFileSync('mystery-file');
const isPNG = buf.slice(0, 8).equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]));

console.log(isPNG ? '✅ This is a real PNG' : '❌ Not a PNG');

  • Every PNG file starts with the same 8 bytes, regardless of its filename or extension.

  • Comparing raw bytes like this is far more reliable than checking if a filename ends in .png — someone could rename any file to photo.png.

3️⃣ Reassembling a File Upload from Streamed Chunks

Large uploads (or any HTTP body) arrive in pieces. Buffers collect and combine them once the full file has arrived.

const http = require('http');

const server = http.createServer((req, res) => {
  const chunks = [];

  req.on('data', chunk => chunks.push(chunk));

  req.on('end', () => {
    const fileBuffer = Buffer.concat(chunks);
    console.log(`📦 Received ${fileBuffer.length} bytes`);
    res.end('Upload received!');
  });
});

server.listen(3000);

  • Each chunk is already a buffer — Buffer.concat() stitches every piece back into one complete file.

  • This pattern underlies how frameworks like Express handle raw file uploads before any parsing library gets involved.

4️⃣ Measuring the Real Size of Text (Emoji-Safe)

.length on a string counts characters, not bytes — that's wrong for anything with emoji or accented characters. A buffer gives the true byte size, which matters for things like database column limits.

const text = 'Hello 👋';

console.log(text.length);                     
// 8 — JS counts the emoji as 2 code units
console.log(Buffer.byteLength(text, 'utf8'));  
// 10 — actual bytes when stored/sent

  • Buffer.byteLength() tells you exactly how much space the text will take up once encoded — the number that actually matters for storage limits or network payload size.

5️⃣ Hashing a File (Checking Integrity)

Verifying a downloaded file wasn't corrupted or tampered with means hashing its raw bytes — not its text — since files aren't always text.

const fs = require('fs');
const crypto = require('crypto');

const fileBuffer = fs.readFileSync('installer.exe');
const hash = crypto.createHash('sha256').update(fileBuffer).digest('hex');

console.log('🔒 SHA-256:', hash);

  • readFileSync without an encoding returns a buffer — exactly what createHash().update() expects.

  • This is the same technique behind the "checksum" values software sites publish next to downloads, so you can confirm your copy matches theirs.

6️⃣ Reading a File's Header to Identify Its Format

Beyond just checking "is this a PNG," buffers let you read structured binary formats byte by byte — like pulling metadata out of a WAV audio file's header.

const fs = require('fs');

const buf = fs.readFileSync('song.wav');

const chunkId = buf.toString('ascii', 0, 4);     
// 'RIFF'
const format = buf.toString('ascii', 8, 12);     
// 'WAVE'
const sampleRate = buf.readUInt32LE(24);          
// e.g. 44100

console.log({ chunkId, format, sampleRate });

  • .toString(encoding, start, end) reads just a slice of the buffer as text — WAV headers store labels like 'RIFF' and 'WAVE' as plain ASCII at fixed positions.

  • .readUInt32LE(offset) reads 4 bytes at a given position as a number (little-endian) — this is how the sample rate is actually stored in the file, as raw bytes rather than text.

  • This same byte-reading approach is how libraries parse image dimensions, video metadata, or any binary file format with a fixed header layout.

7️⃣ Compressing Data Before Sending It

Compression algorithms work on raw bytes, so buffers are the natural input and output for something like gzip.

const zlib = require('zlib');

const original = Buffer.from('a'.repeat(1000)); 
// 1000 bytes
const compressed = zlib.gzipSync(original);

console.log(`Original: ${original.length} bytes`);
console.log(`Compressed: ${compressed.length} bytes`);

const restored = zlib.gunzipSync(compressed);
console.log(restored.toString().length); 
// 1000 — back to the original

  • gzipSync takes a buffer in and returns a smaller, compressed buffer out — this is exactly what happens when a server gzips a response before sending it over the network.

  • gunzipSync reverses it — a browser or client does this automatically when it receives a Content-Encoding: gzip response.