Ai Engineering Core / Transformers, Tokens & Context Windows

Understanding Parameters and Scaling Laws

📌 What Are Parameters?

Think of parameters as tiny learned settings inside an AI model — internal values the model picks up during training that help it recognise patterns, store knowledge, and generate outputs.

No single parameter does anything impressive on its own. But combine billions of them together, and you get something that behaves remarkably like intelligence.

🧩 One parameter = meaningless. A billion parameters working together = a language model that can write, reason, and code.

📈 How Parameter Counts Exploded

Early ML models were tiny — a credit score predictor might have had 20 to 200 parameters. Deep learning changed that dramatically, and GPT models show just how fast things scaled:

Model

Parameters

🌱 GPT-1

117 Million

📈 GPT-2

1.5 Billion

🌊 GPT-3

175 Billion

🧠 GPT-4

~1.76 Trillion

🔥 Modern Frontier Models

Possibly tens of trillions

Each jump wasn't just bigger — it unlocked entirely new capabilities.

💡 Bigger Isn't Always Better

For a long time, the dominant strategy was simple: more parameters = smarter model. That's still broadly true, but modern AI has complicated the picture.

Google's Gemma model has just 270 million parameters — smaller than GPT-2 — yet outperforms it significantly. Better architecture, smarter training, and improved optimisation techniques mean models can now do more with less.

This is also why modern LLMs come in multiple sizes:

Model Family

Sizes Available

🤖 GPT

Nano → Mini → Full

🦙 Llama

3B → 8B → 70B+

💜 Claude

Haiku → Sonnet → Opus

Larger = more powerful, but also more expensive to run.

Smaller = faster and cheaper, but with some trade-offs in capability.

🏋️ Two Ways to Make AI Better

Modern AI improves through two distinct but complementary strategies:

⚡ Strategy 1: Training Time Scaling

Make the model bigger and train it on more data.

The idea is straightforward — a larger model trained on more data learns more. This was the dominant approach for years and is summarised by the Chinchilla Scaling Laws: as you increase parameters, you need to proportionally increase training data, otherwise that extra capacity goes to waste.

🚀 Strategy 2: Inference Time Scaling

Make the model think smarter when it's actually being used.

Rather than just building a bigger model, this approach improves performance at the moment of use. Two powerful techniques:

🧠 Chain-of-Thought Reasoning — Instead of jumping straight to an answer, the model reasons through the problem step by step first. Simply adding "think step-by-step" to a prompt can meaningfully improve accuracy.

🧩 Context Engineering — Feed the model exactly what it needs: relevant data, documents, policies, or tool outputs. A travel assistant given live prices and schedules will always outperform one working from memory alone. This is also the foundation of RAG (Retrieval-Augmented Generation) — where the model retrieves fresh information and injects it into context before responding.

💡 Training scaling builds a bigger brain. Inference scaling teaches it to think better.

🧠 Mixture of Experts (MoE)

As models grew to hundreds of billions of parameters, running the entire model for every single query became impractical. The solution? Don't.

Mixture of Experts (MoE) splits a large model into specialised sub-models — "experts" — and only activates the ones relevant to the task at hand:

Coding question   →   coding experts activate
Maths question    →   maths experts activate
Writing task      →   language experts activate

This makes massive models far more efficient — you get the capability of a trillion-parameter model without paying the full compute cost every time.

Models like DeepSeek (~671B parameters) use this approach, achieving enormous scale while remaining practically usable.

🔥 The Full Evolution

Traditional ML
   ↓
Neural Networks
   ↓
Deep Learning
   ↓
Large Transformers
   ↓
Training Scaling
   ↓
Inference Scaling
   ↓
Reasoning + Agents + Context Systems

AI is no longer just about building bigger models — it's about building smarter systems.