Ai Engineering Core / Transformers, Tokens & Context Windows

From LSTMs to Transformer: Evolution of Modern AI

📌 Before Transformers — The LSTM Era

Before transformers took over, the go-to architecture was the LSTM (Long Short-Term Memory) — a type of Recurrent Neural Network (RNN).

These models read text the way you'd read a sentence out loud — one word at a time, in order:

💡 How Transformers Fixed This

Transformers threw out the step-by-step approach entirely. Instead of processing words one after another, they look at everything at once.

This is exactly what the famous 2017 paper "Attention Is All You Need" argued — you don't need complex recurrent mechanisms. Simple attention + large-scale parallelization is enough to get better results.

💡 Even though LSTMs were arguably more sophisticated in design, transformers won simply because they were faster, cheaper, and easier to scale.

🌍 When AI Hit the Mainstream

When ChatGPT exploded in popularity, the public reaction was a mix of shock, amazement, and confusion. Terms like transformer, GPT, and LLM suddenly entered everyday conversation.

But not everyone was impressed. Some researchers pushed back with a sharp criticism:

🦜 The "Stochastic Parrots" Argument

The concern was that LLMs aren't actually thinking — they're just very good at predicting which word comes next. Critics worried this would lead to:

  • Misinformation spreading at scale

  • AI hallucinating confidently wrong answers

  • People blindly trusting AI outputs

✨ The Surprise Nobody Expected — Emergent Intelligence

Everyone assumed bigger AI models would just get better at sounding convincing. What actually happened was far more surprising — they started getting things genuinely right.

Not just text that looks like an answer, but actual correct solutions to maths problems, working code, logical plans, and accurate summaries.

Ask it 25 × 12 and instead of text that sounds like an answer, you get 300. The right answer.

This phenomenon is called Emergent Intelligence — as models grow larger and train on more data, they begin displaying capabilities that were never explicitly programmed into them.

🔍 Researchers understand the math and the training process — but why scale alone produces intelligence-like behaviour remains one of AI's biggest unsolved mysteries.

🧩 Prompt Engineering → Context Engineering

💼 The Prompt Engineering Phase

For a while, knowing how to talk to AI effectively was genuinely a high-paying skill. Good prompts included:

  • ✅ Background context

  • ✅ Clear instructions

  • ✅ Desired style and format

  • ✅ Examples and constraints

🔄 How It Evolved

Over time, everyone naturally picked up prompting — developers, writers, students, businesses. It stopped being a specialist skill and became just... how people use AI.

This matured into something broader called Context Engineering (RAG) — the practice of giving an LLM the best possible information and tools to solve a task.

Think of a travel chatbot. If you feed it live ticket prices, schedules, and policies — it gives you accurate, useful answers. Better inputs → Better outputs. That's context engineering in a nutshell.

🤝 The Rise of Copilots

The next big leap was AI Copilots — tools like Microsoft Copilot and GitHub Copilot designed for human-AI collaboration rather than just conversation.

Copilots could:

  • Generate and review code

  • Draft and summarise documents

  • Assist with tasks and workflows

  • Accelerate everyday work

🤖 Agentic AI — The Current Frontier

Today's hottest topic in AI is Agentic AI — and for good reason.

What Makes AI "Agentic"?

A regular LLM responds to your input. An agentic AI takes initiative. It can:

  • Break a goal into a plan

  • Choose which tools to use

  • Execute tasks autonomously

  • Evaluate results and adjust

  • Keep going until the job is done

This is often described as LLM + Loop + Tools — the model thinks, acts, checks the result, and repeats:

⚡ The Key Word: Autonomy

The AI doesn't just answer — it decides what happens next. It picks tools, manages workflows, and self-corrects along the way. Real-world examples include coding agents, research assistants, and multi-step workflow orchestrators.

Even so, at its core, every agentic system still works the same fundamental way — input goes in, output comes out. The difference is that those outputs now include actions, plans, and decisions, not just text.

🔥 The Full Evolution at a Glance