From Bag of Words to Transformers: The Evolution of Natural Language Processing

Computers process numbers. Humans communicate in discrete, context-dependent symbols. That mismatch is the entire problem NLP has been trying to solve for the last two decades — how do you convert raw text into a mathematical space that actually captures meaning, order, and context?

The path from early statistical text matching to modern LLMs wasn’t a straight line of “bigger models, better results.” It was a chain of specific engineering fixes — each new architecture built to patch the exact failure mode of the one before it. Tracing that chain is, I think, the fastest way to actually understand why Transformers look the way they do.

1. Bag of Words: the naive baseline

The earliest statistical approach to NLP treated a document as a literal “bag” of words — throw away grammar, throw away order, just count.

Build a fixed vocabulary from your corpus, then represent each document as a vector where each dimension is the frequency of one word.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
 
corpus = [
    "AI models learn from data",
    "Data engineers build pipelines"
]
 
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
print("Vocabulary:", vectorizer.get_feature_names_out())
print("BoW Matrix:\n", X.toarray())

This was a real step forward — before BoW, NLP leaned on hand-crafted regex and rule systems. A simple numeric vector space meant Naive Bayes, Logistic Regression, and SVMs could suddenly do statistical text classification and spam filtering.

But it breaks down fast:

• Sparsity. A 100,000-word vocabulary means every document is a 100,000-dimensional vector that’s almost entirely zeros.
• No word order. “The model was accurate, not slow” and “The model was slow, not accurate” produce the exact same vector.
• Frequency bias. Filler words like “the” and “is” dominate the counts purely by showing up often, not by carrying meaning.

  2. TF-IDF: weighting words by how much they actually tell you

TF-IDF (Term Frequency–Inverse Document Frequency) fixes the frequency-bias problem directly, by scaling a word’s count against how rare it is across the whole corpus:

A word that shows up in nearly every document — “the,” “is,” “at” — gets an IDF score near zero, which neutralizes it. A domain-specific term like “transformer” or “hyperparameter,” used often in one document but rarely elsewhere, gets its weight boosted instead.

from sklearn.feature_extraction.text import TfidfVectorizer
 
tfidf = TfidfVectorizer()
matrix = tfidf.fit_transform(corpus)

This became the backbone of early search engines and text-classification baselines, because it surfaces distinctive keywords automatically — no manual stopword lists required.

What it still can’t do: understand meaning. In TF-IDF space, “cat” and “feline” are completely orthogonal vectors (cosine similarity of zero) — the model has no idea they mean the same thing. Syntax, ordering, and negation are all still invisible to it.

3. Word2Vec: giving words a geometry

In 2013, Mikolov et al. at Google introduced Word2Vec, and this is really where NLP starts to feel like modern deep learning — words get projected into a dense, continuous vector space (typically $\mathbb{R}^{300}$).

The idea leans on J.R. Firth’s old linguistic principle: “You shall know a word by the company it keeps.” Word2Vec trains a shallow two-layer network on one of two tasks:

• CBOW — predict a target word from its surrounding context.
• Skip-gram — predict the surrounding context from a target word.

  Skip-gram Objective:
  Input: "learning"  ──►  Predict context: ["machine", "deep", "models", "data"]
  

The hidden-layer weights learned during training are the embeddings.

Two things made this a genuine breakthrough:

1. Compact, dense vectors — 300 dimensions instead of 100,000+ sparse ones.
2. Semantic geometry. Similar words cluster together, and vector arithmetic starts working in ways that feel almost magical the first time you see it: \(\text{Vector("King")} - \text{Vector("Man")} + \text{Vector("Woman")} \approx \text{Vector("Queen")}\) The catch: every word gets exactly one vector, no matter the context. “Bank” has an identical embedding in “river bank” and “investment bank.” Polysemy just isn’t modeled at all.

4. RNNs and LSTMs: adding memory over a sequence

To let meaning shift based on surrounding words, the field moved to Recurrent Neural Networks and LSTMs. Instead of treating each word in isolation, an RNN walks through a sentence token by token, carrying a hidden state forward like a running memory:

LSTMs add dedicated memory cells with gating mechanisms (forget, input, output) that control how much of the past actually sticks around.

Step 1: "The"      ──► [LSTM Cell] ──► h_1
                           │
Step 2: "model"    ──► [LSTM Cell] ──► h_2
                           │
Step 3: "computed" ──► [LSTM Cell] ──► h_3

This gave models a genuinely dynamic representation — a word’s meaning now depends on everything that came before it in the sentence, and variable-length structure is finally captured.

Two bottlenecks showed up almost immediately, though:

• No parallelization. Step t strictly depends on step t−1, so a 1,000-word document means 1,000 sequential operations. GPUs are built for parallel work; RNNs can’t use that.
• Compression bottleneck. In encoder-decoder setups (translation, summarization), the entire input sequence had to be squeezed into a single final vector before the decoder ever saw it.

  5. Attention: stop compressing everything into one vector

Bahdanau and Luong’s Attention Mechanism (2014–2015) targeted the compression problem directly. Instead of the decoder relying only on the encoder’s final hidden state, attention gives it a direct line to all intermediate encoder states — at each decoding step, the model computes a score for how relevant each source word is right now.

Decoder Step: "Bonjour"
Attention Weights ──► High focus on Encoder State for "Hello"
                  ──► Low focus on Encoder State for "World"

This was a big jump for machine translation quality — decoders could finally reach back across long sentences and grab exactly the tokens that mattered.

It didn’t fix everything, though. Attention was bolted onto RNNs, and the underlying recurrent loop was still there — training was still sequential, still slow.

6. Transformers: drop the recurrence, keep the attention

In 2017, Vaswani et al. published “Attention Is All You Need,” and the core idea was almost brazen in its simplicity: what if you got rid of recurrence entirely and relied only on self-attention?

A Transformer computes relationships between every pair of tokens in a sequence at once, using scaled dot-product self-attention:

Each token gets projected into three vectors:

• Query (Q) — what this token is looking for
• Key (K) — what this token offers
• Value (V) — the information actually passed along on a match ```python import torch import torch.nn.functional as F

def self_attention(Q, K, V): d_k = Q.size(-1) # Compute attention matrix (all pairs at once) scores = torch.matmul(Q, K.transpose(-2, -1)) / torch.sqrt(torch.tensor(d_k, dtype=torch.float32)) attn_weights = F.softmax(scores, dim=-1) # Weighted sum of values output = torch.matmul(attn_weights, V) return output, attn_weights ```

Since there’s no recurrence, order has to be injected another way — Transformers add positional encodings directly to the input embeddings.

Why this mattered so much:

1. Full GPU parallelization — no more waiting on sequential time steps; it’s matrix multiplication across every token at once.
2. O(1) long-range connections — any token can attend directly to any other token in a single step, no gradient decay over distance.
3. It scales. Self-attention is what let architectures grow to billions of parameters — BERT, GPT, Claude, Llama, all of it.

   Architectural evolution at a glance

Where this leaves us

None of these shifts added complexity for its own sake — each one existed to remove a specific, named bottleneck in what came before it. That’s a useful lens for understanding today’s LLM engineering too: KV-caching, RoPE positional embeddings, and sparse-attention tricks like FlashAttention are all, in effect, the next round of patches — this time aimed at the Transformer’s own O(N²) memory cost.