Why Transformers Replaced RNNs

Before ChatGPT-style models, the default for translation and summarization was an RNN encoder–decoder: read the source left to right, squeeze it into a state, then emit the target one token at a time. Those systems shipped products. They also hit a wall on length, parallelism, and credit assignment — a wall the Transformer climbed over by replacing recurrence with attention. If you understand why that shift happened, every later lesson on Q/K/V, masking, and decoding will feel like engineering, not magic.

Intuition

Imagine summarizing a 40-page brief into one sticky note, then asking someone else to reconstruct every clause from that note. That is the classical context-vector bottleneck: the encoder’s final hidden state had to carry the whole input. Early words — subject gender, negation, rare entities — got overwritten as the RNN marched forward.

Attention (Bahdanau-style) was the first relief valve: keep all encoder states and let the decoder look back with soft weights. The Transformer went further. Instead of walking the sequence step by step, every token can directly compare itself to every other token in one parallel mesh. Long-range links no longer require surviving dozens of recurrent steps; they are a single hop in the attention graph.

The interview-ready punchline: RNNs are great at state over time; Transformers are great at relations over a set, with position injected separately. Modern LLMs are mostly decoder-only Transformers precisely because that mesh scales on GPUs and models rich context for next-token prediction.

How it works

Classical seq2seq. An encoder RNN updates h_t = f(h_{t-1}, x_t) until the last state h_T (or a pool of states) is handed to the decoder. Training must unroll time: step t depends on t-1, so GPUs sit partly idle. Gradients flow backward through that long chain and often vanish or explode.

Attention as a spotlight. Alignment weights let the decoder build a fresh context each step:

score_ti   = score(decoder_state_t, encoder_h_i)
alpha_ti   = softmax_i(score_ti)
c_t        = sum_i alpha_ti * encoder_h_i

When translating “cat” → “chat”, alphas might peak on the source word “cat” instead of relying on a single compressed vector.

Transformer leap. Self-attention drops the recurrent chain. For a sequence of embeddings X, each position computes queries, keys, and values and mixes information in matrix form:

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

Dependency path length between any two tokens becomes O(1) through attention (plus depth through stacked layers), versus O(n) along an RNN. Training can process all positions of a sequence in parallel (with causal masks in decoders — covered later). Multi-head attention runs several of these mixers side by side so different heads specialize (syntax vs coreference vs locality).

flowchart TB subgraph RNN["RNN seq2seq"] R1[Token t depends on t-1] R2[Final state bottleneck] R3[Limited GPU parallelism] end subgraph ATT["Attention era"] A1[Keep all encoder states] A2[Soft alignment per decode step] end subgraph TR["Transformer"] T1[Self-attention mesh] T2[Parallel token batching] T3[O(1) pairwise links] end RNN --> ATT --> TR

Why GenAI standardized on this. Pre-training needs huge corpora and huge batches. Parallelism over sequence length (and batch) made scaling laws practical. The same block — attention + feed-forward + residual + norm — stacks into GPT-style decoders and encoder–decoder models (T5, classic Transformer NMT). When someone says “the model attended to the system prompt,” they mean this mechanism, not a metaphor.

In code

A tiny demo of the bottleneck: an RNN-like running average that overwrites early signal, versus a soft attention mix that can still emphasize the first token.

import numpy as np

rng = np.random.default_rng(0)
# Three token embeddings in 4-D; token 0 carries a unique spike
X = rng.normal(size=(3, 4))
X[0] = np.array([5.0, 0.0, 0.0, 0.0])

# Fake "RNN final state": exponential moving average (late tokens dominate)
h = np.zeros(4)
for t in range(3):
    h = 0.2 * h + 0.8 * X[t]
print("RNN-like final state ~", np.round(h, 3))
print("spike retained?", h[0] > 1.0)

# Soft attention from a query that looks for the spike
q = np.array([1.0, 0.0, 0.0, 0.0])
scores = X @ q
weights = np.exp(scores - scores.max())
weights = weights / weights.sum()
ctx = weights @ X
print("attention weights", np.round(weights, 3))
print("attention context ~", np.round(ctx, 3))

You should see the moving average dilute the spike while attention puts most mass on token 0 and recovers a large first component. That is the architectural moral in four lines of NumPy.

What goes wrong

One-line summary

Transformers replaced RNN seq2seq by swapping sequential state and fixed bottlenecks for parallel self-attention, giving constant-hop token interactions that scale on modern hardware.

Key terms