What this is: Before ChatGPT-style models, translation and summarization used RNN encoder–decoder systems. They worked in production — but hit hard limits on length, speed, and long-range memory.
Why it matters: The Transformer (2017, Attention Is All You Need) replaced slow recurrent loops with parallel self-attention. If you understand why that shift happened, every later lesson on Q/K/V, masking, and decoding will feel like engineering, not magic.
Imagine summarizing a 40-page brief onto one sticky note, then asking someone to reconstruct every clause from that note alone. That is the context vector bottleneck: the encoder's final hidden state had to carry the whole input. Early words — subject gender, negation, rare names — got overwritten as the RNN marched forward.
Bahdanau cross-attention was the first relief: keep all encoder states and let the decoder look back with soft weights at each step. The Transformer went further — drop the RNN entirely. Every token can directly compare itself to every other token in one parallel pass. Long-range links no longer need to survive dozens of recurrent steps; they are a single hop in the attention graph.
| Problem | Plain-English idea |
|---|---|
| Fixed-length context bottleneck | Squeezing a long paragraph into one vector (e.g. 512 numbers) loses information as sequences grow. |
| Vanishing / exploding gradients | Learning signals must travel backward through every time step. They either shrink to zero or blow up, stalling training. |
| Poor long-range dependencies | Information from early tokens must pass through many intermediate hidden states. By the time it reaches distant positions, it is heavily diluted. |
| No parallelism in training | Step t cannot start until step t-1 finishes. GPUs sit partly idle on a single-lane pipeline. |
| Uniform treatment of input | Early networks had no way to focus on the most important words. Every token got equal structural weight. |
| Property | Cross-attention (RNN era) | Self-attention (Transformer) |
|---|---|---|
| Direction | Decoder looks back at encoder (between two sequences) | Tokens query other tokens in the same sequence |
| Dependency path | Sequential — O(n) steps through time | Instant — O(1) link across the full matrix |
| GPU parallelism | Highly restricted (sequential dependency) | Fully parallel — entire sentences processed at once |
Cross-attention spotlight (Bahdanau):
score_ti = score(decoder_state_t, encoder_h_i)
alpha_ti = softmax_i(score_ti)
c_t = sum_i alpha_ti * encoder_h_iWhen translating "cat" → "chat", weights might peak on source word "cat" instead of a single compressed vector.
The Transformer leap — self-attention:
Attention(Q, K, V) = softmax(Q * K^T / sqrt(d_k)) * VDependency path length between any two tokens becomes O(1) through attention (plus depth through stacked layers), versus O(n) along an RNN. Training can process all positions in parallel (with causal masks in decoders — covered later). Multi-head attention (MHA) runs several of these mixers side by side so different heads can track syntax, coreference, and locality at the same time.
Pre-training needs huge text corpora and huge batches. Parallelism over sequence length (and batch size) made scaling laws practical. The same block — attention + feed-forward network (FFN) + residual connection + layer normalization (LN) — stacks into GPT-style decoders and encoder–decoder models (T5, classic Transformer machine translation). When someone says "the model attended to the system prompt," they mean this mechanism, not a metaphor.
A tiny demo of the bottleneck: an RNN-like running average that overwrites early signal, versus a soft attention mix that can still emphasize the first token.
import numpy as np
rng = np.random.default_rng(0)
# Three token embeddings in 4-D; token 0 carries a unique spike
X = rng.normal(size=(3, 4))
X[0] = np.array([5.0, 0.0, 0.0, 0.0])
# Fake "RNN final state": exponential moving average (late tokens dominate)
h = np.zeros(4)
for t in range(3):
h = 0.2 * h + 0.8 * X[t]
print("RNN-like final state ~", np.round(h, 3))
print("spike retained?", h[0] > 1.0)
# Soft attention from a query that looks for the spike
q = np.array([1.0, 0.0, 0.0, 0.0])
scores = X @ q
weights = np.exp(scores - scores.max())
weights = weights / weights.sum()
ctx = weights @ X
print("attention weights", np.round(weights, 3))
print("attention context ~", np.round(ctx, 3))The moving average dilutes the spike while attention puts most weight on token 0 and recovers a large first component. That is the architectural lesson in four lines of NumPy.
Transformers replaced RNN seq2seq by swapping sequential state and fixed bottlenecks for parallel self-attention, giving constant-hop token interactions that scale on modern hardware.