Word Representations — One-Hot and Embeddings

Before a neural net can “read,” words must become numbers. The two classic encodings are one-hot vectors and embeddings. One-hots are honest but barren: every word is an orthogonal spike with no notion of similarity. Embeddings are dense, learned coordinates where cat and kitten can sit near each other. Every modern language model starts with an embedding table—so this short lesson is the bridge from vocabulary IDs to the geometry that later layers actually compute on.

Intuition

Imagine a dictionary with 10,000 entries. A one-hot encoding gives each word a 10,000-dimensional vector that is all zeros except a single 1 at that word’s index. “Dog” and “puppy” look as unrelated as “dog” and “quantum”—their dot product is zero. The model must rediscover every synonym from scratch through later weights.

An embedding is a short dense vector (say 64 or 768 numbers) looked up by word ID. Training nudges those vectors so words that appear in similar contexts drift together. Distance and direction start to mean something: analogies, clustering of sentiment words, and smoother generalization to rare synonyms. You are no longer storing a flag; you are storing a point in meaning-space.

Think of one-hot as a locker number and embedding as a GPS coordinate for the idea behind the word. Lockers don’t know which neighbors are related; GPS does.

How it works

Vocabulary and IDs. Map each unique token to an integer 0 … V-1. Unknown words often share an <unk> id.

One-hot. For token id i and vocab size V:

one_hot[j] = 1 if j == i else 0

Length is always V. Sparse, high-dimensional, no learned parameters in the encoding itself.

Embedding lookup. Maintain a matrix E of shape (V, d) where d is the embedding dimension. The representation of token i is simply row i:

x = E[i]          # shape (d,)

During training, gradients update E[i] (and only rows that appear in the batch). Downstream layers see dense vectors; the one-hot is never materialized in practice—lookup is “multiply by a one-hot” in matrix form.

Why geometry helps. If two rows of E are close (cosine similarity high), linear layers and attention treat those tokens more alike. That is the seed of transfer: learning about run helps with running.

In code

import numpy as np

vocab = {"the": 0, "cat": 1, "sat": 2, "mat": 3, "<unk>": 4}
V, d = len(vocab), 8
rng = np.random.default_rng(0)
E = rng.normal(0, 0.1, size=(V, d))  # embedding table

def one_hot(token_id, V):
    v = np.zeros(V)
    v[token_id] = 1.0
    return v

def embed(token, E, vocab):
    tid = vocab.get(token, vocab["<unk>"])
    return E[tid]  # same as one_hot(tid) @ E

print("one-hot 'cat':", one_hot(vocab["cat"], V))
print("embed 'cat' shape:", embed("cat", E, vocab).shape)
print("dot(one-hot cat, one-hot sat):",
      one_hot(1, V) @ one_hot(2, V))  # always 0 for distinct words

In frameworks this is nn.Embedding(V, d). LLMs use the same idea at token (subword) level, not whole words.

What goes wrong

Huge one-hots in the forward pass. Materializing V-length vectors for every position wastes memory; always use embedding lookup.

Frozen random embeddings. If you never train E (and have no pretrained table), later layers get arbitrary coordinates—learning is much harder.

Vocab mismatch. A token id from a different tokenizer indexes the wrong row. Always keep tokenizer and embedding table paired.

Thinking embeddings “are” meaning. They encode distributional similarity from data, including bias and corpus quirks—not a clean dictionary definition.

One-line summary

One-hot marks which word you mean; embeddings learn where that word lives in a dense space so similar words can share structure.

Key terms