CLIP learns a shared space where a matching image and caption sit close together, and unrelated pairs sit far apart.
Plain sentence: CLIP answers “Does this text describe this image?” — not “Write me a paragraph about this image.”
A fixed classifier needs a new labeled head for every new category. CLIP can use a text description as the label at test time, so you can classify new categories without retraining on those labels.
Picture two separate encoders — one for images, one for text — meeting in a shared embedding room:
That is contrastive learning: learn by comparison, not by memorizing one correct essay per image.
| Tower | What it does |
|---|---|
| Image tower | ResNet or ViT → one global image feature → project to shared dimension |
| Text tower | Transformer + BPE tokenizer → text representation (often from an end-of-sequence position) |
The towers do not do token-by-token cross-attention during encoding. Each modality is encoded independently — efficient for large-scale retrieval.
Batch picture: 4 images and 4 captions → a 4 × 4 similarity matrix.
| Caption 0 | Caption 1 | Caption 2 | Caption 3 | |
|---|---|---|---|---|
| Image 0 | ✓ match | ✗ negative | ✗ negative | ✗ negative |
| Image 1 | ✗ | ✓ | ✗ | ✗ |
| Image 2 | ✗ | ✗ | ✓ | ✗ |
| Image 3 | ✗ | ✗ | ✗ | ✓ |
Training job for image 0’s row: “Which caption belongs to me?” — caption 0 should win the softmax.
Core idea — two symmetric tasks:
Steps in plain words:
τ — smaller τ makes the softmax sharper (punishes near-ties harder).Formula sketch: s_ij = (image_i · text_j) / τ
Tiny score example for one dog image row:
| Caption | Score | After softmax intuition |
|---|---|---|
| “a photo of a dog” | 0.91 | Should dominate |
| “a photo of a bus” | 0.18 | Should stay low |
| Other in-batch captions | lower | In-batch negatives |
Temperature intuition: τ = 0.5 vs τ = 0.05 — the smaller value makes the model punish “almost tied” scores much harder.
image_features = image_encoder(images)
text_features = text_encoder(captions)
image_features = l2_normalize(project_image(image_features))
text_features = l2_normalize(project_text(text_features))
scores = image_features @ text_features.T / temperature
labels = torch.arange(batch_size) # diagonal = correct pairs
loss = 0.5 * (
cross_entropy(scores, labels) + cross_entropy(scores.T, labels)
)More negatives without changing the encoders? Increase effective batch size (more GPUs, gradient accumulation, or feature queues). A batch of 256 gives 255 negatives per image instead of 3 in a batch of 4.
At inference — no new classifier head:
Walk a tiny example:
dog, cat, busPrompt wording matters: “dog” vs “a photo of a dog” can produce different text embeddings. Teams often average several templates.
| Method | What you train at test time |
|---|---|
| Zero-shot | Nothing on the target set — language defines classes |
| Linear probing | Freeze CLIP encoder; train a small labeled head on top |
Strengths: open-vocabulary tagging, image–text retrieval, strong visual backbone, natural-language categories.
Uses: search, tagging, moderation, similarity ranking, guidance for image generation.
Limits: coarse semantics, weak counting and spatial relations, small text in images, prompt sensitivity, web-data bias, no free-form explanation.
CLIP aligns image and text in a shared embedding space with a symmetric contrastive loss, enabling zero-shot classification from language-defined labels.