Image search finds matching images from a description or another image.
It is useful for product catalogs, stock-photo libraries, reverse image search, and content moderation.
Suppose you have 200,000 product photos and a shopper types "black waterproof hiking backpack."
The instinctive solution is to write a caption for every photo, then run ordinary text search over the captions. That does work, and for years it was how image search was built. It has two stubborn problems.
First, the caption is written before anyone asks anything. Whoever wrote it decided which details mattered. If the caption says "black backpack" and the shopper cares about side pockets, that information was thrown away at indexing time and no amount of clever searching recovers it.
Second, it puts a translation step between the question and the evidence. Every mistake in the caption becomes a permanent, invisible error in the index.
Modern image search does not translate pictures into text at all. It converts both the photo and the query into vectors, and places them in the same space — so they can be compared directly, without either becoming words first.
That is the entire trick, and everything else in this lesson follows from it. A CLIP-style model is what makes it possible, because it was trained specifically so that a picture and its caption land in the same region.
The user types:
Snow-capped mountains at sunset
The system returns matching images, even when nobody added that exact caption to them.
The user supplies a reference image. The system returns visually similar images.
Both directions can use the same image index. Only the query encoder changes.
CLIP learns with matched image-text pairs. Its image encoder and text encoder place both kinds of content in a shared vector space.
Because text and image vectors are comparable, a text description can search image vectors directly. The system does not need to write a caption for every image first.
Before search:
For each user query:
query_vector = clip.encode_text(
"black waterproof hiking backpack with side pockets"
)
image_ids = image_index.search(query_vector, top_k=10)
return catalog.images(image_ids)For image-to-image search, replace encode_text with the image encoder.
A shared embedding is good at broad visual meaning. It may not reliably preserve an exact SKU, size, brand, or every spatial detail.
For product search, combine signals:
For example, filter by brand = Acme and then rank the remaining images by similarity to “red waterproof backpack.”
Common benchmarks include Flickr30K and MS COCO.
If 80 of 100 queries find their correct image in the first five, Recall@5 is 80%.
Benchmark scores depend on the model, data, direction, and test rules. Compare systems under the same protocol.
“A red car” is easier than “the small red sedan facing left in the third row.”
A request with colour, object type, direction, position, and relation may lose one or more constraints in a single vector.
Very detailed queries can exceed what the text encoder learned to represent well.
The training data and searchable collection affect which images appear “most relevant.” Test different cultures, environments, and long-tail examples.
Image-to-image retrieval may focus on background, colour, or style instead of the specific object feature the user intended.
Controls include metadata filters, reranking, user feedback, and task-specific evaluation.
CLIP-style image search compares text and images in one vector space, while filters and reranking protect exact or fine-grained requirements.