Visual Search System Design: Search by Photo in a Fashion App

Text search fails, because the shopper has no words for what they saw and the seller used different ones. A photo holds colour, pattern, cut and embroidery all at once. Visual search system design lets the photo itself be the query.

Picture a common setup. A fashion shopping app sells ethnic and western wear from about 2,500 sellers, with roughly 4 lakh (400,000) products. The setup is made up, but the problem is real.

Every week, the app's support team gets about 1,800 chats that start with a screenshot. It might be a wedding photo, a web series still or a stranger's kurta.

Then the shopper tries the search box. They type pink dress and get 9,000 results. They type pink anarkali with gold border and get dupattas, sarees and a pink phone cover.

The seller of the matching anarkali had tagged it rani colour, zari work, festive. None of those words came from the shopper. Text search matches words, so the right product sat on page 40.

The product team asks the search team for a camera button in the search bar: upload a photo, see similar products. This guide follows that design, from the simple first version to the parts that break at scale.

You need five words: embedding, vector, cosine similarity, index and recall at k. Each one is explained below with the app's own example.

Embedding. An embedding is a list of numbers that describes what an image looks like, produced by a trained model. The app's model turns each photo into 256 numbers, and similar-looking kurtas get similar lists.

Vector. A vector is the same list of numbers treated as a point in space. Two vectors close together mean two images that look alike.

Cosine similarity. This is the score used to measure closeness. It runs from minus 1 to 1 and says how much two vectors point the same way, ignoring their length.

Index. An index is a data structure built ahead of time so searching is fast. The app's vector index holds every catalogue vector, arranged so the closest ones can be found quickly.

Recall at k. Recall at k is the share of test searches where the correct product appears in the top k results. A recall at 10 of 0.7 means the right kurta is in the top 10 for 7 searches out of 10.

Visual search system design is the plan for two pipelines that share one encoder. An online pipeline answers a shopper's photo in about a second, and an offline pipeline prepares every catalogue photo ahead of time.

Start, as in any design, by writing down what the feature must do and how well. Here is the team's list.

• A shopper uploads a photo or screenshot and sees up to 40 similar products, best match first, each product once.
• If the photo shows several garments, the shopper can tap the one they want.
• Sold-out products never appear, and new products show up within an hour of listing.
• Results appear within 1 second for most searches on a 4G phone.
• The feature costs less to run than the extra sales it brings in.

Next come rough numbers. The app has 4 lakh products with about 3 photos each, so 12 lakh (1.2 million) catalogue photos. The team expects 30,000 photo searches a day, rising to 15 a second at the peak of a festive sale.

Notice that the requirement says 40 products, not 40 photos. That small word matters later, because one product has three photos and all three look alike.

A trained image model, called an encoder, reads the pixels and outputs an embedding. Photos of similar garments get embeddings that sit close together, so comparing photos becomes comparing lists of numbers.

Think of an old haberdashery shop near Surat's textile lanes, full of thread spools and button cards. A customer brings in a fabric swatch and asks for matching thread.

The owner never reads a label. He looks at the swatch and notices a few things at once: the shade, the sheen, the weave, whether there is a print. Then he picks spools that share those qualities.

• The swatch is the shopper's photo.
• The owner's eye is the encoder.
• The qualities he notices are the numbers in the vector.
• The spools on the wall are the catalogue vectors.
• Picking spools that share the qualities is finding the nearest vectors.

Where the picture breaks: the owner can name what he notices. No single number in an embedding means green or paisley, and the encoder only knows what its training photos taught it.

Why not compare pixels directly? The same kurta photographed on a hanger and on a person shares almost no pixels in the same places. A trained encoder learns to ignore pose, background and lighting, and keep what makes the garment itself.

Cosine similarity then gives one score per pair. Two photos of the same maroon sharara might score 0.92, and a maroon sharara against a blue denim jacket might score 0.15 (made up).

You run an object detector first: a model that draws a box around each garment and names its type, such as kurta, dupatta or handbag. Then you embed only the crop the shopper cares about, not the whole picture.

A wedding photo has five people, a flower wall and a plate of sweets. Embed the whole image and the vector describes the scene, so results come back full of floral prints. Cropping first makes the vector describe one garment.

Pinterest's Shop The Look paper, from KDD 2020, a major data mining conference, works this way. Its detector outputs a box and a category label for each object, and each detected object becomes a query that shoppers can shop from.

Detection also gives you a category for free. If the box says sharara, you can search only among shararas. eBay's KDD 2017 paper searched only inside the top predicted leaf categories, the most specific ones. Shop The Look kept separate indexes per category.

Here is the online query path. Follow it from the phone. Notice that the vector search is only one box of eight, and the filter after it matters as much.

You use approximate nearest neighbour search, or ANN: an index that finds vectors that are very likely among the closest, without comparing the query with every vector. You give up a little accuracy for a large gain in speed.

The simple version: exact search

Start with exact search, also called brute force, which compares the query with every catalogue vector. It is the right first version and, later, the measuring stick. Here it is in Python with numpy.

import numpy as np

def normalise(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

def exact_search(query_vec, catalogue_unit, product_of_photo, in_stock, k=40, fetch=500):
    # catalogue_unit: one row per photo, already scaled to length 1
    scores = catalogue_unit @ normalise(query_vec)       # cosine similarity
    scores[~in_stock[product_of_photo]] = -np.inf          # hide sold-out products
    fetch = min(fetch, len(scores))
    cand = np.argpartition(-scores, fetch - 1)[:fetch]      # best 500, unsorted
    cand = cand[np.argsort(-scores[cand])]                  # sort just those
    best = {}
    for i in cand:                                         # keep each product's best photo
        if scores[i] == -np.inf:
            break
        best.setdefault(int(product_of_photo[i]), round(float(scores[i]), 3))
    return list(best.items())[:k]

rng = np.random.default_rng(7)
photos, products = 9, 3
cat = normalise(rng.standard_normal((photos, 8)).astype(np.float32))
product_of_photo = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])
in_stock = np.array([True, False, True])        # product 1 is sold out
print(exact_search(cat[0], cat, product_of_photo, in_stock))

Running it prints [(0, 1.0), (2, 0.327)]. Product 0 matches itself with a score of 1.0, product 1 is hidden because it is sold out, and product 2 comes next.

What just happened? The @ line gives the cosine similarity with every photo at once, because all vectors were scaled to length 1 beforehand. Sold-out photos get minus infinity, so they sink and the loop stops before reaching them.

Then numpy's argpartition picks the best 500 without sorting all 12 lakh scores, and only those 500 are sorted. The loop keeps each product's best photo, so one kurta with three photos fills one slot, not three.

The memory maths

12 lakh photos times 256 numbers times 4 bytes is 1,228,800,000 bytes, about 1.23 GB (1.14 GiB, the binary unit many tools show). Check it yourself: python3 -c "print(1_200_000 * 256 * 4)" prints 1228800000.

That fits in one server's memory. On our shared 2-core test machine, one exact search over 12 lakh photos took 0.1 to 0.25 seconds (numpy 2.5.4). For today's catalogue, brute force works.

The trouble is growth. The team plans to add review photos and 20 lakh more products, about 10 times the vectors, or 1.2 crore (12 million). Exact search does 10 times the work per query.

Memory traffic grows with it. Run one at a time, every exact search reads the whole 1.23 GB table. So 15 searches a second already reads about 18 GB a second, and 10 times the vectors means 10 times that.

Approximate search: HNSW and compressed indexes

Back to the haberdashery. The owner does not hold the swatch against all 3,000 spools. He glances at the greens, then the dark greens, then compares the last ten closely. That is the idea behind ANN.

HNSW, short for Hierarchical Navigable Small World, is a widely used ANN index. It links each vector to a few close neighbours, in several layers. Upper layers hold few points and long jumps, lower layers hold more points and short steps.

A search enters at the top layer, hops towards the query, drops down a layer and repeats. The 2016 paper by Malkov and Yashunin reports logarithmic complexity scaling: the work grows slowly as the data grows.

The other common family is IVF with product quantization, or IVF-PQ. IVF groups vectors into clusters and searches only the nearest few. Product quantization stores each vector as a short code, here 32 bytes instead of 1,024.

The team picks HNSW for next year and keeps exact search as the measuring stick. It runs the same test searches through both and counts how many true matches the fast index misses.

You train on pairs: a real shopper photo and the catalogue photo of the same product. The model learns to pull each pair together and push other products away. Training only on studio photos leaves a gap the model never learns to cross.

Researchers call this gap street-to-shop, or consumer-to-shop. A shopper's photo is dim, tilted, half-hidden by a dupatta and taken under a yellow bulb. The catalogue photo is bright, front-on and against a plain wall.

The DeepFashion dataset, from the CVPR 2016 computer vision conference, was built for this. It holds over 800,000 fashion images, from posed shop photos to unconstrained consumer photos, with over 300,000 cross-pose and cross-domain pairs. One of its four benchmarks is consumer-to-shop clothes retrieval.

Indian ethnic wear looks different from most public data, so the team builds pairs from the app's own data, in three ways.

• Review photos: buyers often upload a photo of themselves wearing the product, already linked to the product id.
• Support chats: each of the 1,800 weekly screenshot chats where an agent found the product gives one pair.
• Purchases after launch: a photo search followed by buying a result gives a weak but cheap pair.

Training uses a triplet loss, a score computed on three images at a time. It is zero when the matching photo is clearly closer than a non-matching one, and above zero otherwise.

1. 1

   Pick the anchor

   Take a shopper's review photo of a mustard kurta. This is the anchor, the image the model must place correctly.

2. 2

   Pick the positive

   Take the catalogue photo of that same mustard kurta. The model should place it close to the anchor.

3. 3

   Pick a hard negative

   Take a different mustard kurta with a different neckline. A black jacket is too easy and teaches the model nothing.

4. 4

   Nudge the model

   The loss is above zero unless the negative is farther than the positive by at least a small margin. The weights then shift slightly to widen that gap.

Two details catch people out. The FaceNet paper notes that easy triplets do not contribute to training, so you must pick hard ones. It also warns that always picking the very hardest negatives can trap training early.

Here is how a new encoder goes from training to the live app. Notice the decision box: a new model reaches shoppers only if it beats the current one on the same test set.

The last box is an A/B test, where some shoppers get the new model and others keep the old one. The measuring section comes back to it.

Keep the test pairs out of training, and check each category on its own. A model can improve on kurtas and quietly get worse on shararas, and one overall number would hide that.

You run an indexing pipeline that embeds each new catalogue photo soon after it is listed, and you check stock at query time. Re-embedding the whole catalogue is needed when the encoder, or the cropping in front of it, changes.

About 6,000 new products arrive on a normal day (made up), and many more before Diwali. Sellers also delete products and sell out of sizes all day. The index has to keep up with all of it.

In the diagram below, the queue lets the pipeline absorb a burst of uploads without slowing the seller app.

Why keep stock out of the index? Stock changes thousands of times an hour, but vectors change only when photos do. Keeping the fast-changing flag in a cache hides a sold-out product within seconds, with no index rebuild.

Big systems work the same way. Pinterest's 2015 paper describes an incremental service that computes features only for new daily batches of images, and builds a new feature store only when a feature's algorithm changes.

Changing the encoder is different. Two separately trained encoders generally place images in unrelated spaces, so a new query vector cannot be compared with old catalogue vectors. The team re-embeds all 12 lakh photos into a fresh index, then switches traffic in one step.

There is one exception worth knowing. Backward-compatible training, from a 2020 paper by Shen and others, trains a new model so its vectors can be compared with the old ones, and skips re-embedding.

You measure in three layers: offline recall on labelled test pairs, people rating whether results look relevant, and live A/B tests on clicks and orders. Each layer catches problems the others miss.

Offline recall is fast and cheap, so it runs on every change. The team keeps 5,000 test pairs of a shopper photo and its correct product, split by category. Here is the calculation, with a tiny example you can run.

def recall_at_k(results, truth, k=10):
    # results: query id -> ranked product ids; truth: query id -> correct product id
    hits = sum(1 for q, right in truth.items() if right in results[q][:k])
    return hits / len(truth)

truth = {"q1": 501, "q2": 777, "q3": 42}
results = {
    "q1": [501, 88, 12],       # right product first: a hit
    "q2": [5, 6, 7, 8, 777],   # right product fifth: a hit at k=10, a miss at k=3
    "q3": [9, 10, 11],         # never found: a miss
}
print(round(recall_at_k(results, truth, k=10), 3), round(recall_at_k(results, truth, k=3), 3))

It prints 0.667 0.333. At k equal to 10, two of the three searches find the right product. At k equal to 3, the second search misses, because the right product came fifth.

Recall misses something, though. A shopper rarely needs the exact product. A very similar kurta in stock at a better price is often a happy result, yet recall counts it as a miss.

So the second layer is human judgement. Each week, two people rate the top 10 results for 200 random searches as exact, similar or wrong. Shop The Look used human raters on its top 5 results, alongside offline tests and online A/B experiments.

The third layer is the business test. In an A/B test, some shoppers get the old system and some the new one, and you compare what they do. The team tracks a short list.

• Click-through: the share of photo searches where the shopper taps any result.
• Add to cart and orders per photo search, the numbers the business cares about.
• Zero-result rate after the stock filter, which should stay near zero.
• Time to first result at the 95th percentile, the time that 95 searches in 100 beat, so slow searches are not hidden by the average.

Real gains are often modest. Pinterest's 2015 paper reports that Visual Related Pins raised total repins (saves) in its Related Pins feature by 2 percent, after three months on 10 percent of eligible live traffic.

Most failures come from photos the model never saw in training, from filters applied too late, and from the index drifting away from the catalogue. Each one has a plain fix you can name.

• Wrong garment: the detector crops the blouse when the shopper wanted the saree. Fix: show tappable boxes and use the biggest garment only as a default.
• Colour shift: a yellow-lit photo matches mustard instead of cream. Fix: train with lighting changes and real shopper photos.
• Empty pages: strict filters after a small fetch leave few results. Fix: filter inside the search, or fetch hundreds of candidates.
• Copies everywhere: ten sellers, one supplier photo. Fix: group near-identical vectors and show each design once.
• Unsafe uploads: some people upload photos that break content rules. Fix: run a content check before storing any shopper photo.

Privacy needs a line in your design too. Shopper photos often show faces and homes. The team keeps them only as long as the search needs, unless the shopper agrees to share them for training.

Two bugs, both silent. An earlier version of this guide used a shorter exact search: it scored every photo, set sold-out photos to minus infinity, sorted all scores and returned the first 40 photos.

Bug one: sold-out photos come back when few are in stock. With 6 in-stock photos and 40 requested, that version returned 40 results, and 34 of them were sold out with a score of minus infinity.

Bug two: one product fills several slots. Its top 5 held the same product three times, once for each of its photos. The shopper sees one kurta three times and loses two real choices.

It was also slow. It rescaled all 12 lakh vectors on every call and sorted every score, taking 1.5 to 2.4 seconds on our test machine. The version above stores scaled vectors and sorts only 500, and took 0.1 to 0.25 seconds.

Check the fixed version yourself. Paste these lines under the first sample; they test both edges with 100 products of 3 photos each.

pop = np.repeat(np.arange(100), 3)                  # 100 products, 3 photos each
cat = normalise(rng.standard_normal((300, 8)).astype(np.float32))
stock = np.zeros(100, dtype=bool)
stock[:6] = True                                    # only 6 products in stock
res = exact_search(cat[0], cat, pop, stock)
print(len(res), all(stock[p] for p, _ in res))     # edge 1: no sold-out results
stock[:] = True
res = exact_search(cat[0], cat, pop, stock)
print(len(res), len({p for p, _ in res}))          # edge 2: 40 different products

After the first sample's line, it prints 6 True, then 40 40. Only the 6 in-stock products come back, and 40 results means 40 different products.

They trade memory, speed and recall in different ways, and your own data decides the winner. We ran both in faiss 1.15.1, an open-source library from Meta, on 2 lakh made-up vectors grouped into clusters.

Made-up vectors are not real photo embeddings, and our queries were near copies of stored vectors. Read these numbers as the shape of the trade-off. Searches ran on one CPU thread, timed over 1,000 queries.

Two settings matter most. In HNSW, efSearch is how far the search explores the graph: higher means better recall and slower queries. In IVF, nprobe is how many clusters it scans.

What surprised us: scanning four times more clusters did not lift IVF-PQ recall at all. The true neighbours were already among the top 100 candidates, but 32-byte codes were too rough to order them.

Re-scoring fixed it, at a price. Keeping the full vectors beside the codes, and re-scoring the top 100 exactly, lifted recall to 0.998 but brought memory back to about 1,071 bytes per vector.

The memory figures match the faiss wiki formulas, where d is the vector length and M is the links per vector. A flat index needs 4 times d bytes, and HNSW adds about M times 8 bytes of links.

IVF-PQ needs its code bytes plus 8 bytes for the id, so 40 bytes here. Our measured 47 adds the cluster centres and codebooks, a fixed cost that shrinks per vector as the index grows.

You pass the allowed ids into the search itself, so the index skips sold-out vectors while it searches. Filtering afterwards works only if you fetch enough spare candidates, and that number grows very large when few items are in stock.

In our faiss test, a median of 4 were left, and the worst query kept none. That is the empty page from the earlier warning, measured.

faiss can filter during the search with an IDSelector, an object that says which ids may be returned. IDSelectorBitmap takes one bit per vector, which suits a stock flag. Here it is, runnable with faiss-cpu 1.15.1.

# pip install faiss-cpu==1.15.1 numpy
import numpy as np, faiss

rng = np.random.default_rng(0)
vecs = rng.standard_normal((20_000, 256)).astype(np.float32)
faiss.normalize_L2(vecs)                       # length 1, so inner product = cosine

index = faiss.IndexHNSWFlat(256, 32, faiss.METRIC_INNER_PRODUCT)
index.add(vecs)

in_stock = rng.random(20_000) < 0.1            # only 1 photo in 10 is in stock
bits = np.packbits(in_stock, bitorder="little")
params = faiss.SearchParametersHNSW(
    sel=faiss.IDSelectorBitmap(bits), efSearch=64)    # covers ids below 8 * len(bits)

_, ids = index.search(vecs[:1], 10, params=params)
found = [i for i in ids[0] if i >= 0]          # -1 means "no result"
print("results:", len(found), "all in stock:", all(in_stock[i] for i in found))

try:
    index.remove_ids(np.array([0], dtype=np.int64))
except RuntimeError as err:
    print("delete failed:", "not implemented" in str(err))

It prints results: 10 all in stock: True, then delete failed: True. The filtered search still returned a full 10 results, all in stock. The bitmap must cover every id in the index, so rebuild it after adding photos.

The second line is the catch. The faiss wiki says its HNSW index does not support removing vectors, because removal would destroy the graph structure. Its flat indexes do support removal, and in our test its IVF-PQ index removed vectors too.

So treat a deleted product like a sold-out one: keep its bit off in the filter, and rebuild the HNSW index on a schedule to reclaim the space. These behaviours are as of 19 October 2026, faiss 1.15.1; pin your version.

Strict filters starve the graph search

The filter has a limit. We built an HNSW index of 50,000 synthetic vectors and made the filter stricter step by step, with efSearch at 64.

• 10 percent of ids allowed: always 10 results, with recall 0.88 to 0.92 against exact search over the allowed ids.
• 1 percent allowed: only 2.4 to 2.8 results on average, and recall below 0.2.
• 0.1 percent allowed: almost no results at all.

Raising efSearch to 1,024 lifted the 1 percent case to recall 0.893, at about 3 ms a query. For small allowed sets, exact search over just those ids was perfect and took under 0.1 ms.

We found no warning about this on the two faiss wiki pages we read, as of 19 October 2026. Treat it as measured behaviour of faiss 1.15.1, not a documented rule, and pin your version.

Explain it as two pipelines that share one encoder: an offline one that indexes the catalogue and an online one that answers a photo. Then walk through the trade-offs in this order.

1. 1

   Clarify the goal

   Ask whether shoppers want the exact product or similar ones, how big the catalogue is, and the target response time.

2. 2

   Estimate the size

   Multiply photos by numbers per vector by 4 bytes. Say whether it fits in one machine's memory, and what peak searches per second look like.

3. 3

   Draw the query path

   Upload, detect, crop, embed, nearest neighbour search, filter, re-rank, show. Explain why cropping comes before embedding.

4. 4

   Draw the indexing path

   New photo event, queue, embedding workers, vector index. Keep stock in a separate fast cache checked at query time.

5. 5

   Pick an index and defend it

   Start with exact search if it fits. Move to HNSW for speed or a compressed index for memory, and measure recall against exact.

6. 6

   Handle filters and deletes

   Filter by stock inside the search, or score a small allowed set exactly. Show each product once, and plan deletes, because faiss's HNSW index cannot remove vectors.

7. 7

   Close the street-to-shop gap

   Train on real shopper photos paired with catalogue photos, with hard negatives. Re-embed everything when the encoder changes.

8. 8

   Measure in three layers

   Recall at k offline, human relevance ratings, then an A/B test on clicks and orders.