How visual search finds a product from an ordinary photo

You see a chair in a café and want the same one. You do not know the brand or the model name, so a text search is mostly guessing. Visual product search starts from the thing you do have: a photo. It is the project the MuskarTech AI Lab is working on right now, and this post explains how the approach works.

The problem with keywords

Text search works when you know what something is called. Products are often the opposite: easy to recognise, hard to name. “Wooden armchair with curved arms” matches thousands of items, and the words a shopper uses rarely match the words in a catalogue. A photo carries far more information than a description, if the system can read it.

Step 1: detect the product

An everyday photo is messy. The chair sits next to a table, a lamp and a window. The first step is object detection: a model finds the product in the frame and draws a box around it, so that everything that follows looks at the chair and not the room.

Detection also decides what the search is about when a photo contains several products. A good system lets the person pick, or searches each one.

Step 2: describe it as an embedding

Next, the cropped product goes through an image encoder, a neural network that turns the picture into an embedding: a list of numbers that captures what the product looks like, its shape, material, colour and proportions.

The useful property of embeddings is that similar-looking products end up close together in that number space, even when the photos differ in angle, lighting or background. Two photos of the same armchair, one in a showroom and one in a café, should land near each other.

Step 3: match against the catalogue

Every product in the catalogue is encoded the same way, ahead of time. A search then becomes a geometry problem: find the catalogue embeddings nearest to the query embedding. Vector indexes make this fast even across large catalogues, and the result is a ranked list of matches with a similarity score for each.

Returning scores, not just a single answer, matters. It lets people judge the result, and it lets the system say “these are close, but none is an exact match” when that is the honest answer.

What makes it hard

The hard part is not the three steps but the photos people actually take:

• Angles and occlusion. Products are photographed from the side, from above, half hidden behind something else.
• Lighting. Warm café light changes how wood and fabric look compared with a studio shot.
• Near-duplicates. Catalogues contain many products that differ only in a detail, such as leg colour or fabric.
• Speed and cost. A model that is accurate but slow or expensive to run will not survive production.

How we measure it

A visual search model is only as good as its evaluation. Before anything reaches users, we test it on a held-out set of real-world photos and check how often the correct product appears among the top results (often called recall at k). That number, and the failure cases behind it, decide what we work on next.

Where it can help

Visual product search is useful anywhere a photo is the most natural starting point: letting shoppers search a store by picture, matching supplier or marketplace photos to an existing catalogue, or identifying an item in a store or warehouse from a quick phone photo.

The model is in closed beta. If you have a catalogue, marketplace or inventory problem that starts with a photo, you can join the beta from the AI Lab page.