Design Pinterest Visual Search

Free with account Updated Nov 16, 2026 · 17 min read
On this page

    Problem Description

    In this interview, we're going to design visual search for a shopping app, similar to Pinterest Lens or Google Lens. A shopper sees something they like, such as a chair in a café, and takes a photo of it or uploads one. The app marks each product it sees in the photo, such as the chair, the table and the lamp, and the shopper taps the one they want. The system then shows products from the app's catalogue that look like it: the same product first if the catalogue sells it, then similar ones, only products in stock. Shoppers tap, save and buy products from the results, and the system keeps learning from what they do. Sellers add new products with photos all day, and those should be found too. You can use existing building blocks: a catalogue service that stores every product with its photos, price and stock and announces every change, and an object detector model that finds the things in a photo and names their category (chair, shoe, lamp). This interview is about finding products that look like the one in the photo, not about the detector or checkout.

    Out of Scope

    • Training the object detector – given; we use its boxes and categories.
    • Searching with a photo and words together ("like this, but in blue") – a good follow-up question.
    • Recognising things that aren't products (plants, landmarks, text) – other products with their own models.
    • Product pages, cart and checkout – the app already has them.
    • Detecting counterfeit listings – a separate trust and safety system.

    Good Clarifying Questions

    1. How big is the catalogue, and how many visual searches? – About 200M products with about 5 photos each, so 1B photos. About 50M visual searches a day, 2,000 a second at peak.
    2. How fast must it feel? – The marked products within 300 ms of the photo arriving, and the matches within 300 ms of the tap, both at p99 on our servers.
    3. Same product or similar ones? – Both: the exact product first when we sell it, then look-alikes. Most phone photos are taken in bad light and at odd angles, unlike the catalogue's studio photos.
    4. How fresh? – A newly listed product should be findable within an hour. The image model can be updated monthly.
    5. What if matching fails? – The shopper must still get somewhere: offer a text search for the category we detected rather than an error.

    Requirements

    Functional Requirements

    1. Find Products in a Photo – Shoppers can take or upload a photo; the app marks each product it sees and names its category.
    2. Show Matching Products – Shoppers can tap one marked product and see 50 products in stock that look like it, the same product first when the catalogue sells it.
    3. Record What Shoppers Do – The system records which products each shopper was shown for each photo, at which position, and whether they tapped, saved or bought them.
    4. Keep Learning – The system trains a new image model from these records and the catalogue's photos, and uses it only after it proves at least as good as the current one.

    Non-Functional Requirements

    1. Scale – 50M visual searches a day: about 600/s on average, 2,000/s at peak, over 1B photos of 200M products.
    2. Latency – Marked products within 300 ms of the photo arriving; matches within 300 ms of the tap; both p99 on our servers.
    3. Freshness – A new product findable within 1 hour; a new image model at least monthly.
    4. Availability – 99.9%. If matching fails, the shopper is offered a text search for the detected category; never an error screen.