Shelf Gap and Planogram Check from Phone Photos for FMCG Distributors

The problem distributors have with shelves they cannot see

FMCG brands and their distributors pay for shelf space and visibility in thousands of small shops, and they rarely know what the shelf actually looks like. Sales reps visit each shop, eyeball the shelf, and fill a form: is our product in stock, how many facings, is the competitor taking our space? The data is slow, inconsistent and easy to fake. Empty facings mean lost sales that nobody notices until the monthly numbers arrive.

Large retailers in rich markets use shelf cameras and robots. Kiryana stores, pharmacies and small supermarkets will not install anything; the only camera available is the rep's phone.

What already exists and the gap

• Trax provides image recognition for retail execution to large consumer goods companies; its researchers also published the SKU-110K dataset (11,743 densely packed shelf images with over 1.7 million annotated products, all labelled as a single class).
• ParallelDots ShelfWatch analyses shelf photos for stock, placement and pricing compliance, with customised enterprise pricing.
• Neurolabs uses synthetic data to recognise products without large manual labelling efforts.

The gap is an affordable tool for regional distributors and mid-sized brands, trained on the look of small South Asian shops, that can add new SKUs from a handful of photos rather than an expensive labelling project.

What you build

A pipeline in three stages plus a phone app. Detection finds every product on the shelf regardless of brand; SKU-110K is ideal for this because it is about dense, overlapping packages. Recognition crops each detection, computes an embedding with a pre-trained vision model, and matches it against a gallery of reference photos of your target SKUs, falling back to "unknown" below a similarity threshold. Shelf reconstruction groups boxes into rows, orders them, estimates the expected width of a facing, and marks gaps. A planogram file describes what should be on each row, and the checker lists differences.

The app guides the rep to take a usable photo (checks blur, tilt and glare before upload), sends it, and shows the result within seconds.

The technically interesting part

Recognising products you have barely seen. Supervised classification needs hundreds of labelled images per SKU and retraining for every new flavour. Embedding retrieval needs only a few reference images per product, which is how real deployments keep up with new packaging. Your key experiment: recognition accuracy as a function of reference images per SKU (1, 3, 5, 10), comparing at least two embedding models, and with and without fine-tuning the embedding on your shelf crops. Then show how often visually similar SKUs (the same brand in two sizes) are confused and what helps, such as adding the detected box's relative size as a feature. The second hard part is gap detection in rows with uneven lighting and angles, which you evaluate separately.

How to evaluate it

• Detection mAP on SKU-110K test and on your local shelf set.
• Top-1 SKU recognition accuracy versus reference images per SKU, per embedding model.
• Gap detection precision and recall against manual annotations on at least 100 local shelf photos.
• Share-of-shelf error compared with a manual count.
• End-to-end time per photo from capture to result, and phone data use.
• A field trial with at least two reps or students acting as reps across 20 shops, comparing tool results with a careful manual audit.

Suggested plan

1. Month 1: SKU-110K detector baseline, shop permissions, photo collection.
2. Month 2: local fine-tuning and labelling of a local shelf set.
3. Month 3: reference gallery, embedding recognition experiments.
4. Month 4: row reconstruction, gap detection and planogram checker.
5. Month 5: phone app with capture quality checks.
6. Month 6: field trial, error analysis, write-up.

Path to a product

The first customer is a regional distributor or a mid-sized food or personal care brand with a field sales team. Price per active rep per month, with a setup fee for building the SKU gallery. Scaling means more SKUs and categories, which the few-shot approach keeps cheap, and dashboards that aggregate shelf health across territories.

Scope warning

Do not try to recognise every product in the shop; only the target brands and their direct competitors. Do not build fixed cameras or robots. Do not collect shop photos without the shopkeeper's permission, and avoid capturing customers' faces.

Learn what you need on Sefism

• Machine Learning roadmap : detection, embeddings and evaluation you will use throughout.
• Mobile roadmap : the capture app and on-device quality checks.
• Machine Learning cheat sheet : quick reference on detection metrics and retrieval accuracy.
• Google Colab : free GPU time for training on SKU-110K.
• Pinterest Engineering Blog : Pinterest has written about visual search with embeddings at scale, the same retrieval idea used here.

Get help with this idea

This is a large project, so the scope boundaries and the evaluation set matter a lot. A mentor can help you choose the target SKUs and plan the field trial. Bring your draft scope, the dataset you plan to use and your evaluation table, and we can look at them together. The build is yours; the guidance is on scope, stack and how to measure it. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session