Daily Online Grocery Price Index for Pakistani Cities with Product Matching

The problem: prices change daily, official numbers arrive weekly

For a family in Pakistan, the price of flour, cooking oil, sugar and milk is the most important economic number there is. The Pakistan Bureau of Statistics publishes the Sensitive Price Indicator every week, covering 51 essential items collected from 50 markets in 17 cities. It is careful and trusted, but it is weekly, aggregate and based on physical markets.

Meanwhile, more people buy groceries online, and online stores publish their prices openly. Those prices change daily, often at the product and store level. Nobody turns them into a clean, documented open dataset that journalists, students and researchers can use to see what happens between official releases.

What already exists and the gap

• The Sensitive Price Indicator and the monthly Consumer Price Index from the Pakistan Bureau of Statistics are the official measures.
• PriceStats, a commercial company that grew out of MIT research, produces daily online inflation indices for many countries for paying clients.
• Price comparison apps show today's cheapest store for a product, not a consistent index over time.

The gap is an open, transparent daily index for Pakistani cities, with product matching and methodology published, so that anyone can check and reuse it.

Inspiration

Cavallo and Rigobon's paper "The Billion Prices Project: Using Online Prices for Measurement and Research" (Journal of Economic Perspectives, 2016) describes how MIT researchers collected prices from online retailers daily and built inflation indices that tracked official figures closely, and in some countries revealed when official numbers were unreliable. Your project applies the same method to Pakistani grocery stores, at student scale.

What you build

• Collectors visit each store's category pages once a day at a polite rate and save product name, pack size, price, discount and stock status.
• The matching step links "Dalda Cooking Oil 1L Pouch" in one store to "Dalda Oil Pouch 1 Litre" in another, and normalises everything to a unit price.
• The index follows a fixed basket aligned to the official item list, chained day to day.
• The dashboard shows the index by city and by item, the official weekly figures alongside, and a downloadable dataset.

The technically interesting part

• Product matching. Names mix English, Roman Urdu, brand abbreviations and inconsistent units. Combine rules (parse sizes and units with regular expressions), fuzzy string matching and sentence embeddings, then measure precision and recall on a set you label by hand. Wrong matches quietly corrupt the index, so prefer leaving an item unmatched over matching it wrongly.
• Index method. Products appear, disappear and go out of stock. Use a chained geometric mean of price relatives (the Jevons index) for matched products, and decide and document how you treat stockouts and discounts.
• Robust collection. Store sites change layout without warning. Build checks that alert you when the number of products collected drops sharply, rather than discovering a week of missing data later.
• Being a good citizen. Collect once a day, identify your collector, respect robots.txt and terms of use, and store only public price information. If a store objects, remove it.

How to evaluate it

• Matching quality: precision and recall on a hand-labelled set of at least 300 product pairs.
• Coverage: share of basket items with a matched online price in each city, each day.
• Agreement: correlation and lag between your weekly-averaged index and the official SPI for the same items.
• Reliability: days with complete collection over the evaluation period, and how quickly breakages were detected.

Suggested plan

• Weeks 1 to 3: choose stores and basket items, check terms of use, build the first collector.
• Weeks 4 to 6: database schema, daily scheduling, size parsing and unit prices.
• Weeks 7 to 9: product matching and the labelled test set.
• Weeks 10 to 12: index calculation, comparison with official figures.
• Weeks 13 to 15: dashboard, open dataset, methods note and report. Collection should run in the background from week 4 so you have enough history.

Path to a product

Newsrooms, think tanks, retailers and fast-moving consumer goods companies all want timely price data. An open public index with a paid data feed for businesses is a known model. The project also shows employers exactly the skills data engineering roles ask for: reliable collection, messy matching, documented methodology.

Scope warning

Start with three stores and twenty basket items, not every product. Do not attempt to collect from apps that require logins or bypass any protection. Do not present the index as an official inflation figure; call it an experimental online price index and say what it does not cover.

Learn what you need on Sefism

• Data engineering roadmap: scheduled pipelines, data quality checks and backfills.
• SQL and databases cheat sheet: schema design for a long price history.
• Apache Airflow Documentation: scheduling and monitoring daily collection.
• Pakistan Bureau of Statistics: the official price statistics you compare against.
• Storytelling with Data (book and blog): presenting an index people understand.

Get help with this idea

The two classic failures are wrong product matches and a collector that silently breaks. Tauseef can help you design the labelled test set, choose an index method you can defend and plan the comparison with official figures. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session