Neighbourhood House Price Index from Open Property Sales Data

The problem with average house prices

Most people judge a neighbourhood's prices from the average or median sale price. That number moves for the wrong reasons. If one month several large houses sell, the median jumps even though no single house became more valuable. In small areas with few sales, the median swings wildly. Buyers, renters negotiating rent, local journalists and small estate agencies all draw conclusions from these noisy numbers.

Statistics offices solve this with quality-adjusted indices, but they publish them at city or region level. The neighbourhood level, where decisions are actually made, is usually missing or hidden behind paid reports.

What already exists and the gap

• UK House Price Index, produced with HM Land Registry and the Office for National Statistics, uses a hedonic regression method and is published down to local authority level.
• HM Land Registry Price Paid Data is the underlying record of residential sales in England and Wales, published as open data under the Open Government Licence.
• Zillow Home Value Index covers the US with its own model; similar private indices exist in other countries.
• Property portals such as Zameen in Pakistan show asking prices and area trends, which are listings rather than completed sales.

The gap: a transparent, reproducible index at neighbourhood level, with visible uncertainty, built from open data and documented well enough that another team could rebuild it. The method transfers to any country that opens up transaction records.

What you build

Roles: visitor (explores areas), analyst (downloads series and methodology), admin (runs monthly updates, reviews outliers).

Flows:

1. Monthly ingestion of new transactions and corrections into a raw store, then cleaned staging models.
2. Address normalisation and property matching to find repeat sales of the same home.
3. Index computation: repeat-sales regression, hedonic regression with property type, tenure and new-build flags, and a median baseline.
4. Small-area handling: areas below a sales threshold are merged with neighbours or shown with a warning.
5. The interface shows index lines with confidence bands, a map of change over a chosen period, and downloadable CSVs.

The technically interesting part

Getting from messy transactions to a defensible index. Repeat-sales matching depends on address quality: flat numbers, building names and street spellings vary. Build a matching step, measure its precision on a hand-checked sample, and show how match errors would bias the index.

Then the statistics: repeat-sales regression compares the same property over time, which controls for quality but only uses homes that sold twice. Hedonic regression uses all sales but depends on the attributes you have. Present both, explain where they disagree, and show the effect of sparse data with confidence intervals. The monthly incremental rebuild (new sales can revise past index values) is a nice data engineering problem to show your evaluators: how you version the published series so users can see revisions.

How to evaluate it

• Matching: precision on 200 hand-checked repeat-sale pairs; share of sales matched.
• Accuracy: compare your index at local authority level with the official index; report correlation and mean absolute difference of annual change.
• Stability: revision size when a new month is added; volatility of your index versus the median series in small areas.
• Performance: full rebuild time on the whole dataset in DuckDB; API latency for an area query.
• Users: five or more people (buyers, a journalist, an agent) interpret a chart; check whether they read uncertainty correctly.

Suggested plan

1. Weeks 1 to 3: read the index methodology papers, download data, profile quality.
2. Weeks 4 to 8: ingestion, staging, incremental updates.
3. Weeks 9 to 13: address matching and repeat-sale linking, quality checks.
4. Weeks 14 to 19: repeat-sales and hedonic models, confidence intervals, small-area rules.
5. Weeks 20 to 24: API, map and charts, methodology page.
6. Weeks 25 to 28: validation against the official index, user tests, write-up.

Path to a product

The first customers are small estate agencies and property journalists who need credible local charts for listings and articles. Offer free public charts and paid API access or white-label widgets. Mortgage brokers, valuers and developers are later buyers. When a new country publishes transaction data, the same pipeline can be pointed at it.

Scope warning

Do not build an automated valuation model for individual homes; that is a different and much harder problem. Do not scrape listing portals. Do not try to cover rentals as well unless you have an open rental dataset.

Learn what you need on Sefism

• Data Analytics roadmap : the path from SQL to statistical analysis.
• Machine Learning cheat sheet : regression basics you need for the hedonic model.
• OpenIntro Statistics : confidence intervals and regression explained clearly.
• DuckDB Documentation : fast rebuilds over tens of millions of rows.
• Storytelling with Data : charts that show uncertainty without confusing readers.

Get help with this idea

The key decisions are the area unit, the minimum sales threshold and which index method you present first. If you want a second opinion on scope, stack or how to evaluate it before you pitch it to your supervisor, a mentor can help you shape it. You still build it; the guidance is about making the plan realistic and the results defensible. New here? create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session