Full-Text Search Engine with BM25 Ranking and Urdu Tokenisation

The problem: search that fails the moment users type in Urdu or Roman Urdu

Try searching a local news site, a university notice board or a government portal for something in Urdu, or in Roman Urdu the way most people actually type on their phones ("dakhla", "dakhila", "admission"). Results are often empty or irrelevant. Urdu text has multiple Unicode forms for the same letter, optional diacritics and joined or split words. Roman Urdu has no standard spelling at all. Most sites plug in whatever search their CMS ships with and never test it in these languages.

Many other languages with non-Latin scripts and informal romanisation have the same problem, from Hindi and Bengali to Arabic dialects.

What already exists and the gap

• Elasticsearch and OpenSearch, built on Apache Lucene, are the industry standard, with analysers for many languages, but they are heavy to run for a small site and their Urdu handling depends on configuration most teams never touch.
• Meilisearch uses its own tokenisation library and focuses on instant, typo-tolerant search for app developers.
• Typesense is a fast, simple search server; its own comparisons note weaker support for languages that need complex tokenisation.
• Tantivy is a Lucene-like library in Rust, a good reference for how a modern engine is structured.

The gap is twofold: an understandable engine a student can build and explain line by line, and a measured answer to how much Urdu and Roman Urdu normalisation actually improves relevance.

What you build

• Indexer: splits documents into fields, tokenises, normalises and writes postings (document id, term frequency, positions) to segment files. New documents go into a new segment; segments merge in the background. Deletes are recorded in a bitmap and cleaned up on merge.
• Analyser chain: Unicode normalisation, folding of Urdu letter variants and diacritics, lower-casing for Latin script, stop words per language, and a Roman Urdu variant map built from a word list you collect.
• Query engine: parses queries, looks up postings, intersects or unions them, checks positions for phrases, scores with BM25 and returns the top results with highlighted snippets.
• Typo tolerance: candidate terms within edit distance 1 or 2, found through a trie or a finite automaton over the term dictionary.
• Demo: a search page over a bilingual corpus with a toggle that switches the analyser chain on and off, so anyone can see the difference.

The technically interesting part

Two things. First, the data structures: postings lists that are fast to intersect, segment merging that does not block queries, and a term dictionary that makes fuzzy lookups cheap. This is the heart of every search engine and ties directly to what CMU's database course teaches about storage and indexing.

Second, and more original, is measuring relevance. Build a judged query set: 50 or more real queries (collect some from people typing naturally), and for each, mark which documents are relevant. Then report nDCG@10 for the plain tokeniser, for Urdu normalisation, and for Roman Urdu variants. If a normalisation step does not improve the numbers, say so. Evaluators rarely see a student project that proves its own design choices with an information retrieval metric.

How to evaluate it

• Relevance: nDCG@10 and recall@10 on the judged query set, per analyser configuration.
• Speed: p50 and p95 query latency for single-term, multi-term, phrase and fuzzy queries on a corpus of at least 100,000 documents.
• Index cost: index size as a percentage of raw text, and indexing throughput.
• Comparison: the same corpus and queries in one existing engine with default settings.
• User check: eight to ten people search the demo in Urdu, Roman Urdu and English and rate whether the top results answer their question.

Suggested plan

1. Weeks 1 to 3: collect the corpus and draft the query set; read how Lucene and Tantivy lay out segments.
2. Weeks 4 to 7: in-memory index, BM25 scoring, boolean queries.
3. Weeks 8 to 11: on-disk segments, merging, deletes, phrase queries.
4. Weeks 12 to 16: Urdu and Roman Urdu analyser chain; first relevance numbers.
5. Weeks 17 to 22: typo tolerance, snippets, HTTP API and demo page.
6. Weeks 23 to 28: benchmarks, comparison, user check and report.

Path to a product

Publish the Urdu and Roman Urdu analyser as an open source plugin for an existing engine; that alone is useful to news sites and e-commerce stores serving Urdu speakers. The engine itself can be sold as embedded site search for small publishers, schools and directories at a monthly fee per site, or offered as a consulting service to tune search for regional-language sites. Scale by adding analysers for other languages with the same romanisation problem.

Scope warning

Do not add vector or semantic search; that is a separate project and hides whether your lexical engine works. Do not build distributed sharding or replication. Do not crawl the web; use a corpus you have permission to index.

Learn what you need on Sefism

• DSA roadmap : tries, heaps for top-k and merging sorted lists are all used directly.
• DSA cheat sheet : quick reference for the data structures inside an index.
• CMU 15-445/645: Intro to Database Systems : storage, indexing and buffer management lectures behind segment design.
• Operating Systems: Three Easy Pieces : file I/O and memory mapping, which decide how fast your index reads.

Get help with this idea

The relevance test set is what turns this from a toy into research-grade work, so start it in week one. If you want a second opinion on scope, stack or how to evaluate it before you pitch it to your supervisor, bring your draft proposal and we will pressure-test it together. You build it; the guidance is on the decisions.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session

New here? Create a free account first.