Aspect-Based Review Insights for Small Sellers in English and Roman Urdu

The problem small sellers have with their own reviews

A small clothing or accessories brand selling through a marketplace, Instagram and WhatsApp gets hundreds of reviews and comments a month. The star rating tells them almost nothing: a 3-star review might say the fabric is lovely but the size runs small and delivery took nine days. Owners read a few, react to the loudest, and miss the pattern that half the complaints this month are about one courier.

Many of these comments are written in Roman Urdu mixed with English ("size chota hai but quality zabardast"), which general sentiment tools handle badly. The seller needs to know, per aspect, what customers like and dislike, and whether it is getting better or worse.

What already exists and the gap

• Amazon's AI-generated review highlights, introduced in 2023, summarise common opinions on a product page for shoppers, but they are for Amazon listings only and aimed at buyers, not the seller's trend view.
• Yotpo and similar review platforms offer AI summaries and topic tagging, priced for established Shopify brands.
• Academic ABSA datasets such as SemEval-2014 Task 4 (laptops and restaurants, over 6,500 annotated sentences) show the task is well studied in English.

The gap is a cheap, per-seller tool that handles code-mixed Roman Urdu, works from a CSV export, and shows aspect trends rather than a one-off summary.

What you build

A pipeline and a dashboard. The seller uploads a CSV or pastes comments. Each review is split into sentences, and each sentence is tagged with zero or more aspects from a fixed list and a polarity for each. The dashboard shows a table of aspects with this week's positive and negative counts, a trend line per aspect, and three representative quotes for each complaint.

You compare two approaches. The first is a fine-tuned small model: XLM-RoBERTa base, which already knows some Urdu, trained as a multi-label classifier on your labelled data. The second is a prompted LLM given the aspect list and a few examples. Comparing them honestly on the same test set, including cost, is a large part of the project's value.

The technically interesting part

Labelling code-mixed text consistently. Roman Urdu has no standard spelling ("acha", "achha", "acha hai"), sarcasm is frequent, and one sentence often contains two aspects with opposite polarity. Most student projects fail here, not in the model. Write clear labelling rules, have two people label the same 200 sentences, measure agreement with Cohen's kappa, and fix the rules where you disagree. Then show how model accuracy changes with training set size (250, 500, 1,000, 1,500 sentences) so your evaluator can see whether more data would still help. The public UCI Roman Urdu sentiment dataset (over 20,000 records) is useful for pre-training polarity, but it has no aspects, so your own labels remain the core contribution.

How to evaluate it

• Aspect detection micro and macro F1 on a held-out test set, reported separately for English, Roman Urdu and mixed sentences.
• Polarity accuracy per aspect for the fine-tuned model and the LLM baseline.
• Inter-annotator agreement on the double-labelled subset.
• Cost per 1,000 reviews and median latency for both approaches.
• A short study with three to five real sellers: can they name their top complaint correctly after five minutes with the dashboard, compared with reading reviews for five minutes?

Suggested plan

1. Weeks 1 to 2: pick the category, collect public reviews and comments (with permission for any private data), draft the aspect list.
2. Weeks 3 to 5: labelling rules, double labelling, agreement check, full labelling.
3. Weeks 6 to 8: fine-tune the small model, build the LLM baseline, run the comparison.
4. Weeks 9 to 11: dashboard, trend alerts and CSV importer.
5. Weeks 12 to 14: seller study and write-up.

Path to a product

The first customers are small fashion and home brands that sell online and already export review data. A low monthly subscription per store, or a free tier with a limit on reviews per month, fits this market. The same engine can be sold to marketplace seller-support teams as an API. Adding categories means new aspect lists and a few hundred labels each, which is a repeatable process you can document.

Scope warning

Do not build scrapers for marketplaces whose terms forbid it; use CSV exports, public datasets and sellers' own data. Do not try to cover every product category, and do not build a review reply bot in the core scope. One category, done well and measured, is the project.

Learn what you need on Sefism

• Machine Learning roadmap : fundamentals of classification, train and test splits and metrics.
• AI Engineering roadmap : how to prompt, evaluate and cost an LLM baseline properly.
• Machine Learning cheat sheet : precision, recall and F1 in one place.
• Google Colab : free GPUs for fine-tuning XLM-RoBERTa.
• Kaggle Datasets : public review datasets to pre-train or sanity-check with.

Get help with this idea

Most of the marks here come from the labelling rules and the honest comparison between the small model and the LLM. A mentor can review your aspect list and test set design before you label 1,500 sentences. Bring your draft scope, the dataset you plan to use and your evaluation table, and we can look at them together. The build is yours; the guidance is on scope, stack and how to measure it. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session