Medicine Box and Strip Identification for Pharmacy Dispensing Double-Checks

The problem: the wrong box in the bag

In a busy neighbourhood pharmacy, an assistant reads a prescription, walks to the shelf and picks a box. Many medicines from the same company share colours and layouts. Names sound alike. The same medicine comes in two strengths with nearly identical boxes. Mistakes happen, and the patient usually cannot tell. The World Health Organization's work on medication safety names look-alike, sound-alike medicines as a known source of error.

Large pharmacy chains use barcode scanning tied to their stock system. Most small pharmacies do not, and many strips cut from a larger pack have no barcode at all. A second person checking every item is the usual safeguard, but at busy times there is no second person.

What already exists and the gap

• Pharmacy point of sale systems with barcode scanning check the product against the sale, where barcodes and stock systems are in place.
• Pill identification tools help identify loose tablets by imprint, colour and shape.
• Research on pill image recognition shows that matching a consumer photo to reference images works well with learned embeddings.

The gap is a simple camera check for small pharmacies that works with boxes and loose strips, does not depend on barcodes and can learn a new product from a few photos.

Inspiration

"The National Library of Medicine Pill Image Recognition Challenge: An Initial Report" by Ziv Yaniv and colleagues, presented at the IEEE Applied Imagery Pattern Recognition Workshop in 2016, describes a challenge to match consumer-quality photos of pills to reference images. The winning entry used a triplet network, a way of learning image embeddings where similar items sit close together. Your project applies the same retrieval idea to boxes and strips.

What you build

• Pharmacy assistants select the prescription line, scan the picked item and see a green match, a red mismatch or an amber "check by hand".
• Pharmacists add new products by photographing them and review a log of mismatches caught.
• Pharmacy owners see how often the check caught a mistake.

The scan should take a few seconds, because a check that slows down a queue will be skipped. The app keeps a short log of each check with the time, the expected item, the scanned item and the result, so a pharmacist can review near misses at the end of the day. A near miss that was caught is useful information about which products need to be shelved apart.

The technically interesting part

• Retrieval, not classification. A classifier must be retrained for every new product. Embedding each catalogue photo with a pretrained model and finding the nearest match lets you add products instantly. Compare a few pretrained models and decide whether light fine-tuning helps.
• Strength is the hard part. Two strengths of the same medicine often look almost identical. Reading the strength text from the box is the deciding check, so combine retrieval with OCR and handle cases where the text is hidden.
• Strips. A cut strip shows only part of the print, often on shiny foil. Measure how well it works and when the app should give up and ask for the box.
• Unsure is a valid answer. False alarms make people ignore a tool. Tune thresholds so the app says "check by hand" rather than guessing.

How to evaluate it

• Top-1 retrieval accuracy on held-out photos taken in a real pharmacy.
• Strength reading accuracy on boxes with two or more strengths.
• Look-alike test: pairs chosen by a pharmacist; how often the app tells them apart.
• False alarm rate on correct picks, and time per check.

Suggested plan

• Weeks 1 to 3: agree with a pharmacy, photograph the catalogue, pick look-alike pairs with the pharmacist.
• Weeks 4 to 7: embedding retrieval and the first accuracy numbers.
• Weeks 8 to 10: strength reading and the decision rules.
• Weeks 11 to 13: the app and a short trial in the pharmacy.
• Weeks 14 to 15: report and demo.

Path to a product

Small pharmacies are a large market across South Asia, and medication safety matters to regulators and to pharmacy chains expanding into smaller towns. A low-cost app with a shared catalogue is a realistic start. The project shows practical computer vision and careful evaluation, both valuable in health technology roles.

Scope warning

This is a double-check, not a replacement for the pharmacist. Say so in the app. Do not read handwritten prescriptions in this project. Keep patient data out of the system entirely: the app only needs the medicine name and strength.

Learn what you need on Sefism

• Machine learning roadmap: evaluation and thresholds for the match decision.
• AI engineering roadmap: embeddings and vector search in a real application.
• torchvision Documentation: pretrained image models for embeddings.
• OpenCV-Python Tutorials: cropping and cleaning photos before matching.
• Computer Vision: Algorithms and Applications, 2nd Edition: image retrieval background.
• FastAPI Tutorial: User Guide: the matching service.

Get help with this idea

The risk is a demo that works on clean photos and fails behind a real counter. Tauseef can help you plan the pharmacy partnership, the look-alike test and the thresholds. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session