MCQ Exam Item Analysis and Faulty Question Finder for University Teachers

The problem: bad questions quietly change students' grades

Multiple-choice exams are everywhere in Pakistani universities, colleges and entry tests. They are quick to mark, but they are only as fair as their questions. A question with an ambiguous stem, two defensible answers or simply the wrong key in the marking sheet penalises the students who knew the material best. In a large class, a few bad questions can move grades for dozens of students.

Psychometrics has known how to catch these problems for decades. Item analysis computes, for each question, its difficulty and whether it separates students who did well overall from those who did not. Questions where strong students choose a wrong option are almost always flawed. Yet most teachers never see these numbers, because their exams are marked on paper or by OMR and results end up in a spreadsheet.

What already exists and the gap

• Moodle's quiz statistics report shows a facility index and a discrimination index for each question, but only for quizzes taken inside Moodle.
• OMR products such as Remark Office OMR and phone-based graders such as ZipGrade include item reports within their own ecosystem.
• Statistical packages such as the R mirt package can fit sophisticated models, but they need someone comfortable with R.

The gap is a simple, open tool that accepts the spreadsheet a teacher already has, from any marking method, and returns a clear review list in plain language.

Inspiration

Classical test theory gives you difficulty, discrimination and the KR-20 reliability coefficient, all of which are simple to compute and well documented in measurement textbooks. For a stretch goal, Chalmers' paper "mirt: A Multidimensional Item Response Theory Package for the R Environment" (Journal of Statistical Software, 2012) describes the item response theory models used in modern testing programmes. Comparing the classical and IRT views of the same exam is a good learning exercise.

What you build

• A teacher uploads a CSV or Excel file: one row per student, one column per question with the chosen option, and a row with the key.
• The tool scores the exam, computes item statistics and checks for likely faults.
• The report lists questions in order of concern, with a short explanation for each: "Option C was chosen by 48 percent of the top quarter of the class; check the key."
• A department head can compare item quality across sections taught by different teachers.

The technically interesting part

• Discrimination done right. Correlate each question with the total score excluding that question; otherwise every question looks better than it is. Compare point-biserial with the simpler upper-lower group method and explain when they disagree.
• Wrong key detection. If a distractor has a strong positive discrimination and the keyed answer has a negative one, the key is probably wrong. Test this rule on simulated exams where you deliberately swap keys, and measure how often the tool catches them.
• Small classes. With 30 students, statistics are noisy. Show uncertainty and avoid flagging a question on weak evidence.
• Plain language. The hardest part is writing explanations that a busy teacher understands without a statistics course.

How to evaluate it

• Detection on simulated data: generate exams with known faulty questions (wrong key, two correct options, a question everyone guesses) and measure how many the tool flags, and how many false flags it raises.
• Real exams: with permission and anonymised data, run the tool on three past exams and ask the teachers whether the flagged questions were really faulty.
• Agreement: compare your difficulty and discrimination values with the R mirt or CTT package results on the same data.
• Time saved: how long a teacher takes to review an exam with and without the report.

Suggested plan

• Weeks 1 to 2: read about item analysis and design the input format.
• Weeks 3 to 5: scoring, difficulty, discrimination, reliability.
• Weeks 6 to 8: distractor analysis, wrong key detection, simulation tests.
• Weeks 9 to 11: report page and upload flow.
• Weeks 12 to 14: trials with real teachers, write-up and demo.

Path to a product

Universities, colleges, testing bodies and coaching academies all run MCQ exams. A free tool for individual teachers, with a paid version for institutions that want a question bank and reports across departments, is a natural model. The project also shows strong practical statistics, which helps for data analyst and education technology roles.

Scope warning

Do not build an OMR scanner in the first version; start from the spreadsheet. Never store student names; replace them with codes on upload. Do not let the tool change grades automatically; it recommends questions to review, and the teacher decides.

Learn what you need on Sefism

• Data analytics roadmap: cleaning results sheets and building reports.
• Data analytics cheat sheet: aggregation and clear presentation.
• OpenIntro Statistics: correlation, sampling error and reliability basics.
• pandas User Guide: reshaping one row per student into per-question statistics.
• Streamlit Documentation: a quick first version of the upload and report screens.

Get help with this idea

The trap is a report full of numbers that teachers ignore. Tauseef can help you design the simulation tests, find teachers willing to try it and write explanations people act on. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session