Submit your papersSubmit Now
For Enquiries: [email protected]
IIARD LogoIIARD

A Solution to Guessing in Multiple-Choice Questions: A New Paradigm for (Medical) Assessment

John J Barnard

Abstract

Multiple-choice questions (items) have undoubtedly become the most popular type of question in assessment. However, the number of options and how to deal with guessing remain contentious issues. Correction for guessing formulae and guessing parameters have been suggested but do not convincingly provide solutions. Whereas modern measurement theories score a response to an item by estimating the probability of selecting the correct option, the scoring rule in Option Probability Theory (OPT) is based on the probability assigned to the correct option. The difference is subtle but significant. Furthermore, dichotomous scoring misses valuable information captured on continuous scales. In addition to an OPT score, a realism index is calculated which indicates how realistic the test taker was about their knowledge. OPT purports to provide a solution to quantifying the amount of guessing or uncertainty which can be useful to increase measurement precision and to identify misconceptions in diagnostic analysis. If a test is compiled with content constraints, a profile of strengths and weaknesses can be generated. Empirical research studies concluded that OPT yields more precise scores than traditional methods since it accounts for guessing whilst also credits for partial knowledge.

Keywords

Multiple-choiceguessingassessmentoption probability theory

References

Andrich, D., Marais, I. & Humphry, S. Using a theorem by Andersen and the dichotomous Rasch model to assess the presence of random guessing in multiple-choice items. Journal of Educational and Behavioural Statistics, 2012; 37: 414. Andrich, D. & Marais, I. Person proficiency estimates in the dichotomous Rasch model when random guessing is removed from difficulty estimates of multiple-choice items. Applied Psychological Measurement, 2014; 38(6): 432-449. Bala, L., Westacott, R.J., Brown, C. & Sam, A.H. Twelve tips for introducing very short answer questions (VSAQs) into your medical curriculum. Medical Teacher, 2023; 45(4), 360- Barnard, J.J. Option Probability Theory: A quest for better measures. American Review of Mathematics and Statistics, 2015; 3 (1): 61-69. Barnard, J.J. Measurement error in assessment. International Journal of Current Research, 2019; 11(8), 6202-6208 Ben-Simon, A., Budescu, D.V. & Nevo, N. A comparative study of measures of partial knowledge in multiple-choice tests. Applied Psychological Measurement, 1997; 21: 65- Bond, A.E., Bodger, O., Skibinski, D.O.F., Jones, D.H., Restall, C.J., Dudley, E., et al. Negatively-Marked MCQ Assessments That Reward Partial Knowledge Do Not Introduce Gender Bias Yet Increase Student Performance and Satisfaction and Reduce Anxiety. 2013; PLoS ONE 8(2). Carneson, J., Delpierre, G. & Masters, K. Designing and managing multiple-choice questions 2016; (2nd ed). De Finetti, B. Methods of discriminating levels of partial knowledge concerning a test item. British Journal of mathematical and statistical psychology, 1965; 18:87-123. Dirkzwager, A. Multiple evaluation: A new testing paradigm that exercises guessing. International Journal of Testing, 2003; 3(4): 333-352. Haladyna, T.M. & Downing, S.M. A taxonomy of multiple-choice item writing rules. Applied Measurement in Education, 1989; 2: 37-50. Haladyna, T.M., Downing, S.M. & Rodriguez, M.C. A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 2002; 15: 309-334. Haladyna, T.M. Developing and validating multiple-choice test items. New Jersey: Lawrence Erlbaum. 2004 Higgins, E. & Tatham, L. Assessing by multiple-choice question tests. ICT Education Resources. August 2003. Hanna, G.S. Incremental reliability and validity of multiple-choice tests with an answer-until- correct procedure. Journal of Educational Measurement, 1975; 12: 175-178. Jaradat, D. & Tollefson, N. The impact of alternative scoring procedures for multiple- choice items on test reliability, validity, and grading. Educational and Psychological Measurement, 1988; 48: 627-635. Raffeld, P. The effects of Guttman weights on the reliability and predictive validity of objective tests when omissions are not differentially weighted. Journal of Educational Measurement, 1975; 12: 179-185. Rodriguez, M.C. Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice. Summer 2005. Shuford, E.H. Jr., Albert, A. & Massengill, H.E. Admissible probability measurement procedures. Psychometrika, 1966; 31:125-145. Tarrant, M., Ware, J., & Mohammed, A.M. An assessment of functioning and non- functioning distractors in multiple-choice questions: A descriptive analysis. BMC Medical Education, 2009: 9(40). Tigelaar, D.E.H., van der Vleuten, C.P.M. Assessment of Professional Competence. In: Billett S., Harteis C., Gruber H. (eds) International Handbook of Research in Professional and Practice-based Learning. Springer International Handbooks of Education. Springer, Dordrecht. 2014 Ventouras, E., Triantis, D. & Stergiopoulos, C. Comparison of examination methods based on multiple-choice and constructed-response questions using personal computers. Computers and Education, 2010; 54(2): 455-461. Trigg, M., Barnard, J.J., Devitt, P. & Pham, H. Comparing examination performance across three online assessment formats in Australian medical students. International Journal of research and Current Development 2016; 1(1), 11-16. Zoanetti, N., Beaves, M., Griffen, P. & Wallace, E.M. Fixed or mixed: A comparison of three, four and mixed-option multiple-choice tests in a Fetal Surveillance Education Program. BMC Medical Education, 2013; 13 (35).

More Articles from INTERNATIONAL JOURNAL OF HEALTH AND PHARMACEUTICAL RESEARCH

Antibacterial Activity of Stem Bark Extract of Boswellia Odorata Against Some Isolates of Wound Among Patients Attending Selected Hospitals in Kano Metropolis

Author: Kamal Abdulkadir Muhammad, Ismaila Ahmed, Lawan Danjuma, Raliya Sulaiman, Dauda, Isah Musa Bebeji, Zubairu Sani Ibrahim, Yahaya Ubah Yau, Binta Kabir, Muhammad, and Mukhtar Aminu Bala, Corresponding author

Prevalence and Prevention of Health Problems Among Inmates in Agodi Gate Correctional Centre, Ibadan

Author: John Betiku, a, Oluwabunmi Hannah Aremo, b, Emmanuel Olusegun Abe, a, Rachael, Omotomilayo Ajayi, c, Adenike Kaosarat Alabi, d, Kelechi Princess John, d, orcid ---