Association between candidate total scores and response pattern in script concordance testing of medical students

Authors

  • Siu Hong Wan School of Medicine, University of Notre Dame. Sydney. Australia
  • P Duggan University of Adelaide
  • E Tor University of Notre Dame
  • J N Hudson University of Adelaide

DOI:

https://doi.org/10.11157/fohpe.v18i2.145

Keywords:

medical education, script concordance, clinical reasoning, assessment, test taking strategies

Abstract

Introduction: The script concordance test (SCT) aims to test clinical decision making and clinical reasoning. This study is a preliminary attempt to understand an alleged test-taking strategy where students avoid extreme response options, potentially threatening the validity of SCT scores. We investigated whether there is a significant association between the propensity to avoid the extreme response options and candidates’ overall SCT scores.

Methods: The SCT scores of 660 clinical-year medical students (six cohorts from 2013–2015) were analysed for a possible association with candidates’ response pattern. The proportion of middle range response options was calculated. Propensity to avoid extreme response options is defined as a response pattern with 15% or more of middle-range responses compared to those of the expert reference panel. The distribution for candidates with propensity to avoid the extreme options was further investigated using chi-square statistics for possible association with their overall SCT results.

Results: Fifty-five percent of the students from the lowest quartile, compared to 30% from the top quartile, had shown a propensity to avoid the extreme options. The differences were statistically significant (p < 0.001) and were consistent among all six cohorts included in this study.

Conclusions: Students whose SCT scores are in the lowest quartile are more likely to avoid the extreme response options in answering SCT questions. For quality assurance in high stakes summative SCTs, it may be worthwhile to select items with expert reference panel’s modal answers covering the full 5-point response options

Author Biography

Siu Hong Wan, School of Medicine, University of Notre Dame. Sydney. Australia

Head of Basic and Clinical Science (BCS) Domain

Associate Professor, Assessment. Medical Education Unit

School of Medicine, Sydney

The University of Notre Dame Australia

References

Boulouffe, C., Doucet, B., Muschart, X., Charlin, B., & Vanpee, D. (2014). Assessing clinical reasoning using a script concordance test with electrocardiogram in an emergency medicine clerkship rotation. EMERGENCY MEDICINE JOURNAL, 31(4), 313-316. doi:10.1136/emermed-2012-201737

Brazeau-Lamontagne, L., Charlin, B., Gagnon, R., Samson, L., & van der Vleuten, C. (2004). Measurement of perception and interpretation skills during radiology training: utility of the script concordance approach. Medical Teacher, 26(4), 326-332. doi:10.1080/01421590410001679000

Carrière, B. (2009). Assessing clinical reasoning in pediatric emergency medicine: validity evidence for a Script Concordance Test. Ann Emerg Med, 53(5), 647-652. doi:10.1016/j.annemergmed.2008.07.024

Chang, T. P., Kessler, D., McAninch, B., Fein, D. M., Scherzer, D. J., Seelbach, E., . . . Education, N. (2014). Script Concordance Testing: Assessing Residents’ Clinical Decision-Making Skills for Infant Lumbar Punctures. Academic Medicine, 89(1), 128-135. doi:10.1097/ACM.0000000000000059

Charlin, B., Roy, L., Brailovsky, C., Goulet, F., & van der Vleuten, C. (2000). The Script Concordance Test: A Tool to Assess the Reflective Clinician. TEACHING AND LEARNING IN MEDICINE, 12(4), 189-195. doi:10.1207/S15328015TLM1204_5

Cullen, M. J., Sackett, P. R., & Lievens, F. (2006). Threats to the Operational Use of Situational Judgment Tests in the College Admission Process. International Journal of Selection and Assessment, 14(2), 142-155. doi:10.1111/j.1468-2389.2006.00340.x

Dawson, T., Comer, L., Kossick, M. A., & Neubrander, J. (2014). Can Script Concordance Testing Be Used in Nursing Education to Accurately Assess Clinical Reasoning Skills? Journal of Nursing Education, 53(5), 281-286. doi:10.3928/0148434-20140321-03

Dory, V., Gagnon, R., Vanpee, D., & Charlin, B. (2012). How to construct and implement script concordance tests: insights from a systematic review. Medical Education, 46(6), 552-563. doi:10.1111/j.1365-2923.2011.04211.x

Drolet, P. (2015). Assessing clinical reasoning in anesthesiology: Making the case for the Script Concordance Test. Anaesthesia Critical Care & Pain Medicine, 34(1), 5-7. doi:10.1016/j.accpm.2015.01.003

Ducos, G., Lejus, C., Sztark, F., Nathan, N., Fourcade, O., Tack, I., . . . Minville, V. (2015). The Script Concordance Test in anesthesiology: Validation of a new tool for assessing clinical reasoning. Anaesthesia Critical Care & Pain Medicine, 34(1), 11-15. doi:10.1016/j.accpm.2014.11.001

Duggan, P., & Charlin, B. (2012). Summative assessment of 5th year medical students' clinical reasoning by script concordance test: requirements and challenges. BMC MEDICAL EDUCATION, 12(1), 29-29. doi:10.1186/1472-6920-12-29

Foucault, A., Dubé, S., Fernandez, N., Gagnon, R., & Charlin, B. (2015). Learning medical professionalism with the online concordance-of-judgment learning tool (CJLT): A pilot study. Medical Teacher, 37(10), 955.

Irfannuddin, I. (2009). Knowledge and critical thinking skills increase clinical reasoning ability in urogenital disorders: a Universitas Sriwijaya Medical Faculty experience. Medical Journal of Indonesia, 18(1), 53-59. doi:10.13181/mji.v18i1.341

Kow, N., Walters, M. D., Karram, M. M., Sarsotti, C. J., & Jelovsek, J. E. (2014). Assessing intraoperative judgment using script concordance testing through the gynecology continuum of practice. Medical Teacher, 36(8), 724-729. doi:10.3109/0142159X.2014.910297

Lambert, C., Gagnon, R., Nguyen, D., & Charlin, B. (2009). The script concordance test in radiation oncology: Validation study of a new tool to assess clinical reasoning. Radiation Oncology, 4(1), 7-7. doi:10.1186/1748-717X-4-7

Lineberry, M., Kreiter, C. D., & Bordage, G. (2013). Threats to validity in the use and interpretation of script concordance test scores. Medical Education, 47(12), 1175-1183.

doi:10.1111/medu.12283

Lubarsky, S., Vleuten, C. P. M. v. d., Charlin, B., Chalk, C., & Cook, D. A. (2011). Script concordance testing: a review of published validity evidence. Medical Education, 45(4), 329-338. doi:10.1111/j.1365-2923.2010.03863.x

McDaniel, M. A., Psotka, J., Legree, P. J., Yost, A. P., & Weekley, J. A. (2011). Toward an understanding of situational judgment item validity and group differences. The Journal of applied psychology, 96(2), 327-336. doi:10.1037/a0021983

Nouh, T., Boutros, M., Gagnon, R., Reid, S., Leslie, K., Pace, D., . . . Meterissian, S. H. (2012). The script concordance test as a measure of clinical reasoning: A national validation study. American Journal of Surgery, 203(4), 530-534. doi:10.1016/j.amjsurg.2011.11.006

Palmer, E. J., Duggan, P., Devitt, P. G., & Russell, R. (2010). The modified essay question: Its exit from the exit examination? Medical Teacher, 32(7), e300-e307. doi:10.3109/0142159X.2010.488705

See, K. C., Tan, K. L., & Lim, T. K. (2014). The script concordance test for clinical reasoning: re-examining its utility and potential weakness. Medical Education, 48(11), 1069-1077. doi:10.1111/medu.12514

Tan, K., Tan, N., Kandiah, N., Samarasekera, D., & Ponnamperuma, G. (2011). A Script Concordance Test for Neurological Localization and Emergencies. NEUROLOGY, 76(9), A268-A268.

Tsai, T.-C., Chen, D.-F., & Lei, S.-M. (2012). The ethics script concordance test in assessing ethical reasoning: really good stuff. Medical Education, 46(5), 527-527. doi:10.1111/j.1365-2923.2012.04252.x

Wan, S. H. (2014). Using Script Concordance Testing (SCT) to Assess Clinical Reasoning- The Progression from Novice to Practising General Practitioner. Medical Education, 48(2), 6.

Wan, S. H. (2015). Using the script concordance test to assess clinical reasoning skills in undergraduate and postgraduate medicine. HONG KONG MEDICAL JOURNAL, 21(5), 455-461. doi:10.12809/hkmj154572

Downloads

Published

2017-08-04

How to Cite

Wan, S. H., Duggan, P., Tor, E., & Hudson, J. N. (2017). Association between candidate total scores and response pattern in script concordance testing of medical students. Focus on Health Professional Education: A Multi-Professional Journal, 18(2), 26–35. https://doi.org/10.11157/fohpe.v18i2.145

Issue

Section

Articles