Association between candidate total scores and response pattern in script concordance testing of medical students
DOI:
https://doi.org/10.11157/fohpe.v18i2.145Keywords:
medical education, script concordance, clinical reasoning, assessment, test taking strategiesAbstract
Introduction: The script concordance test (SCT) aims to test clinical decision making and clinical reasoning. This study is a preliminary attempt to understand an alleged test-taking strategy where students avoid extreme response options, potentially threatening the validity of SCT scores. We investigated whether there is a significant association between the propensity to avoid the extreme response options and candidates’ overall SCT scores.
Methods: The SCT scores of 660 clinical-year medical students (six cohorts from 2013–2015) were analysed for a possible association with candidates’ response pattern. The proportion of middle range response options was calculated. Propensity to avoid extreme response options is defined as a response pattern with 15% or more of middle-range responses compared to those of the expert reference panel. The distribution for candidates with propensity to avoid the extreme options was further investigated using chi-square statistics for possible association with their overall SCT results.
Results: Fifty-five percent of the students from the lowest quartile, compared to 30% from the top quartile, had shown a propensity to avoid the extreme options. The differences were statistically significant (p < 0.001) and were consistent among all six cohorts included in this study.
Conclusions: Students whose SCT scores are in the lowest quartile are more likely to avoid the extreme response options in answering SCT questions. For quality assurance in high stakes summative SCTs, it may be worthwhile to select items with expert reference panel’s modal answers covering the full 5-point response options
References
Boulouffe, C., Doucet, B., Muschart, X., Charlin, B., & Vanpee, D. (2014). Assessing clinical reasoning using a script concordance test with electrocardiogram in an emergency medicine clerkship rotation. EMERGENCY MEDICINE JOURNAL, 31(4), 313-316. doi:10.1136/emermed-2012-201737
Brazeau-Lamontagne, L., Charlin, B., Gagnon, R., Samson, L., & van der Vleuten, C. (2004). Measurement of perception and interpretation skills during radiology training: utility of the script concordance approach. Medical Teacher, 26(4), 326-332. doi:10.1080/01421590410001679000
Carrière, B. (2009). Assessing clinical reasoning in pediatric emergency medicine: validity evidence for a Script Concordance Test. Ann Emerg Med, 53(5), 647-652. doi:10.1016/j.annemergmed.2008.07.024
Chang, T. P., Kessler, D., McAninch, B., Fein, D. M., Scherzer, D. J., Seelbach, E., . . . Education, N. (2014). Script Concordance Testing: Assessing Residents’ Clinical Decision-Making Skills for Infant Lumbar Punctures. Academic Medicine, 89(1), 128-135. doi:10.1097/ACM.0000000000000059
Charlin, B., Roy, L., Brailovsky, C., Goulet, F., & van der Vleuten, C. (2000). The Script Concordance Test: A Tool to Assess the Reflective Clinician. TEACHING AND LEARNING IN MEDICINE, 12(4), 189-195. doi:10.1207/S15328015TLM1204_5
Cullen, M. J., Sackett, P. R., & Lievens, F. (2006). Threats to the Operational Use of Situational Judgment Tests in the College Admission Process. International Journal of Selection and Assessment, 14(2), 142-155. doi:10.1111/j.1468-2389.2006.00340.x
Dawson, T., Comer, L., Kossick, M. A., & Neubrander, J. (2014). Can Script Concordance Testing Be Used in Nursing Education to Accurately Assess Clinical Reasoning Skills? Journal of Nursing Education, 53(5), 281-286. doi:10.3928/0148434-20140321-03
Dory, V., Gagnon, R., Vanpee, D., & Charlin, B. (2012). How to construct and implement script concordance tests: insights from a systematic review. Medical Education, 46(6), 552-563. doi:10.1111/j.1365-2923.2011.04211.x
Drolet, P. (2015). Assessing clinical reasoning in anesthesiology: Making the case for the Script Concordance Test. Anaesthesia Critical Care & Pain Medicine, 34(1), 5-7. doi:10.1016/j.accpm.2015.01.003
Ducos, G., Lejus, C., Sztark, F., Nathan, N., Fourcade, O., Tack, I., . . . Minville, V. (2015). The Script Concordance Test in anesthesiology: Validation of a new tool for assessing clinical reasoning. Anaesthesia Critical Care & Pain Medicine, 34(1), 11-15. doi:10.1016/j.accpm.2014.11.001
Duggan, P., & Charlin, B. (2012). Summative assessment of 5th year medical students' clinical reasoning by script concordance test: requirements and challenges. BMC MEDICAL EDUCATION, 12(1), 29-29. doi:10.1186/1472-6920-12-29
Foucault, A., Dubé, S., Fernandez, N., Gagnon, R., & Charlin, B. (2015). Learning medical professionalism with the online concordance-of-judgment learning tool (CJLT): A pilot study. Medical Teacher, 37(10), 955.
Irfannuddin, I. (2009). Knowledge and critical thinking skills increase clinical reasoning ability in urogenital disorders: a Universitas Sriwijaya Medical Faculty experience. Medical Journal of Indonesia, 18(1), 53-59. doi:10.13181/mji.v18i1.341
Kow, N., Walters, M. D., Karram, M. M., Sarsotti, C. J., & Jelovsek, J. E. (2014). Assessing intraoperative judgment using script concordance testing through the gynecology continuum of practice. Medical Teacher, 36(8), 724-729. doi:10.3109/0142159X.2014.910297
Lambert, C., Gagnon, R., Nguyen, D., & Charlin, B. (2009). The script concordance test in radiation oncology: Validation study of a new tool to assess clinical reasoning. Radiation Oncology, 4(1), 7-7. doi:10.1186/1748-717X-4-7
Lineberry, M., Kreiter, C. D., & Bordage, G. (2013). Threats to validity in the use and interpretation of script concordance test scores. Medical Education, 47(12), 1175-1183.
doi:10.1111/medu.12283
Lubarsky, S., Vleuten, C. P. M. v. d., Charlin, B., Chalk, C., & Cook, D. A. (2011). Script concordance testing: a review of published validity evidence. Medical Education, 45(4), 329-338. doi:10.1111/j.1365-2923.2010.03863.x
McDaniel, M. A., Psotka, J., Legree, P. J., Yost, A. P., & Weekley, J. A. (2011). Toward an understanding of situational judgment item validity and group differences. The Journal of applied psychology, 96(2), 327-336. doi:10.1037/a0021983
Nouh, T., Boutros, M., Gagnon, R., Reid, S., Leslie, K., Pace, D., . . . Meterissian, S. H. (2012). The script concordance test as a measure of clinical reasoning: A national validation study. American Journal of Surgery, 203(4), 530-534. doi:10.1016/j.amjsurg.2011.11.006
Palmer, E. J., Duggan, P., Devitt, P. G., & Russell, R. (2010). The modified essay question: Its exit from the exit examination? Medical Teacher, 32(7), e300-e307. doi:10.3109/0142159X.2010.488705
See, K. C., Tan, K. L., & Lim, T. K. (2014). The script concordance test for clinical reasoning: re-examining its utility and potential weakness. Medical Education, 48(11), 1069-1077. doi:10.1111/medu.12514
Tan, K., Tan, N., Kandiah, N., Samarasekera, D., & Ponnamperuma, G. (2011). A Script Concordance Test for Neurological Localization and Emergencies. NEUROLOGY, 76(9), A268-A268.
Tsai, T.-C., Chen, D.-F., & Lei, S.-M. (2012). The ethics script concordance test in assessing ethical reasoning: really good stuff. Medical Education, 46(5), 527-527. doi:10.1111/j.1365-2923.2012.04252.x
Wan, S. H. (2014). Using Script Concordance Testing (SCT) to Assess Clinical Reasoning- The Progression from Novice to Practising General Practitioner. Medical Education, 48(2), 6.
Wan, S. H. (2015). Using the script concordance test to assess clinical reasoning skills in undergraduate and postgraduate medicine. HONG KONG MEDICAL JOURNAL, 21(5), 455-461. doi:10.12809/hkmj154572
Downloads
Published
How to Cite
Issue
Section
License
On acceptance for publication in FoHPE the copyright of the manuscript is signed over to ANZAHPE, the publisher of FoHPE.
