Credible and defensible assessment of entry-level clinical competence: Insights from a modified Delphi study

Rachel Bacon; Lauren Williams; Laurie Grealish; Maggie Jamieson

doi:10.11157/fohpe.v16i3.86

Authors

Rachel Bacon University of Canberra
Lauren Williams Griffith University
Laurie Grealish Griffith University
Maggie Jamieson University of Canberra

DOI:

https://doi.org/10.11157/fohpe.v16i3.86

Keywords:

assessment, clinical placements, Delphi study, competence

Abstract

Background: During clinical placements, supervisors repeatedly assess health and medical students for competence. Quality assessment is dependent upon the supervisors having a rich understanding of entry-level standards and an assessment approach that is sufficiently dynamic to accommodate the changing healthcare system. This study aimed to assess whether consensus could be gained by supervisors when assessing the performance of student dietitians during clinical placements and to establish a shared interpretation of entry-level clinical competence.

Methods: A modified 3-round Delphi study with a focus group discussion was conducted with eight supervisors. Participants were required to assess the performance of student dietitians from audiovisual recordings of authentic student‒client consultations in aged-care and outpatient settings.

Results: Consensus was achieved for 2/11 assessments after one Delphi round, 6/11 assessments after two rounds and 10/11 assessments after the third and final round. During the focus group discussion, the expert panel expressed a shared understanding of entry-level performance, however this was not transferred into a shared assessment of entry-level performance in the Delphi task.

Conclusions: Dialogue amongst supervisors leads to a more reliable interpretation of the competency standards. A shared responsibility for assessment, with continuous and open negotiation of meaning, is required to ensure quality assessments of entry- level practice.

References

Albanese, M. (2000). Challenges in using rating judgements in medical education. Journal of Evaluation of Clinical Practice, 6(3), 305‒319.

Academy of Nutrition and Dietetics. (2013). Revised 2012 standards of practice in nutrition care and standards of professional performance for registered dietitians. Journal of the Academy of Nutrition & Dietetics, 113(6), s29‒s45.

Ash, S. (1995). The development of an assessment strategy related to the Dietitian’s National Competency Standards. Canberra, Australia: Dieticians Association of Australia.

Ash, S., Dowding, K., & Phillips, S. (2010). Mixed methods research approach to the development and review of competency standards for dietitians. Nutrition & Dietetics, 68(4), 305‒315.

Ash, S., & Phillips, S. (2000). What is dietetic competence? Competency standards, competence and competency explained. Australian Journal of Nutrition & Dietetics, 57(3), 147‒151.

Bacon, R., Williams, L., & Grealish, L. (2015). Aged care facilities and primary health- care clinics provide appropriate settings for dietetic students to demonstrate individual case management clinical competence. Nutrition & Dietetics, 72(1), 54‒62.

Benner, P. (1984). From novice to expert: Excellence and power in clinical nursing practice. Menlo Park, CA: Addison-Wesley Publishing Company.

Bowles, N. (1999). The Delphi technique. Nursing Standard, 45(13), 32‒36.

Brace-Govan, J., Farrell, S., Joy, S., Luxton, S., & Davey, I. (2001). Delphi revisited: A concise method for industry consultation on curriculum. Australian New Zealand Journal of Vocational Education Research, 9(1), 1‒19.

Chapman, J. (1998). Agonising about assessment. In D. Fish & C. Colin (Eds.),

Developing professional judgement in health care: Learning through the critical appreciation of practice (pp. 157‒181). Boston, MA: Butterworth-Heineman.

Cox, K. (2000). Examining and recording clinical performance: A critique and some recommendations. Education and Health, 13(1), 44‒54.

Dalton, M. (2009). Development of the clinical assessment of physiotherapy skills (CAPS): A standardised and valid approach to assessment of clinical competence in physiotherapy. Brisbane, Australia: Australian Learning and Teaching Council.

Dalton, M., Keating, J., & Davidson, M. (2009). The assessment of physiotherapy (APA) instrument clinical educator resource manual. Melbourne, Australia: Australian Learning and Teaching Council.

Davies, J., & Clarke, D. (2004). Clinical skills acquisition in children’s nursing: An international perspective. Paediatric Nursing, 16(2), 23‒26.

Dietitians Association of Australia (DAA). (2011). Manual for accreditation of dietetics education programs Version 1.2 (October 2011). Canberra, Australia: Author.

Dietitians Association of Australia (DAA). (2009). National competency standards for entry-level dietitians in Australia. Canberra, Australia: Author.

Dreyfus, H., & Dreyfus, S. (1980). A five-stage model of the mental activities involved in direct skills acquisition. Washington, DC: Storming Media.

Epstein, R., & Hundert, E. (2002). Defining and assessing professional competence. Journal of the American Medical Association, 287(2), 226‒235.

Eraut, M. (1994). Developing professional knowledge and competence. London, England: The Falmer Press.

Govaerts, M., & van der Vleuten, C. (2013). Validity in work-based assessment: Expanding our horizons. Medical Education, 47, 1164‒1174.

Govaerts M, van der Vleuten C, & Schuwirth L. (2002). Optimising the reproducibility of a performance-based test in midwifery education. Advances in Health Science Education, 7, 133‒145.

Grealish, L. (2009). Crafting competence: The governance of multiplicity in nursing. Unpublished doctoral dissertation, University of Canberra—Canberra, Australia.

Hughes, R., & Desbrow, B. (2010). An evaluation of clinical dietetics student placement case-mix exposure, service delivery and supervisory burden. Nutrition & Dietetics, 67(4), 287‒293.

Johnsson, M., & Hager, P. (2008). Navigating the wilderness of becoming professional. Journal of Workplace Learning, 20(7), 526‒536.

Lennie, S., & Juwah, C. (2010). Exploring assessment for learning during dietetics practice placements. Nutrition & Dietetics, 23(3), 217‒223.

Levett-Jones, T., Gresbach, J., Arthur, C., & Roche, J. (2011). Implementing a clinical competency assessment model that promotes critical reflection and ensures nursing graduates’ readiness for professional practice. Nurse Education in Practice, 11(1), 64‒69.

Ling, P. (1999, July). Assessing competency. Paper presented at the HERDSA Annual International Conference, Melbourne, Australia.

National Health Workforce Taskforce. (2008). Data, capacity and clinical placements across Australia: A discussion paper. Canberra, Australia: Author.

McAllister, S., Lincoln, M., & Ferguson, A. (2010). Issues in developing valid assessments of speech pathology students’ performance in the workplace. International Journal of Language Communciation Disorders, 45(1), 1‒14.

McAllister, S., Lincoln, M., Ferguson, A., & McAllister, L. (2011). A systematic program of research: The assessment of speech-language pathology competencies. International Journal of Speech Language Pathology, 13(6), 469‒479.

McAllister, S., Lincoln, M., Ferguson, A., & McAllister, L. (2006). Competency assessment in speech pathology. Melbourne, Australia: Speech Pathology of Australia.

McKinley, R., Fraser, R., & Baker, R. (2001). Model of directly assessing and improving competence and performance in revalidation of clinicians. British Medical Journal, 322, 721‒714.

Palermo, C., Capra, S., Ash, S., Beck, E., Truby, H., & Jolly, B. (2014). Professional competence standards, learning outcomes and assessment: Designing a valid strategy for nutrition and dietetics. Sydney, Australia: Office for Learning and Teaching, Australian Government.

Powell, C. (2003). The Delphi technique: Myths and realities. Journal of Advanced Nursing, 41(4), 376‒382.

Schuwirth, L. V., & van der Vleuten, C. (2003). ABC of learning and teaching in medicine: Written assessments. British Medical Journal, 326, 643‒645.

van Manen, M. (1984). Practising phenomenological writing. Phenomenology & Pedagogy, 2(1), 36‒68.

von der Gracht, H. (2012). Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technological Forecasting and Social Change, 79(7), 1525‒1536.

Williams, P., & Beck, E. (2011). Final report on Simulated Learning Environment Project conducted for Health Workforce Australia. Canberra, Australia: Dietitians Association of Australia.

Yin, R. (2011). Qualitative research from start to finish. New York, NY: Guilford Press.