Inter-rater Reliability of a Clinical Practice Assessment Rubric for Physical Therapy Students: Mahidol University
Keywords:
Clinical practice training, Rubric assessment, Inter-rater reliability, Physical therapyAbstract
The rubric for clinical practice training is content-accurate, however a student can be assessed more than one clinical instructor by occasion. This may be affected to achieve the learning objectives and outcomes. It is thus this study aim to investigate inter-rater reliability of a clinical practice assessment rubric. The researcher compiled data of learners who were evaluated in competency by more than one clinical instructor during the same period, in the academic years 2020- 2022. The assessment rubric includes three aspects of competency: 1 item in the knowledge aspect, 10 items in the skill aspect, and 9 items in the professional behavior aspect, for statistical analysis and to consider the Intraclass Correlation Coefficient (ICC 2,2). From a dataset of 158 scores, good inter-rater reliability was observed in questions related to patient care standards (ICC = 0.91), self-improvement in knowledge (ICC = 0.83), and effective communication (ICC = 0.79). Moderate reliability was found in most skill-related items, such as hypothesis formulation (ICC = 0.73) and physical examination skills (ICC = 0.56). However, four professional behavior items and three skill-related items exhibited low reliability, indicating a need for further refinement. The agreement on conditions of assessment and the process of training evaluators needs to be reviewed to guarantee the rubric's reliability among clinical instructors. Additionally, the operational definition should be refined to provide reliable and objective competency assessments of physical therapy students, leading to more effective and consistent evaluations.
References
Andrade, H. G. (2005). Teaching with rubrics: The good, the bad, and the ugly. College Teaching, 53(1), 27–31.
Dawson, P. (2017). Assessment rubrics: towards clearer and more replicable design, research and practice. Assessment & Evaluation in Higher Education, 42(3), 347–360.
Jonson, A. S. G. (2007). The use of scoring rubrics: reliability, validity and educational consequences. Educational Research Review, 2, 130–144.
Julie, J. I., & Annette, S. S. (2009). Rubrics for clinical evaluation: Objectifying the subjective experience. Nurse Education in Practice, 9(2), 134–140.
Kohn, K. A. M. (1995). Grading practices used by clinical instructors in physical therapy in the United States. Physiotherapy Theory and Practice, 11(3), 175–181.
Kongoun, S., Sermpon, N., Suraprapapich, P., Nuntapornsak, A., Klomjai, W., & Vongsirinavarat, M. (2019). The Development of the Clinical Practice Rubric Assessment Form for Undergraduate Physical Therapy Students at Mahidol University. Journal of Research Methodology, 20(1).
Koo, T. K., & Li, M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, 15(2), 155–163.
Li, J., & Qian, W. (2021). Development and validation of a rating scale for summarization as an integrated task. Asian-Pacific Journal of Second and Foreign Language Education.
Malini, R. Y., & Heidi, A. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435–448.
O'Donnell, J. A., Oakley, M., Haney, S., O'Neill, P., & Taylor, D. (2011). Rubrics 101: A Primer for Rubric Development in Dental Education. Journal of Dental Education, 75(9), 1163–1175.
Smith, K., Simon, G., & Austin, M. (2016). The Use of Rubrics in the Clinical Evaluation of Podiatric Medical Students. Journal of the American Podiatric Medical Association, 106(1).
The Physical Therapy Profession Act, B.E. 2547. (2004).
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
