Yes/No Angoff Standard Setting on an English Proficiency Test: Setting Cut-Off Scores and Analyzing Standard Setting Validation

Abstract

This research aimed to determine cut-off scores based on the CEFR framework levels A2, B1, and B2 using the Yes/No Angoff method for an English proficiency test developed by the English Department at Chiang Mai Rajabhat University, and to analyze the validation of the standard setting and the determined cut-off scores. There were ten judges in this study. The research instruments included an English proficiency test with a total score of 100, a CEFR standard comprehension test, a survey of judges’ opinions on the pre-standard setting training, the Yes/No Angoff evaluation form, and a survey of judges’ opinions on the standard setting process. The research findings indicated that the cut-off scores for the aforementioned English proficiency test, based on the CEFR framework, were set at 26 points for level A2, 64 points for level B1, and 84 points for level B2.The internal validation of the standard setting process and the assigned cut-off scores was verified through the Intraclass Correlation Coefficient (ICC) calculations, showing results of .92 for A2, .92 for B1, and .81 for B2.External validation, on the other hand, was examined using three methods: the first method involved focus group discussions on the reasonableness of the cut-off scores with representatives of curriculum administers and lecturers, the second method compared the grouping of test takers according to the CEFR levels by an experienced lecturer with the grouping based on the determined cut-off scores (Pearson correlation = .63), and the third method assessed the procedural validation by asking judges for their opinions on the standard setting process they participated in. It was found that the judges strongly agreed (mean = 3.78) with the overall standard setting process.