Comparison of CAT Procedures at Low Ability Levels: A Simulation Study and Analysis in the Context of Students with Disabilities

Abstract

The estimation of extreme abilities in computerized adaptive testing (CAT) is more biased and less accurate than that of intermediate abilities. This situation contradicts the structure of CAT, which targets all ability levels. This research aims to determine the procedures that perform better at lower skill levels, in accordance with other ability levels, by comparing the performances of various CAT procedures. In addition, a large-scale test examined whether the determined procedures would show similar performance in the ability levels of students with disabilities, as a group unfortunately more often of extreme abilities and that CAT will offer advantages in many respects. A pool of 1000 items and 1000 examinees with standard normal ability distribution were simulated with Monte Carlo. The CAT performances of 36 conditions consisting of different item selection methods, ability estimation methods and termination rules were compared. As a result of the research, the precision criterion termination rule used together with the maximum likelihood ability estimation method, Kullbak-Leibler information item selection rule, and precision criterion termination rule with test length limit (20 items) performed better and more consistently in terms of CAT performance across the ability levels. These procedures show high performance in the ability levels of students with disabilities, also in real data.