Exploring the effects of small item pools on examinee achievement estimates for computer-adaptive tests: a simulation study

Abstract

We explored the practicality of relatively small item pools in the context of low-stakes Computer-Adaptive Testing (CAT), such as CAT procedures that might be used for quick diagnostic or screening exams. We used a basic CAT algorithm without content balancing and exposure control restrictions to reflect low stakes testing scenarios. We examined the effects of small item pools under various testing conditions using a series of Monte Carlo simulations. We examined the effects of these conditions on the accuracy, precision, and stability of examinee achievement estimates. Our results showed that the effects of item pool size are strongest when there is less-precise targeting between item and person location parameters. Our findings suggest that small item pools can effectively support satisfactory performance in CAT, particularly when examinee targeting is adequate and a variable-length test with a standard error stopping rule is implemented. We consider implications for research and practice.