APPLICATION OF RASCH MODEL IN DEVELOPING COMPUTERIZED ADAPTIVE TESTING ITEM BANK

Abstract

Background: The item bank and adaptive testing algorithm are the two most important components of a Computerized Adaptive Testing (CAT) system. The measurement quality of adaptive testing depends on the quality of the calibrated set of test items included in the assessment. This study focuses on evaluating the functionality and effectiveness of the UMPItemBank to improve the UMPCAT adaptive testing system.

Objective: To evaluate the functionality and operational efficiency of the UMPItemBank software, which utilizes the Rasch model to develop a CAT item bank for assessing initial knowledge levels of learners.

Methods: UMPItemBank software is primarily designed for question management and test generation. It is developed using PHP programming language and MySQL database, operating on Windows 11 with hybrid cloud storage. Security is ensured through OAuth 2.0 authentication and authorization, and WAF firewall. All item content were developed based on constructivist learning theory, undergo a rigorous review and pilot testing process, integrated with the Rasch model for CAT. Processing formative evaluation and case study methods are employed to provide continuous feedback in real-world contexts for software research and development. The data were managed and analyzed using Excel MS 365, JASP 0.19.1, and Heuristic Lab Optimizer 3.3.16.1786.

Results: The study indicated that the item bank had operated as designed, effective and compatible with the CAT algorithm, providing evidence of content validity, bank size and format variety, fairness, cost-effectiveness, feasibility, and potential applications in testing management and computer adaptive testing. However, there were some issues to be addressed, such as threshold control of standard error of measurement, item exposure rate, differential item functioning analysis, and user interface.

Conclusion: The research results indicate the effectiveness and efficiency of the UMPItemBank algorithm as well as some suggestions for improving the quality of the item bank.

Keywords: item bank; formative evaluation; computer adaptive testing; Rasch model; item response theory; genetic algorithm; symbolic regression