Computer Adaptive Testing: Methodology (09)

Chair: Siu-Ying Chen, Thursday 23rd July, 9.40 - 11.00, Lowercroft, School of Pythagoras. 

Shu-Ying Chen, Department of Psychology, National Chung-Cheng University, Taiwan. A new procedure for controlling test overlap in Computerized Adaptive Testing. (010)

Hsuan-Jui Shih and Wen-Chung Wang, National Chung Cheng University, Taiwan. Computerized Adaptive Testing Using the two parameter logistic model with ability-based guessing. (242)

Matthieu Brinkhuis and Gunter Maris, Cito, Dutch National Institute for Educational Measurement, Arnhem, The Netherlands. Dynamic estimation. (203)

ABSTRACTS

A new procedure for controlling test overlap in Computerized Adaptive Testing. (010)
Shu-Ying Chen
To date, exposure control procedures that are designed to control test overlap in computerized adaptive tests (CATs) are generated based on the assumption of item sharing between pairs of examinees. However, examinees may obtain test information from more than one previous test taker in practice. This larger scope of information sharing needs to be taken into account in refining exposure control procedures. To control item exposure and test overlap among a group of examinees larger than two (denoted as general test overlap), Chen and Lei (in press) provided some useful directions. In their study, the relationships between item exposure rate and general test overlap rate have been investigated thoroughly. The derived relationships could be applied to develop procedures to control item exposure and general test overlap simultaneously. The purpose of this study is to propose a new test overlap control method—the SHGT procedure based on the relationships derived in Chen and Lei (in press). The SHGT procedure is an extension of the Sympson and Hetter (1985) procedure and designed to control the proportion of overlapping items encountered by an examinee with a group of examinees who have previously taken the test (general test overlap rate). A simulation study was conducted to investigate the effects of the SHGT procedure on test overlap control and measurement precision. Results indicated that item exposure rate and general test overlap rate could be simultaneously controlled by implementing the SHGT procedure. In addition, these two indices were controlled on the fly without any iterative simulations conducted prior to operational CATs. Thus, the SHGT procedure would be an efficient procedure for improving test security in CATs.

Computerized Adaptive Testing Using the two parameter logistic model with ability-based guessing. (242)
Hsuan-Jui Shih and Wen-Chung Wang
The two-parameter logistic model with ability-based guessing (2PL-AG) model was proposed in this study, which is extended from the 1PL-AG model which not only incorporate the chances of a correct guess being dependent on ability and difficulty (Martín, del Pino, & De Boeck, 2006), but comprise a discrimination parameter in the items. To estimate the parameters of the model, Markov chain Monte Carlo (MCMC) method was used in this study. Furthermore, a computerized adaptive testing (CAT) algorithm based on the 2PL-AG model (called 2PL-AG-CAT) was constructed. Through Monte Carlo simulation study, the performance on latent trait estimation of 2PL-AG-CAT was compared to that based on 1PL-AG model (called 1PL-AG-CAT) and 3PL model (called 3PL -CAT). The results indicated that the 2PL-AG was most efficient among these models in term of slightly lower bias and root mean square error (RMSE).

Dynamic estimation. (203)
Matthieu Brinkhuis and Gunter Maris
Point estimation is eminently suited in situations where parameters remain constant over time. If however, for one reason or another, parameters do not remain constant over time, point estimation does a poor job at detecting and adapting to changes. For such situations, an alternative estimation method will be considered here. Specifically, Markov chain Monte Carlo methods will be employed to construct Markov chains with known invariant distributions about the true values of the parameters. A new state for the Markov chain is constructed given the current state added with new information. New information can for instance be the response to an additional item. This approach to parameter estimation has a long tradition in the field of measurement of chess expertise, where the Elo rating system is widely used. The dynamic estimation methods introduced here are closely related to the Elo rating system, but allow for more statistical control. Such methods have applications in the context of monitoring growth of students over time, and in the detection of item drift. Both types of applications will be illustrated with real data.