Computerized Adaptive Testing: Item selection (11)

Chair: Hua hua Chang, Friday 24th July, 9.30 - 10.50, Boys Smith Room, Fisher Building.

Hua-Hua Chang, Chun Wang and Keith Boughton, Department of Psychology, University of Illinois at Urbana-Champaign, USA. A Simplified KL Information Index (SKI) for multidimensional computerized adaptive tests. (104)

Marjan Bakker and Hua-Hua Chang, University of Amsterdam, The Netherlands, Constrained item selection in computerized classification testing. (197) ♥

Chia-Ling Hsu and Shu-Ying Chen, Department of Psychology, National Chung-Cheng University, Taiwan. Effects of practical constraints on item selection rules in computerized classification testing. (135)

Richard J. Swartz and Seung W. Choi, The University of Texas M.D. Anderson Cancer Center, Houston, USA. A greedy and burdened CAT: a method to include response burden in the minimum expected posterior variance item selection method for computerized adaptive tests. (237)

ABSTRACTS

A Simplified KL Information Index (SKI) for multidimensional computerized adaptive tests. (104)
Hua-Hua Chang, Chun Wang and Keith Boughton
This study proposed a simplified version of the Kullback-Leibler information index (SKI) for item selection in Multidimensional Computerized Adaptive Tests. In unidimensioanl IRT it is well known that KL information provides a global information measure and yields more robust estimation accuracy at early stages of adaptive tests (e.g., Chang and Ying, 1996). Veldkamp and van der Linden (2002) generalized the KL information index (KI) from unidimensioal to multidimensional applications and proposed to use a central symmetric domain in multiple integrations. However, the multiple integrations make KI computational intensive. Therefore, it is desirable to simplify the calculation without sacrificing estimation accuracy. Wang & Chang (2009) showed that the size of KI depends on the function of the item multidimensional discrimination. They also analytically demonstrated that the multi-dimensional KI reaches its maximum when the item difficulty equals to a certain linear combination of theta elements. According to the theoretical results, a simplified KL information index (SKI), which is expressed as a closed analytical form of the item parameters and the ability estimates, is proposed. It greatly reduces the computational intensity by avoiding the multiple integrations. In addition, to balance the item exposure rates on-the-fly, we also incorporate the continuous a-stratification index (CAI) into SKI. CAI evolves from the a-stratification idea, but enforces the item selection in such a way that a-parameters of the chosen item are in a strict ascending order item-by-item. In conclusion, the SKI includes three parts: the function of item discrimination, the function of item difficulty and CAI. A large-scale Simulation studies was conducted and the results favored SKI over KI in that SKI provided a better trade-off between estimation accuracy and exposure control, while maintained computationally easy.  

Constrained item selection in computerized classification testing. (197)
Marjan Bakker and Hua-Hua Chang
Examinees can be classified into different categories by use of computerized classification testing (CCT). Hereby, sequential probability ratio test (SPRT) is used to classify an examinee. However, non-statistical constraints must be considered too. Different constraints are for example, content balancing, answer key balancing, and conflicting items. A special constraint is exposure control. In this study different methods for constraint management and item exposure control, together with different item selection methods are compared. Both constraint handling methods, the maximum priority index (MPI) and the weighted deviation model (WDM), handle constraints well. MPI controls item exposure more adequate than the Sympson and Hetter (SH) method, but requires a few more items. When item exposure control is included, estimate based (EB) item selection is more efficient than cut-score based (CB) item selection. Especially MPI with EB Fisher information (FI) item selection manage constraints and controls item exposure in a simple and adequate way.  

Effects of practical constraints on item selection rules in computerized classification testing. (135)
Chia-Ling Hsu and Shu-Ying Chen
Instead of providing trait estimation, computerized classification testing (CCT) can classify examinees into categories efficiently. Two commonly used item selection rules in CCT are sequential probability ratio test (SPRT) with cut-score based (CB) item selection and sequential Bayes procedure (SBP) with trait estimate based (EB) item selection. According to previous research (Spray & Reckase, 1996; Thompson, 2008), the SPRT with CB item selection tended to require fewer items than SBP with EB item selection to achieve the same level of classification accuracy. The superior of the SPRT, however, was observed in an idealistic CCT setting in which information was the only criterion used in item selection. In more realistic CCT situations, where content balancing and item exposure control are criteria for item selection, in addition to information, the relationships between the two procedures may be different. The purpose of this study is to compare the effects of the two procedures under three conditions: (1) using only the item information as the item selection criterion; (2) using both the item information and content balancing; and (3) using the item information, content balancing, and item exposure control. Results indicated that SPRT tended to perform better than SBP with respect to test efficiency in condition 1 and condition 2. The differences between the two procedures in test efficiency, however, were negligible when item exposure control was taken into account in item selection.  

A greedy and burdened CAT: a method to include response burden in the minimum expected posterior variance item selection method for computerized adaptive tests. (237)
Richard J. Swartz and Seung W. Choi
Computerized adaptive testing (CAT) adaptively selects and administers items to estimate a person’s unknown latent trait. Many current CAT item selection algorithms myopically select the best item to administer next given the response history. This allows for shorter and more precise tests.Although successful in reducing response burden (the length of the test), many CAT algorithms use ad hoc rules – such as limiting the maximum number of items to be administered – instead of formally considering response burden in the item selection process. This study proposes a Bayesian decision theoretic approach using loss functions to develop an item selection method that extends the minimum expected posterior variance item selection method to incorporate burden (MEPV-b). The MEPV-b using varying degrees of burden was compared to the standard MEPV method. Both real and simulated response data from an item bank of 62 polytomous items measuring depressive symptoms were used to compare the different methods. The MEPV-b algorithm for item selection results in tests that are on average shorter than those obtained from the standard MEPV method without severely affecting the standard error of measurement. This is very useful in assessment settings where burden is a concern.