Person fit analysis (38)
Chair: Wim van der Linden, Tuesday 21st July, 10.50 - 12.10, Uppercroft, School of Pythagoras.
Sebastien Beland, Gilles Raiche, Patricia Brassard Dorantes, Universite du Quebec a Montreal, Canada,
David Magis, KU Leuven, Belgium. Correlates of personal fluctuation, pseudo-guessing, and inattention subject parameters: Relations with parametric and non-parametric person-fit statistics. (025)
Gilles Raiche, Departement d’education et pedagogie, Universite du Quebec a Montreal, Canada. David Magis, Katholieke Universiteit Leuven, Belgium, Jean-Guy Blais, Universite de Montreal, Canada. Multidimensional fluctuation, pseudo-guessing and carelessness IRT person parameters models. (067)
Judith M. Conijn, Marcel A.L.M. van Assen, Wilco H.M. Emons and Klaas Sijtsma, Department of Methodology and Statistics, Tilburg, University, The Netherlands. Person-fit analysis using multilevel logistic regression. (162)
Wim J. van der Linden, CTB/McGraw Hill, Monterey CA USA. A bivariate lognormal response-time model for the detection of collusion between test takers. (015)
ABSTRACTS
Correlates of personal fluctuation, pseudo-guessing, and inattention subject parameters: Relations with parametric and non-parametric person-fit statistics. (025)
Sebastien Beland, Gilles Raiche, Patricia Brassard Dorantes and David Magis
Several methods have been proposed to detect examinees who present inappropriate test behaviour like cheating or guessing. Since the 1970’s, the use of Person-fit statistics seems to be one of the most interesting alternative since they have a high detection rate. Raiche (2008) proposed multidimensional item response models adding new person parameters (fluctuation, pseudo-guessing and inattention) to the level of proficiency of the candidate. Because these models are more realistic, we simulate 1000 responses sets under 4 test lengths (5,10,20,40). Then, we compute the canonical correlations between the person parameters and some parametric and non-parametric person-fit statistics. In this study, we obtained two results. First, the discrimination person parameters are strongly related with all the person-fit statistics. Second, the pseudo-guessing and the inattention person parameters represent new aspects and can be considered as new person-fit statistics.
Multidimensional fluctuation, pseudo-guessing and carelessness IRT person parameters models. (067)
Gilles Raiche, David Magis and Jean-Guy Blais
Multidimensional item response models adding new person parameters to the level of proficiency of the candidate are proposed. In the spirit of Trabin and Weiss person response curves, like the discrimination, pseudo-guessing and an eventual upper asymptote item parameters, these models offer fluctuation, pseudo-guessing and carelessness person parameters. There are cases where the candidates miss attention, motivation or preparation and show underachievement or overachievement. Their result does not correspond any more to their true potential, an inappropriate response pattern being obtained. These multidimensional models circumvent these situations and diminish considerably the associated proficiency level bias. Estimation methods, results from simulation showing the efficacy of these models and recommendations for the design of testing situations will be presented.
Person-fit analysis using multilevel logistic regression. (162)
Judith M. Conijn, Marcel A.L.M. van Assen, Wilco H.M. Emons and Klaas Sijtsma
Person-fit research concerns the detection of response patterns that do not fit the hypothesized item response theory (IRT) model or that deviate from the majority of response patterns in a sample. The person-response function (PRF) models the probability of a correct response as a function of the item difficulties. Reise (2000) proposed to use the slope parameter of the PRF as an indicator of person fit and to estimate the PRF by means of multilevel logistic regression. An advantage of the multilevel logistic regression approach of Reise is that person fit can be related to individual characteristics such as gender or age. This study focuses on the usefulness of multilevel logistic regression for estimating and interpreting PRFs in a person-fit analysis. First, we argue that using the slope of the PRF as an indicator of person misfit may be problematic. Second, we show that one of the assumptions of the multilevel logistic regression model is incompatible with the PRF approach. Third, we evaluate by means of a simulation study how seriously parameter estimates are biased when multilevel logistic regression is used to estimate the PRF. Finally, we discuss the implications of our results for using multilevel logistic regression in the context of person-fit analysis.
A bivariate lognormal response-time model for the detection of collusion between test takers. (015)
Wim J. van der Linden
A bivariate lognormal model for the distribution of the response times on a test by a pair of test takers is presented. As the model has parameters for the item effects on the response times, its correlation parameter automatically corrects for the spuriousness in the observed correlation between the response times of different test takers due to variation in the time intensities of the items. This feature suggests using the model in a routine check of response-time patterns for possible collusion between test takers using an estimate of the correlation parameter or a statistical test of an hypothesis about it. Closed-form expressions for the MLEs of the model parameters and a Lagrange multiplier test for the correlation parameter are presented. As in any type of statistical decision making, results from such procedures should be corroborated by evidence from other sources, e.g., results from a response-based analysis or observations during the test session. The effectiveness of the model in removing the spuriousness from correlated response times is illustrated using empirical response-time data.