Methodological potpourri (49)
Chair: Marie Wiberg, Wednesday 22nd July, 11.45 - 13.05, Dirac Room, Fisher Building.
Marieke van Onna and Anton Béguin, Psychometric Research Center, Cito, Arnhem, The Netherlands. Optimizing the measurement accuracy of a test using correlated scales. (075)
Hsiu-Ting Yu, Institute of Psychology, Unit Methodology and Statistics, Graduate School for Social and Behavior Science, Leiden University, The Netherlands. Model selection methods based on the concept of predictability for longitudinal data. (136)
Brian P. Flaherty, Department of Psychology, University of Washington, Seattle, USA. Model selection in latent class and mixture models: What’s to be preferred? (140)
Shing On Leung,
Faculty of Education,
ABSTRACTS
Optimizing the measurement accuracy of a test using correlated scales. (075)
Marieke van Onna and Anton Béguin
In an assessment, multiple separate tests may be administered to a person. An accurate estimate of a person’s position on each of the traits can be attained by increasing the length of each of the tests. However, results on correlated tests also inform us on the focus trait position. For example, a high score on a science test predicts math ability. If we use this common information efficiently, the full assessment may require fewer items, and may consequently be a lesser burden to the assessed person. This general idea is applied by a number of researchers (e.g.Wainer et al., 2001;Yao & Boughton, 2007). In the current study we propose to fit a unidimensional IRT model (OPLM, Verhelst, Glas & Verstralen, 1994) on each of the relatively short unidimensional tests. Once one scale is fixed, items of correlated traits are added to the focus scale. Their discrimination parameters on the focus trait will in general be less than the discrimination parameters of the original items. However, the accuracy of the focus trait estimate may be increased, while still measuring the same latent trait. The advantage of this approach over multidimensional IRT modeling is that the meaning of the scales is already established in the test construction phase. References: Verhelst, N. D., Glas, C. A. W., & Verstralen, H. H. F. M.(1994). OPLM: computer program and manual. Arnhem: Cito., Wainer, H., Vevea, J. L., Camacho, F., Reeve, B. B., Rosa, K., Nelson, L., et al. (2001). Augmented scores-‘‘Borrowing strength’’ to compute scores based on small numbers of items. In D. Thissen & H. Wainer (Eds.), Test scoring (pp. 343-387). Mahwah, NJ: Lawrence Erlbaum., Yao, L. & Boughton, K.A. (2007). A multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Applied Psychological Measurement, 31, 83–105.
Model selection methods based on the concept of predictability for longitudinal data. (136)
Hsiu-Ting Yu
This research concerns the model selection methods for analyzing longitudinal data. Traditional model selection methods based on likelihood concept such as the Akaike's information criterion (AIC), Bayesian information criterion (BIC) and their generalizations do not take into account the sequential ordering of the data. These likelihood-based methods for model selection ignore the fact that the measurements in longitudinal data are collected in a sequential order; however, the existing temporal ordering of the data can provide essential and vital information about the model ¯t based on the concept of predictability. We compare two classes of model selection methods relating to the concept of the predictability: methods based on cross-validation and the methods based on prediction errors. Motivations, justi¯cations, and discussions of theoretical and practical aspects of these methods are presented. Series of simulation studies are conducted to investigate the performance and e®ectiveness of the various model selection methods. Guidelines and recommendations will be provided concerning the issue of model selection for modeling longitudinal data.
Model selection in latent class and mixture models: What’s to be preferred? (140)
Brian P. Flaherty
Different types of research (e.g., exploratory or confirmatory;
descriptive or theoretically driven) have different goals. Research
approaches such as null hypothesis testing or model fitting entail
different preferences and assumptions. Furthermore, research is
typically guided by common principles, such as parsimony and
generalizability. Following a brief review of these topics, typical
model selection practice used with latent class and mixture models is
located in this research context. It is argued that confirmatory and
null-hypothesis testing approaches have been applied automatically and
inappropriately in the growing body of latent class and mixture model
applications. Current practice appears to follow from a null-hypothesis
testing tradition, emphasizing parsimony and simplicity. However,
latent class and mixture models follow more naturally from a model
fitting framework, which defaults toward more complexity, rather than
simplicity. A rationale for favoring complexity and a hypothesis
generative approach to the use of these techniques is given. An
empirical illustration employing US national data on tobacco use
patterns will be discussed. Motivating features of this example include
the fact that a sub-group analysis is theoretically motivated, yet the
state of the substantive literature provides little specific guidance
on what sub-groups to expect.
Computational issues in parametric bootstrapping of limited information statistics for 2pcontingency tables. (259)
Shing On Leung
The goodness-of-fit problems on high way contingency tables are well
recognized. Recent works of Cai, Maydeu-Olivares, Coffman & Thissen
(2006) and Maydeu-Olivares & Joe (2005) revised Bartholomew
and Leung’s (2002) Y-statistics to incorporate Jacobian under composite
null hypothesis. However, the inversion and computation of the
information matrix pose a controversial problem. This paper suggests
using only the diagonal elements to bypass the singularity problem, and
at the same time uses moment matching methods to approximate
p-values. We parametric bootstrap data from latent trait models with 5
binary items, and apply the method to previously analyzed data set. The
whole procedure is repeated with 8 binaryitems and another real data
set. Results suggested that limited information statistics are always
better under sparseness. And, it is better when more items are used.
Computational issues involved in bootstrapping, maximum likelihood
estimations, calculation of limited information statistics are
discussed. We suggest directions for further works, e.g. two factor
analysis, over-fitting, under-fitting, power analysis, etc.