Hierarchical models and structure (33)

Chair: Ab Mooijaart, Tuesday 21st July, 10.50 - 12.10, Castlereagh Room, Fisher Building 

Victor L. Willson and Ross Larsen, Texas A&M University, USA. Modeling second level cluster dependency in multilevel models. (080)

Ralph Carlson and Hilda Medrano, Department of Psychology, University of Texas Pan American, Edinburg, USA. Nested/hierarchical measures. (127)

Yasuo Miyazaki, Virginia Polytechnic Institute and State University, Blacksburg, USA. Impact of ignoring nested data structure in IRT models (173)

Jorge González, Ernesto San Martín and Jorge Manzi, Centro de Medicion MIDE UC, Escuela de Psicologia, Pontificia Universidad Católica de Chile. Value-added modeling of school performance under large school variability and high selectivity (036)

ABSTRACT

Modeling second level cluster dependency in multilevel models. (080)
Victor L. Willson and Ross Larsen
Multilevel models can have a structure in which dependencies in the data occur not at the first level but at the second. We present a covariance model that addresses this condition and evaluate four programs that typically are used to analyze multilevel data: MPLUS, LISREL, SAS, and R. We generated a 2x2x2x3x2 factorial design varying cluster size, second level covariate presence, first level covariate presence, number of time points, and AR presence with data simulated in SAS Proc IML using a lag-1 autoregressive function for 4 or 8 time points. Results indicate only SAS correctly estimated the parameters using a REML procedure. MPLUS failed to converge, and LISREL gave incorrect AR estimates, although its mean and variance estimates were adequate. Some issues in extending the work to SEM problems are presented.

Nested/hierarchical measures. (127)
Ralph Carlson and Hilda Medrano
Measures are often organized by subtests or factors into nested/hierarchical structures. Anastasi and Urbina (1997) indicate that the current trend in ability testing is toward hierarchical scores and that this condition will likely characterize measurement in the twenty-first century. Measures are considered nested within a hierarchical structure if each level of one measure is within one and only one level of another measure. When there are significant differences within a lower level of a nested hierarchical measure, all successive levels are not interpretable due to a lack of "cohesion"; therefore, there is a problem with loss of information at the higher levels. This study presents a model that adjusts for nested effects in nested/hierarchical measures. Thus, this model provides a solution by alleviating the loss of information due to a lack of "cohesion" across successive levels of nested/hierarchical measures. Nested variance is transitive and interpretation of nested effects must be from bottom-up. The current study presents an example of the decomposition and adjustment of nested/hierarchical effects in WISC-IV data. These data were obtained from a random sample of 9 year old bilingual Hispanic children. This study presents a refinement in thinking, analysis, and interpretation of nested/hierarchical measures.

Impact of ignoring nested data structure in IRT models (173)
Yasuo Miyazaki
Item response model (IRT) is an advanced psychometric methodology that is frequently used in testing. IRT model assumes a simple random sampling, i.e., item responses from one person is independent of those from another person. In many educational testing settings, however, this assumption of independence is less likely to be met because of its sampling design used. In education, two-stage sampling is frequently used where schools are sampled first and then from each school selected students are sampled. If the IRT models are applied to the data that are obtained from a two-stage sampling, then it is expected that there will be some impacts on the item parameter estimates such as item difficulty. Thus, the goal of the current study is to evaluate the impacts of ignoring the nested data structure on the IRT parameter estimates. Specifically, a simulation study is conducted in differing conditions of factors such as number of items, number of subjects, number of schools, and intraclass correlations to compare the results from standard IRT to those from multilevel IRT. The preliminary results suggest that ignoring the nested data structure in fact has some negative impacts to varying degree. The implications to the practice will be discussed.

Value-added modeling of school performance under large school variability and high selectivity (036)
Jorge González, Ernesto San Martín and Jorge Manzi
Value-added indexes can help schools and policy makers acting as a way to control school or teacher effectiveness. They can also be useful for parents who need to decide which school their children should attend. However, the usefulness of value-added models depends on statistical issues and specific decisions made during implementation. In this paper we consider value-added models of school assessment and their implementation in Chile. Two special features can be distinguished in the Chilean context: a) many schools have a selectivity process to admit students and b) there exists high variability across levels of socioeconomic status (see, e.g., PISA 2006). Regarding a), it is shown that using the previous score (intake) as the only single explanatory variable in the model (as it is normally considered by various value-added systems in the world), is not enough for controlling selectivity. To address this issue, we provide a compositional effect which successfully account for school selectivity. In addressing b), we propose a hierarchical linear mixed model which takes into account the variability, allowing for heteroskedasticity at SES level. Residual analyses and model comparison statistics show that the heteroskedastic model fits better than the simpler model.