Psychometric Society

Benjamin Nagengast, Department of Education, University of Oxford, UK, and Rolf Steyer. Generalized ANCOVA for quasi-experimental multilevel designs with treatment assignment at the cluster-level. ♥

ABSTRACTS

Symposium overview
Ulf Kröhne and Rolf Steyer
The symposium presents recent developments in the estimation of average causal effects with model based adjustment procedures. Different extensions are necessary to bring classical analysis of covariance in line with certain specifics of quasi-experimental designs, for example interactions, stochasticity of predictors, heteroscedasticity and multi-level structure. Fiege et al. introduce the theory of causal effects and causal dependencies (Steyer et al., forthcoming) and present examples of recent applications for the evaluation of school effects.  Based on a replication of a within-study-comparison for evaluating the effects of a math and language training, Pohl et al. compare the results of different adjustment methods. Kröhne et al. focus on the consequences of stochastic predictors for unbiasedness of the OLS estimates of the average causal effect, when interactions are present. Nagengast et al. further generalize the ANCOVA to quasi-experimental multi-level designs with treatment assignment at the cluster-level. The generalized analysis of covariance investigated in this symposium, if embedded in multi-group structural equation models, offers great flexibility for expressing specific hypothesis and for handling missing values and measurement error of covariates.

Fair Comparisons for the Evaluation of School Effects: An Application of the Theory of Causal Effects
Christiane Fiege and Rolf Steyer
School effectiveness studies aim at providing system knowledge by comparing the achievement scores of different classes or schools. Therefore, adjustment procedures that account for pre-existing differences in students’ competencies have to be implemented to generate fair comparisons. These adjustment procedures and the selection of covariates can be critically studied with respect to the theory of causal effects (Steyer et al., forthcoming). In this presentation, the basic concepts of the theory and the necessary assumptions for causal interpretations of the comparison scores are introduced. Then, the adjustment strategy used by the Thuringian Competency Test – a state wide assessment system for monitoring students’ achievement in the German state of Thuringia – is analysed with respect to the causal interpretation of the reported comparisons. In the Thuringian Competency Test, the deviation of the class-specific mean from the adjusted comparison score is interpreted as adjusted descriptive measure of instruction. This measure should not be causally interpreted. But it helps to rule out alternative reasons for differences in the achievement scores, thus it reveals information that assists teachers to identify critical aspects of their instructional performance and it supports school improvement.

Causal unbiasedness from an observational study
Steffi Pohl, Peter M. Steiner, Jens Eisermann, Renate Soellner and Thomas D. Cook
In order to approximate the results of random experiments with non-experimental designs, adjustment methods like propensity score (PS) and ANCOVA are often used. Shadish, Clark and Steiner (in press) used a within-study comparison for investigating how well these adjustments work in practice. They randomly assigned students to participating either in a randomized or in a nonrandomized experiment. Treatment effects were then estimated in the experiment and compared to the adjusted non-experiment using both ANCOVA and PS. Most of the selection bias in the non-experiment was reduced by adjusting for the covariates whatever the form of data analysis. The present study replicates the study of Shadish et al. (in press) and obtains basically the same results despite differences in the design and the amount and direction of initial selection bias. The results show that the selection of covariates matters considerably for bias reduction in non-experiments, but, the choice of a PS or ANCOVA analysis matters less. 

Generalized analysis of covariance with interactions and stochastic predictors: Unconditional inference about the average causal effect.
Ulf Kröhne and Benjamin Nagengast
The general linear model is usually developed under the assumption of fixed predictors. In quasi-experimental designs however, predictor variables are appropriately considered as random variables whose values vary from sample to sample. While the OLS estimators of regression parameters under the fixed regressor assumption are conditionally and marginally unbiased if predictors are stochastic, the differentiation between fixed and random predictors affects inference about the average causal effect obtained from the generalized analysis of covariance with covariate-treatment interactions. Although the OLS estimator of the non-linear function representing the average causal effect is consistent and unbiased, its unconditional variance can be obtained only under certain conditions from the OLS estimates. For the most general case (stochastic predictors and interaction), inferences under the simple fixed regressor assumption are shown to be invalid, because the unconditional variance of the average causal effect is underestimated. The consequences of this finding for causal inference based on the general linear hypothesis or based on mean-centering in moderated regression models are discussed. Finally, multigroup structural equation models are presented as an alternative, which allows correct unconditional inferences about the average causal effect under a multivariate normality assumption. The analytic findings are illustrated with a Monte Carlo Simulation. 

Generalized ANCOVA for quasi-experimental multilevel designs with treatment assignment at the cluster-level
Benjamin Nagengast and Rolf Steyer
In quasi-experimental multilevel designs with treatment assignment at the cluster-level, whole clusters instead of units self-select to treatment conditions. The general theory of causal effects and dependencies (Steyer et al., 2009) is shown to naturally account for the specific properties of these designs such as interactions between the unit and the cluster. Conventional approaches to the analysis of covariance in multilevel designs have failed to uniquely identify the average causal effect in the presence of interactions and have not adequately modelled contextual effects. Furthermore, statistical implementations using the linear mixed model usually assume fixed predictors. Based on the general theory of causal effects and dependencies, the generalized ANCOVA for quasi-experimental designs with treatment assignment at the cluster-level is developed to identify the average causal effect in the presence of interactions. Two statistical implementations of the generalized ANCOVA are compared in a simulation study: The conventional linear mixed model leads to biased estimates of the average effect and underestimates its variability in the presence of interactions. On contrast, the multigroup multilevel structural equation model, as implemented in Mplus (Muthén & Muthén, 1998-2007) yields unbiased average effect estimates in the presence of contextual effects and adequate standard errors. (111)