Measurement invariance in experimental psychology, personality research, and genetics
Symposium organised by Jelte Wicherts, Department of Psychological Methods,
Chair: Jelte Wicherts, Tuesday 21st July, 10.50 - 12.10, Palmerston Lecture Theatre, Fisher Building.
Jelte Wicherts, Department of Psychological Methods,
Annemarie Zand Scholten and Denny Borsboom, Psychological Methods Department,
Conor Dolan, Iris Smits, and Harrie Vorst, Psychological Methods Department,
Sophie van der Sluis, Danielle Posthuma, and Matthijs Verhage, Department of Biological Psychology, Free University of
ABSTRACTS
Symposium overview: Issues related to the study of measurement invariance in experimental psychology, personality research, and genetics
The study of measurement invariance across groups originated from the need to study the fairness of cognitive tests in education and in personnel selection, but the psychometric framework of measurement invariance can be quite helpful in the study of various psychological phenomena in other settings. In this symposium, the presenters will discuss some recent developments in the study of measurement invariance in various fields of psychological science, namely experimental psychology, personality research, and genetics. Each of these applications of measurement invariance raises specific statistical and psychometric issues that may direct the future technical advancements in the field. In the first talk, Jelte Wicherts will discuss the relevance of measurement invariance to experimental research. He will focus on the use of multi-group confirmatory factor analysis (MGCFA) with mean structure in between-subject experimental designs and discuss issues related to statistical power. Next, Annemarie Zand Scholten and Denny Borsboom will focus on the common use of sum scores in the two-way ANOVA model. They will show how, even under measurement invariance, spurious interactions may emerge. In the third talk, Conor Dolan and colleagues will present the results of a large-scale study of sex differences in personality. As personality scales often do not conform to restrictions of simple structure, they employed a recently-developed exploratory common factor model with target rotation in their study. Finally, Sophie van der Sluis and colleagues will discuss the relevance of measurement invariance to genetics research. She will present results of simulations study on the consequences of violations of measurement invariance in this area.
Measurement invariance in psychological experiments
Jelte M. Wicherts
Because psychological constructs are not directly observable, researchers have to choose operationalizations of these constructs in their experimental studies. The data from psychological experiments are normally submitted to Analysis of Variance (ANOVA) and the mean effects on the observed scores are often treated as if they accurately reflect mean differences in the underlying construct. However, the experimental manipulation may also affect the measurement characteristics of the measure of the experimental effect, which will result in a violation of measurement invariance across conditions. For instance, in a study on the cognitive behavioral treatment of anxiety, a self-report measure of anxiety may show a reactive self-report change (Shadish, Cook, &
Legitimate inferences based on meaningless statistics
Annemarie Zand Scholten and Denny Borsboom
Psychologists typically like to conclude something about psychological properties based on measurements. When the truth-value of such a conclusion depends on the scale that is chosen (i.e. a ‘permissible’ transformation that leaves intact the qualitative relations), it is called an empirically meaningless or illegitimate inference. This is why conclusions based on parametric statistics performed on ordinal scores are considered uninterpretable in terms of the underlying psychological property. Although such tests are strictly meaningless, they do allow for unambiguous interpretation of group differences under certain circumstances. Several authors have investigated these circumstances for situations where a latent interval property is known or assumed. For example, Loftus (1978) specifies what type of effects in 2-way ANOVA designs are legitimate when the ordinal observed scores are monotonically related to a latent interval property (see also Davison and Sharma (1988, 1990, 1994). Kang and Waller (2005) investigated the influence of model parameters test length and difficulty on the detection of spurious interaction effects using a Rasch model. This study extends these findings by investigating the discrimination parameter and using both an additive and a multiplicative model. We show how spurious interactions can be detected and how they are related to measurement invariance.
Measurement invariance of a (Big 5) personality test with respect to cohort over a 25 year period
Conor V. Dolan, Iris Smits, and Harrie C. M. Vorst
We investigated of measurement invariance with respect to sex and cohort of an 70 items (7 point Likert scales) personality test purporting to measure the Big 5 personality dimensions. The data were obtained in first year psychology students at the
Measurement invariance in the context of genetics
Sophie van der Sluis, Danielle Posthuma, & Matthijs Verhage
The aim of genetic association studies is to identify genetic variants that are systematically associated with a phenotype of interest. To enhance the statistical power to detect genes of small effect, it is common practice to combine samples from different labs and/or different countries. In addition, samples from other labs/countries are used as replication samples. Prior to pooling or comparing data from different subpopulations, one should ideally have shown that the phenotypic instrument is measurement invariant (MI) across these subpopulations, i.e., that the mathematical relation between observed and latent trait scores is equal across all involved groups. For many instruments, however, MI has not been established at all, or only with respect to some subgroups (e.g., with respect to gender). Bias resulting from violations of MI has been considered in the context of family-based heritability studies, but not in the context of genetic association studies. As most genetic association studies utilize sum-scores, it is important to study the degree to which observed sum-scores need reflect mean differences in the underlying latent trait. Here we present the first results of a series of (simulation) studies into the consequences of violations of MI for the outcomes of genetic studies. (207)