Cognitive diagnosis: Applications (17)

Chair: Andreas Frey, Thursday 23rd July, 14.55 - 16.15, Lowercroft, School of Pythagoras.

Marian Hickendorff, Institute of Psychology, Leiden University, The Netherlands. Estimating the accuracy of solution strategies in cognitive tasks: An explanatory IRT modeling approach. (081)

Claus Carstensen, University of Bamberg, Germany. An application of a general diagnostic model to competency profiles from standard assessments. (212)

Yang Tao, Yu Na, Faculty of Education, Beijing Normal University, China, Yang Tao, Li Lingyan, National Assessment of Educational Quality, Beijing Normal University, China. Compilation of diagnostic math tests for grade 4 and grade 5 based on the rule space model. (102)

Gabriela Gonzalez Marin, Samantha Bouwmeester, Erasmus University, A. Marije Boonstra, PsyQ, The Hague, The Netherlands. Understanding Planning Ability Measured by the Tower of London, a Contribution to the Construct Validity. (255) ♥

ABSTRACTS

Estimating the accuracy of solution strategies in cognitive tasks: An explanatory IRT modeling approach. (081)
Marian Hickendorff
Solution strategies are an often-studied topic in cognitive and in educational psychology. Strategy use is mostly measured in nominal categories, and is often repeatedly observed within a subject. For example, a student takes a mathematics test of several items, and for each item the strategy used is coded into one of several distinct categories. When one is interested in the success rate or accuracy of the different strategies, various sources of variability play a role. First, persons can vary: they vary in underlying ability and they may use several strategies on a set of items. Second, items can vary: they will differ in general difficulty level and in the extent to which they elicit each particular strategy. In this presentation, I will discuss how IRT models can deal with these sources of variation simultaneously. Specifically, I propose to use the solution strategy as a person-by-item predictor variable in explanatory IRT models. In this approach, the estimated probability to get a correct answer is a function of the solution strategy used, while accounting for the person’s latent ability and the item difficulty. This flexible framework also allows for testing of various restrictions and interaction effects, as will be shown by two real-data examples of primary school mathematics achievement.  

An application of a general diagnostic model to competency profiles from standard assessments. (212)
Claus Carstensen
The German federal states implemented educational standards for school subject areas like mathematics and several levels of graduation to improve classroom instruction and student performances. In a data collection with n=9577 grade 9 students responses to a test organized according to the standards by five big ideas, six competencies and three levels of complexity, according to the standards for mathematics, were obtained and analysed in this paper. The question addressed in this paper is whether and how results can be reported with respect to the big ideas as well as on the content related competencies. What is the differential information one may obtain from complex competency profiles, how reliable are the reported results and how complex can profiles reliably be reported? After obtaining competency profiles it may be asked whether mixture distributions with regard to these profiles can be identified. The data were analysed using multidimensional item response models to obtain competency profiles. Using the General Diagnostic Models approach (von Davier, 2005) mixture distribution models were estimated. Either for big ideas and competencies, latent classes basically distinguish between high and low achieving students; some competencies seem to have more discriminative power than other competencies.  

Compilation of diagnostic math tests for grade 4 and grade 5 based on the rule space model. (102)
Yang Tao, Yu Na, Yang Tao and Li Lingyan
Based on the rule space model (RSM), this study investigated the procedure to compile diagnostic math test with the Q matrix as the blueprint and compiled the diagnostic test for grader 4 and grader 5. The application of RSM for the diagnosis is to analyze the test items to get the Q matrix which describes the attributes involved in each item, and deduce the adjacent matrix. In the compilation of diagnostic test, the process is form adjacent matrix to the Q matrix. The procedure includes three steps, first is to identify the important attributes within the domain, second is the to specify the attribute combination pattern., third is to develop items according the attribute combination pattern. The results revealed that the test not only demonstrated good property on reliability and validity but also provided abundant diagnostic information. According to the diagnostic results from the test which administered to 1159 students, the grader 4 and grader 5 performed well on whole number, elementary operation and application; however, the attributes such as measure, statistics, searching for pattern and advanced operation were not well mastered. What is more, those not-well-mastered attributes demonstrated the fastest growth between grade 4 and grade 5.

Understanding Planning Ability Measured by the Tower of London, a Contribution to the Construct Validity. (255)
Gabriela Gonzalez Marin, Samantha Bouwmeester, A. Marije Boonstra
The Tower of London (TOL) is a widely used instrument for assessing planning ability. Inhibition and (spatial) working memory are assumed to contribute to performance on the TOL, but findings about the relationship between these cognitive processes are often inconsistent. Moreover, because most research on planning is done by using a total move score as measure for planning performance, the influence of specific properties of TOL-problems on cognitive processes and difficulty level is ignored. Furthermore, it may be expected that several planning strategies can be distinguished which can not be extracted from the total score. In this study a factor analysis was used to investigate the abilities involved in solving twelve TOL-problems that differed on properties. It turned out that the 1-factor model fitted best, although only 16 percent of the variance was explained. A latent class regression analyses was performed with planning time as dependent variable and problem properties as independent variables. The results showed that four strategy groups could be distinguished which differed with respect to average preplanning time. The effect of problem properties differed for the four strategy groups. Additional analyses showed that the strategy groups differed with respect to average planning performance but that there were no significant differences between inhibition and spatial working memory performance.