Development and analysis of scales in education (48)

Chair: Sayaka Arai, Friday 24th July, 9.30 - 10.50, Dirac Room , Fisher Building.

Jyh-Sheng Lin, Institute of Education, National Taiwan Ocean University, Taiwan. The development of the ‘Learning and Adaptive Stress Scale’. (201)

Yi-Ling Chen, Ying-Yao Cheng, Kun-Shia Liu, Hsueh-Pin Tsai and Yi-Ling Chen, Graduate Institute of Education, National Sun Yat Sen University, Taiwan. Development of the Critical Thinking Disposition Inventory. (202)

Satoshi Usami, University of Tokyo, Japan. Statistical analysis of essay test data: Investigation of measurement problems with controlling length of essays. (002)

Sayaka Arai and Shin-ichi Mayekawa, Japan Center for Examination Research, Tokyo, Japan. Estimation of abilities by the weighted total scores using the globally optimal scoring weights under polytomous IRT model. (193)

ABSTRACTS

The development of the ‘Learning and Adaptive Stress Scale’. (201)
Jyh-Sheng Lin
The ‘Learning and Adaptive Stress Scale’ (LASS) has been developed to measure Taiwanese university students’ level of stress (Lin, 2005). This present paper is to adapt the scale to measuring the level of stress on vocational high school students in Taiwan. An initial ‘scoping’ interview, involving 6 teachers and 12 students, was used to modify a 60-item prototype LASS for administering 2678 students in 3 marine and fishery high schools. Based on the Principal Component Analysis with direct oblimin rotation, the final 40-item scale revealed five dimensions: (1) Career development, (2) Living expenses, (3) Social Relationships, (4) Learning adaptation and (5) Teacher-student interactions. These accounted respectively for the following percentages of the scale variance: 28.434, 7.500, 5.389, 4.228 and 3.407. In addition to this normative survey data, technical specifications relating to reliabilities and types of validity were ascertained. This scale development indicates three promising results. Firstly, individual students are able to identify their various sources and levels of stress. Secondly, school authorities can locate the source of stress, so that they can provide practical solutions to ease their students’ stress. Finally, educational authorities enable to examine the causes of stress, to improve student counselling or guidance programmes.

Development of the Critical Thinking Disposition Inventory. (202)
Yi-Ling Chen, Ying-Yao Cheng, Kun-Shia Liu, Hsueh-Pin Tsai and Yi-Ling Chen
This research aimed to develop an instrument for measuring undergraduates’disposition toward critical thinking. Items of the Critical Thinking Disposition Inventory (CTDI) were generated by research team members and checked by field experts. Two samples of undergraduates were employed in instrument development, one for determining the number of underlying constructs in the CTDI and the other for factor confirmation. Exploratory factor analysis identified five factors: (a) Truth-seeking, (b) Open-mindedness, (c) Evidence-finding, (d) Systematic thinking, and (e) Reflective thinking. Confirmatory factor analysis (CFA) indicated that the five-factor
model had an adequate model-data fit and reliability. Multigroup CFA showed that the fitted  model exhibited measurement invariance across genders. Results suggest that the CTDI provides a good tool for measuring undergraduates’critical thinking disposition.

Statistical analysis of essay test data: Investigation of measurement problems with controlling length of essays. (002)
Satoshi Usami
This study investigated measurement problems of essay test data with controlling length of essays. Two sets of data (A・B) were obtained from 303 high school students. Students were divided into 2 groups : one group (N=155) took essays A and B within 400 and 800 words respectively, and vice versa for another (N=148). 4 raters evaluated all essays both holistically and analytically (11-12 items). From covariance structure analysis to analytically-evaluated data, it was statistically confirmed that 2 factor (“linguistic ability factor” and “writing ability factor”) model was valid regardless of the length of essays, raters and essays. Correlation analysis between raters showed that correlations were higher, particularly at holistic evaluation, “phrase” ,“interpretation of contents”, ”summary” and “format”, and that length of essays indicated different effects on different items. The result based on multivariate generalizability theory indicated that (1)adding essays is more effective than raters, (2)4 or 5 raters were enough, and adding more raters showed little improvement, (3)to obtain generalizability coefficient more than 0.8, more than 10 essays are needed if the number of raters is 1. Propensity score analysis, with analytically-evaluated scores as covariates, showed that “beauty of handwriting” and “direction of opinion” might bias holistic scores.

Estimation of abilities by the weighted total scores using the globally optimal scoring weights under polytomous IRT model. (193)
Sayaka Arai and Shin-ichi Mayekawa
Under item response theory (IRT), the ability parameter is estimated by either the MLE or EAP method from the response patterns to the test items for each individual. However, it is possible to estimate the ability on the basis of the observed weighted total score. The weighted total score is easy to calculate, although statistical properties of the score depend on the weights. Mayekawa (2008) developed the globally optimal weights, which maximize the expected test information, and showed that the globally optimal weights reduced the posterior variance when evaluating the EAP given the weighted total score. In this research, we appled the globally optimal weights to a real polytomous item data and compared the efficiency of our method to some other possible weights.