Scoring Second Language Performance Assessments; Knoch, Fairbairn; Yan

Ute Knoch [+–]

University of Melbourne

Ute Knoch is the Director of the Language Testing Research Centre at the University of Melbourne. Her research interests are in the area of writing assessment, rating processes, assessing languages for academic and professional purposes, and placement testing. She was the Co-president of the Association for Language Testing and Assessment of Australian and New Zealand (ALTAANZ) between 2015-2016 and currently serves on the Executive Board of the International Language Testing Association (ILTA). In 2014, Dr Knoch was awarded the TOEFL Outstanding Young Scholar Award by the Educational Testing Service (Princeton, US), recognizing her contribution to language assessment.

Judith Fairbairn [+–]

British Council

Judith Fairbairn leads on assessment development projects for the British Council’s Global Assessments team, which includes project management, assessment design, item writing, rating scale development, test quality assurance and developing bespoke assessment solutions for clients. Areas of expertise include research into rating speaking and writing tests, online rater training, rating scale development, and equality, diversity and inclusion in testing. Judith has an MA in Language Testing (Lancaster University) and is studying a PhD at University College London, Institute of Education focusing on rater decision making processes when marking speaking tests.

Yan Jin [+–]

Shanghai Jiao Tong University, China

Dr. Yan Jin is a professor of linguistics and applied linguistics at the School of Foreign Languages, Shanghai Jiao Tong University, China. Her research interests include the development and validation of large-scale, high-stakes language assessments. She is currently chair of the National College English Testing Committee of China and co-president of the Asian Association for Language Assessment. She is co-editor-in-chief of the Springer open-access journal Language Testing in Asia and is also on the editorial boards of Language Testing, Language Assessment Quarterly, International Journal of Computer-Assisted Language Learning, and a number of academic journals published in China.

The ability to speak and write effectively is widely recognized as an important skill in many contexts and for many purposes, both personal, educational and professional. Because these skills are considered important in second and foreign language learning contexts, they are often included in performance assessments. The scoring of such performances is, however, a complex undertaking and has attracted much attention, both in first and second language learning contexts. The increasing use of automated scoring systems has added to this complexity in recent years. It is therefore all the more surprising that there is no book available that provides an overview of this topic area – the scoring of second language performances. This monograph fills this gap, by drawing together the latest literature in the area. It focusses on issues relating to both rater-mediated assessments and sets out consideration in relation to automated scoring systems (and other technology) which are increasingly used in our field. This monograph provides a useful introduction to graduate students, researchers, test developers, other practitioners and teachers to this topic which has in many ways dominated the field of language assessment over many decades.

Series: British Council Monographs on Modern Language Testing

Preliminaries

List of Figures [+–] vi

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

List of Tables [+–] vii

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

Introduction

Introduction [+–] 1-6

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

Introduction to the book outlining the chapters.

Chapter 1

Understanding Raters and Ratings [+–] 7-28

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

In Chapter 1, we start by introducing the topic of raters and ratings. We provide a visual model that shows the complexity of the factors that influence rating quality in rater-mediated assessments, as well as some issues that are relevant to automated scoring. This model also provides an overview of many of the issues we introduce in the book. We then discuss the various rater effects that have been described in the literature and explore factors that may affect rating quality.

Chapter 2

Measuring Rating Quality [+–] 29-49

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

Chapter 2 focusses on measuring rater quality. We provide a detailed framework for classifying measures of rating quality and discuss the most commonly used indices and measurement techniques in more detail. The chapter concludes with a summary of what detailed information each rater index can provide about the rater effects described in Chapter 1.

Chapter 3

Rater Cognition [+–] 50-64

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

In Chapter 3, we explore more qualitative aspects of the rating process – rater cognition. The chapter examines methodological considerations in collecting rater cognition data and then focusses on studies exploring rater decision-making.

Chapter 4

Approaches to Enhancing Rating and Score Quality [+–] 65-78

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

Chapter 4 introduces approaches to enhancing rating and score quality, including work that has looked at rater recruitment, training, ongoing standardisation and monitoring of raters. The chapter also discusses various score resolution techniques that have been used and researched in language assessment contexts in some detail.

Chapter 5

Rater-mediated Judgment with or without Rating Scales [+–] 79-101

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

In Chapter 5, we look at scoring, in particular by using rating scales. We introduce various rating scale types and also examine rating scale development methods described in the literature. The chapter concludes by looking at one method increasingly used when scoring without specific scale descriptors – comparative judgement.

Chapter 6

Technology in Scoring [+–] 102-132

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

Technology in scoring is becoming an increasingly important topic, in particular through the introduction of automated scoring and feedback systems. In Chapter 6, we focus on such technological innovations. The chapter first examines the application of online marking, and then turns to automated scoring systems used for both speech and writing evaluation. The chapter outlines how such scoring systems work, and considers their advantages and disadvantages. Research on automated scoring systems is also presented to outline the current status of thinking in the field.

Chapter 7

Validating Scoring Processes [+–] 133-149

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

In the final chapter, Chapter 7, we focus on the validation of scoring processes. More precisely, we examine how the conceptualisation of the validation of scoring processes has changed historically, introduce some key validation frameworks and then focus on some specific validation contexts: the validation of automatically-scored assessments, and the validation of classroom-based assessments.

Conclusion

Conclusions [+–] 150-152

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

In this monograph we have summarised the latest knowledge-base on scoring L2 performance assessments. Essentially, this text is about validation and the various factors that influence the quality of the scores we report to learners and the inferences that can be made about learners based on these scores. We have presented a complex picture, which is bound to evolve in future years as research in this area is likely to continue to proliferate. More work in the various areas we have introduced is clearly needed. It is likely that, in particular, the focus on automated scoring systems will increase in the future, and we as researchers and practitioners need to ensure that the emphasis on producing and reporting valid scores is continued. This view is essentially inward-facing, that is, we are focusing attention on the test and how scores are produced.

End Matter

References [+–] 153-186

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

Index [+–] 187-194

Ute Knoch,Judith Fairbairn,Yan Jin

University of Melbourne

British Council

Shanghai Jiao Tong University, China

Reviews

By distilling and synthesising a vast and complex body of research findings into an approachable volume, this compact book will become an essential reference for researchers of language assessment and language testers, providing a quick and easy understanding of how many of variables identified are associated with scoring outcomes and impact on rating process.
Assessing Writing

Scoring Second Language Spoken and Written Performance offers a concise and much needed overview of issues in rater-mediated assessment. Drawing on their rich expertise in language testing research and development, the authors managed to untangle the complexities associated with rating productive language performance in a highly useful manner for graduate students, researchers, and practitioners in language testing.
Language Testing

.

Books and Journals in Humanities, Social Science and Performing Arts

Scoring Second Language Spoken and Written Performance

Issues, Options and Directions

Ute Knoch [+–]

University of Melbourne

Judith Fairbairn [+–]

British Council

Yan Jin [+–]

Shanghai Jiao Tong University, China

Table of Contents

Reviews

Related Journal

Related Interest

Search Equinox

Subjects