Comprehensibility in Language Assessment
A Broader Perspective
Parvaneh Tavakoli [+–]
University of Reading
View Website
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
Sheryl Cooke [+–]
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
Comprehensibility is considered central to successful communication (Munro & Derwing, 1995a, 2007). Yet, despite the crucial role it plays in communication and the contribution it makes towards the assessment of spoken language ability, comprehensibility has occupied an ambiguous position in language testing. At times, it is feature-driven, taking a linguistically-atomistic approach with little reference to context or communicative purpose, treating comprehensibility as an abstract construct made up only of linguistic components. At other times, the assessment of comprehensibility is an intuitive action, relying on a holistic sense of understanding by the assessor and rarely going beyond the speaker’s utterances to include listener characteristics. The lack of a perspective that encompasses broader linguistic and socio-pragmatic factors that contribute to achieving meaning in spoken language has motivated us to propose the current manuscript as an approach to understanding, defining and assessing comprehensibility. In this monograph, we argue that conceptualising comprehensibility as a multidimensional construct and adopting a broader perspective to understanding and analysing it for communication purposes would benefit the fields of second language assessment and second language acquisition.
This work is licensed under https://creativecommons.org/licenses/by-nc-nd/4.0/
The ebook is free of charge and can be accessed from https://equinoxonlinelibrary.com/book/58244/comprehensibility-in-language-assessment
Open Access funded by University of Reading
Series: British Council Monographs on Modern Language Testing
Table of Contents
Prelims
List of Figures vii
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
List of Tables viii
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
Chapter 1
Introduction [+–] 1-21
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
This chapter provides an overview of the scope and aims of the book and presents a summary of current approaches to the construct of comprehensibility and the related concepts of intelligibility and accentedness (Levis, 2005; Munro & Derwing, 2011; Isaacs, 2014). A brief synopsis of previous research articles is followed by consideration of the difficulties inherent in defining, teaching and measuring a concept that is characterised by complexity and further confounded by broader socio-linguistic factors such as the role of the listener and the existence of multiple variations of languages (Kachru, 1985; Lindemann, 2017; Harding & Mcnamara, 2017) . The outline of a framework for analysing comprehensibility from a micro- to macro-level is presented. Following chapters focus on describing and providing a rationale for this new approach.
Chapter 2
Comprehensibility at a Phonological Level [+–] 22-52
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
This chapter focuses on the more traditional scope of the comprehensibility/intelligibility construct and takes an analytical approach to examining evidence related to phonemic and prosodic linguistic components affecting comprehensibility of spoken language. An overview of the features that have been identified by previous studies is presented and categorised (e.g. Isaacs & Harding, 2017; Isaacs & Trofimovich, 2012), including those occurring at phoneme level (articulation), word level (segmental units, word stress) and sentence level (intonation, pitch, fluency). The features are then examined from three key perspectives: a) reported impact on comprehensibility b) how the features are reflected in rating scales c) whether the features are L1-dependent
Chapter 3
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
Spoken texts and the features associated with longer turns of monologic speech are the focus of this chapter. These typically include monologues such as presentations or speeches, narration and extended turns that might include characteristics of argumentation or description. Potentially impactful elements identified in the literature (e.g. Isaacs & Trofimovich, 2012) and in research are examined and, as in chapter 2, analysed from three perspectives: a) reported impact on comprehensibility b) how the features are reflected in rating scales c) impact of L1 discourse conventions on comprehensibility
Chapter 4
Comprehensibility at a Pragmatic Level [+–] 74-95
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
This chapter focuses on the pragmatic aspect of comprehensibility, i.e., when the speaker’s sounds and words are clear, but their intended meaning is not because of their limited pragmatic knowledge. The discussions of this chapter will include the speaker’s perspective and highlights the dynamic nature of comprehensibility. After defining comprehensibility from a communication and intercultural perspective, the chapter will discuss the key aspects of pragmatic knowledge that affect comprehensibility. Summarising research in this area (e.g., Purpura, 2004; Rover, 2011, Taguchi, 2005, 2007, 2012) we will discuss issues such as use of formulaic expressions, implicatures and indirect speech acts as some potential areas in which comprehensibility issues arise. We will also focus on cultural norms and differences in pragmatic aspects of language use, e.g., politeness and backchannelling, that affect comprehensibility. More importantly, we will argue that to have a full understanding of the effects of pragmatic knowledge on comprehensibility, it is necessary to examine the speaker’s participation in extended discourse during both monologic and dialogic types of performance (Tavakoli, 2016; Tavakoli, 2018).
Chapter 5
Comprehensibility and Fluency [+–] 96-113
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
The chapter’s main aim is to discuss the relationship between fluency and comprehensibility. After defining the construct of fluency in terms of cognitive, perceived and utterance fluency (Segalowitz, 2010; Tavakoli & Wright, 2020) and measureable features of speed, breakdown and repair fluency (Skehan, 2003, Tavakoli, et al., 2020), the chapter discusses in what ways these features of speech affect comprehensibility. More specifically, the chapter will discuss the effects of fluency on listeners and their judgements. By providing examples of how these different features of fluency are represented in rating descriptors of different language tests, the chapter will argue that the relationship between fluency and comprehensibility is not clearly represented in rating descriptors and rating materials.
Chapter 6
Technology and Comprehensibility [+–] 114-135
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
Technological advances are having a significant impact on language assessment in general and this chapter examines the opportunities machines present to better measure comprehensibility, as well as the threats that constrained tasks demanded by automated assessment might present to the construct. An overview of current models for automated rating of speech is presented (e.g. Xi, et al. 2008; Van Moere & Downey, 2016; Isaacs, 2014) and the approaches are examined from the perspective of the comprehensibility framework presented in the book, drawing on the features extrapolated in each of the preceding chapters to examine how technology does and does not offer solutions for automated assessment and feedback for learning on particular components of the comprehensibility inventory.
Chapter 7
Teaching towards a Comprehensibility Goal [+–] 136-152
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
This chapter provides a teaching perspective to the discussion of comprehensibility. Highlighting the importance of teaching the concept to L2 teachers, it will raise some key questions about the extent to which L2 teaching curricula focus on comprehensibility, or whether comprehensibility is actually taught and practiced in L2 classrooms. Analysing some national curricula for English language teaching, we will examine the focus on comprehensibility in these documents and strengthen our arguments by research in this area. We will provide some examples of what L2 teachers can do to highlight the role of comprehensibility.
Chapter 8
Conclusion [+–] 153-169
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
In this chapter, we will provide an overview of issues related to comprehensibility and its key features that are central to successful communication. This chapter will also draw together the analytical framework discussed in the monograph to demonstrate the key aspects of the framework that benefit different language professionals. We highlight the contexts that potentially influence comprehensibility in significant ways and highlight the dynamic relationship between speakers and listeners. We will argue that adopting the broader perspective we have suggested will enable language professionals to have a more objective assessment of comprehensibility.
References
References 170-195
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
Appendices
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
Appendix 2: IELTS Descriptors 197-198
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
End Matter
Index 201-213
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is an Assessment Researcher with the British Council Assessment Research Group. Sheryl has 20 years’ experience in various areas of language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on the assessment of spoken English and the potential implications for English as a Lingua Franca. Her research interests include assessment of speaking, the use of new technologies in language testing and the ethics of language assessment in the global context.
