Assessment Across Online Language Education; Stephanie Link; Jinrong Li

Edited by

Stephanie Link [+–]

Oklahoma State University

Stephanie Link is an Associate Professor of TESOL/Applied Linguistics and Director of International Composition at Oklahoma State University. She earned her Ph.D. from Iowa State University and a dual Masters degree from Winona State University in Minnesota, USA and Tamkang University in Taiwan. She primarily teaches graduate-level courses in TESL, grammatical analysis, language and technology, and research methods. Her research interests include the study of emerging technologies for language learning and assessment, written genre analysis, and L2 pedagogy. Her most recent work is on automated writing evaluation and can be found in the Journal of English for Specific Purposes, Journal of Second Language Writing, Language Learning & Technology, System, and CALICO Journal.

Jinrong Li [+–]

Georgia Southern University

Jinrong Li is an Assistant Professor in the Department of Writing and Linguistics at Georgia Southern University. Before that, she has taught English in Beijing and ESL in Boston University and Iowa State University. Her research interests include Computer-Assisted Language Learning (CALL), the instruction and assessment of Second Language (L2) writing, and information literacy and writing pedagogy. She has presented at CALICO, Second Language Research Forum, and AAAL, and her key work has appeared in CALICO Journal, Language Learning & Technology, Journal of Second Language Writing, Computers and Composition, and Assessing Writing.

Despite the challenges in assessing language learning online, the editors of this volume believe the wide range of online language learning opportunities has brought new tools and methods to both strengthen assessment and inform pedagogical decisions in online language teaching. In terms of assessment, technology first provides researchers and practitioners with more options to document learners’ language use in different contexts and their progress over time. The instances of learners’ actual use of language will complement any achievement and proficiency measures of language learning outcomes. Moreover, the use of technology motivates researchers and practitioners to re-conceptualize assessment of online language learning. More importantly, technologies make it possible for the assessment to be incorporated for the purpose of learning (e.g., adaptive learning) and teaching (e.g., technology-mediated dynamic assessment and teacher intervention).

Assessment Across Online Language Education examines these challenges emerged in online language teaching and learning, explores the new opportunities for language teachers and learners, and provides suggestions for future research on assessment and learning in online language education.

Series: Advances in CALL Research and Practice

Preliminaries

Acknowledgments [+–] vii

Stephanie Link,Jinrong Li FREE

Oklahoma State University

Georgia Southern University

With the expansion of online language learning opportunities, language teachers and learners are presented with an increasingly diverse range of tools to facilitate language learning in various contexts. However, CALL researchers and practitioners often have limited knowledge about the effectiveness of online language learning primarily due to a lack of research on online language learning outcomes and on valid assessment measures. Despite the challenges in assessing language learning online, the editors of this volume believe the wide range of online language learning opportunities has brought new tools and methods to both strengthen assessment and inform pedagogical decisions in online language teaching. In terms of assessment, technology first provides researchers and practitioners with more options to document learners’ language use in different contexts and their progress over time. The instances of learners’ actual use of language will complement any achievement and proficiency measures of language learning outcomes. Moreover, the use of technology motivates researchers and practitioners to re-conceptualize assessment of online language learning. More importantly, technologies make it possible for the assessment to be incorporated for the purpose of learning (e.g., adaptive learning) and teaching (e.g., technology-mediated dynamic assessment and teacher intervention). Assessment Across Online Language Education examines these challenges emerged in online language teaching and learning, explores the new opportunities for language teachers and learners, and provides suggestions for future research on assessment and learning in online language education.

Introduction

1. The Online Language Learning Imperative: Maximizing Assessment Practices to Ensure Student Success [+–] 1-18

Stephanie Link,Jinrong Li £17.50

Oklahoma State University

Georgia Southern University

The expansion of online language learning opportunities has provided language teachers and learners with an increasingly diverse range of tools aiming to facilitate language learning in various contexts. However, CALL researchers and practitioners do not have adequate knowledge about the effectiveness of online language learning; and controversies and challenges exist in the on-going efforts to better understand aspects of assessment in language teaching online. Therefore, the purpose of this introduction chapter to the edited volume is to engage language teachers, researchers, and practitioners in the conversation of opportunities and challenges regarding assessments in online language education, to further the discussion on how to re-conceptualize the need and use of assessments in the online environment, and finally to contribute to a better understanding of how assessment tools can affect online language teaching and learning.

PART ONE: ASSESSING LEARNER PROGRESS AND DEVELOPMENT

2. Assessing Language and Intercultural Learning during Telecollaboration [+–] 21-48

Senta Goertler,Theresa Schenker,Carly M. Lesoski,Sonja Brunsmeier £17.50

Michigan State University

Senta Goertler is an Associate Professor of Second Language Studies and German at Michigan State University. She is the co-editor of the 2008 CALICO book on online education and a co-recipient of a CALICO Journal Article of the Year Award. Her co-authored article on planned online language education versus crisis-prompted online language teaching is one of the top most cited articles in the Foreign Language Annals. Recently she co-wrote a book on education abroad.

Yale University

Theresa Schenker is Senior Lector II and Director of the Language Program in the German department at Yale University. She is also the editor of the Unterrichtspraxis. Her research focuses on technology-mediated language learning and community language programs.

Wayne State University

Carly M. Lesoski is an e-Learning Specialist at Wayne State University. She has successfully defended her dissertation in German Studies on the topic of identity and capital during telecollaboration.

University of Education, Freiburg

Sonja Brunsmeier is Instructor of English at the University of Education in Freiburg. Her research focuses on telecollaboration.

In this chapter, we present a telecollaboration project between a pre-service class on intercultural competence for teachers of English as a foreign or second language at a German University of Education and a US undergraduate course focusing on developing advanced language skills in combination with learning about linguistics and second language acquisition (Goertler & Lesoski, 2015). The purpose of the study is twofold: to (1) evaluate the instruments and assessments used for telecollaboration, and to (2) provide recommendations for assessment of telecollaboration in the future. Correlation between scoring rubrics intended to measure the same constructs were low, and so was inter-rater reliability. Specific recommendations for improvement of assessment mechanism and instruments are given in the discussion section.

3. Assessing the Effect of Pedagogical Interventions on Success Rates and Students’ Perceptions of Connectedness Online [+–] 49-70

Victoria Russell £17.50

Valdosta State University

Victoria Russell is an Associate Professor of Spanish and Foreign Language Education at Valdosta State University. She earned a Ph.D. in Second Language Acquisition and Instructional Technology from the University of South Florida, and her research interests include online language pedagogy, pragmatics, and foreign language teacher preparation. Her work has appeared in journals such as Foreign Language Annals, The Internet and Higher Education, and Dimension.

This chapter describes a grant project, the purpose of which was to increase students’ success rates, as measured by final course grades, in a third semester university-level online Spanish course. In order to meet this goal, two pedagogical interventions were implemented: online peer support groups and synchronous conversations with native speakers. In addition, four objectives for the project were set forth as follows: (1) increase student-student interaction, (2) increase student-content interaction, (3) reduce students’ foreign language anxiety, and (4) correct students’ false beliefs about language learning. The first two objectives were intended to increase learners’ perceptions of connectedness with each other and with the course content. The second two objectives were intended to decrease grades of Below Average (D), Failure (F) and Withdrawal (W), thereby increasing students’ chances of successfully completing the course. The chapter provides a review of the relevant literature related to the project, a full description of the two pedagogical interventions, the instruments that were used to measure the objectives of the project, and the results. The discussion includes suggestions for future research.

4. Language MOOCs: Assessing Student Knowledge and Comprehension of Clinical Terminology [+–] 71-90

Carrie Demmans Epp,Rae Mancilla,Valerie Swigart £17.50

University of Pittsburgh

View Website

Carrie Demmans Epp is a research associate at the University of Pittsburgh where she studies language learning and the use of technology to support learning. Prior to this, Carrie completed her PhD in mobile-assisted language learning at the University of Toronto, and she earned her MSc (University of Saskatchewan) in computer science with a focus on applying artificial intelligence techniques within computer-assisted language learning. Her work focuses on how to effectively use technology in educational settings.

University of Pittsburgh

Rae L. Mancilla, Ed.D. ([email protected]) is an Instructional Designer in the Center for Teaching and Learning at the University of Pittsburgh. She works in the division of online learning and has published in numerous SLA and CALL journals including Applied Linguistics, Learning Languages Journal, and Language Learning and Technology. Her research focuses on written syntactic complexity, technology-enhanced language learning, and professional development for Instructional Designers.

University of Pittsburgh

Valerie Swigart, PhD, MSN, MEd, CRNP is a full professor of nursing at the University of Pittsburgh. Her academic career has focused on the interface of technology, health care, ethics, and human behavior. She has completed two NIH funded studies with large qualitative databases. Her research has supported better understanding of the processes of surrogate decision-making regarding life-sustaining technology, the impact of long-term mechanical ventilation, homeless persons’ health care seeking behaviors, Middle Eastern women’s experience of sexual harassment and domestic abuse, and the use of MOOCs in the flipped classroom. Teaching interests include international research and collaboration, cross cultural communication and ethical/social justice issues. She is currently teaching a massive open online course about clinical terminology that has reached thousands of students over the globe.

This chapter examines whether a massive open online course (MOOC) can support specialized vocabulary learning by combining situated-learning and learning-by-testing approaches. This exploration of how student (n = 1,265) performance on MOOC assessments varies based on group membership and the activities students performed within the online course showed students achieved high performance standards within the MOOC. Those who were additionally enrolled in a nursing program (n = 103) demonstrated improved vocabulary knowledge following MOOC completion, and their self-reports indicate the online course and the learning materials it provided facilitated students’ aural comprehension of clinical terms outside the MOOC. The results of the automatically graded assessments, their relationship to student actions, and student completion of practice quizzes indicate both graded and ungraded assessment should be used to enable learning in online courses. Additional implications for online assessment and course design are highlighted within the chapter.

PART TWO: ASSESSING ONLINE TEACHERS

5. Issues and Challenges in the Assessment of Online Language Teacher Performance [+–] 93-115

Barbara A. Lafford,Carmen King de Ramirez,James E Wermers £17.50

Arizona State University

Barbara A. Lafford (PhD,
Cornell University) is a Professor of Spanish Linguistics
at Arizona State University and is Head of the Faculty of Languages and Cultures at the
Phoenix Downtown Campus. Her recent research interests include the study of the effects
of social and cognitive factors on the acquisition of second languages in classroom and
study-abroad contexts, languages for professional purposes, and computer-assisted
language learning. She has served as Chair of the CALICO board and is currently the
Editor for the Monograph/Focus Volume Series of The Modern Language Journal.
Her publications also include Spanish Second Language Acquisition: State of
the Science (Georgetown, 2003) and The Art of Teaching Spanish:
Second Language Acquisition from Research to Praxis (Georgetown, 2006), both
co-edited with Rafael Salaberry.

University of Arizona

Carmen King de Ramirez is an Assistant Professor of Spanish at the University of Arizona. She has worked as an online coordinator of Spanish courses at two major state universities, where she designed and launched course content for new Spanish programs. She serves as a certified online course reviewer for Quality Matters and is engaged in developing online pedagogy training for foreign language (FL) faculty and graduate students. Her research interests/publications include Online FL Pedagogy, Language for Specific Purposes (LSP), and Service Learning.

Arizona State University

James E. Wermers is the Digital Humanities Course Manager for the Languages & Culture faculty in the College of Integrative Sciences and a Faculty Fellow in the Center for the Study of Race and Democracy at Arizona State University. James coordinates training, development, and deployment of digital pedagogy and digital initiatives for a 50-member interdisciplinary faculty at ASU, teaches courses in composition, film, gender studies, literature, philosophy, and religious studies, and works on innovative and engaging community program, including the “philosophy and Film Series” and “Words on Wheels.”

This chapter will focus on four issues involved in the assessment of online language teaching (uneven training/uneven institutional expectations, faculty resistance to online teaching and to their assessment as online instructors, diverse backgrounds among online instructors, and instruments used to assess the performance of online faculty) the challenges these factors present for carrying out such assessments, and possible solutions for meeting each of those challenges. This chapter will also include brief findings from a pilot study that provided a rationale for the focus on these four issues associated with online instructor assessment.

6. Evaluating Teacher Tech Literacies Using an Argument-based Approach [+–] 116-137

Jesse Gleason,Elena Schmitt £17.50

Southern Connecticut State University

Jesse Gleason, PhD, is an associate professor of applied linguistics and Spanish at Southern Connecticut State University. Her teaching and research interests include bilingualism/biliteracy development and technology-mediated instruction. She has presented her research internationally in Australia, Belgium, Brazil, Canada, Chile, Germany and the U.S. as well as published research in a number of refereed journals, such as CALICO, System, and Language, Culture and Curriculum.

Southern Connecticut State University

Elena Schmitt is a Professor of Applied Linguistics and TESOL in the Department of World Languages and Literatures at Southern Connecticut State University in New Haven, CT. She received a Ph.D in Linguistics from the University of South Carolina. Elena’s research interest include first language attrition, heritage language studies, bilingualism, and language teacher training. Her recent publications appeared in the International Journal of Bilingualism, Bilingualism: Language and Cognition, and Linguistic Encyclopedia of Slavic Languages.

Across the US, the need for language teachers with technology-specific literacies is climbing, yet many teachers may be underprepared for such positions due to inadequate experience with or training in computer-assisted language learning (CALL) (Godsey, 2015; Lynch, 2016). What is more, language teacher education programs may struggle to equip teacher candidates with the tools they need to approach an ever-changing multimodal landscape (Borthwick & Gallagher-Brett, 2014; Nami, Marandi & Sotoudehnama, 2016; Sehlaoui & Albrecht, 2009). In the Northeastern US, in a region with critical need for language teacher training, faculty in an MS Bilingual Education & TESOL program took up the challenge to train PK-12 teachers using state-of-the-art videoconferencing (VC) technology. The present study showcases data taken from one course in this program, and uses an argument-based approach to validity (Chapelle, Enright & Jamieson, 2008; Kane, 2006) to make the claims that teachers in this course (a) were able to transform their traditional face-to-face (f2f) teaching practices to sophisticated technology-mediated teaching practices within a synchronous online environment and (b) had equivalent content learning opportunities to those in the on ground courses.

7. Face-to-Face Teacher to Online Course Developer [+–] 138-164

David Donnarumma,Sarah Hamilton £17.50

BPP University

David Donnarumma is Head of English for BPP University. He has been involved in the field of English Language Teaching for 15 years as a manager, academic, teacher educator, test developer and materials developer both online and in print. He is also an author of a Communication Skills textbook. He has presented at several international conferences and written in the areas of ELF communication, assessment and online learning. In his current role he is responsible for English language development and delivery at the University. He also holds external examining positions at the Universities of Wolverhampton and Exeter.

BPP University

Sarah Hamilton is the Online Programme Leader for the BPP University School of English Language and Foundation Studies. Prior to joining the school Sarah worked both as a tutor in the Business School and as a member of the Universities central Learning and Teaching team focussing on Assessment Enhancement activities. She is also an online author of Business Communication skills and Human Resources eLearning materials for accountancy students. In her current role she is responsible for online curriculum design and supporting the teaching team in their move from face to face to online.

Even though many teacher development programs now include online learning, the question of whether these teachers are ready for online course development still remains. It has been well documented that language teachers need a working knowledge of information and communications technology (Gilbert, 2013; Stickler & Emke, 2015). This leads to the challenge of how best to prepare teachers for online course design. Most studies have focused on what the teacher does during the online course with much emphasis to date being placed on technological competence. This small scale study seeks to show that readiness for online is as much about the values and beliefs of the teacher as it is about their digital literacy skills (technological competence) and the implications this has for assessing a teacher’s readiness for online course development.

PART THREE: ASSESSMENT TOOLS FOR ONLINE ENVIRONMENTS

8. Innovative Implementation of a Web-Based Rating System for Individualizing Online English Speaking Instruction [+–] 167-183

Hyejin Yang,Elena Cotos £17.50

Iowa State University

Hyejin Yang is a lecturer in universities in Seoul, South Korea. She obtained her Ph.D. degree in Applied Linguistics and Technology at Iowa State University. Her academic interests lie in speaking language assessment, automated writing evaluation, and computer assisted language learning.

Iowa State University

Elena Cotos is an associate professor in the Applied Linguistics Program and graduate faculty in the Human Computer Interaction Interdisciplinary Program at Iowa State University. She is also the Director of the Center for Communication Excellence of the Graduate College. She investigates written and spoken discourse, automated writing evaluation, the application of corpus-based genre analysis to applied natural language processing tasks, and computer-assisted language learning and assessment.

Technology offers unique features for integrating assessment of and for language learning in online instructional environments. This chapter focuses on validating the diagnostic uses of a web-based rating platform called R-PLAT, which captures multiple aspects of language performance that can serve as diagnostic evidence needed to inform transitioning from a face-to-face speaking course to a more individual ability tailored online mode. Empirical evidence obtained from quantitative and qualitative data supports the judgmental assumption that the diagnostic descriptors recorded via R-PLAT are indicative of target speaking ability levels used for placement into proficiency-based levels of the course. Overall, this work presents an example of how theoretically-grounded evaluation frameworks can serve to overarch the validation of the uses of computer-based assessment tools for the purpose of shaping a strong learner fit quality for online language teaching.

9. A Systematic Approach to Vetting Reading Comprehension Items for Inclusion in Cloud-based Assessments [+–] 184-197

Fabiana MacMillan £17.50

Rosetta Stone

Fabiana MacMillan, Ph.D., serves as Head of Assessment at Rosetta Stone®. She has focused her research on the interface of language assessment, particularly reading assessment, and written discourse analysis. She has taught English as a second or foreign language in Brazil and Canada and has been involved in language test development in several languages for over fifteen years.

Fabiana MacMillan reports the trial of a process for vetting reading comprehension items for inclusion in online placement and achievement assessments—the Systematic Approach to Item Vetting (SAIV). The chapter begins with an outline of the SAIV system used to perform a qualitative analysis of all items in the dataset prior to field testing. Next is a brief general description of the dataset and the tests for which the items were developed. This is followed by a description of the quantitative analysis of learner response data using the Rasch model (Rasch, 1980) to obtain difficulty parameter estimates and discrimination indices for each item. The penultimate section discusses results of the analyses and the extent to which qualitative data correlates with field test outcomes. Finally, her chapter concludes by showing that the SAIV process may help predict which items will be successful after field testing and has promise in also predicting item difficulty.

10. The Lingo of Language Learning Startups: Congruency Between Claims, Affordances, and SLA Theory [+–] 198-218

Gabriel A Guillen,Thor Sawin,Sarah E. Springer £17.50

Middlebury Institute of International Studies

Gabriel Guillen is an Assistant Professor in Spanish at the Middlebury Institute of International Studies (MIIS) in Monterey. At MIIS he teaches content-based language courses such as Social Entrepreneurship in the Hispanic World and Spanish in the Community, connecting his Spanish students with English learners in the county of Monterey. His dissertation focused on online intercultural exchanges and the use of language learning social networks in the context of hybrid language education.

Middlebury Institute of International Studies

Thor Sawin is an Assistant Professor of Applied Linguistics at the Middlebury Institute of International Studies (MIIS) in Monterey with a background in second language learning and linguistic anthropology. His research in language learning among cross-cultural workers on the front lines of globalization sparked his interest in mobile devices as a low-cost and low-barrier means of connecting autonomous learners with authentic language in diverse and superdiverse environments.

Middlebury Institute of International Studies

Sarah Springer works at the intersection of design, research, teaching, and
technology as senior staff and adjunct faculty member at the Middlebury
Institute of International Studies (MIIS) at Monterey. Based in the San
Francisco Bay area, she leads workshops, creates materials, and facilitates
experiences that integrate human-centered design practices within projects and organizations. Her increasing engagement with the world of entrepreneurship is driven by a recognition that, as with teaching, a deep understanding of human needs, motivations, and behaviors is at its core.

The presence of 541 language learning startups on AngelList evidences the industry’s momentum. While language learners may welcome this proliferation of options, the authors hypothesized that some claims made by these startups reflect ignorance about the findings of Second Language Acquisition (SLA). Thus, learners may find these startups unable to deliver on their bold claims and give up on language learning altogether. To determine the effects of this, we analyzed the congruence between the claims of six archetypical startups (Memrise, Verbling, BliuBliu, Duolingo, Elsa, and ChattingCat), the actual affordances of their platforms, and recommendations from the field of instructed language acquisition. Commonly-held language fallacies (the minimal effort myth, a preference for native speakers, or a focus on simplistic conceptions of language as isolated easily-translatable words) were found, together with more realistic promises. Companies truly after measurable social impact should start by using more honest, transparent, and accurate language in their marketing literature.

PART FOUR: FUTURE DIRECTIONS FOR ONLINE LANGUAGE ASSESSMENT

11. Toward Technology-enhanced Alternative Assessment for Online Language Education [+–] 221-238

Zhi Li,Stephanie Link £17.50

University of Saskatchewan

Zhi Li is an assistant professor in the Department of Linguistics at the University of Saskatchewan, Canada. His research interests include language assessment, Technology-supported language teaching and learning, and corpus linguistics. His research papers have been published in System, CALICO Journal, and Language Learning & Technology.

Oklahoma State University

Latest developments in technologically rich environments (TREs) for language learning and teaching have prompted the need for innovative assessments to measure and support whole learning experiences, including not only cognitive but also social, affective, and contextual dynamics that influence learners’ developmental trajectories. To address this need, we argue for a reform in traditional online assessment practices that often rely solely on quizzes, discussion boards, and course grades. We instead promote technology-enhanced alternative assessments to help account for the dynamic, multifaceted experiences taking place in TREs. Supporting our call is a discussion of the current roles of technology in designing and developing alternative language assessments. We then provide considerations for implementing technology-enhanced alternative assessments, including theoretical recommendations and evidence-based model discovery. A more comprehensive understanding of learning experiences in TREs has the potential to complement traditional approaches to assessment and aid in efforts to simultaneously integrate learning and assessment into online language education.

12. Argument-based Approach to Validation in Online Language Education [+–] 239-256

Erik Voss £17.50

Northeastern University

Erik Voss is an Associate Teaching Professor at Northeastern University. His research focuses on validation research, corpus linguistics, and language assessment and technology. He has presented on technology use in the language classroom and language assessment research at domestic and international conferences and workshops.

The goal of learning-oriented assessment is to provide feedback to support language learning. In fact, Popham (2014) suggests that teachers who use effective educational assessments will become better teachers. However, in order to be confident that the language assessment scores are appropriate for their intended purpose and correct decisions are being made based on the interpretation of the scores, such assessments need to be evaluated through a validation process. This chapter will highlight the importance of validation of learner-oriented assessment practices. Next, it will present the framework for the argument-based approach to validation (Chapelle, Enright, & Jamieson, 2008; Kane, 2006). The chapter will conclude with a consideration how the argument-based framework can provide effective assessment information for evaluation of language learning and pedagogical practices in online language environments.

End Matter

Author Index [+–] 257-265

Stephanie Link,Jinrong Li FREE

Oklahoma State University

Georgia Southern University

Subject Index [+–] 266-269

Stephanie Link FREE

Oklahoma State University

Reviews

This is the third book from the Advances in CALL Research and Practice series. It is therefore fair to say that it is the work of—and intended for—CALL enthusiasts. However, it deserves a far wider audience as it raises important questions about the future of assessment. In many ways, this collection breaks new ground.
IATEFL Voices

.

Books and Journals in Humanities, Social Science and Performing Arts

Assessment Across Online Language Education

Stephanie Link [+–]

Oklahoma State University

Jinrong Li [+–]

Georgia Southern University

Table of Contents

Reviews

Related Journal

Related Interest

Search Equinox

Subjects