Multimodal Transcription and Text Analysis; A Multimodal Toolkit and Coursebook with Associated On-line Course; Anthony Baldry; Paul Thibault

Anthony Baldry [+–]

University of Messina

Anthony Baldry was formerly Full Professor in English Language and Translation, Faculty of Political Sciences, University of Messina.

Paul J. Thibault [+–]

University of Agder

Paul J. Thibault is professor in linguistics and communication studies in the Faculty of Humanities and Education, University of Agder, Kristiansand, Norway. He also currently holds the posts of Honorary Professor in the School of Foreign Languages and Literatures at Beijing Normal University and Honorary Associate Professor in the Faculty of Education, University of Hong Kong. He has held full-time appointments in the University of Hong Kong (2009-2012), Lingnan University, Hong Kong (2002), the University of Venice (1994-2005), the University of Padua (1992-1994), the University of Bologna (1984-1986, 1990-1992), and the University of Sydney (1986-1988), and Murdoch University (1982-1983). He completed his Ph.D., which was supervised by Professor M. A. K. Halliday and Professor Roger Fowler, at the University of Sydney in 1985. He is the recipient of various honours and awards, including, most recently, a University of Cambridge/University of Hong Kong Doris Zimmern research fellowship at Hughes Hall, University of Cambridge (2011) and in September 2012 he was appointed Associate Editor of Language Sciences. He was a member of the international Organizing Committee of the 1st International Conference on Interactivity, Language and Cognition (CILC2012) held at the University of Southern Denmark, Odense campus, 12th-14th September 2012. He is co-editor (with Anthony Baldry) of the book series English Linguistics and ELT, published by Equinox, London. His research interests include: distributed language and cognition, discourse analysis, functional grammar and semantics, educational linguistics, language development, multimodality and multimodal corpora, social theory, the bodily basis of cognition and semiosis, narrative theory, and philosophy of mind. His published books include: Social Semiotics as Praxis (Minnesota, 1991), Re-reading Saussure (Routledge, 1997), Discussing Conversation Analysis: The work of Emanuel A. Schegloff (ed., Benjamins, 2003), Language and Interaction: Discussions with John J. Gumperz (ed., Benjamins, 2003), Brain, Mind, and the Signifying Body: An ecosocial semiotic theory (Continuum, 2004), Agency and Consciousness in Discourse: Self-other dynamics as a complex system (Continuum, 2004), Multimodal Transcription and Multimodal Text Analysis (with Anthony Baldry) (Equinox, 2006) together with articles and book chapters. He is currently working on two new book-length projects: (1) Language, Body, World: A critical rereading of Hjelmslev; and (2) Distributed Language: The extended human ecology.

In its attempt to provide answers to the questions raised above, and many others, this book proposes concrete solutions to the problems of multimodal text analysis and transcription of printed texts, websites and film. As such, it constitutes a much needed course in multimodal text transcription and analysis. It also suggests ways in which multimodal discourse analysis can help both educators and students understand how meaning is made in the e-learning environments that now play such an important role in our lives. As befits a coursebook, the individual chapters of the book are carefully organised in such a way as to provide a step-by-step progression in theoretical and descriptive complexity.

Series: Equinox Textbooks and Surveys in Linguistics

Preliminaries

Foreword [+–] xi

Jay Lemke

University of Michigan

What are multimodal texts? How can we transcribe and analyse them? How can multimedia and internet help us in multimodal discourse analysis? What postproduction and authoring skills are needed to analyse a multimodal text or to develop a corpus of multimodal texts? How does integrating multimedia meaning-making resources into hypertext multiply our meaning-making potential? How does the study of language relate to multimodality and multimedia, in particular in the e-learning age? How, and to what extent, will multimodal discourse analysis re-shape linguistics? In its attempt to provide answers to the questions raised above, and many others, this book proposes concrete solutions to the problems of multimodal text analysis and transcription of printed texts, websites and film. As such, it constitutes a much needed course in multimodal text transcription and analysis. It also suggests ways in which multimodal discourse analysis can help both educators and students understand how meaning is made in the e-learning environments that now play such an important role in our lives. In both these respects, readers are encouraged to use the book in conjunction with an associated and freely accessible website which provides many illustrations and exercises that further contextualise and exemplify the insights and descriptions provided by the book. As befits a coursebook, the individual chapters of the book are carefully organised in such a way as to provide a step-by-step progression in theoretical and descriptive complexity.

Acknowledgements [+–] xiii-xiv

Anthony Baldry,Paul J. Thibault

University of Messina

Anthony Baldry was formerly Full Professor in English Language and Translation, Faculty of Political Sciences, University of Messina.

University of Agder

Preface [+–] xv-xviii

Anthony Baldry,Paul J. Thibault

University of Messina

Anthony Baldry is Professor in English Linguistics, Faculty of Political Sciences, University of Messina and continues to teach at the University of Pavia where he was Associate Professor at the Faculty of Medicine from 1998 to 2008.

University of Agder

Chapter 1

Introduction: Multimodal texts and genres [+–] 1-56

Anthony Baldry,Paul J. Thibault

University of Messina

Anthony Baldry was formerly Full Professor in English Language and Translation, Faculty of Political Sciences, University of Messina.

University of Agder

In this chapter, we begin our journey by looking at the ways in which resources typically combine to make meaning, characterising, in particular what we term the resource integration principle (Inset 3, pp. 18-19), which lies at the heart of multimodality. We do this mainly in relation to cluster analysis (Inset 5, p. 31) and phasal analysis (Inset 7 , p. 47). In particular, this will help us understand the relationship between individual multimodal texts and multimodal genres, or to put the matter in slightly different terms between instance and type (see Inset 13 , pp. 172-173). In this respect it is appropriate, as our very first step, to examine the relationship between text and society in terms of the links between context of situation and context of culture (Inset 1, p. 2) and text (Inset 2 , p. 3), a step which will help us subsequently to characterise the relevance of metafunctions (Inset 4, pp. 22-23) and primary and secondary genres (Inset 6, p. 43) in our approach to multimodal text analysis and transcription. 1.0 Introduction 1.1 Multimodal texts and the resource integration principle 1.1.1 Resource integration and the transcription of printed cartoons 1.1.2 Multimodal transcription of cartoon narratives and the question of the metafunctions 1.1.3 Sources of meaning in multimodal texts 1.2 Cluster analysis and the transcription of static multimodal texts 1.2.1 Multimodal transcription and questions of genre 1.3 Textual properties of short printed cartoons 1.4 Printed advertisements and their exemplification of the metafunctions 1.4.1 Metafunctions in relation to genre 1.5 Web pages and their transcription 1.6 Film texts and their transcription 1.6.1 The soundtrack 1.7 Conclusion

Chapter 2

The printed page [+–] 57-102

Anthony Baldry,Paul J. Thibault

University of Messina

Anthony Baldry was formerly Full Professor in English Language and Translation, Faculty of Political Sciences, University of Messina.

University of Agder

In this chapter we link some of the general principles outlined in the previous one to the analysis and transcription of the printed page, the scientific printed page in particular, in an approach which moves from general to increasingly specialist considerations. Specifically, in the first part of the chapter we consider the evolution of the page as a textual unit in its own right in contemporary society, characterising the role played by such resources as tables, charts and diagrams in this process and comparing texts from the 19th century with their counterparts from the end of the 20th century. The later sections of the chapter deal with the way in which scientific meanings are communicated to children in biology textbooks. In the course of the chapter, we will provide a detailed account, both in terms of text analysis and multimodal transcription, of the ways in which a metafunctional framework (see 1.4, pp. 38-44) can help us understand many aspects of the multimodal printed page, such as the spatial arrangement of items on the page, the relation between images and linguistic text, the relations between reader and multimodal text, and so on. 2.0 Introduction 2.1 The printed page and its evolution 2.2 The resource integration principle in the scientific page 2.2.1 How can we study tables systematically? 2.2.2 How does the page communicate? 2.3 Science textbooks and multimodal meaning making 2.4 Visual, verbal and actional semiotic resources in a table 2.4.1 Visual and verbal resources 2.4.2 Thematic development of the page: hierarchies of textual periodicity 2.4.3 Actional semiotic resources 2.5 Blood under the microscope: multimodality in a photographic display 2.6 Integration of scientific photographs and verbal text 2.6.1 The textual metafunction 2.6.2 The ideational (experiential and logical) metafunctions 2.6.3 The interpersonal metafunction 2.7 The Italian texts: differences with respect to the Australian texts 2.7.1 Reading paths 2.7.2 The use of colour 2.8 Expertise and authority vs. comprehensibility and accessibility 2.8.1 Linguistic resources 2.8.2 Visual resources 2.9 Conclusion

Chapter 3

The web page [+–] 103-164

Anthony Baldry,Paul J. Thibault

University of Messina

Anthony Baldry was formerly Full Professor in English Language and Translation, Faculty of Political Sciences, University of Messina.

University of Agder

How can we go about describing a website? Are web pages in fact pages? Or is the idea of the page just a metaphor? How do we account for the web page in terms of resources and the kinds of meanings that people make? And what about the web page as genre? Can we describe web pages in terms of different genres? How can we describe a particular pathway through a website and how can we relate a particular pathway to the virtual resources of a specific website as a whole? And what transcription techniques can we develop? How do websites work? How does this connect back to the notion of transcription? To what extent does the transcription speak for itself as a description of the web page? As we can see from these questions, there are many possible starting points in the analysis of web pages. Our chosen approach is to make some general observations about the nature of websites, which we illustrate with some examples. We then discuss a number of websites in detail applying the technique of multimodal transcription and text analysis to them. 3.0 Introduction 3.1 Page or screen? 3.2 Decoupling of material support and information on the computer screen 3.3 The relationship between web page, website, web users and web genres 3.4 The home page 3.5 The Nasa Kids home page 3.6 Creating a hypertext pathway 3.7 The British Museum Children’s COMPASS website 3.7.1 Children’s COMPASS home page: description of multimodal objects 3.8 A multimodal hypertextual thematic formation: daily life in Asia 3.8.1 Thematic system analysis: preliminary observations and an example 3.8.2 Multimodal thematic system development along a hypertext pathway 3.9 The action potential of hypertext objects 3.9.1 Experiential meaning 3.9.2 Interpersonal meaning . 3.9.3 Textual meaning 3.10 The virtual world of hypertext 3.11 Community or social network of users and practices? 3.12 The WWW as technological infrastructure and meaning-making resource 3.13 Conclusion

Chapter 4

Film texts and genres [+–] 165-249

Anthony Baldry,Paul J. Thibault

University of Messina

Anthony Baldry was formerly Full Professor in English Language and Translation, Faculty of Political Sciences, University of Messina.

University of Agder

Chapter 4 examines some instances of television advertisements and develops detailed transcription techniques for the further analysis of entire advertisements as a form of dynamic text. In this chapter, we return to, and further develop, some key questions about the discourse level of organisation that we first explored in Chapter 1. Chapter 4 also develops some specific ideas concerning the expression (material) stratum of multimodal texts. 4.0 Introduction 4.1 The Eskimo text: a macro-analytical approach to transcription 4.2 The Westpac text: an integrated approach to transcription 4.3 Etic and emic criteria in multimodal transcription 4.4 Phases, subphases and transitions 4.5 Column 1: Row number and time specification 4.6 Column 2: The visual frame 4.6.1 Visual frames and shots 4.6.2 Information structure: Given and New 4.6.3 Sequencing and relations of interdependency between shots 4.7 Column 3: The visual image 4.7.1 Specifying visual information 4.7.2 Perspective 4.7.3 Distance 4.7.4 Visual collocation 4.7.5 Visual salience 4.7.6 Colour 4.7.7 Coding orientation 4.7.8 Visual focus or gaze of participants 4.8 Column 4: Kinesic action 4.8.1 The meaning of movement 4.8.2 Interpersonal modification of movement 4.8.3 General observations on the notation of movement 4.9 Column 5: The soundtrack 4.9.1 Integrating auditory phenomena 4.9.2 Sound acts and sound events 4.9.3 Dialogic relations among sound events 4.9.4 A brief comment on the notation of the soundtrack 4.9.5 The rhythm of sound events 4.9.6 Accented rhythmic units 4.9.7 Rhythm groups 4.9.8 Degree of loudness 4.9.9 Duration of syllable, musical note, sound event 4.9.10 Tempo 4.9.11 Continuity and pausing 4.9.12 Dyadic relations among auditory voices: sequentiality, overlap, turntaking 4.9.13 Vocal register 4.10 Column 6: Metafunctional interpretation 4.10.1 Metafunctional notation in relation to Column 6 4.11 Display and depiction: two sides of the same semiotic coin in visual texts 4.11.1 Multimodal discourse analysis: the Mitsubishi Carisma advertisement revisited 4.11.2 From delimited optic array to visual text: the stratification of the visual sign 4.11.3 Transformations in the optic array: some examples from the Mitsubishi Carisma text 4.11.4 Visual transitivity frames and experiential meaning 4.11.5 Identity chains in visual semiosis 4.11.6 Dependency relations in the Mitsubishi Carisma text: implications for visual texts 4.11.7.Some sources of coherence in the Mitsubishi Carisma advertisement: Phase 1 4.11.8. Counter-expectancy and hypertext in the Mitsubishi Carisma advertisement 4.12 Conclusion: the shape of things to come

End Matter

References [+–] 251-260

Anthony Baldry,Paul J. Thibault

University of Messina

Anthony Baldry was formerly Full Professor in English Language and Translation, Faculty of Political Sciences, University of Messina.

University of Agder

Appendices [+–] 261-264

Anthony Baldry,Paul J. Thibault

University of Messina

Anthony Baldry was formerly Full Professor in English Language and Translation, Faculty of Political Sciences, University of Messina.

University of Agder

Index [+–] 265-270

Anthony Baldry,Paul J. Thibault

University of Messina

Anthony Baldry was formerly Full Professor in English Language and Translation, Faculty of Political Sciences, University of Messina.

University of Agder

Reviews

It is difficult to bring out the full semiotic potential of multimodal texts in the linear structure of a book, but Baldry and Thibault’s book succeeds magnificently. …The potential audience for this book and course is wide, spanning advanced undergraduates, lecturers and researchers in linguistics, media studies and related field, and media professionals: each group will come away with new insights.
Baal News, Autumn 2007

Multimodal Transcription and Text Analysis is an indispensable resource for the analysis and interpretation of multimodal texts as they unfold through time; the book develops a rich array of practical, theoretically principled and computationally enabled tools for deconstructing the interaction of verbiage, image and sound across registers including work on cartoons, web pages, textbooks and film. This is the single most important contribution to multimodal discourse analysis since O’Toole’s The Language of Displayed Art and Kress & van Leeuwen’s Reading Images created the field.

James R. Martin, Professor of Linguistics, Department of Linguistics, University of Sydney

After reading Multimodal Transcription and Text Analysis you will never look at texts monomodally, i.e. simply as texts, any longer. This book offers an excellent and comprehensive view and understanding of the ways our communication has changed in the age of Internet and other media and offers an extensive toolbox for analysing these integrated meanings in printed pages, web pages and films. The book encourages students and researchers as regards interdisciplinarity which is of vital importance when attempting to understand the complex communicative procedures used in today’s technological world.
Eija Ventola, Professor of English Philology, University of Helsinki

Multimodal Transcription and Text Analysis is a book that many of us have been looking for: a readable “how to” manual for analyzing images, websites, video and film, cartoons, magazine layouts, advertisements, textbooks, television programs, and computer games. Paul Thibault and Anthony Baldry strike just the right balance between rich examples and accessible explanations of the concepts that lie behind their practical methods. This book is, however, far more than a “how to” manual: it is a comprehensive introduction to the field of multimedia analysis.
from the Foreword by Professor Jay Lemke, University of Michigan

.

Books and Journals in Humanities, Social Science and Performing Arts

Multimodal Transcription and Text Analysis

A Multimodal Toolkit and Coursebook

Anthony Baldry [+–]

University of Messina

Paul J. Thibault [+–]

University of Agder

Table of Contents

Reviews

Related Journal

Related Interest

Search Equinox

Subjects