Dr. Tuomo Hiippala
I do research on multimodality, that is, how language, images, layout and other forms of communication interact with each other in different documents and situations. You can find my projects, publications, papers in preparation and teaching resources below.
I currently work as a post-doctoral researcher at the Centre for Applied Language Studies, University of Jyväskylä, but I am also affiliated with the University of Helsinki through a number of research projects.
Using computer vision for multimodal analysis
I am currently researching how computer vision techniques may be used to describe, sort and analyze the content of photographs taken and uploaded to Instagram by tourists around Helsinki, Finland. The data are retrieved automatically from specific geographical areas within a certain timeframe. The initial work has been supported by City of Helsinki Urban Facts (Helsingin kaupungin tietokeskus).
Information graphics: knowledge, its transformation and impact
I am currently applying for funding for my next research project, which focuses on contemporary information graphics. Although information graphics have become increasingly common, relatively little is known about how they combine language and images into unified wholes. I intend to use multimodal analysis to uncover and describe their structure, while also considering how the information graphics contextualise knowledge using Legitimation Code Theory.
Expressions of interest in collaboration are very welcome: feel free to send me an e-mail.
Representation of Helsinki in tourist brochures from 1850s to present day
Together with my colleague Dr. Salla Jokela, we are working on a nonfiction book on how Helsinki, the capital of Finland, has been portrayed in tourist brochures between 1850 and the present day. We both studied the Helsinki tourist brochures in our doctoral dissertations and drew similar conclusions: the city has been represented in different ways during different times, reflecting changes in the society and in the city. In this book, we tell the reader how and why in an accessible manner. Our initial work has been supported by the Uusimaa Regional Fund of the Finnish Cultural Foundation.
Combining computer vision and multimodal analysis: a case study of layout symmetry in bilingual in-flight magazines
In Building Bridges for Multimodal Research: International Perspectives on Theories and Practices of Multimodal Analysis / Peter Lang / 2015
In this article, I examine how bilingual documents use layout to signal the reader that their contents are semantically equivalent. Recently, I’ve been interested in computer vision and machine learning: I apply some of these techniques to sketch a method for studying layout symmetry in larger data sets. The article will be published in a volume edited by Janina Wildfeuer (Bremen University).
The Structure of Multimodal Documents: An Empirical Approach
Routledge / 2015
This book, which is largely based on my doctoral dissertation, develops a framework for studying how language, images and other modes of communication work together in page-based documents. In addition to revising and rewriting each chapter, two new chapters have been added, which focus on the page as a unit in a multimodal document and show how the framework can be applied to digital media. In addition to conducting an extensive, empirical study of the Helsinki tourist brochures published between 1967 and 2008, I examine how digital longform journalism, exemplified by publications such as New York Times’ “Snow Fall”, combines photography, short videos, cinematic transitions and written language.
Multimodal genre analysis
In Interactions, Images, and Texts: A Reader in Multimodality / De Gruyter Mouton / 2014
Drawing on the research conducted for my doctoral dissertation, I also contributed a chapter to a handbook, which shows how the concept of genre may be used to describe and compare different documents. My argument is that in order for the concept of genre to be useful, it needs to be tied to document structure. I then show how the visualisation tools developed for my dissertation can reveal different types of structure commonly found in documents. These structures form patterns, which arise from the communicative function of the document. These patterns help the user to recognise the document and interact with it.
Modelling the structure of a multimodal artefact
University of Helsinki / 2013
My doctoral dissertation studied how language, image and other communicative resources work together in tourist brochures. I compiled a set of tourist brochures published between 1967 and 2008, which advertised the city of Helsinki, into a multimodal corpus, describing their content, layout, rhetorical organisation and appearance using the Genre and Multimodality model. I then searched the corpus for patterns, which characterise the tourist brochures as a document. In addition, I explored how these patterns and their combinations have changed over time as a result of new production technologies.
The interface between rhetoric and layout in multimodal artefacts
In Literary & Linguistic Computing / 2013
In 2011, I gave a two-minute lightning talk about my research at the InterFace 2011 digital humanities conference. On the basis of my presentation, I was invited to contribute to a special section of Literary & Linguistic Computing. This paper develops some ideas about how the organisation of content as a part of a layout signals how it should be interpreted. I later explored these issues in more detail using the data set collected for my doctoral dissertation.
Reading paths and visual perception in multimodal research, psychology and brain sciences
In Journal of Pragmatics / 2012
This article emerged from a side project during my doctoral studies. After reading several articles on visual perception, while simultaneously annotating a multimodal corpus containing a rich description of document structure, I wanted to find a way to make the information stored in the corpus accessible to an eye-tracker. The article proposes a way of interfacing a corpus with an eye-tracker and outlines some research questions that could be investigated using such a combination. Unfortunately, so far I’ve been unable to test this approach in practice.
The localisation of advertising print media as a multimodal process
In Multimodal Texts from Around the World: Linguistic and Cultural Insights / Palgrave Macmillan / 2012
My first publication, a chapter in a volume edited by Wendy Bowcher (Sun Yat-Sen University). The chapter addresses questions of localisation, that is, what happens to documents when they are translated into another language and adapted for the target language’s culture. I argue that this encompasses far more than simply translating the text: attention must be paid to aspects of visual communication as well.
WORK IN PROGRESS
Aspects of multimodality in higher education monographs
In Multimodality in Higher Education / Brill / Forthcoming in 2015/2016
Learning to read academic texts is a key skill to any student or researcher. In this paper, I show how multimodal analysis can be used to reveal different types of structure present in academic texts, how these structures can be made explicit to the students, and how the content carried by these structures can be recontextualised in other learning materials. The paper will be published in a volume edited by Arlene Archer (University of Cape Town) and Esther Breuer (University of Cologne).
Communicating corporate social responsibility in in-flight magazines
A case study of Finnair’s “Blue Wings” in-flight magazine
Together with my colleague Ivan Berazhny from HAAGA-HELIA University of Applied Sciences, I have been studying how Finnair’s in-flight magazine “Blue Wings” presents the content related to corporate social responsibility. Applying tools developed for linguistic and multimodal analysis, we show how different communicative strategies are reflected in the structure of Blue Wings. We are currently revising the paper for resubmission.
TEACHING RESOURCES / OPPIMATERIAALI
Lyhyt johdanto tekstilingvistiikkaan kevään 2015 luentokurssilta. Aineisto, joka esittelee tekstilingvistiikan keskeisiä käsitteitä, on lisensoitu Creative Commons Nimeä – Ei Kaupallinen 4.0 Kansainvälinen -lisenssillä. Voit siis vapaasti ladata ja jakaa aineistoa, sekä muokata sitä tarpeen mukaan, kunhan mainitset aineiston alkuperän.