About Us
Concept
The analysis, storage and retrieval of information from the visual images and video texts proliferating through the rapid advance of digital technology is a major problem for government, industry and business. Efficient practices for managing data in text-based documents have been developed. However, visual images (e.g. photographs, drawings and diagrams) and video texts remain elusive with regards to management, coding and searching for information. The problem intensifies as video-sharing websites become the standard forum for exchanging information. Dynamic digital texts clearly require theoretical concepts and analytical approaches which extend beyond those developed for text-based data.
Social scientists and computer scientists do not usually work together to solve problems. However, the two groups of researchers are working together in the Multimodal Analysis Lab to develop new approaches to multimodal and multimedia analysis.
Multimodal social semiotics provides a theoretical basis for integrating the two domains, because interactive digital technology is multimodal social semiotic technology. That is, interactive digital technology provides a common platform for semiotic resources (e.g. language, visual imagery, gesture, movement, music, sound etc) to combine and unfold in new and innovative ways. The functionality of semiotic resources and the ways in which semiotic choices combine for the construction of knowledge and events in the process-based environment of interactive digital technology is investigated in the Multimodal Analysis Lab.
The research collaboration between social scientists and computer scientists in the Multimodal Analysis Lab, each sharing, developing and expanding their own knowledge base and expertise, will advance existing approaches to multimodal and multimedia analysis.
For further information see Multimodal Analysis Lab and O’Halloran, K. L. (2007) Multimodal Analysis and Digital Technology (download pdf file) in A. Baldry and E. Montagna (eds.) Interdisciplinary Perspectives on Multimodality: Theory and Practice. Campobasso: Palladino.
