Multimodalities

Introduction

If you have ever heard your professors mention multi-modal writing, they are likely referring to writing that uses text, as with traditional essays, and some other media, such as images or video. Multi-modal writing is becoming increasingly popular in college classes because this kind of writing is becoming more popular in professions.

Mode vs Media

A mode is a means of communicating. A medium is the channel or system through which communications are conveyed. The plural form of medium is media. So, for example, if we want to communicate in the linguistic mode, we might choose the medium of print. If we want to communicate in the aural mode, we might choose the medium of a podcast. Both print and podcasts are forms of media.

When analyzing or producing multimodal compositions, it is important to recognize the operation of multiple modes within artifacts. But it is also useful to think about which mode generally predominates in any given medium. Both photographs and films, for instance, employ the visual mode. Films differ from photographs, however, in that they involve movement of bodies and objects through space (spatial mode). We might say, then, that the visual mode predominates in photographs, the spatial mode in film. Being aware of dominant modes within a medium will prove helpful later when choosing powerful claims and persuasive evidence for composing your own multimodal argument.

Modalities

In semiotics, a modality is a particular way in which information is to be encoded for presentation to humans, i.e. to the type of sign and to the status of reality ascribed to or claimed by a sign, text or genre. It is more closely associated with the semiotics of Charles Peirce (1839–1914) than Ferdinand de Saussure (1857–1913) because meaning is conceived as an effect of a set of signs. In the Peircean model, a reference is made to an object when the sign (or representamen) is interpreted recursively by another sign (which becomes its interpretant), a conception of meaning that does in fact imply a classification of sign types.

The psychology of perception suggests the existence of a common cognitive system which treats all or most sensorily conveyed meanings in the same way. If all signs must also be objects of perception, there is every reason to believe that their modality will determine at least part of their nature. Thus, the sensory modalities will be visual, auditory, tactile, olfactory, gustatory, kinesthetic, etc. A list of sign types would include: writing, symbol, index, image, map, graph, diagram, etc. Some combinations of signs can be multi-modal, i.e. different types of signs grouped together for effect.

But the distinction between a medium and a modality should be clarified:

1. text is a medium for presenting the modality of natural language;

2. image is both a medium and a modality;

3. music is a modality for the auditory media.

So, the modality refers to a certain type of information and/or the representation format in which information is stored. The medium is the means whereby this information is delivered to the senses of the interpreter.

Natural language is the primary modality, having many invariant properties across the auditory media as spoken language, the visual media as written language, the tactile media as Braille, and kinetic media as sign language. When meaning is conveyed by spoken language, it is converted into sound waves broadcast by the speaker and received by another’s ears. Yet this stimulus cannot be divorced from the visual evidence of the speaker’s manner and gestures, and the general awareness of the physical location and its possible connotative significance. Similarly, meaning that is contained in a visual form cannot be divorced from the iconicity and implications of the form. If handwritten, is the writing neat or does it evidence emotion in its style. What type of paper is used, what colour ink, what kind of writing instrument: all such questions are relevant to an interpretation of the significance of what is represented. But images are distinguishable from natural language. For Roland Barthes (1915–80), language functions with relatively determinate meanings whereas images “say” nothing. Nevertheless, there is a rhetoric for arranging the parts which are to signify, and an emerging, if not yet generally accepted, syntax that articulates their parts and binds them into an effective whole. Rhetorician Thomas Rosteck defined rhetoric “as the use of language and other symbolic systems to make sense of our experiences, construct our personal and collective identities, produce meaning, and prompt action in the world.”

Examples of multimodalities include websites and/or podcasts that incorporate images, graphs, and narrations.

Summary

Multimodality describes communication practices in terms of the textual, aural, linguistic, spatial, and visual resources – or modes – used to compose messages. Where media are concerned, multimodality is the use of several modes (media) to create a single artifact. As the world is becoming increasingly more digital with communication and presentation, incorporating multimodal presentations in your college career will assist you in embedding your knowledge of digital literacy in your workplace.

Sources:

“Multi-modal writing.” By Excelsior Online Writing Lab. Retrieved from: https://owl.excelsior.edu/online-writing-and-presentations/multi-modal-writing/ Licensed under: CC-BY