Cross-modal correspondence: Different modes, common codes? Investigating musical engagement with an Ecological cognitive approach
Context and background
Our senses are interconnected, we constantly make sense of the information that is available in our environment. We give meaning to the information based on our knowledge and experience (mental models). Something we seem to know from ordinary life in that moods have colours, colours suggest flavours, sounds can be heavy or light and personalities can be spiky. We take for granted that the stage lighting for a concert is the “right” stage lighting and we seldom experience mismatched senses. When we do, it´s strange: a piece of slow pace music paired with yellow, rapidly flashing lights, for example, would seem wrong to some individuals.
The work in this project with Bjørn Dreyer and Olivier Lartillot touches a little on these kinds of topics but in a less dramatic way. I wanted to inquire into how the things we pay attention to affect the way we hear things. Artistic researcher and guitarist Bjørn wanted to share his cross-sensory experience with the audience; researcher Olivier created visualisation in the concert. He also wanted to make an app called “Synesthesizer” that allowed listeners to assign colours to the music. I was able to work with these amazing people to find out how the audience reacted to the sounds and colours. I also asked people who weren´t at Bjørn´s concert about their sensory reactions. My main roles here were to design the interactions, to conduct user research, and to develop data collection and data analysis methods and evaluate the data with the 4E framework. There were two phases of experiments (concert and non-concert settings). Do music listeners share common codes when they listen to the music in different modes?
My research questions were:
RQ 0: How is music perception affected by the sensory channels a subject attends to?
RQ 1: What is the relation of the listening conditions to the way the music content is understood in a concert setting?
RQ 2: What is the relation of the listening conditions to the way the music content is understood in a non-concert setting?
RQ 3: What is the difference between these two sets of results (RQ1) and (RQ2)?
Short project summary
If you are interested in the topic, read my whole thesis!
This project explored music listeners’ cross-modal associations. A 2-stage experiment was conducted in a live concert setting and an offline non-concert setting. An ecological cognitive approach was applied as a foundation of the project. Data collection methods such as interviews, observations and open-ended surveys were used. Due to the complexity of participants’ responses, I used content analysis and coding to categorise the variables into themes and tags. Four major codes were generated: 1) emotional, 2) colour, 3)conceptual and 4) attentional. These suggest aspects of cross-modal music listening that research can look into further. A mixed-methods analysis is recommended for future research such as the correlation analysis to investigate individual feature-to-feature correspondence. Moreover, I found that timbre (i.e. the sound of a guitar or plucking string instrument) affords a calm sensation and elicits the cooler tone colour (i.e. blue, yellow and green). Also, the use of a C major, quiet in dynamic and slow in tempo appeared to elicit imagination to the themes such as Nature and Narrative such as walking in nature or a forest. This cross-modal mental imagery effect of music suggested that listeners could use slow, quiet, guitar/plucking strings music in combination with cool tone colours as an emotional regulation strategy. In HCI, this could further be tested and implemented as a music therapeutic tool. Another finding between music and imagined finger movements and colour suggested that using colour and music or even letters to teach motor skills to young children may be feasible.
The general conclusion I draw from the findings of this work suggests that cross-modal correspondence is a potentially rich topic of exploration that the small-scale nature of this project only touched upon. The relation of music to vision and emotion is a three-way relation of three complex phenomena with numerous variables. Part of the phenomena are based on physical causal relations and part of them are inevitably cultural and related not to causation but to meaning and interpretation. This obvious conclusion is that research in music requires all the tools available to researchers to adequately do justice to music´s physical, cultural and emotional aspects.
Graphical summary
