Advanced collaborative spaces, tele-immersive systems and the Nidarøs Sculpture
In the MCT elective course ‘Sonification and Sound Design’ curated by Anna Xambó we got exposed to many different approaches within the context of the sonification field from the arts and sciences. Sonification is concerned with research functions, e.g. the uncovering of patterns that lie outside our usual sphere of consciousness and are difficult to grasp visually. But sonification also functions in artistic and playful auditory forms. The lectures were in that sense elaborative, innovative and visionary without claiming an ownership on sonification and it’s ‘real’ purpose. Even though some academic practices certainly attach more importance to firm definitions other practices don’t give it such a prominent place even though the approach is the same.
The art of science or the science in art
I was facilitating the talk of Leif Arne Rønningen, our second guest lecturer, a professor emeritus from the department of telematics at NTNU (today the department of information security and communication technology). Not only is he drawing knowledge from academia but he also held several job positions, from being a diving instructor- and marketing manager to research director at the Department of Signal Processing and System Design at SINTEF. Leif Arne Rønningen consulted also for one of the biggest Norwegian tele-communication companies and was involved in digital TV solutions for Europe. Topics, that are on the radar in the moment are for example audio-visual installations in the context of music-drama and advanced city planning.
In his guest lecture he introduced us to ‘Advanced Collaboration Spaces, requirements and possible realisations’ and to the ‘Nidarø Sulpture’, a dynamic vision and audio sculpture. In both parts Leifs main research areas on tele-immersive collaboration systems and low-latency networks are on the forfront. Leif Arne provides us an in the nutshell overview on camera lense constructions required for immersive experiences through multi-perspective image transmissions and showed areas where it could be applied: office space, travel scenarios among other examples. The Nidarøs Sculpture is an architectural piece and encompasses a lot of Leif Arne’s technical concepts. Presented in a artistic manner it includes the sonification of a short clip he made to demonstrate this architectural idea.
Generating sound from vision, method of sonification
The sonification process Leif chose was generated from changes in the image in the short demo-clip on the Nidarø sculpture. Up to that time point we have not come across that method in the course. He explains how changes in a sequence of video images can be temporal or spatial. The sounds he mapped to the sequences are inspired by an opera, three music instruments are used for melody, chord and drums. To reduce processing which as explained is very heavy in this undertaking, Leif focussed to measure a subset of pixels. The changes in the intensity for the selected set of pixels were measured. When switching from one image frame to the next he then was drawing different variables from those in a RGB format. The input from the image frames was then mapped to operational if-then rules.
Questions, answers and reflections
The Q&A’s from the students though were mainly addressed to his visions and ideas of tele-immersive systems and how distant they are from now, in years. One question was if mixed reality and collaborative spaces would then compete in some way, but Leif stresses their complementary role towards each other. It is about studying and finding the connecting dots, a collaboration of different systems and approaches. Another questions was addressed to one of the applied examples of these so called ‘multi-view-architectures’, it is the concept of the travel bureau: Beside the near-natural-virtual communication environment through a multi-view perspective on the landscape and sensoral integration into the happenings of an touristic event, it would of course also include a corresponding soundscape. From an engineering perspective Leif-Arne Rønningen’s ‘hands-on’ concepts are so advanced but practical at the same time that he is usually at least 20 years ahead of the current practice. But they are technically already possible he explained, only too expensive for a mass market. External factors could eventually accelerate research, and the days in which these “vacation scenarios” become real are most likely closer than we think and want.
To classify as a contribution for the portal the article should have been much longer, it has a lot to do with what is trying to be established in our study program. But already in the late 1980ties Leif-Arne Rønningen started researching on low latency image and video transmission in collaborative environments. What seems new and innovative to a lot of us MCT people in terms of telematic practices was thought of quite before our time and remained often a lot of distant theory. Therefore it was an honor to be exposed to an overview of in depth on Advanced Collaboration Spaces. It could be very interesting to extend the possibilities of telematic musicking with Leifs technical concepts and see where it goes. Especially to learn about the perceptual aspects that would accompany such technical developments. The approach to sonification was very refreshing and new to me, as I like to work with visuals and video, Leifs method is a nice inspiration.