Abstract

During my master’s thesis, I have designed and developed a tech platform where a mobile application creates slideshows from multimedia content uploaded in a web application titled “Multimedia Slideshow Maker” (MSM). The project is carried out for an external partner, Alight AS, for a project called Alight. Alight is a mobile tech platform aiding caregivers in sending personalised video sessions to patients with dementia. The thesis aimed to determine to what degree MSM can be used independently by a caregiver, without instructions from others or prior experience in video editing.

The MSM was designed and developed in iterations based on feedback from experienced evaluators and tested with end-users. The system has been compared to similar applications, where both strengths and limitations of the MSM were discovered. Participants were to a large extent able to use MSM without instructions from others, where MSM enabled them to create a multimedia slideshow without requiring experience in video editing. The sample size, however, might not account for all differences in abilities among caregivers.

Background

The number of people in Norway and the rest of the world with dementia is increasing every year. Dementia is an incurable disease and does not only affect people with dementia but also their caregivers, families and the entire society (World Health Organization, 2019). Music and reminiscence therapy has been used to improve the quality of life for people with dementia and their caregivers (Istvandity, 2017). As the number of people with dementia is rising, we need to be able to provide care in the patients’ own homes and be able to care for more people at once.

A company called Alight AS is working on a project called Alight. Alight is currently a mobile tech platform for healthcare workers at nursing homes and the home healthcare service. The platform is connecting the healthcare worker to a patient, aiding and sending personalised video sessions to patients with mild to moderate dementia. These so-called “personalised video sessions” are developed with the intention of providing music and reminiscence therapy in a mobile solution, where the structure and content of a video session consist of:

  • Personalised image slideshow with personalised music
  • Video of the healthcare worker (exercise, live playing, breathing exercise etc.), or other personalised videos.

In the current state of Alight, the video sessions are manually made by employees at Alight. Moving forward, Alight’s vision is to let any caretaker use the tech platform independently to care for people with mild to moderate dementia. A caretaker can be a therapist, home nurse, family, friend, anyone with close relations to a person with dementia.

The purpose of the thesis

To help with Alight’s vision, this project has involved the creation of a system that creates multimedia slideshows (video sessions) from content uploaded by a caregiver in MSM. A caregiver can be just about anyone; old, young, tech-savvy, inexperienced in using web apps, with potentially no experience in video editing. This meant that the system had to be designed for a high degree of usability, making the system itself do most of the editing. The aim was then to determine to what degree the MSM can be used independently by a caregiver, without instructions from others or prior experience in video editing.

Music and reminiscence theory

“Music therapy is recognized as an established health profession that uses music to facilitate therapeutic processes. Even without a professional music therapist, many patients and clinicians listen to or play music to manage stress, anxiety, and pain in clinical settings.” (Kemper & Danhauer, 2005, p. 283).

Tone Sæther Kvamme (2008) argues that music therapy is a vital necessity for people with dementia. that music can give people with dementia access to feelings, help them express themselves, awake memories, strengthen their identity, and also give a sense of achievement and affiliation. Even Ruud (2008) argues that the experience of music and how it affects us depends on our musical background, influence, our choice of music, and the situation we are in while experiencing the music.

“Reminiscence therapy involves the discussion of memories and past experiences with other people using tangible prompts such as photographs or music to evoke memories and stimulate conversation” (Woods et al., 2018, p. 1).

A systematic review by Istvandity (2017) found positive effects in combining music and reminiscence therapy, with effects on the mental well-being of people with dementia, especially stress, anxiety and depression. The study could not determine what is the successful delivery of the combination of music and reminiscence therapy.

The research shows the benefits of using both music and reminiscence therapy to improve the quality of life for patients with dementia, but there is less knowledge of how these therapies should be delivered to be successful. It would be interesting to investigate the effects of music and reminiscence therapy delivered by any caregiver through a mobile app, but that is beyond the scope of this project.

The system

The system consists of a web app (MSM) and a mobile app. These platforms are connected through a backend service called Firebase. The intention is that a caregiver will upload content (images, music and video) in the MSM and then send it to an account belonging to the person with dementia. The person with dementia can then sign in to the mobile app and view the multimedia slideshow that is created in real-time in the app. Currently, the system connects the content that is uploaded, to the same account, letting the caregiver preview the multimedia slideshow when signing in to the app. Letting the caregiver send the content to another account will be implemented when it potentially will be integrated with the platform of Alight.

System overview

The MSM

The MSM was developed with JavaScript, using a library called React and Firebase as a backend service. React was used to build the user interface, making functionalities for uploading content (music, images and video), and for letting the user rearrange the images to their choosing. Firebase was used as a service for authentication (login), database (metadata) and storage (multimedia content).

Technical overview of MSM

The mobile app

The mobile app was made to both create and present a multimedia slideshow in real-time. The reason for making the mobile app do this was the intention to include a music streaming service at a later stage. When a caregiver has to choose music to be consumed by another person it can quickly become a problem with sharing what may often be copyrighted material. Using a music streaming service can deal with this problem.

The mobile app was developed using a cross-platform development tool called Flutter. Cross-platform means that I could write code that works for both Android and IOS, making the development both easier and faster. Flutter is quite a new tool, with the first stable version released in December 2018. As with many new products, bugs can be encountered, and of course, I did. A day of work could go by, trying to solve a bug. With the help of Even Brenna, an experienced developer at Alight AS, and a strong community of developers at Stack Overflow, it was possible to fix and proceed.

The functionality of making a multimedia slideshow was developed with the help of three awesome libraries:

The programming language of Flutter is called Dart. This was used to handle fetching of data from Firebase, to add images to cache memory, and to schedule all events, making it look something like this:

Example of multimedia slideshow

Designing the user interface for a high degree of usability

“Usability is most often defined as the ease of use and acceptability of a system for a particular class of users carrying out specific tasks in a specific environment” (Holzinger, 2005, p. 71).

The biggest part of the project was to develop a user interface with a high degree of usability. This was achieved through the use of two methods. The first method used is called Heuristic evaluation (HE), a method from what Holzinger (2005) describes as User Inspection Methods. HE involves usability specialists judging the system, providing feedback to whether the system follows established usability principles or not. I used some popular usability principles called Nielsen’s Usability Heuristics.

I had a total of four design iterations where I received feedback from people with experience in user interaction, constantly improving the MSM based on the feedback received.

It went from being a single-page app, looking like this:

First iteration

To a multistep form, looking like this:

Current design

The image above shows the step where a user can add and rearrange images by dragging and dropping them in the white container box. The checkmarks above the box are there to illustrate progression in the form. In the previous steps, a title has been added to the multimedia slideshow, and a song has been uploaded. The final step is to add a video, and then submit all the content to Firebase. The content is then connected to the account, where it can be retrieved from Firebase in the mobile app, and a multimedia slideshow is created.

User tests and results

The second method used was to test with potential caregivers (end-users). User tests were carried out with one-on-one video conferencing sessions with a total of eight participants, using a service called Whereby. Each participant was guided to download a beta release of the mobile app, and to download multimedia content (music, video and images) which they could use during testing of MSM.

During the user test, the participant was instructed to register and sign in to MSM and to create a multimedia slideshow by following the steps instructed by the application itself. While the participant was using the application, I used two methods: Thinking aloud and field observation. Thinking aloud means that the participant verbalises his/her thoughts while going through the process. Field observation involved the participant sharing the screen, making it possible for me to see every action the participant did and if problems or errors occurred. After the test session, I performed an interview and got feedback on the participant’s thoughts about MSM’s usability and the problems that had appeared.

The tests showed several usability problems, which affects Holzinger’s (2005) five usability characteristics; Learnability, efficiency, memorability, low error rate and satisfaction. No critical errors appeared, but there were some unhandled events with missing or insufficient feedback to the participant. There was one element that stood out as a confusing and non-intuitive, and that was the plus/cross-icon at the home page, which you can see at the bottom of the page in the image below.

Confusing element

This is the icon that a user had to press to start making a multimedia slideshow. One user had to be told to press it to get started, and three other users asked if they should press it to proceed. The problem with the element is that it does not say “click here to make a multimedia slideshow”. There should be a label, and it should be made to look more “clickable”. There is a problem with “perceived affordance”, a term coined by Norman (2004), based on the original term “affordance” by J.J. Gibson. One participant did not perceive that an action was possible at all, and it was not clear to the other three participants what action it afforded. It can be argued that the three participants who asked if they should press the element would have done so on their own without me being present as an observer, and would have learned its action. Still, it should not be necessary for participants to learn the action’s outcome, and therefore it should be improved.

Several other usability problems surfaced, like image previews being too small for some users, an image appearing in a large format while dragging in Firefox, placeholder text being a bit confusing, and partly unclear illustration of progression. Images being too small is quite a serious problem, where one participant was observed to move closer to the screen to view them. As MSM is meant to be used by any caregiver, the application needs to account for older people with reduced eyesight. This has to be improved in the future version.

Most problems affected the overall satisfaction of use. This does not mean that it should not be addressed, it very much should. Most participants went through the application and completed all necessary steps on their own. Even with its usability problems, the participants reported that MSM was easy to use, easy to learn and that it was a quick process.

Comparison with similar applications

There already exit web-based applications for making multimedia slideshows. Three commercial applications that I can compare MSM to, are:

  • Magisto; markets itself as an AI (artificial intelligence) based video creation and editing platform.

  • Adobe Spark Video (ASV); markets itself as a video maker that lets the user create videos without design skills.

  • Animoto; markets itself as an easy to use, quick to learn, video slideshow maker.

All of these share a common goal: To make a video maker that is easy to use. they all edit and generate a video file from the content you provide as a user. They provide a selection of editing styles (transitions, filters and effects), but the number of editing possibilities varies across the applications. Magisto leaves most of the editing to an AI-based Emotion Sense Technology. ASV includes options such as video trimming, reordering of media, in-browser microphone recordings and more. Animoto has the most editing capabilities, including every feature present in ASV and more.

What separates MSM most from these applications is that MSM is made for a single purpose. MSM follows the predefined structure, defined by Alight, where it starts with displaying images while the song is playing, and when the song is finished, the video is displayed (as you can see in the video further up in the blog). Since the videos that are added can be a greeting from a caregiver, musical performances, explanations of exercises and so on, it was important to stop the music while the video is shown. In all the aforementioned applications, the music will play over all the images and videos that are added, and you have no option to change this.

Magisto is the application that is closest to MSM, where most of the editing is left to the system itself. MSM and Magisto are the only two applications where the system will decide the display time of images. In ASV and Animoto, the user has to manually set the display time on every image.

Many of the editing capabilities found in these applications will be interesting to add to MSM, and may very well be added in a future version, but still, it is very important to keep MSM as easy to use for a user without experience in video editing.

Comparison with a similar application with the same goal and purpose

An app proposed by Imtiaz et al. (2018) shows similarities in both system and purpose. The application is described as “a mobile multimedia reminiscence therapy application to reduce behavioural and psychological symptoms in persons with Alzheimer’s”. Alzheimer’s disease is the most common cause of dementia.

The biggest difference from MSM is that they propose a single mobile application for both the caregiver and the person suffering from dementia, where the caregiver uploads multimedia content in the app on the device, and the patient watches the created multimedia slideshow on the same device. In my opinion, the part of the application that involves the presentation of the multimedia slideshow should be easily shared between caregivers. Let’s say a health worker puts together a multimedia show for a patient living at home. If one caregiver could send the multimedia show over the Internet to another account, any caregiver; friend, family, a health worker, can view the presentation together with the person with dementia.

Imtiaz et al. (2018) focus more on assessing the effects of individualised music therapy and reminiscence therapy delivered through a mobile app, and not on assessing the usability of the tool for a caregiver. The research has not gotten to the point where they have been able to test this in a clinical setting, so sadly I cannot say what therapeutic effect this application has on people with dementia.

Conclusion and future work

Participants were to a large extent able to use MSM without instructions from others, where MSM enabled them to create a multimedia slideshow without requiring experience in video editing. The sample size, however, might not account for all differences in abilities among caregivers. The MSM has both strengths and limitations compared to similar applications. MSM does close to all editing itself and is tailored to the structure defined by Alight. Still, it can be improved in some areas, giving the users more freedom with added features, and improving the design to account for the usability problems that were encountered.

I would like to investigate the effect of music and reminiscence therapy on people with dementia, delivered through a mobile app from any caregiver, not only those that are educated as a music or reminiscence therapist. When the mobile app is designed and developed for a person with dementia, this can be assessed in clinical trials. I think it is important to assess both the therapeutic effectiveness and usability by using both qualitative and quantitative methods.

If you want to read a more detailed description of the project, you will at some point be able to find the thesis at NTNU Open. Search for Multimedia Slideshow Maker.

References

Holzinger, A. (2005). Usability engineering methods for software developers. Communications of the ACM, 48(1), 71–74. https://doi.org/10.1145/1039539.1039541

Imtiaz, D., Khan, A., & Seelye, A. (2018). A Mobile Multimedia Reminiscence Therapy Application to Reduce Behavioral and Psychological Symptoms in Persons with Alzheimer’s. Journal of Healthcare Engineering, 2018, 1–9. https://doi.org/10.1155/2018/1536316

Istvandity, L. (2017). Combining music and reminiscence therapy interventions for wellbeing in elderly populations: A systematic review. Complementary Therapies in Clinical Practice, 28, 18–25. https://doi.org/10.1016/j.ctcp.2017.03.003

Kemper, K. J., & Danhauer, S. C. (2005). Music as Therapy. Southern Medical Journal, 98(3), 282–288. https://doi.org/10.1097/01.SMJ.0000154773.11986.39

Kvamme, T. S. (2008). Musikk for demensrammede – en livsnødvendighet? In Perspektiver på musikk og helse: 30 år med norsk musikkterapi (pp. 487–497). Norges musikkhøgskole.

Norman, D. A. (2004). Affordances and Design. 5.

Ruud, E. (2008). Et humanistisk perspektiv på norsk musikkterapi. In Perspektiver på musikk og helse: 30 år med norsk musikkterapi (pp. 5–28). Norges musikkhøgskole.

Woods, B., O’Philbin, L., Farrell, E. M., Spector, A. E., & Orrell, M. (2018). Reminiscence therapy for dementia. Cochrane Database of Systematic Reviews. https://doi.org/10.1002/14651858.CD001120.pub3

World Health Organization. (2019). Dementia. https://www.who.int/news-room/fact-sheets/detail/dementia