The Sound of Immersion. Understanding the Impact of Audio in Immersive Journalism

By Noëlle S. Lebernegg
Supervisor: Dr. Sanne Kruikemeier

Introduction

Virtual reality journalism is on the rise. By employing affective digital storytelling techniques, journalists are aiming for increased audience engagement, due to stronger involvement (i.e., immersion) in the news story (Papacharissi, 2015; Wahl-Jorgensen, 2013). As a result of first-person experiences of news content, it is assumed that immersive journalism (IJ) may instigate emotional involvement among audiences, which was found to increase the feeling of actually being in the depicted environment (i.e., the sensation of presence) (Slater, 2009; Baños et al., 2004; Riva et al., 2007). Specifically, first-person experiences were found to increase emotional reactions (i.e., physiological responses) to the depicted content, leading to increased sensations of presence, which in turn increases emotional engagement of users and, by that, positively affects the applied effectiveness of IJ. Previous studies, however, emphasized conceptualizations of IJ and in that mainly the determinants of this sensation of presence in IJ (de la Peña et al., 2010; Slater, 2009, Cummings & Bailenson, 2016).

Nevertheless, several important questions remain concerning the determinants of presence on the one hand, as well as the role of emotions in a conceptual framework of IJ, on the other hand. Hence, the present project examined the extent to which auditory cues (i.e., the use of narration or music) in IJ productions elicit emotional arousal, and how this relates to presence, emotional engagement (i.e., empathic perspective-taking), and the applied effectiveness (i.e., news subject understanding) of IJ. As such, this project addresses essential gaps in the literature on IJ and moves beyond current literature by assessing auditory determinants of presence and doing so using physiological measurements for arousal during exposure additionally to traditional self-reports.

Determinants of Presence: The Impact of Auditory Cues

Previous research on determinants of the sensation of presence in virtual environments focused on interaction potential as well as technological features. The latter mainly concerns the impact of different devices and virtually recreated embodiment (Cummings & Bailenson, 2016).

The relevance and impact of auditory cues have not yet gained considerable attention, which is particularly surprising as these additional technological audio features are almost always used in IJ (De Bruin, De Haan, Goutier, Kruikemeier, & Lecheler, 2019). Evidence from related fields, however, suggests a considerable impact of additional auditory cues such as music and narration due to the emotional arousal they elicit (e.g., Andreano et al., 2009; Jørgensen, 2010; Baumeister et al., 2004; Sloboda and O’Neill, 2001). Hence, the first hypothesis concerns the emotionality of auditory cues. Particularly, (1a) music as well as (1b) narration are assumed to elicit stronger emotional arousal (i.e., physiological responses) in a virtual environment in comparison to respective environments equipped with ambient sound only.

Two Roles of Emotions in IJ

An issue that occurs when trying to determine the role of emotions in immersive journalism is the inconsistent and somewhat misleading use of measurements for the perception of presence. That is, some consider measurements for presence, that are used as an indication of emotional engagement by others (Cummings & Bailenson, 2016). As such, many studies employ a way broader definition of presence, not only including the illusion of place and plausibility, but also engagement in the environment, and even empathic perspective-taking (ibid.), which clearly exceeds the feeling of presence in the virtual environment, but touches upon involvement, and emotional engagement. To examine the different roles of emotions in immersive environments, presence as a concept should be defined more clearly, and reduced to what it is: the subjective sensation of being in an environment and the perceived realism of events occurring in the same (Slater, 2009). Respectively, research thus far – whether intended or not suggests a manifold relationship between presence and emotions, while findings include emotional arousal (i.e., physiological responses) as well as emotional engagement with the environment depicted (i.e., empathic perspective-taking), while the first was found to mediate the effect between immersion and presence (Baños et al., 2004; Makowski et al., 2017; Riva et al., 2007), and the second as a mediator between the sensation of presence and the applied effectiveness (i.e., news subject understanding) of IJ (Beckett & Deuze, 2016; Sundar et al., 2017). Accordingly, hypothesis 2 proposed increased sensations of presence in virtual environments equipped with music (a) or narration (b) in comparison to ambient sound only due to the increased emotional arousal they elicit in comparison to VR environments with ambient sound only. Hypotheses 3 and 4 state an indirect effect of additional auditory cues on the applied effectiveness (i.e., news subject understanding) of IJ productions: Both, music (H3) as well as narration (H4) are assumed increase the applied effectiveness of IJ in comparison to ambient sound only due to (a) the emotional arousal they elicit, (b) the sensation of presence resulting, and (c) subsequently increased emotional engagement (i.e., empathic perspective-taking) with the environment.

Research Design and Procedure

To examine the impact of audio and emotions in IJ, two virtual reality experiments were conducted using between-subjects designs. Two different VR experiences were used. In study one, participants were exposed to the 360° video “Support Refugees – Refugee Crisis”, a virtual experience about the beginnings of a refugee’s journey, depicting a family’s run from their home town due to an upcoming war.

In the second study, the video “My Home Shatila” was shown, a VR experience produced by the Amsterdam based virtual reality production studio “VR Gorillas”, with the aim of increasing awareness and empathy towards refugees. In this video, a girl named Fadia guides the user through the camp Shatila in Lebanon, outlining the living circumstances in refugee camps rather than the previous escape experience.

Both studies were conducted as part of one joint study; hence, participants (N = 131, Mage = 21.41; 70% female) were the same in both studies. Subjects were randomly assigned to one of three conditions (voice/music/control) with emotional arousal, presence, empathy, and news subject understanding as dependent variables. During exposure, electrodermal activity (=EDA) was measured using the E4 wristband, a non-invasive tool to measure changes in skin conductance. After exposure, participants were asked to fill in an online survey on QUALTRICS, indicating the levels of arousal using two more measurement tools, as well as the sensation of presence, empathic perspective-taking, and news subject understanding.

Main Findings

The first aim of the outlined studies was to test the impact of sound on the sensation of presence in IJ due to the emotional arousal they elicit. In contrast to previous studies (e.g., Andreano et al., 2009; Jørgensen, 2010; Baumeister et al., 2004; Sloboda and O’Neill, 2001), emotion-evoking potential of music and narration cannot be confirmed based on the present study. Neither a significant relationship between music or narration on emotional arousal nor effects on presence or news subject understanding were found.

The second goal of this project was a further clarification of the differential roles of emotions in a conceptual framework of IJ. In line with previous research that found emotional arousal triggering the sensation of presence (Baños et al., 2004; Riva et al., 2007), the present studies results suggest a direct positive effect of experienced emotional arousal on perceived presence. Additionally, news subject understanding seems to be mediated by emotional arousal, presence as well as empathic perspective-taking. Hence, the findings suggest no significant effects of sound on emotional arousal. Nevertheless, the role of emotions in IJ proved tremendous for its efficacy. Thus, rather than focusing on the sensation of presence only, future research and conceptualization attempts of IJ should invest effort in a further examination of the different roles of emotions, it’s determinants as well as consequences.

References

  • Andreano, J., Liang, K., Kong, L., Hubbard, D., Wiederhold, B. K., & Wiederhold, M. D. (2009). Auditory cues increase the hippocampal response to unimodal virtual reality. Cyberpsychology & behavior : the impact of the Internet, multimedia and virtual reality on behavior and society, 12(3), 309–313. doi: 10.1089/cpb.2009.0104
  • Baños, R. M., Botella, C., Alcañiz, M., Liaño, V., Guerrero, B., & Rey, B. (2004). Immersion and emotion: Their impact on the sense of presence. Cyberpsychology and Behavior, 7(6), 734–741. doi: 10.1089/cpb.2004.7.734
  • Baumeister, R. F., Zhang, L., & Vohs, K. D. (2004). Gossip as cultural learning (Vol. 8) (No. 2). doi: 10.1037/1089-2680.8.2.111
  • Beckett, C., & Deuze, M. (2016). On the Role of Emotion in the Future of Journalism. Social Media and Society, 2(3). doi: 10.1177/2056305116662395
  • Cummings, J. J., & Bailenson, J. N. (2016). How Immersive Is Enough? A Meta-Analysis of the Effect of Immersive Technology on User Presence. Media Psychology, 19(2), 272–309. doi: 10.1080/15213269.2015.1015740
  • De Bruin, K., De Haan, Y., Goutier, N., Kruikemeier, S., & Lecheler, S. (2019). Soaking Up the News? A Content Analysis of Immersive Journalism. Presented at the annual meeting of the International Communication Association in Washington DC, United States.
  • De la Peña, N., Weil, P., Llobera, J., Giannopoulos, E., Pomés, A., Spanlang, B., … Slater, M. (2010). Immersive journalism: Immersive virtual reality for the first-person experience of news. Presence: Teleoperators and Virtual Environments, 19(4), 291–301. doi: 10.1162/PRESa00005
    Jørgensen, K. (2010). Time for new terminology? Diegetic and non-diegetic sounds in computer games revisited. doi: 10.4018/978-1-61692-828-5.ch005
  • Makowski, D., Sperduti, M., Nicolas, S., & Piolino, P. (2017). “Being there” and remembering it: Presence improves memory encoding. Consciousness and Cognition, 53(May 2016), 194–202. doi: 10.1016/j.concog.2017.06.015
  • Papacharissi, Z. (2015). Toward New Journalism(s): Affective news, hybridity, and liminal spaces. Journalism Studies, 16(1), 27–40. doi: 10.1080/1461670X.2014.890328
  • Wahl-Jorgensen, K. (2013). The strategic ritual of emotionality: A case study of Pulitzer Prize-winning articles. Journalism, 14(1), 129–145. doi: 10.1177/1464884912448918
  • Riva, G., Mantovani, F., Capideville, C. S., Preziosa, A., Morganti, F., Villani, D., … Alcañiz, M. (2007). Affective interactions using virtual reality: The link between presence and emotions. Cyberpsychology and Behavior, 10(1), 45–56. doi: 10.1089/cpb.2006.9993
  • Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549–3557. doi: 10.1098/rstb.2009.0138
  • Sloboda, J. A., & O’Neill, S. A. (2001). Emotions in everyday listening to music BT Music and emotion: Theory and research. In Music and emotion: Theory and research (pp. 415–430).
  • Sundar, S. S., Kang, J., Oprean, D., Shyam Sundar, S., Kang, J., & Oprean, D. (2017). Being There in the Midst of the Story: How Immersive Journalism Affects Our Perceptions and Cognitions. Cyberpsychology, Behavior, and Social Networking, 20(11), 672–682. doi: 10.1089/cyber.2017.0271