Your brain on VR
The psychology of VR, how to design immersive worlds and the positive and negative effects of the medium.
The psychology of Virtual Reality
The buzz around Virtual Reality, VR, is everywhere. The technological advances are starting to make scenarios reserved for sci-fi a reality. Spanning the likes of The Matrix’s fully immersive worlds, Star Trek’s holodeck and Minority Report’s personalised and targeted advertising.
Described by Forbes as ‘technology’s next big wave’¹, Google, Facebook and now Apple are investing heavily in the sector. With Google’s cardboard VR simulators and Facebook’s purchase of Oculus Rift last year, VR devices are starting to make their way into our homes and classrooms². Undoubtedly, it’s an exciting field.
What is VR?
In it’s simplest definition, VR is an immersive and interactive experience based on real-time 3D graphics. They are experienced through a headset, with Google cardboards on one end of the scale and a fully tracked and integrated headsets on the other.
Designing VR with psychology in mind
How to create immersion and presence in VR?
As you dip into VR literature, ‘immersion’ and ‘presence’ are highly prominent. But, what exactly are they and how can we design to optimise for them?
Studies into VR and VR professionals alike agree that the most convincing VR worlds are those which appear seamless. Seamless worlds are a mixture of what is termed immersion (the ‘technological quality of the media’) and presence (the ‘technological quality of the “experience of being there”’). They both play a role in maintaining the fabric and believability of a virtual world.
A lot has been written about how to effectively use immersion and optimise for presence. The model by the psychologist Wirth³ is the clearest and structures presence as a two step process:
“The user must draw upon spatial cues” which create the appearance of a “plausible space”, creating a “sense of self-location”.
The user must feel like they are located in that space. Which in turn presents the possibility that they can act and interact within that space.
What are the cues of presence?
When creating VR worlds, Wirth’s model gives the framework for determining spatial cues. They include: Static monocular cues
These are elements that help give a perception of depth within the VR world, including the use of perspective, relative size of objects, and use of textures.⁴ Here’s a great article that explains it in full.
The movement of objects, environments and the avatar can enhance presence, by utilising the layering effects of motion parallax and cast shadow motion when objects are moving.⁵
This is a great paper for more on Binocular cues.
How to create immersion?
Immersion is based primarily around the application of the technology. Slater and Wilbur’s system outlines the three elements needed to ensure immersion:
A world that offers high fidelity and multi-sensory simulations.
Which finely maps the actions of a person’s VR body to their physical body.
And uses self-contained plots and narratives in the VR world so that they become detached from what is happening in the external world.
What are the technological features of immersion?
Immersion is determined by a using a balance of 7 features.
1. Tracking level
Tracking level is the degrees of freedom the user is tracked whilst in the VR world. The better the tracking, the better the sense of self-location and feedback.
2. Stereoscopic vision
Use of either monoscopic or stereoscopic visuals. Stereoscopic visuals present two slightly different images to each eye, aiding depth perception in the array. Monoscopic on the other hand presents the same image to both eyes, so doesn’t create as immersive experiences.
3. Image quality
Both the design of the image aesthetics and technological feasibilities affect the image quality. From the realism and detailed features of the display to the technical aspects of flicker rate, lighting types and resolution.
4. Field of view
How much the user can see in the environment’s visuals.
5. Sound quality
A world which is devoid of sound is less believable, and a little eerie. Using sound within a VR world increases user feedback and makes the space more believable. There are a number of sound channels and effects that can increase immersion. These include ambient sound of generalised sound effects and sounds that happen within view – for example character voices, footsteps, and specialised sounds.
6. Update rate
The rate the virtual environment is rendered.
7. User perspective
Shifting the perspective from 1st person, where a person embodies their avatar, to 3rd person, where a person can view their avatar.
With the dependence on technology as being the main determinant in creating a believable VR world, it leads to the question.
How immersive is enough?
Glancing at the list, it is only natural to assume that the more technology and immersive quality results a higher level of presence.
A study by psychologists James Cummings and Jeremy Bailenson⁶ suggests that this is only part of the story. In their study, they tested which elements resulted in higher presence. They found that three main elements produced higher presence. These are high levels of user-tracking, the use of stereoscopic visuals instead of monoscopic and wider fields of view of the visual displays.
Reeves and Nass also found that a ‘high fidelity of visuals have no impact of user attention, recognition or subjective experience.’⁷ What appears to be important are functional graphics. By all accounts, people are still able to extract the spatial cues from the simplest of graphics as they are from the top of the range effects.
How do you determine ‘place’ in a VR space?
to the connotation of ‘place’. In a VR context it is the difference between thinking you are in a VR world, and being in the VR world. When you remove physical walls and buildings, how can the same attribution of ‘place’ be created in VR worlds?
Balakrishnan and Sundar⁸ provide the answer. In their study, they show that ‘media factors’ otherwise known as the affordances of the technology affects a person’s ability to define space and place. Both the design of the VR display and the interactive capabilities cause the shift over from simple space creation to the idea of one being in a place.
Three main elements produced higher presence:
High levels of user-tracking
The use of stereoscopic visuals instead of monoscopic
Wider fields of view of the visual displays.
Virtual touch in a Virtual world
Touching is intrinsic to human happiness. It allows us to physically connect with the world around us and everything within it.
The ability to touch a virtual world not only makes it more immersive, but it is a huge element of creating a believable and enjoyable world. So, understanding how people interact with each other is crucial in designing a virtual world.
The subtleties of how humans interact will have to be mirrored in a virtual world⁹. We wouldn’t want to be left with the VR equivalent of a limp handshake or indeed a misalignment of the wide array of social touching that people integrate on a daily basis. It could easily become awkward. Psychologists Jeremy Bailenson and Nick Yee examined touch, both in the physical and virtual reality¹⁰.
It appears people use a lighter touch when touching different areas of the body. People touch faces with a more gentle touch than the torso or limbs. The same light touch was used when people touched representations of others, with a heavier touch used for inanimate objects.
They also recorded a difference in how male and female avatars were touched. Male avatars were touched with more force than female representations, by both sexes.
The affordances of the technology affects a person’s ability to define space and place.
Design of a VR avatar
Turning the focus from the VR environment to the design of the avatar bring to light some interesting psychological research.
The avatar is a powerful representation of self. Where it is the equivalent of our VR twin or representation of our choosing, any manipulation in our avatar has psychological impact on our real world selves. Some may argue that we have used a number of avatars in gaming – anyone that created a Mii character can contest to how attached we become to them. The difference being we embody a VR avatar. Being able to both see and feel the cause and effects of our behaviour. Interacting in a VR world and the effect of getting the sensory feedback means we are intrinsically linked to these representations.
This embodiment can have both positive and negative effects on our real world selves. Bailenson’s study on Doppelgängers shows how this psychological bridge can be used for good.¹¹
Virtual Reality as a technology is unique in its integration and connection with the person under the headset. The psychological bridge of immersion and presence is an area of study that interests many different types of psychologists, from cognitive to behavioural and sociological. How these are bridged is due to our perception. The study of how we perceive and interpret the world around us is called perception psychology.
Psychology of perception
Perception psychology is the analysis of sensory information within the brain. As we go through our day, we are surrounded by the rich stimuli of modern life and we rely heavily on our senses to tell us about where we are within that world. Through perception, we get a description of our surroundings and what they mean.
Processing Visual Information in the Brain
We process information three ways. Bottom-up sensory processing¹², sensorimotor self-awareness frameworks¹³ and top-down processing¹⁴. All depict the flow of information a person processes when seeing and interacting with objects in both the real and virtual world.
Top-down processing depicts how prior knowledge and past experiences help us to establish accurate perceptions of the world around us. These includes our memories, heuristics and emotions.
Bottom-up processing takes the view that all the information required for perception comes from our sensory inputs. From the images entering our eyes to the body’s weight, position, temperature, touch, skin response etc. The brain then combines all these information to give your body a sense of place in the world.
For VR it means both how we design the scenes, and craft the viewers internal and emotional responses are as important.
When the body receives a number of sensory feedbacks that correlate - the brain is more likely to believe that information as true. For VR, this means more immersion.
When conflicting information is presented, it causes the brain to reject certain sensory information.¹⁵ This is one of the main causes of simulator sickness. When simulator sickness occurs, the visual input might indicate movement, while the vestibular system does not.
To combat this we can optimise peripheral visual content to increase the number of sensory points that the brain can refer to.¹⁶ This is especially useful when you have a high movement sequence. Another way is to reinforce opportunities to activate multi-sensory neurons - neurons that have more than one sensory input¹⁷. In short, combining visual with auditory feedback.
Sensorimotor frameworks help the brain judge whether the information it is processing is correct. These compare the internal representations of the actual sensory input, the desired input and the predicted reaction in the world after an action has been taken.
Meaning, they give frameworks of how the brain judges the information it is perceiving for reliability, with what the best action the body can take to react to whatever it is experiencing.
When the sensory input matches what it expects to experience then it is more likely to infer that the sensory input is correct.
This explains how VR is so believable, both in passive and active VR experiences. It creates immersion in passive use - with 1840 stereoscope or in modern 360 videos. It also explains why people engage in believable active VR, with intentional interaction with the environment and exploration. The effect is even stronger when a person is represented by a VR avatar. To merge the sensory expectation with the visual feedback of an avatars movement of their arm or head. When designing for VR, it is useful to merge these frameworks into how we layer the information and what feedbacks we give to people.
Sensory frameworks relate to a person’s top-down processing. When sensory input into the brain fits actual, desired and predicted response frameworks it primes the brain to believe the world more. As a result, more top processing is carried out to reinforce the VR illusion.
A study by Haggard¹⁸ found that action binding mechanisms are perceived closer together. The action of pressing a button, and the delayed feedback of the buzzer sound. In effect, the brain is choosing to ignore information that doesn’t conform to the sensory framework, or it manipulates it to make it fit.
The brain’s self-selectiveness is useful to overcome technological limitations, including latency and object grasping. When the VR world has a lag, a person doesn’t notice and it doesn’t affect a person’s immersion in that experience. With object grasping, the brain ignores delayed tactile feedback and target drift.
When conflicting information is presented, it causes the brain to reject certain sensory information. This is one of the main causes of simulator sickness.
What are the positive and negative effects of VR on real world behaviour?
VR has been proven to have a positive effect on a person’s real world self, from saving more trees to saving for their future. It sounds like an amazing new technology, but we must heed the warnings. Worryingly, it can also make people meaner and possibly more violent.
Summarised below are the latest findings from the top psychologists.
The positive effects of VR
The Superhero effect
Have you ever imagined yourself as a superhero? VR can give you the opportunity to step into the shoes of Superman, Wonder Woman or your hero of choice. By immersing a person in a superhero world, every good deed simulated in VR can increase pro-social behaviour in the real world. A study into Virtual Superheroes by Rosenberg et al¹⁹ recreated Superman in VR, giving participants the power of flight. Their mission was to find and save a lost diabetic child by flying around the metropolitan city. Those with the experience of ‘super flight’ were more pro-social post experience, helping the experimenter pick up more pens when they ‘accidentally’ spilt them over the floor. Interestingly, they were quicker to help and helped for longer than those who didn’t get the Superman experience.
Observing your VR avatar engaging in exercise increases your real world belief that you can successfully exercise and achieve your health aims. For the most of us, sticking to a new exercise schedule is difficult. Not least because it takes a long time to start to see the physical effects of all your hard work. In the VR study, for every minute a person ran in the physical world, their avatar got physically slimmer. Directly linking the cause and effect reinforces the positive behaviour. Participants engaged in 10 times more exercise and stuck to their goals more than the research group that didn’t use the avatar²⁰.
The time travel effect
Looking towards the future is difficult. Humans are notoriously bad at carrying out behaviours that would benefit their future selves. Whether it be saving for retirement or drinking less at the weekend. This is until they are confronted with their future selves in VR. The ability to take an avatar and age it in front of a person’s eyes makes them more future conscious. Connecting the behaviours of now with the visible effects of the future increases self-preserving behaviours. In Ersner-Hershfield’s study, feeling more connected to our future selves means that we are more likely to save for old age²¹.
The possible negative effects of VR
Can VR make us violent?
With the immersive aspect of VR, are the fears many associated with gaming relevant to VR? A mass of research shows that violent media increases aggression and violent behaviour, both in the short and long term²². The same can be said for violent activities in VR²³. There is only a small amount of research into the effects of VR and aggression behaviour. But the few studies agree that it does increase arousal²⁴ and aggressive thoughts²⁵ due to the first person, immersive interaction with the behaviour.
Creating false memories
Humans are innately creative beings, and no more so than when it comes to our recall of past events. It leads to an odd phenomenon called ‘false memories’. Memories that make us not only think, but 100% believe that we have been somewhere or done something when we haven’t. They can be a simple distortion of event recall or a completely fabricated memory. It can be caused by something as easy as looking at a photograph, adding in different details that can be mixed into your memory of something²⁶. Segovia took the idea of false memories and wondered what impact VR would have. Interestingly, Segovia found that when we witness a doppelgänger engaging in a certain behaviour or activity, we are more likely to create a false memory and think we did the same behaviour in the real world.
Worryingly, children are more susceptible to the creation of false memories²⁷.
Segovia took a group of children on a day trip to VR Sea World, where the children swam with whales. When asked about their trip a week later, the children not only believed they swam with whales in the real world, they also elaborated with recall stories to include details not in the simulation, like what they ate beforehand.
Other real world effects
Adverts are increasingly getting smarter and more personalised. Could we be heading to a ‘Minority Report’ style of advertising? Where adverts call out our own names and sell to us directly? With VR, that is a real possibility.²⁸
Personalisation of a VR environment and the psychology of self-endorsing could be the next step in advertising. Studies have shown that what a person wears and uses in a VR world can affect their opinions in the physical world. If your avatar wears a top emblazoned with a brand’s logo, you are more likely to recall and prefer that brand in the physical world.²⁹
Your avatar acts as an extension of yourself. So when you see your avatar acting as a product endorser, the ‘self-referencing effect’ means you are then more likely to want that brand later on.
Gerber, R. Four Big Technology Trends For 2016
Statt, N. Google is offering its virtual reality classroom system to schools for free. 2015
Wirth, W. Hatmann, T. Böcking, S. Vorderer, P. Klimmt, C. Schramm, H. A process model of the formation of spatial presence experiences. 2007
Hu, B. Knill, D. Binocular and Monocular Depth Cues in Online Feedback Control of 3-D Pointing Movement. 2011
Cummings, J. Bailenson, J. How Immersive Is Enough? A Meta-Analysis of the Effect of Immersive Technology on User Presence. 2016
Balakrishnan, B. Sundar, S. Where am I? How can I get there? Impact of navigability and narrative transportation on spatial presence. 2011
Jakobsson, M. Virtual Worlds & Social Interaction Design. 2006
Bailenson, J. Nick Yee, N. Virtual interpersonal touch: Haptic interaction and co-presence in collaborative virtual environments. 2008
Bailenson, J. Doppelgängers – a new form of self? 2012
Calvert, G., Spence, C., and Stein, B. E. The Handbook of Multisensory Processes. 2004
Blanke, O. Multi-sensory brain mechanisms of bodily self-consciousness. 2012
Haggard, P., Clark, S., and Kalogeras, J. Voluntary action and conscious awareness. 2002
Dijkstra, N. Zeidman, P. Ondobaka, S. Distinct Top-down and Bottom-up Brain Connectivity During Visual Perception and Imagery. 2017
Xiao, R., and Benko, H. “Augmenting the field-of-view of head-mounted displays with sparse peripheral displays,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2016
Stein, B. E., and Stanford, T. R. Multi-sensory integration: current issues from the perspective of the single neuron. 2008
Haggard, P., Clark, S., and Kalogeras, J. Voluntary action and conscious awareness. 2002
Rosenberg, R. Baughman, S. Bailenson, J. Virtual Superheroes: Using Superpowers in Virtual Reality to Encourage Prosocial Behavior. 2013
Bailenson, J. Doppelgängers – a new form of self? 2012
Ersner-Hershfield, H., Bailenson, J, & Carstensen, L Feeling more connected to your future self: Using immersive virtual reality to increase retirement saving. 2008
American Psychological Association Violence in the Media - Psychologists Study TV and Video Game Violence for Potential Harmful Effects 2013
Calvert, S. Tan, S-L.Impact of Virtual Reality on Young Adults’ Physiological Arousal and Aggressive Thoughts: Interaction Versus Observation. 1994
Rovira, A. Swapp, D. Spanlang, B. Slater, M. The Use of Virtual Reality in the Study of People’s Responses to Violent Incidents. 2009
Tamborini, Ron, Matthew S. Eastin, Paul Skalski, and Kenneth Lachlan. Violent virtual video games and hostile thoughts. 2004
Loftus, E. Pickrell, J. The Formation of False Memories. Psychiatric Annals. 1995
Brainerd , C. Reyna, V. Forrest, T. Are young children susceptible to the false-memory illusion? 2002
Bailenson, J. Doppelgängers – a new form of self? 2012