Virtual Experience
Humans are interacting species by nature. We love to meet, talk and exchange ideas. However, the current pandemic situation has forced us to find innovative ways to communicate and interact with each other.
At work, we used to meet physically in conference rooms, and more often than not, people used to forget to start a virtual conference for folks that could not attend in person. It was not a necessity unless requested by someone. I used to feel very weird having an entire meeting on a video call.
However, after the pandemic hit, in-person business communication came to a screeching halt, and virtual interaction became imperative. Everybody was forced to experience “the virtual world”.
What exactly is a virtual experience? I do not want to confuse virtual and online. Virtual is not necessarily online.
A virtual experience is an experience in the absence of a real one (sometimes in the lack of a real one). That experience could be anything; it could be a conversation, an adventure, or even something physically impossible.
I like to illustrate this with the following schematic:
There is a user, and there is an experience. There lies a wall between the user and the experience. The more transparent (or invisible) the wall is, the more engaged the user will be in the experience.
My theory is that the more senses a person uses to enjoy the experience, the more transparent the wall will be. If the user just sees the experience, there is a certain level of engagement. But if the user sees, hears, talks, touches and walks within an experience, there is a whole new level of engagement.
We can classify virtual experiences based on the types of senses the person uses to enjoy them.
Let’s look at these experiences one by one.
If you want the user just to see the experience, you can present a text article or a picture. A picture is obviously more engaging since the user does not need to do any word processing in their brain. Reading a text article or looking at a picture is a virtual experience connecting to the writer/creator through their creation.
Here are a couple of experiences. The first one is a text excerpt from a dictionary explaining what “Subjective” means.
The next one is an image that vividly illustrates how “everything is subjective”. Naturally, the image conveys the meaning of “subjectivity” more quickly and amusingly.
The next level of user engagement can be achieved by letting the user hear the experience. Hearing something rather than reading or seeing has a little higher level of engagement because it is possible to multi-task while listening to something. We often listen to audiobooks or podcasts while driving, rather than sitting someplace, reading something.
There are several technological advancements in sound. They are stereo sound, surround sound, and, very recently, spatial sound (Dolby Atoms). Here is a demo of spatial sound. Please be sure to put your headphones on.
If the user is seeing and hearing the experience, there is an additional level of engagement. Now, more than one sense is involved in allowing the user to enjoy the experience. Most currently available online learning falls into this category.
LinkedIn Learning (previously Lynda.com) and Udemy have video tutorials and courses in which the instructor engages the user in video chapters of a particular training course. The user sees what the instructor is showing on the video, and listens to the instructor's voice.
Online movie streaming is seen and heard, too, and there is some quite engaging material out there for sure.
There is an interesting approach where the user sees and touches the experience. Most science museum exhibits fall into this category.
For example, I have seen a sandbox at a tech museum where the user touches the sand and makes sand structures while a topography map is projected onto the sand. When the sand is restructured, the map changes in real-time. It is a very effective exhibit to understand how topography maps work.
Pop-up books fall into this category as well. Instead of just holding a book to read, you can touch the 3D elements in the book.
Topography in a Sandbox
Pop-up Books
This is a widely used category of virtual experience where the user sees, hears and talks within it. All live webinars and online meetings with back and forth interactions between attendees are in this category. Naturally, the user is more engaged than in the See and Hear virtual experience because the user is now communicating with others rather than just enjoying the inflow of information.
There is some online narrative content in this category as well. For example, Steven Soderbergh directed a web series called ‘Mosaic’ for HBO in 2017. Before releasing it on HBO, the series was actually released on its own app.
The app worked like an interactive film, where users could choose from which perspective they viewed the plot and learn different facets of the plot without affecting it. Based on the selection, the story would change.
Similarly, Netflix released a film called ‘Bandersnatch’ with a similar experience. They would pause the movie after certain actions were taken and ask the user to make a choice. Then, the movie would progress in a particular direction based on which choice the audience made.
It was an innovative effort. However, it was not hugely successful or entertaining. Mostly, I think this is the case because there was no interaction with a live person from the other side.
Most of the 4D theme park rides fall into this category. You are stationed in one seat, mostly not moving at all. However, just through the pictures that are shown and special audio and sensory effects surrounding you, a virtual experience is created like no other. The Transformers 4D at Universal or Soarin in Disneyland are prime examples of this type.
Any online classes where you need to interact with physical learning material also fall into this category. An online painting class is one example. During the pandemic, most schools were online, and kids were supposed to interact with teachers over a video call and also do their homework by interacting with physical learning materials. Interactive digital books or advanced video games also fall into this category.
Most augmented reality applications fall into this category. For example, Genesis cars have an app that you can open on your iPad and point towards the car. Then through the camera, you can look at the car live, and the app would show you how to do the oil change.
Or in manufacturing, technicians can wear a headset with glasses in front of their eyes. The work instructions for their jobs are projected onto the glasses, telling them exactly what to do.
Another example is the ability to walk up to the display in the Lego store with your lego kit, and the display will show you what the assembly will look like.
Or when you buy furniture on Amazon, sometimes it has an option to “view in your room”. It turns on the camera and shows you how the furniture will look in your room. You can walk around and keep looking through the camera to understand how it will look from every angle.
This is a special and very innovative category where the user experiences something that would be physically impossible. For example, in the Franklin Institute in Philadelphia, there is a giant heart that you can walk into. Now, obviously, this is not possible in reality, but because this display is designed as a walk-in experience, users can understand how the heart functions.
Another example would be the Van Gogh exhibit, recently in San Francisco, where all of his paintings were projected on the walls, floor and ceiling, and you can walk in and through Van Gogh's paintings. This is a clear example of See and Walk, and is a completely immersive experience.
One popular belief is that art is created organically, as though very little pre-planning is involved. However, I strongly believe that art creation can be liberated by classifying and standardizing the elements of each art form, because the more structured a process becomes, the easier it is to bring the experience to life.
To achieve that structure in the creation of a virtual experience, I have ventured to create a classification system that incorporates human senses so that a person can pick and choose from them to create a virtual experience. By thinking about incorporating and combining senses, virtual experiences can be created in a more efficient manner.
