Virtual reality is not filmmaking

Let’s agree on something: 360-degree video presented inside a virtual reality headset isn’t really virtual reality. It’s a novel, sometimes powerful experience, but it’s not what makes VR arguably the most exciting breakthrough in storytelling since moving images.

While much has been made of immersion as a defining trait of virtual reality, a key component of normal (i.e. non-virtual) reality is painfully absent in the live-action and pre-rendered experiences I see being confused with true VR: interactivity.

In real life, you make decisions. Those decisions change the world around you — or your experience of it — sometimes in profound, unexpected ways. Real life decisions aren’t presented to you as tidy forks in the road. Real life decisions are messy. You often don’t realize you’ve even made a decision until you’re reeling from its consequences.

Real life decisions emerge from continuous behavior within an environment — not discrete, binary choices.

Virtual reality, then, should strive to bring that same level of interaction to a user. Doing so implies (at least) the following:

1. Virtual reality is not filmmaking.

Whenever a new medium comes along, we spend years trying to cram old media into it. Jesse Schell, who’s been working with virtual reality since the first wave of VR in the 1990s, puts it nicely in his Gamasutra post:

Throughout the history of entertainment, the first impulse of those who create in a new medium is to imitate what came before. Early movies were just filmed stage plays, with no cuts, close ups, or camera motion. Early internet videos were attempts to imitate the format of television (30 and 60 minute shows).

In all cases the pattern is the same — the new medium is derided for not being as good as the old one, but then, gradually, after many experiments by pioneers, the unique strengths of each medium come to light, showing it to be powerful in a way that was not possible before.

An LA Times article covering the release of Oculus Story Studio’s latest VR project, Henry, voices the concerns of some traditional filmmakers:

As VR cinema skeptics have noted, [interactivity] can undermine big story moments. No filmmaker wants to have Michael Corleone shoot Sollozzo and McCluskey in a restaurant while an audience member is in the kitchen ogling the appetizers.

This “argument” is simply a failure to understand that VR is not a new vehicle for old experiences (i.e. films) — it is a new medium entirely. Will you have the same emotional reaction to a VR retelling of The Godfather that the original film delivered? Probably not. Can you have a different — possibly even more profound — emotional reaction to some other VR experience? Absolutely.

As Ramiro Lopez Dau, the director of Henry, says later in the same LA Times article:

“What I’ve learned with ‘Henry’ is that it’s not really a movie anymore. It’s storytelling in a new medium, and we’re trying to figure out what to call it,” he said. “A movie is the closest name because it’s the most familiar. But if we keep adding layers of complexity, we’ll come up with a new one.”

2. Virtual reality is real-time.

In order to have a truly VR experience, the virtual world must be rendered in real-time.

360° pre-rendered experiences — whether live action or CG — presented in virtual reality headsets are enjoying a spike of novelty now, but once VR becomes more widespread, that novelty will quickly fade. Rapidly improving real-time technologies will make “live action VR” feel comparatively dull, even lifeless.

Don’t buy it?

Imagine this simple scenario:

You’re standing in a concrete cell lit only by feeble light from cracks in the cast iron door before you. You tug on the door’s handle, but it doesn’t budge. Locked tight.

You hear a chirping sound coming from the corner of the room. It seems to be emanating from under a small table there — really just an overturned wooden crate.

You kneel down and peer into the crate. Something glints in the shadows. On your hands and knees, you crawl closer, peering into the darkness. You can just make out the twitching whiskers of a mangy rat. As it turns to face you, you see a golden key clamped firmly in its mouth.

Slowly, you reach out and lift the crate, exposing the rodent. To your surprise, it doesn’t scurry away. Instead, it crawls eagerly toward you, gingerly droping the key in your other hand.

Freedom!

The mix of fear, wonder and ultimately exhilaration that such an experience could create would be a direct result of being able to explore your surroundings in real time. In a passive version of the experience, you would enjoy the same level of sensory immersion, but without being able to crouch down, move toward an object and interact with it, you would be a catatonic observer, locked in a paralytic trance.

You wouldn’t feel a sense of ownership over the outcome of the situation because your actions would be limited to simply rotating in space and “gazing” at things. While this gazing activity could conceivably trigger a branching narrative, the experience would be “on rails,” a finite set of vignettes that shatters any sense of agency in the virtual world and instead forces a narrative upon you.

Here’s where game developers have a massive advantage over traditional filmmakers. They’ve been crafting experiences in real-time engines like Unreal and Unity (and many others) for decades. Real-time gaming platforms provide all the graphical and interactive scaffolding you need to create interactive worlds — and immersive storytelling experiences.

That’s not to say that VR storytelling and gaming are the same. They’re not. But from a purely technical perspective, game-making is far ahead of filmmaking in understanding the challenges — and potential solutions — that face VR storytelling.

“So pre-rendered VR is a waste of time?”

No, not at all.

But live action and pre-rendered storytelling are inherently limited to a set of experiences that I would call “passive VR.” If you’ve enjoyed any of Chris Milk’s VR works, you know that these experiences can be quite powerful. (My personal favorite is “Evolution of Verse,” created in partnership with VFX powerhouse Digital Domain. Download the VRSE app to watch the film.)

Passive VR is pushing the boundaries of what can be achieved without interactivity — and it appears that these boundaries reach farther than most imagined. It’s important, however, to distinguish between passive VR and true VR.

Why? Because from both technical and creative standpoints, true VR is much, much more difficult than passive VR. Equating passive VR and true VR shortchanges the exciting challenges ahead for the latter — and the attendant discoveries that will be made along the way.

3. Virtual reality means giving up control.

When it comes to VR as a storytelling medium, one of the emerging masters is former Pixar animator and director Saschka Unseld.

Unseld now works with Oculus Story Studio, where he directs short-form interactive experiences like Lost, which debuted at this year’s Tribeca Film Festival to rave reviews. In a recent blog post (required reading for anyone interested in VR storytelling), Unseld explains his team’s gradual acceptance that trying to control the user’s experience was holding them back:

One of the most powerful aspects of film is the total control over the shot. We talked endlessly about how to regain that control. How could we make sure the viewer always looks in the correct direction?

We tried guiding the audience’s view through audio cues. We had a bird fly by the viewer to capture their attention and guide their gaze towards a point in the scene. We also tried to design the set in a way that guides the viewer’s gaze to the right areas.

However, each time we implemented one of these dictatorial tools too heavy handedly, the storytelling started to feel forced, staged, and artificial.

Over time we stepped away from this sort of thinking. To embrace VR as its own unique medium, we have to let go of our almighty control of what the audience sees. Instead of instantly pushing the story onto the viewer, we take a step back for a while and let the viewer take part in discovering the story themselves.

The impulse to “tell” a story is loaded with authorial intent to control the user’s experience. Telling, after all, is a mostly one-way experience. But good storytellers respond to their audiences, skipping details or embellishing others in order to achieve the greatest emotional impact. They read the room and making adjustments to their material on the fly.

Perhaps “telling” a story is the wrong starting point altogether.

Recent games like The Stanley Parable and Gone Home present their narratives through a series of discoveries. They aren’t telling stories so much as presenting all the pieces of a story and letting the user assemble them however they wish. While there is a master narrative, the specific experience of it is unique for each user.

Materials from Motionographer.