How do you go about creating stories in VR?

RedSharkNews

Immersive storytelling is a more profound shift than from BW to colour or 2D to 3D. There’s something fundamentally different about how the audience consumes this media arguably making it a new medium. So how do you go about creating stories in VR? 

Article here 

Matt Celia, the co-founder and creative director of Light Sail VR, has offered up his creative techniques and technical tips for filmmakers new to immersive entertainment.  The primer, streamed in 180° virtual reality, features some excellent takeaways for cinematographers and directors looking to try their first VR project.

It’s also a marketing tool for Canon and its EOS VR system including new Dual Fisheye lens which RedShark covered here: but Sail’s lessons from experiments in VR 360 and VR 180 are informative. He’s done some hard yards to give you a head start.

Sail considers immersive storytelling a new medium and boils this down to one thing: agency. “With immersive storytelling we are handing over some of that traditional control of narrative back to the audience. I find that incredibly empowering. Now they can experience it and therefore form a stronger connection to the work.” 

We can now look at visual storytelling on a spectrum with 2D cinema on one end the full holodeck with ‘experiencers’ in motion track suits, AI and sensors at the other. At this end you are inside the story.

Sail is concentrating on the middle of these extremes. VR is a still a lean back experience but I can still be an artist and filmmaker by gaining some control whilst transporting someone to a new environment.”

One can of course view VR (or 180) content on a smartphone but Sail’s focus here is on headsets.

“For me, VR begins when we can block out everything in the world we are in and replace with something else,” he says. “This is about presence – the idea of being in a particular place.”

In order to achieve that sense of presence clearly some traditional 2D film techniques don’t wont so well. There’s no CU, mid or wide that filmmakers rely on to tell the story.

“Instead, you have to think about where’s the best spot to stand in the room,” he says. “We’re trying to replicate the feeling of being there but it’s a fixed perspective. We see everything from our eyes and from our POV and we want to make it feel very authentic and natural.”

Some VR pieces have attempted to change the perspective in a scene, with over the shoulder coverage, for example, and use more traditional directorial format.  But this can feel stuck in between the two mediums, not taking advantage of what VR offers.

Camera movement in VR

Another aspect of modern cinematography that may not translate well into VR is camera movement. This is for the simple reason that if our eyes are moving but our body is not that induces nausea.

And clearly in conventional 2D filmmaking you can art direct each frame of the cinematic experience. You can hide the crew. You control focus. 360 is the opposite. All elements of control are handed to the viewer who can experience the frame however they wish. All of this makes it a lot harder for a director to know where their audience is looking. Consequently, the VR filmmaker has to work harder to direct your attention to a particular point in the space.

“Even though the viewers can look anywhere we do only look at one area at one time but the filmmaker must think about the whole 360 frame.”

For these reasons Sail thinks 180-degree VR strikes “a cool balance because I can control the front half of what the audience is looking at and the audience has some agency to look around. The stereo 3D is usually better in 180 and the viewer sees the action in front so you can hide crew and lights, generally operating a little more comfortably as a filmmaker.”

The difference between 180 and 360 VR

There is potential to mix the two panoramas. His general advice is: “180 is a medium of people and 360 is storytelling through place.” In other words, use 180 when there is intimate action between actors and 360 for establishing environments.

Other advantage of 180 (and bear in mind this is ultimately a marketing pitch for Canon’s system): There’s no stitching required to piece the angles together so post is a lot faster. It’s higher resolution than 360 too. The Canon shoots 8K worth of pixels to shoot 180 degrees of picture as opposed to filling 360-degrees worth of pixels (outputting half the resolution). This means the 180 imagery looks better in non-VR applications too.

Sail suggests different ways of thinking about the viewer. They could be passive - simply looking and not interacting. Their presence could be acknowledged as part of the story by the other actors (though you can’t actually interact with the narrative) or they could be ‘guests’ which breaks the fourth wall that creates a connection with the story. He prefers this style and likens it to a close up because of the feeling of intimacy it evokes. 

When it comes to planning a set-up in 180 he suggests that the strongest stereo effect is in the front 120 degrees: “The 3D effect will fall away at the sides.”

Aside from placing the action in that 120 front view the distance the camera is to the subject is also important. For this there is a rule of thumb used by stereographers which is 30 to 1.

Whatever your inter-pupillary distance (IPD) – the gap between the centres of your  dual lenses is multiply by 30 and that is your minimum stereo distance from camera to subject. 

The Canon fisheye has an IPD of 60mm we learn, which is close to most people’s own IPD.

“That’s important in conveying the idea of presence – of the virtual world being seen through someone’s eyes,” Sail says. Applying the 1 to 30 rule equates to about 6ft but Sail thinks 3.5ft to 15ft is the optimum range. He mentions 15ft since he advises not staging the action too far away from the camera since the action will look lost.

Find creative ways to light your scenes. Practicals can be great tools to add dimension to your scenes.  You can also paint out lights by shooting clean plates.

Frame rates: 24p is not comfortable in a headset – there’s too much motion blur. 30fps is where you should start, he recommends. The Canon can record 8K 30fps or 4K 60fps.

Use 2D cameras to capture close ups for picture-in-picture frames. This technique allows you to vary your framing when shooting VR.

Be careful with motion or you risk making viewers sick. Keep a level horizon and use smooth, slow forward or upward motion.

Use blocking and art direction to tell the story: Without usual cinema grammar or choice of different lenses or camera movement what have you got to play with? Well, you can set up the world to draw the audience’s attention to certain aspects of the story. You’ll also need to be aware of the cropping for viewing on things like YouTube VR (the Canon system has a guide on its LCD display for monitoring this).

This workshop was part of the virtual Sundance Film Festival.