IBC
VFX Oscar winner and Magic Leap SVP Creative Strategy John
Gaeta explains how a dimensional internet could lead to CG experiences into
real world spaces.
A whale leaps out of the room in front of you. Penguins
falls off your table. You are holding an elephant in the palm of your hand.
This is mixed reality devised by Magic Leap and if you believe its head of
creative strategy then these gimmicks herald the dawn of a new age of computing
in which we will all share.
“Blended reality will be part of our everyday life,” John
Gaeta explained to IBC365 at the View Conference for VFX, animation and
immersive media, in Turin.
“So much so that we will create memories of incidents and
people which are part real, part simulation. That experience will change the
way we think, the way we work, how we interact socially and in turn, it will
change the way artists create.”
Gaeta has led research pushing at the creative interface
between human and machine for over 25 years for Microsoft, Lucasfilm and now
Magic Leap, along the way devising the ground-breaking ‘bullet time’ visual
effect in The Matrix. He describes himself as “a design and experience person”
rather than technologist, though futurist might be more apt.
He believes the internet is about to lift into dimensionality
because the technology exists to integrate believable computer-generated
experiences into the world around us.
Apple and Google also believe Augmented Reality to be the
next stage of the internet but Magic Leap has placed the biggest bet. Google,
Warner Bros. and Alibaba among other investors have pumped a mammoth $2.44
billion into the Florida-headquartered firm to build a platform for spatial
computing.
“Spatial computing governs the interaction between the
physical and the virtual world,” Gaeta explains. “There will be various and
infinite types of digital overlays on the real world. These layers or (uni)verses
will bridge the physical and the digital worlds.”
The digital overlay will be viewable by users wearing
goggles, similar to Google Glass and Microsoft Hololens. What is different
about Magic Leap – at least in the vision articulated by Gaeta – is the
jaw-dropping scale of this connectivity which will be “full spectrum” meaning
that layers can be accessed interchangeably from mobile AR to full blown
head-mounted virtual reality.
He takes his cue from the immersive experiences imagined by
Walt Disney in the 1950s. Alongside the Magic Kingdom which would transform
animated characters into three dimensions, Disney’s original plans also
included an Experimental Prototype Community of Tomorrow’ (EPCOT) intended to
serve as a test bed for new city living innovations. Magic Leap can be thought
of as an extension of both but on an eye-popping scale.
Beyond the theme park
“The future of Disneyland and that of every other theme park
will be a hybrid of beautiful CG environments and story-based characters
intermingled with physical sets, living content and performers,” Gaeta says.
Indeed, Gaeta has already worked on high concept proposals
at ILM – which is owned by Disney – for “wrapping Disney World in VR”.
“Disney’s vision of immersive and blended reality will move
beyond the theme park into everyday life,” he predicts. “Smart cities will be
architected, built and explored by citizens using blended reality.”
Magic Leap has trademarked the term Magicverse to describe
the layers which it thinks will populate the space between the real and the
digital world.
While these verses can relate to any industry, sector or
business from education, transportation and medicine to the military, it is the
future of entertainment which exercises Gaeta. He says a new approach is needed
for creating narrative content.
“Cinema and TV are built on linear storytelling and on the
intuitive language of editing and cinematography,” he says.
“You don’t need to explain what a series of cuts and camera
angles mean. We just comprehend it. But absent that in a direct orbiting
[360-degree] view and the nature of the canvas on which you want to tell a
story changes entirely. It requires a completely new grammar for communicating
story.”
One consequence of this, he suggests, is that the
traditional authors of a creative work will be replaced in large part by the
audience itself. The “participant” or “visitor”, as Gaeta prefers to term the
user experience, will shape the story simply by entering the virtual world.
“Writers, directors and other narrative artists draw on
their life experience as a source of inspiration which they reframe as metaphor
when they create a work of fiction,” he argues. “I think that the participatory
nature of a virtual universe will mean that each visitor is their own original
source such that we can collectively create a more powerful and more direct
form of storytelling.”
It’s not as if Gaeta’s mixed reality world is entirely
unmediated by artists or corporations, however. Everything is controlled,
albeit benevolently, much like the Steve Jobs’ god-like character Halliday from
Ready Player One. Again, as in Ernest Cline’s novel, Gaeta theorises that
multiple virtual worlds or verses will be created as destinations for people to
visit.
It’s an idea he has already spent several years developing
at Disney’s immersive entertainment arm ILMxLAB as the mouse house kick-started
world building around the Star Wars franchise.
“In the Star Wars universe you have countless destinations
each with histories, storylines and characters that are criss-crossed all the
time,” Gaeta explains. “But it is the destination which attracts people in the
first place. With both VR and MR you can expose visitors to a logic of events
and ongoing situations. That’s why for me it is more interesting to produce the
destination than it is to control the narrative.”
He even suggests that “visitors” could play a role on the
edge of some gigantic Star Wars themed VR world.
“When whole scenes or entire films are rendered photoreal
using game engines then the digital universe of the story can be made
accessible as a layer in the Magicverse. You could pass through any given frame
like a portal and look at everything around you in perfect 3D, like a
volumetric sculpture. And while you are there the backdrop would not be a
static thing but dynamic reacting to your presence. You can find yourself on
the periphery of these stories. You could even be dragged into the story
itself.”
That’s because Magic Leap promises your own personal AI.
Sensors on the headset capturing your senses (sight and sound at first, but
conceivably touch, smell, and taste going forward) will help the AI understand
and document your engagement with spatial computing.
“The verse will exhibit behaviours in reaction to your
presence and you can interact with the environment. The product of that
interaction turns out to be the story,” he says.
It is no coincidence that the Wired journalist and author of
cyberpunk classic Snow Crash, Neal Stephenson, is Magic Leap’s Chief Futurist.
Steven Spielberg, director of futurist fantasies Minority Report and Ready
Player One is reportedly an investor.
“Spatial computing is the ability to have content persist
over the world geosynchronously and in flawless volumetric 3D,” he says.
“It means the application has to be reactive to the content,
to what is happening, to people who are interacting with it.
“I don’t want to sound trite. This is not going to happen in
the near term.”
This involves massive data transmission, massive multi-user
networking and years of work, but Gaeta thinks 5G might kick-start deployment.
Specifically, in urban centres where the broadband connectivity necessary for
high-fidelity realtime graphics will be fastest.
“Whole economic models will be born out of the dimensional
internet,” he says. “People can socialise in verses like Second Life. They will
transact through these applications.”
Not surprisingly he predicts that issues of intellectual
property and user ID will generate considerable debate although blockchain may
play a role in safeguarding digital personas.
“A new language will emerge as we all become a little more
familiar with forms of simulated interaction,” he says. “In just a few years we
have become comfortable holding face to face conversations via a screen. It is
only a short step to having a holographic co-presence with us all the time.”
He is no fan of VR, blaming the failure on headset makers
(like Sony and Facebook) to stimulate the home market “long before people could
comprehend it”.
“VR is a car wreck,” he says. “You have to make VR first to
understand it. I am concerned that VR could become isolated. On the other hand,
it could be the most mind-bending computer interface. But MR is an easier path
to imagine use cases emerging.”
Magic Leap’s project has critics too, having taken five
years in much-hyped secrecy to get to a point where it has only just launched a
headset for developers.
Gaeta defends its record, pointing out that in 2011 the
start-up’s digital projections could only be viewed via a machine the size and
weight of a fridge. Now all the processing is condensed into goggles not much
bigger than sunglasses and a lightweight PC.
“It costs billions of dollars to have the world’s greatest
engineers miniaturise this,” he says. “And it’s only generation one. Mixed
Reality could go mainstream if the ergonomics are worked out and it looks cool.
The point is that all development in this area has been led by older people –
like me.
“We probably won’t see a step change in the technology or
the media until those younger generations who’ve grown-up completely comfortable
interacting by gesture and voice with a computer, take hold of the baton.”
Even so, goggles or wearable haptics of any description are
surely a rudimentary means of mapping digital assets to reality. Much like The
Matrix with all its attendant risk of paranoia, science may evolve to being
able to program our brains directly.
“This is just a matter of time,” Gaeta believes. Indeed,
researchers at the University of California, Berkeley, have used a process
called Functional Magnetic Resonance Imaging to decode and reconstruct movie
clips that people had already viewed just by using brain scans.
“A more advanced version of the technology could reproduce
our dreams and memories in a visual format that anyone could share,” he
theorises.
The concept parallels Douglas Trumbull’s 1983 sci-fi feature
Brainstorm, in which scientists recorded a person’s sensations so that others
could experience them.
That would bring Gaeta’s story full circle.
John Gaeta: profile
Born in 1965, the New Yorker went to NYU’s Tisch School of
the Arts to study film after being inspired by the work of Stanley Kubrick. A
few years after graduation he took a job as production assistant at the
Massachusetts lab of Douglas Trumbull, the special effects wizard behind
Kubrick’s 2001: A Space Odyssey.
He went from sweeping floors to shooting scenes for
Trumbull’s experimental high frame rate, large format theme park rides. When
Trumbull left to help launch IMAX as a commercial theatre operation, Gaeta and
the small team Trumbull had hired were given the run of the lab to continue
research into computer vision and digital compositing.
Because Trumbull preferred to work as an outsider
deliberately out of sight of Hollywood, it was the indie filmmaking community
which got wind of Gaeta’s R&D.
Director Vincent Ward asked him to co-supervise development
of innovative 3-D paint effect stylisations and LIDAR laser scanning for What
Dreams May Come(1998). Then the Wachowski brothers gave him his first solo
supervision for The Matrix (1999). Both films went on to win the VFX
Oscar, disrupting the awards dominance of George Lucas’ Industrial Light and
Magic.
“The Matrix ‘bullet time’ sequence is primitive compared to
what you can do today but it proved the concept of universal or volumetric
capture in which multiple cameras record every angle of a scene,” he says.
“It’s the principal behind a lot of current VR.”
Gaeta coined the term ‘virtual cinematography’ to describe
the process used in The Matrix and its two sequels, but credits Kubrick’s use
of Steadicam on The Shining as the genesis of the idea “that a camera need not
be fixed to anything physical and could be placed anywhere”.
After returning to make live action anime Speed Racer (2008)
for the Wachowski’s he left the film business.
“I got disenchanted because I felt the business was moving
in a direction that was not great for innovation. Studios wanted to globalise
with fewer mega-budget blockbusters and to spend less and less on innovation.
The reason we won [Oscar] for The Matrix was because the filmmakers weren’t
afraid to take a risk.”
Creating content
The debut Magic Leap One Creator Edition headset, was
released in August costing $2,295 and targeted at content developers. The firm
has put aside funds to invest in content ahead of launching a consumer version
next year.
Magic Leap One
Nor will it restrict development to Magic Leap hardware
saying it will allow phones, tablets, computers, VR headsets, and future
devices to interoperate with Magic Leap content.
Headline grabbing content partners include Peter Jackson’s
Weta Workshop, Gaeta’s old firm ILMxLAB and VFX house Framestore, which has
created a trivia game for the headset on behalf of client Air New Zealand.
AT&T plans to launch a DIRECTV NOW beta to Magic Leap One in 2019.
Among the more interesting applications is one from Andy Serkis’
Imaginarium Studios. It has hooked up live performance capture to an
AR-generated CG character viewable in Magic Leap, as an example of real-time
previsualization for feature production.
On a technical front, Magic Leap’s goggles have multiple
lenses mounted on the exterior, including the cameras and light sensors, that
image the world in front of you, so AR experiences can be layered over it. The
field of view has a 4:3 aspect ratio and the frame rate is 60p. It is powered
by Nvidia’s Tegra X2 processor with six CPU cores and 256 GPU cores.
