IBC
Framestore’s CTO Lincoln Wallen says the creative
industries should acknowledge the art of math to craft new interactive
AI-driven social experiences.
article here
The future of media and entertainment will be interactive,
social, and fuelled by AI with companies versed in blending tech and art at the
forefront of development, according to Framestore’s Chief Technology Officer
(CTO), Lincoln Wallen.
“It’s going to be a slow burn but the technological basis of
it has already been built,” he says. “Imagine experiences in which consumers
can interact but are nevertheless driven by narrative visual storytelling. It’s
a mixture of active lean forward and passive lean back. That is a media type
that I expect to explode over this next period.”
Wallen joined Framestore and Company 3 Group (FC3) last
summer from Improbable, a tech developer building platforms for multiplayer
games and battlefield simulations which was valued at $3bn in 2022. He
previously spent four years at Electronic Arts building its mobile games
business before entering the film industry at DreamWorks, initially as Head of
R&D from 2008, and then promoted to CTO in 2012.
That career journey has given Wallen an orbital perspective
of future media in which technology and content, both IRL location-based and
virtual, merge to create new forms of real-time communication and
storytelling.
“If you want to spot the metaverse as it emerges it’s going
to come in interactive experiences in which large numbers of consumers come
together,” he explains. “We do this today in social environments when out
shopping, at sporting events, on the street, at festivals and conferences, but
the digital world doesn’t properly support that sense of social engagement.
“The only markets in which people really come together
digitally are video conferencing and video games but those experiences are
designed for very specific numbers of players, not for crowds. Being able to
interact with people meaningfully in a digital environment changes the game.”
Technology that facilitates simultaneous multi-player
interactions at scale will underlie future media says Wallen, who believes this
has been cracked by Improbable (of which he remains a Non-Executive Director).
The next step is to evolve the user experience.
“VR really hasn’t left the laboratory in terms of the
consumer experience. It still isolates the user. But the metaverse will come
back as all truly disruptive ideas do,” he says.
An example of what he means is the work Improbable has done
with Major League Baseball (MLB). Here baseball fans can gather in an MLB
virtual ballpark and interact simultaneously in a single place, at a
naturalistic scale. It claims to be the first service to host such large
interactive sports experiences.
Another example includes fashion brands that might organise
a virtual runway show inviting people to the physical catwalk and
thousands more people to virtually inhabit the same space. Wallen believes this
type of experience will eventually happen daily as physical interfaces like
mixed reality goggles improve, but that the content itself won’t work unless
crafted with storytelling expertise.
“The audiovisual world of linear media has built up an
enormous bank of techniques creating adverts, TV shows or movies to grab a
consumers’ attention and to stimulate emotions but those techniques are not
used today within interactive environments,” he insists.
“Those working in linear media will call it ‘immersive’ as a
way of trying to signal that this is something you’ve seen in VR experiences or
in location-based entertainment. What we haven’t yet seen is that experience
realised in a fully digital environment. It’s neither gaming nor linear but
something in between. The potential of this middle ground is enormous.”
“The linearised model of film production is going to
collapse into a much more iterative holistic model” Lincoln Wallen, Framestore
If Hollywood studios are too entrenched in their own
structures to pivot fast enough, companies that are operating across market
sectors will be able to synthesise these capabilities and exploit them.
Framestore is one. Its work ranges from the VFX on Gladiator II to designing an
interactive exhibit for the Science Museum.
“That isn’t to say that Framestore is suddenly going to
become a content company,” Wallen stresses. “It means identifying those brands
and IPs that are going to pioneer this new media and serve their existing
customers in a different way.”
Framestore launched in 1986 at the birth of the computer
graphics revolution. It remains headquartered in Soho where it has navigated
and led the industry-wide transformation of post-production from an
afterthought into an integral partner spanning the creative industries.
“The linearised model of film production is going to
collapse into a much more iterative holistic model,” Wallen says. “Right now,
the previz phase and the production phase feel separate. I think AI is going to
bring the back end right up to the front entrance and make certain complex
processes real-time.”
Bridging the uncanny valley
For Wallen, the key to the future of interactive media is
the ability to create digital representations that talk, behave, respond and
‘think’ like a human. Such avatars could be our own online personas or the
embodiment of a corporation. AI is already supercharging such developments in
video games where AI-driven non-player characters (NPCs) are being introduced
to interact in a more lifelike manner with game players.
“You are going to see NPCs having existential dialogue with
game players,” Wallen says. “This is one of the most exciting things to me.
Large language models (LLMs) have exposed a degree of structure in natural
language that makes dialogue possible at a level that it hasn’t been in the
past.
“When people look back on this era it is going to be the
impact on language that will be seen as the most profound. Natural language
will become a much more effective interface to [digital] systems and this will
make them more accessible, which will be an enormous economic benefit.”
There is a problem though. The increasing sophistication of
LLMs must be matched by sophistication in visual behaviour, emotion and
characterisation. The uncanny valley is yet to be bridged.
“We don’t have a language of performance. There’s nothing to
correlate spoken language and non-verbal gestures to make your NPC seem alive.
Animators study human behaviour and know exactly how to produce that effect but
we don’t have tools that do that automatically. You can’t take the animator out
of the equation because they understand what human gestures mean, but the way
they’re created right now is painstakingly frame by frame. We have to build
that capability.”
Wallen believes that the creative industries should
acknowledge the art of math to craft new interactive AI-driven social
experiences.
The EU-funded research project PRESENT (Photoreal
REaltime Sentient ENTity) of which Framestore is a partner aims to create
virtual sentient agents that look and behave entirely naturalistically. The
aims are to demonstrate emotional sensitivity, establish an engaging dialogue,
add sense to the experience, and act as trustworthy guardians and guides in the
interfaces for AR, VR, and more traditional forms of media.
“It’s only the high-end media industries that actually know
how to produce digital performances, that that make people feel that NPCs are
alive,” Wallen says.
Understanding how the world works
“There’s a very ambivalent relationship in the humanities
between systematisation and what they think the humanities are all about, which
is inspiration” Lincoln Wallen, Framestore
Wallen has a PhD in artificial intelligence from the
University of Edinburgh, and subsequently taught at the Oxford University
Computing Laboratory before moving into the media industry. He is also a
non-exec board member of Finnish VR/MR headset maker Varjo, Pearson - the
FTSE100 global education company, an advisor to bp and a 25-year member of the
maths and AI strategic consultancy Smith Institute. Few people understand the
intersection of science with the humanities as well as he and that means his view
on AI as an artistic tool is not binary.
“All through my career the driver has been the interplay
between modelling the world and activating it through computational systems,”
he says. “Understanding how the world works is a passion.”
What strikes him about the current AI wave is the tension it
has caused between inspiration and predictive calculation, but it’s a tension
that is not new, and the two are not mutually exclusive.
“Making movies is a balance between algorithmic processes
and artistic choice. That’s a very natural combination that AI is forcing to be
recognised.”
He explains that neural networks, a mathematical construct
designed to mimic human thought, has reached a level of maturity that
demonstrates structure in natural language.
Stevie (Alba Baptista) will star as the protagonist of
FLITE, Framestore’s experimental short film set in a futuristic London of 2053
“We now understand that there’s an awful lot of structure in
writing. It’s not just a grammatical structure. Literary critics have argued
this for centuries, but the humanities oscillate between seeing these patterns
and then running away from them because it seems to be contradictory to
creativity and choice. There’s a very ambivalent relationship in the humanities
between systematisation and what they think the humanities are all about, which
is inspiration.”
He argues that there is an art to mechanisation:
designing-in specific outcomes that preserve human creative agency, with plenty
of scope for machine learning technology.
“LLMs don’t manipulate intent,” he insists. “Humans create
intent. They communicate and collaborate to organise their efforts to produce
fantastic outputs, whether that’s a video game or a film. Without human
dialogue, without the imagination that’s really framing the outcomes, you don’t
get the products.”
We can use advanced AI techniques to try to realise that
imagination, he says.
“Computational techniques are forcing recognition that there
are structures, including within narrative visual storytelling.”
Language of intent
Wallen thinks the industry needs a standardised way for
artists and filmmakers to communicate with each other in the age of AI, using
AI tools. Current-generation AI tools, he says, are not intuitive and actually
hinder artistic collaboration.
“GenAI tools work at a very low level. They don’t help you
express what you’re trying to do. The only way to get the machine to respond is
to give it instructions bit by bit of what you want to see but not why you want
to see it.
“What we want is a language of intent,” he argues. “We want
what the artist is trying to achieve made more directly expressible and
reflected by the interfaces of these tools. If we had these languages we would
be able to build technologies that really turbocharged people’s creative
capabilities. That I think is what the end result of this AI wave will be but
only if the creative industries start looking at their systemisation techniques
and their content [Wallen uses the term folklore) and turning it into activatable
language.”
This is something that Wallen is exploring at Framestore but
it’s too vast a challenge for one company alone.
“I’d hesitate to say that this is a century of development
but it’s also not easy. What I’m trying to do is stimulate academics and
creatives to think about the problem. Even small areas of advancement in this
area can lead to enormous benefits to the creative process. It is one of my
personal motivations for joining Framestore.
“Whether it’s your digital proxy, virtual playmates, NPCs in
games, a retail concierge, or part of the crowd in a virtual stadium doesn’t
matter. There are so many applications of LLMs that are going to explode when
coupled with character.”
Wallen completed a thesis on Mechanisation in Mathematical
Art during the 1980s and suggests that the challenge is as old as the
Scientific Revolution itself. He refers to early computers which worked by
punching sequences into cards.
“By attending to the design of, and access to, the tools
with which we ‘punch the cards’ today we can preserve the poets and even pursue
‘poetical science,’” he says.
No comments:
Post a Comment