How AI is reinventing Visual Effects

IBC

AI/ML and deep learning is having a huge impact in computer graphics research with potential to transform VFX production.

https://www.ibc.org/production/how-ai-is-reinventing-visual-effects/4060.article

In Avengers Endgame, Josh Brolin’s performance was flawlessly rendered into the 9ft super-mutant Thanos by teams of animators at Weta Digital and Digital Domain. In a sequence from 2018’s Solo: A Star Wars Story, the 76-year old Harrison Ford appears pretty realistically as his 35-year old self playing Han Solo in 1977.

Both examples were produced using artificial intelligence and machine learning tools to automate parts of the process but while one was made with the full force of Hollywood, the other was produced apparently by one person and uploaded to the Derpfakes YouTube channel.

Both demonstrate that AI/ML can not only revolutionise the VFX creation for blockbusters but put sophisticated VFX techniques into the hands of anyone.

“A combination of physics simulation with AI/ML generated results and the leading eye and hand of expert artists and content creators will lead to a big shift in how VFX work is done,” says Michael Smit, CCO of software makers Ziva Dynamics. “Over the long-term, these technologies will radically change how content is created.”

Simon Robinson, co-founder at VFX tools developer Foundry says: “The change in pace, the greater predictability of resources and timing, plus improved analytics will be transformational to how we run a show.”

Over the past decade 3D animations, simulations and renderings have reached a fidelity in terms of photorealism or art-direction that is near perfection to the audience. There are very few effects that are impossible to create, given sufficient resources (artists, money), including challenges such as crossing the uncanny valley for photorealistic faces.

More recently the VFX industry has focussed most of its efforts on creating more cost-effective, efficient, and flexible pipelines in order to meet the demands for increased VFX film production.

For a while, many of the most labour intensive and repetitive tasks such as match move, tracking, rotoscoping, compositing and animation, were outsourced to cheaper foreign studios, but with the recent progress in deep learning, many of these tasks can be not only fully automated, but also performed at no cost and extremely fast.

As Smit explains: “Data is the foundational element, and whether that’s in your character simulation and animation workflow, your render pipeline, or your project planning, innovations are granting the capability to implement learning systems that are able to add to the quality of work and, perhaps, the predictability of output.”

Manual to automatic
Matchmoving, for example, allows CGI to be inserted into live-action footage while keeping scale and motion correct. It can be a frustrating process because tracking camera placement within a scene is typically a manual process and can sap more than 5% of the total time spent on the entire VFX pipeline.

Software developer Foundry has a new approach using algorithms to more accurately track camera movement using metadata from the camera at the point of acquisition (lens type, how fast the camera is moving etc). Lead software engineer Alastair Barber says the results have improved the matchmoving process by 20% and proved the concept by training the algorithm on data from DNEG, one of the world’s largest facilities.

For wider adoption studios will have to convince clients to let them delve into their data. Barber reckons this shouldn’t be too difficult. “A lot of this comes down to the relationship between client and studio,” he says. “If a studio has good access to what is happening on set, it’s easier to explain what they need and why without causing alarm.”

Rotoscoping, another labour-intensive task, is being tackled by Australian company Kognat’s Rotobot. Using its AI, the company says a frame can be processed in as little as 5-20 seconds. The accuracy is limited to the quality of the deep learning model behind Rotobot but will improve as it feeds on more data.

Other companies are exploring similar image processing techniques. Arraiy has written an AI that can add photorealistic CGI objects to scenes, even when both the camera and the object itself are moving. An example of its work has been showcased by The Mill.

Software first developed at Peter Jackson’s digital studio Weta for The Planet of the Apes films has been adapted in California by Ziva to create CG characters in a fraction of the time and cost of traditional VFX. Ziva’s algorithms are trained on physics, anatomy and kinesiology data sets to simulate natural body movements including soft tissue movements like skin elasticity and layers of fat.

“Because of our reliance on physics simulation algorithms to drive the dynamics of Ziva creatures, that even in 10,000 years when a new species of aliens rule the earth and humans are long gone, if they can ‘open’ our files they’d be able to use and understand the assets,” says Smit. “That’s a bit dark for humans but also really exciting that work done today could have unlimited production efficiency and creative legacy.”

Smit estimates that a studio would probably need to create fewer than five basic ‘archetypes’ to cover all of the creatures required for the majority of VFX jobs.

“Conventional techniques require experts, some with decades of experience, to be far too ‘hands-on’ with specific shot creation and corrective efforts,” he argues. “This often demands that they apply their artistic eye to replicate something as nuanced as the physical movement or motion of a secondary element in the story. Whereas we know that simulation and data-driven generative content can in fact do that job, freeing up the artist to focus more on bigger more important things.”

Democratising mocap
Similar change is transforming motion capture, another traditionally expensive exercise requiring specialised hardware, suits, trackers, controlled studio environments and an army of experts to make it all work.

RADiCAL has set out to create a motion capture AI-driven solution with no physical features at all. It aims to make it as easy as recording video of an actor, even from a smartphone, and uploading it to the Cloud where the firm’s AI will send back motion-captured animation of the movements. The latest version promises 20x faster processing and a dramatic increase in the range of motion from athletic to combat.

San Francisco’s DeepMotion also uses AI to re-target and post-process motion-capture data. Its cloud application, Neuron, allows developers to upload and train their own 3D characters — choosing from hundreds of interactive motions available via an online library. The service is also claimed to free up time for artists to focus on the more expressive details of an animation.

Pinscreen is also making waves. It is working on algorithms capable of building a photo-realistic 3D animatable avatar based on just a single still image. This is radically different to VFX simulations where scanning, modelling, texturing and lighting are painstakingly achieved such as ILM’s posthumous recreation of Carrie Fisher as Princess Leia or by MPC’s re-generation of the character Rachel in Blade Runner: 2049.

“Our latest technologies allow anyone to generate high-fidelity 3D avatars out of a single picture and create animations in real-time,” says Pinscreen’s Hao Lin. “Until a year ago, this was unthinkable.”

Pinscreen’s facial simulation AI tool is based on Generative Adversarial Networks, a technique for creating new, believable 2D and 3D imagery from a dataset of millions of real 2D photo inputs. One striking example on synthesising photoreal human faces can be seen at thispersondoesnotexist.com.

Such solutions are building towards what Ziva’s Smit calls “a heightened creative class”.

On the one hand this will enable professional VFX artists and animators to assign the technical work to automation in theory permitting more freedom for human creativity and on the other hand democratize the entire VFX industry by putting AI tools in the hands of anyone.

The videos posted at Derpfakes, of which Solo: A Star Wars Story is one, demonstrate the capabilities of image processing using deep learning. An AI has analysed a large collection of photos of a person (Ford in this case) and compiled a database of them in a variety of positions and poses. Then it can perform an automatic face replacement on a selected clip.

Touch of a button
Recent work at USC focusses on generating anime illustrations from massively trained artworks from thousands of artists. “Our algorithm is even capable of distinguishing the drawing technique and style from these artists and generating content that was never seen before using a similar style,” Lin reveals. “I see how this direction of synthesising content will progress to complex animations, and arbitrary content in the near future.”

Progress in this field is rapid, especially given the openness in the ML and Computer Vision community as well as the success of open source publication platforms such as arXiv. Further research needs to be done to develop learning efficient 3D representations, as well as interpretations of higher-level semantics.

“Right now, the AI/ML for VFX production is in its infancy, and while it can already automate many pipeline related challenges, it has the potential to really change how high-quality content will be created in the future, and how it is going to be accessible to end-users,” says Lin.

Human touch
While AI/ML algorithms, can synthesise very complex, photorealistic, and even stylised image and video content simply sticking a ‘machine-learning’ label on a tool isn’t enough.

“There’s a lot of potential to remove drudge work from the creative process but none of this is going to remove the need for human craft skill,” Robinson insists. “The algorithmic landscape of modern VFX is already astonishing by the standards of twenty years ago; and so much has been achieved to accelerate getting a great picture, but we still need the artist in the loop.

“Any algorithmic-generated content needs to be iterated on and tuned by human skill. We’re not in the business of telling a director that content can only be what it is because the algorithm has the last word. But we are going to see a greater range of creative options on a reduced timescale.” 

The future of filmmaking is AI and Realtime

A proof of concept led by facility The Mill showcased the potential for real-time processes in broadcast, film and commercials productions.

‘The Human Race’ combined Epic’s Unreal game engine, The Mill’s virtual production toolkit Cyclops and Blackbird, an adjustable car rig that captures environmental and motion data.

On the shoot Cyclops stitched 360-degree camera footage and transmitted this live to the Unreal engine producing an augmented reality image of the virtual object tracked and composited into the scene using computer vision technology from Arraiy. The director could see the virtual car on location and was able to react live to lighting and environment changes, customising the scene with photo-real graphics on the fly.

The technology is being promoted to automotive brands as a sales tool in car showrooms, but its uses go far beyond advertising. Filmmakers can use the tech to visualise a virtual object or character in any live action environment.

A short film using the technology is claimed as the first to blend live action filmmaking with Realtime game engine processing.