AI Motion Pictures Are Almost Ready for Their Close Up

NAB

Independent filmmakers are experimenting with AI tools today. While they are not yet ready for their big screen close-up it won’t be long until these technologies become widely adopted in Hollywood.

article here

The most high profile text to image AI is DALLE-2, released by Open AI.  The model does not offer motion picture sequences – but the odds are that is soon will. Open AI is likely working on this as we speak.

LA based director Paul Trillo has been creating stop-motion animations using it.

AI art is in its infancy and making fledgling attempts at ‘temporal coherence’  the ability to make something move as we expect it to move in film and video (not forgetting that film is a set of still images replayed 24 times a second).

Deforum is a text to motion AI tool based on the AI model Stable Diffusion (by Stability AI). AI artist Michael Carychao has used this to show how AI tools can re-create famous actors.

“In a couple of years, we’ll be able to write ‘Brad Pitt dancing like James Brown’ and be able to have a screen-ready coherent result,” reckons Pinar Seyhan Demirdag, co-founder of AI based content developers Seyhan Lee, blogging at medium.

Another example using Deforum is provided by an artist known as Pharmapsychotic. The animated sample in this tweet is claimed to be raw output with no post processing or interpolation.

“Give it a couple of years, and you’ll be able to film a scene of four random people walking down an aisle, and to turn them into Dorothy and the gang in the Wizard of Oz,” comments Seyhan Demirdag. “Arguably, you can do this right now, but it will be wonky, smudgy, and missing 8K details, so not ready for the mass audience.”

There are two ways of transferring a style right now. One with a pre-defined style, for example, a Van Gogh painting, the other by using text-to-image based models such as Disco Diffusion and VQGAN+CLIP, where you guide the style with words, referred to as ‘prompts.’

“These prompts are your most significant creative assets, and many people who make art with text-to-image tools also call themselves ‘prompt artists’.”

There are even sites suggesting the best prompts to work with specific AIs – like this one for DALLE-2

Considerable work is being done to incorporate generative art models into games engines.

Daniel Skaale, who works for Khora VR has posted this sample, where he has carried a 2D image that he has created in the text to image AI Midjourney into the Unity games engine.

As good as this is, generating in real-time in Unity or Unreal Engine remains an unexplored territory with huge potential, says Seyhan Demirdag.

Face replacement

@Todd_Spence made this mashup of Willem Dafoe as Julia Roberts in

Pretty Woman. Just for fun of course, but examples like this using AI apps like Reface give us a glimpse into how AI will help optimize production in future.

“Soon, studios will simply need to rent Brad Pitt’s face value rights for him to appear in the upcoming blockbuster film without having to leave the comfort of his couch,” says Seyhan Demirdag.

Similar models have already been used. For example, Focus Pictures’ Roadrunner: A Film About Anthony Bourdain used Deep Fake for Bourdain to say things he didn't (this was controversial mainly because the AI wasn’t acknowledged up front). The Andy Warhol Diaries also used AI to mimic Warhol’s narration but since this was credited in the title sequence up front this Netfflix doc received plaudits for its innovation.

As with any other technology in its infancy, AI art still misses temporal coherence, which is our capacity to make jumping jacks or walk down the street. 

“Right now, you can produce mind-bending, never-before-seen sequences with AI, but you cannot do everything (yet).

She adds, “In a few years, we’ll be able to generate coherent and screen-ready full features that are entirely generated. If you are a producer, director, studio owner, or VFX artist who wants to stay ahead of the curve, now is the time to invest in this technology; otherwise, your competition will be generating headlines, not you.”