Now We Have an AI That Mimics Iconic Film Directors

NAB

An AI that mimics the directorial style of Quentin Tarantino – or literally any other auteur you care to think of? The possibilities are tantalizing and within reach.

article here

A group of researchers out of Aalto University in Finland have devised a tool that generates video in the style of specific directors.

It’s so good that when put to the test and audiences can tell which directors’ style is being mimicked.

Cine-AI is in fact targeted at the automated creation of cutscenes in video games – but its applicability to generating cine-literate visual storytelling that emulate the film language of famous directors - alive or dead - is clear.

It would also only be a hop, skip and a jump to apply the same process to auto-generate synthetic cinematography as if lensed by Vittorio Storaro or Roger Deakins. Anathema as it sounds – this will be possible at a convincingly photoreal level sooner than we think.

The particular problem that the Inan Evin, Perttu Hämäläinen and Christian Guckelsberger sought to crack is laid out in their white paper published in August.

In-game cutscenes are non-interactive sequences in a video game that pause and break up gameplay. In high-quality (AAA) productions especially, cutscenes feature elaborate character animations, complex scene composition and extensive cinematography for which games developers may need to hire dedicated directors, cinematographers, and entire movie productions teams.

“Cutscenes form an integral part of many video games, but their creation is costly, time-consuming, and requires skills that many game developers lack,” they explains. “While AI tools have been used to semi-automate cutscene production, the results, typically lack the internal consistency and uniformity in style that is characteristic of professional human directors.

“We aim to realise procedural cinematography, focusing on how camera placement, shot continuity and composition can be brought together by algorithmic means.”

To that end they have devised Cine-AI, an open-source procedural cinematography toolset capable of generating in-game cutscenes “in the style of eminent human directors.”

Implemented in the game engine Unity, Cine-AI features a novel timeline and storyboard interface for design-time manipulation, combined with runtime cinematography automation.

Besides Tarantino they also chose to train their AI on the films of Guy Ritchie (Lock Stock…, Snatch, Revolver, Sherlock Holmes) explaining that not only did both directors have recognisable and unique shooting styles, but they were well known to a wider audience. That was important when it came to assessing the results.

Arguably also Guy Ritchie’s kinetic style works well for action-based video games while Tarantino’s works best for more sedate dialogue heavy scenes.

As the basis for the dataset they extracted 80 one-minute clips from the most highly rated movies of each director on IMDB. For added representativity, half of the clips chosen were action-heavy, and the other half to be strong on dialogue. Each clip was also assessed in terms of its dramatisation level and the scene’s pace, encoded on a scale with high values given to a scene that unfolds quickly for example.

Importantly, these cutscenes are not baked-into the game but designed to playback dynamically based on the actual game state, “rendering the production of different, static cutscenes for each possible gameplay outcome obsolete.”

With the finished clips they arranged for viewers to take the Pepsi challenge and to judge which was in the style of Ritchie and which a Tarantino. Eighty percent of the responses were correct.

The team is aware of their system’s shortcomings - and its potential. They note, for instance, that not all directors solely focus on camera work to express their style.

“Michael Bay, for example, notoriously employs many post-processing effects such as lens flares and god rays.”

Future enhancements to the software might introduce additional cinematography techniques including post-processing effects.

They would like to open Cine-AI up to more genres outside of action and to account for more nuanced - less iconic – directorial styles.

“Both directors are moreover white and western men, and future efforts in extending the dataset should focus on increasing director diversity,” they state adding they would be intrigued to see how well Cine-AI might reproduce the style of horror and action director Timo Tjahjanto.

Other directors cited as providing “a worthwhile challenge to procedural cinematography” include Spike Lee (who “focuses on color and race relations and is well known for his frequent use of dolly shots to let characters ‘float’ through their surroundings” they say) and French director Agnes Varda (“praised for her unique style of using the camera ‘as a pen’”).

The researchers don’t expect Cine-AI to “completely replace” actual human directors in video game creation but - like other developers of AI tools – see it as a co-creative in the process.

“The cinematography requirements of AAA games tend to get extremely sophisticated in terms of style, scene duration and size,” they explain. “While AAA companies will likely continue relying on dedicated production teams to achieve the desired level of cinematographic quality, they can utilise Cine-AI to create prototypes for their cinematography design, automatically generate shots to inspire new ideas, or use the storyboard feature to quickly iterate on possible shots, following a specific directorial style.”

Both the proof-of-concept dataset and the source code is now publicly available under an open-source license. The Finnish team is inviting other researchers, game developers and film enthusiasts to join them in taking the project further.