Contextual advertising proves its worth

IBC

Automated contextual advertising uses AI to help brands fine-tune the contextual relevance of their addressable campaigns.

article here

ITV and Sky Media have both introduced contextual ads which allow advertisers to buy spots based on the specific content within a programme, rather than more broadly against the type of show or film genre. New contextual targeting capabilities will enable content advertisers to align ad placement using written, audio and visual metadata created using AI.

In a world where consumers are increasingly ad-savvy, context is king, with 72% of consumers saying their perception of an ad is influenced by the surrounding content (Integral Ad Science, The Context Effect, September 2021).

It’s worth reminding ourselves that contextual targeting is nothing new in the world of television. In fact, it was a key component of how many advertisers and buyers targeted television in the past.

“Establishing the appropriate editorial moments for advertising adjacency was pretty much beaten into a television buyer when they joined an agency,” says Rhys McLachlan, Director of Advanced Advertising at UK commercial broadcaster ITV.

“So it’s not new in that respect. What we’re finding in the digital space is that contextual can be done in a much more sophisticated way. And as ITV as a business migrates at a rate of knots from a linear push broadcast ecosystem to an over-the-top-distributed and consumed IP system, the enablement of AI or machine learning-powered contextual targeting makes the canvas just so much greater.”

UK pay TV operator Sky’s ad sales arm Sky Media says more than 5,000 pieces of VOD content had been scanned and tagged against ad-tech standards organisation the Interactive Advertising Bureau (IAB)’s content taxonomy (which runs to more than 700 categories). Each tag includes a sentiment score, identifying positive and negative moments with which advertisers may choose to align messages. While Sky’s service is in test, ITV Adlabs launched Automated Contextual Targeting (ACT) two years ago. ACT uses AI technology to scan through every show across ITV Hub, categorising every scene in them by mood, object or moment.

Key themes

The technology, which has the ability to recognise facial expressions and words spoken within shows, initially allowed the team to draw up three key themes for the pilot: Food, Drink and Mealtimes; Moments of Joy; and Beauty and Cosmetics, with pharmacy chain Boots and supermarket Sainsbury’s as pilot brands.

“The tests helped us understand more about the complexities which we anticipated would occur in a live environment,” says McLachlan.

Algorithms must accurately identify the content’s meaning, nuances, and intent to deliver relevant ads. Inaccurate content and context interpretation risks irrelevant or misplaced ads and a poor user experience.

ITV withdrew ACT from the market and put it back into the labs to work with digital intelligence platform Captify to enhance the proposition.

ACT relaunched a year ago, claiming – in a world first – to identify emotions during ITV programmes for advertiser targeting. Captify’s natural language processing and machine learning models are used to extract more precise show themes which are categorised under an extensive contextual taxonomy. Buyable segments are then created from themes and made available for selection in ITV’s ad platform Planet V.

The launch campaign was the DHSC Every Mind Matters campaign (via OmniGOV and Wavemaker) to coincide with World Mental Health Day. For this campaign ACT was used to identify emotional moments in ITVX shows that were “uplifting or joyful”, to find audiences who are more likely to be feeling anxious or stressed, and to direct viewers to the Every Mind Matters online portal.

OmniGOV were reportedly delighted with the results. “It has enabled us to tap into the core audience behaviours and deliver a timely, supportive message to those on their own mental health journey,” said Louise Turpin, Head of Client & Implementational Planning, OmniGOV.

Pete Markey, CMO at Boots said, “Aligning with contextual moments is hugely important in our marketing activity, but particularly at Christmas, as aligning with those moments of joy is integral to our strategy.”

Nearly 200 campaigns have since been run through the ACT system. “We’ve got 18 cohorts available for ACT on ITVX via Planet V,” adds McLachlan

“From an ITV perspective, ACT has been instrumental in enabling us to unlock budgets for brands or for targeting cohorts that we wouldn’t otherwise be able to,” he said.

Food advertising

The example that constantly crops up when ITV illustrates ACT is food.

“We’re not big on food programming,” McLachlan explains. “The BBC and Channel 4 typically dominate the food magazine shows. If you were to analyse our output you’d probably find that in any year we would do a couple of hundred hours of food programming. But using our machine learning proposition we can scan the catalogue and categorise all the moments where food is consumed.

“There’s a mealtime in every drama and soap or reality show. When you combine them and do that analysis you establish that we’ve got tens of thousands of hours of moments where food’s being consumed. That’s where we start to unlock those brands that are looking to target food environments.”

He also highlights “real success” over the course of the summer with an Olympics ACT proposition. “Evidently, we’re not the broadcast partner for the Olympics but we do have substantial editorialisation of the Olympics across our channels. We were able to find moments that were ‘Olympic adjacent’ in our channels and attracted Olympic partner brands looking to advertise in and around Olympic-adjacent content, or content with an Olympic sentiment. This is where it’s starting to be really smart for us.”

There might be questions as to how much an AI can decipher what’s going on in a video. Is someone on screen really cooking or are they doing something else (a lot of murders take place in kitchens…).

McLachlan says ITV’s AI is scanning the imagery and, critically, the subtitles. “It’s not just the visual components presented on screen that are being evaluated. We’re also using the audio description and subtitles function, so we know that food is being prepared or the scene is around the meal table. Imagery, audio descriptors and subtitles are combined to give us trusted results.”

Desired brand building

Commercial broadcasters claim that adjacency to relevant TV content improves advertising relevance, boosting a campaign’s brand effect. Some advertisers might expect a greater measure of performance, of bang for their buck, with such a targeted solution but McLachlan says they still very much desire brand building.

“The way our addressable advertising works with our advertiser partners is there’ll be several lines on a plan. The first line will be a mass reach proposition with a ‘widely defined addressable audience’ using our data. Then it works down through the funnel through some ACT proposition so the brand is adjacent to the moments that matter and that make editorial sense with the tone or other aspects of the advertising copy. Then you’ll get even more granular addressable audiences about people who have previously bought the products or people who are buying competitive products from all the data sources that we have. It’s very rare that we will have an ACT-only campaign.

“What we’re able to do now is provide a myriad of jigsaw pieces that come together to build a cohesive campaign that works across all the addressable targeting components that we have. ACT is one of them.”

It is also possible to automate the placement of adverts within programming but this is not something deemed primetime for ITV.

“We are not using AI to brute force breaks in content immediately adjacent to relevancy, and that will be our policy for the foreseeable future,” says McLachlan.

He says he finds it “quite remarkable” how SVODs with ad-tiers “are struggling to establish policy” around where and how breaks should be presented in editorial content, such as feature films, that were produced without advertising in mind.

“We will continue to ensure that from an editorial perspective, we are developing and adhering to best practice and break management rather than using the tail to wag the dog.”

Adobe unveils editing upgrades as Firefly labelled first GenAI video model “safe for commercial use”

IBC

article here

Adobe says it has worked closely with professional video creators to advance the Firefly Video Model with a particular emphasis on generative editing

Adobe has progressed its generative AI video product with further tools to enhance the editing process launching in beta.

First previewed earlier this year, the Firefly video model’s capabilities launching in beta this week are designed to help video editors fill gaps in a timeline and add new elements to existing footage.

“Video is hard,” says Alexandru Costin, Vice President, Generative AI and Sensei at Adobe. “We’ve been working on it for a while but we weren’t happy with the quality. Our research department has done wonders to handle 100 times more pixels, 100 times more data using thousands and thousands of GPUs to make sure we master the research. We’re now proud of the quality we’re delivering.”

A 17-year Adobe veteran, Costin is charged with building and integrating generative AI models into the firm’s creative tools, and building AI and ML data, training and inference technologies and infrastructure for Adobe Research. He has helped launch generative AI models for imaging, vectors and design, which are integrated into products accessible from Creative Cloud.

The company says some 13 billion images have been created using Firefly to date. Customers like toy maker Mattel are using it to refine packaging ideas. Drinks brand Gatorade just activated a marketing campaign which encourages customers to design their own virtual bottles using a version of Firefly on its website.

Now the focus is on video generation using text-to-video and image-to-video prompts. Adobe customers though want to use AI smarts to speed up and improve editing rather than for pure video generation.

“Video was a big ask from our customers since video is now a very prevalent medium for content creation,” Costin says. “The most use we get from a Firefly image is Generative Fill [in which users can add, remove, or modify images using simple text prompts inside Photoshop] because we’re serving an actual customer workflow. More than 70% of our use cases for Firefly are in editing versus pure creation. Generative editing is the most important thing our customers are telling us in terms of what they need.”

Generative Extend

Generative editing essentially means helping video creators extend and enhance the original camera footage they already have.

Costin explains: “Most video post-production is about assembling clips, making sure you match the soundtrack and the sounds with the actual clips. One big problem customers have is that sometimes they do not have the perfect shot and cannot match it up with the audio timeline.”

Generative Extend in Premiere Pro is a new tool in beta that allows users to extend any clip by several seconds to cover gaps in footage, smooth out transitions, or hold on shots longer. Not only is the video extended but so too is the audio track.

“We’re extending the ambient ‘room tone’ to smooth out audio edits. You can even extend sound effects that are cut off too early. It’s an amazing technology. We’ve already heard from customers that they’re super excited about this application.”

Generative Extend won’t create or extend spoken dialogue, so it’ll be muted. Music is also not supported due to potential copyright issues, but you can automatically lengthen and shorten tracks with the existing Remix tool.

Also available in beta are Firefly-powered Text-to-Video and Image-to-Video capabilities. The former includes generating video from text prompts, accessing a wide variety of camera controls such as angle, motion and zoom to finetune videos and referencing images to generate B-Roll that fills gaps in a timeline. With Image-to-Video, you can also utilise a reference image alongside a text prompt to create a complementary shot for existing content, such as a close-up, by uploading a single frame from your video. Or you could create new b-roll from existing still photography.

Costin reiterates the importance of Adobe’s “editing first” focus. “We want to make sure customers bring their own assets and use generative AI to continue editing those assets or derive new videos from images because we’ve heard this is the most important thing they need.”

Control is another important attribute that creators are asking for.

Within the Firefly video application users can already write a detailed prompt to which is now added a “first wave of control mechanisms” for calibration of the shot size, motion and camera angle

Controlled prompts

“Our customers are very picky. They want to be able to control the virtual camera and make sure that their Prompt is well understood. They want to make sure we can generate high-quality videos that they can actually use not only in ideation but also in production.”

Within the Firefly video application users can already write a detailed prompt to which is now added a “first wave of control mechanisms” for calibration of the shot size, motion and camera angle.

“Those are very important control points that will help video creators to generate new clips using image-to-video or text-to-video to basically direct their shots so they can tell the story they want. We have many more research capabilities in control to come but we’re very proud of this first step and we’re going to keep investing in it.”

Another generative editing use case is for overlays in which editors can add visual depth to existing footage by overlaying atmospheric elements like fire, smoke, dust particles and water inside Premiere Pro and After Effects.

“We’re also focusing our model to learn both imaginary worlds and the real world so that the quality [of output] of the imaginary worlds is as high as for the real world.”

Another generative editing use case is for overlays in which editors can add visual depth to existing footage by overlaying atmospheric elements like fire, smoke, dust particles and water inside Premiere Pro and After Effects

You can even change the original motion or intent of your shot in some cases. For example, if your clip has a specific action and you’re an editor who wishes to pitch a reshoot to a director, you can help to visualise how the update will help the story while maintaining the same look.

Or if production misses a key establishing shot you can generate an insert with camera motion, like landscapes, plants or animals.

Generative Extend has some limitations in beta. It is limited to 1920x1080 or 1280x720 resolutions; 16:9 aspect ratio; 12-30fps; 8 bit SDR and mono and stereo audio

“We’re rapidly innovating and expanding its capabilities for professional use cases with user feedback. We want to hear how it’s working or not working for you.”

Adobe advises that editors can use can unique identifiers known as ‘seeds’ to quickly iterate new variations without starting from scratch.

It suggests using as many words as possible to be specific about lighting, cinematography, colour grade, mood, and style. Users should avoid ambiguity in prompts by defining actions with specific verbs and adverbs. Using lots of descriptive adjectives is a plus as are use of temporal elements like time of day or weather. “Bring in camera movements as necessary,” Adobe advises. “Iterate!”

Content authenticity

“The Firefly model is only trained on Adobe stock data, and this is data we have rights to train on.” Alexandru Costin, Adobe

For all the focus on editing features, Adobe is insistent that its “responsible” approach to AI differentiates it from companies like OpenAI where few if any guardrails on copyright infringement are acknowledged.

It claims Firefly is “the first generative video model designed to be safe for commercial use” and says this is what its customers want more than anything.

“Our community has told us loud and clear that they needed first and foremost to make sure the model is usable commercially, is trained responsibly and designed to minimise human bias,” Costin says.

“The Firefly model is only trained on Adobe stock data, and this is data we have rights to train on. We don’t train on customer data and we don’t train on data we scrape from the internet. We only train on Adobe stock and public domain data which gives us the confidence and comfort to ensure our customers that we cannot infringe IP.”

It is why Adobe offers its largest [Enterprise] customers indemnification. “We offer them protection from any potential IP infringement.”

Costin also points to the Content Authenticity Initiative (CAI), a cross-industry community of major media and technology companies co-founded by Adobe in 2019 to promote a new kind of tamper-evident metadata. Leica, Nikon, Qualcomm. Microsoft, Pixelstream, SmartFrame, the BBC and even OpenAI are among the 2,500 members.

“We’re getting more and more companies to join the consortium. All the assets that are generated or edited with GenAI in our products are tagged with content credentials. We offer visibility in how content is created so consumers can make good decisions on what to trust on the internet.”

Plus, Content Credentials can be included on export from Adobe Premiere Pro after using Generative Extend.

“We’re also respecting principles of accountability and transparency. In terms of accountability, we have this feedback mechanism in Firefly where we ask for and act on customer feedback. This is what helps us design and improve the guard rails that minimise bias and harm and minimise the potential of defects for our model. We’re proud of the approach we took in building AI responsibly and we know this is a key differentiator that makes our models trusted and usable in real workflows.”

 

NAB Show New York: US broadcasters tackle challenges of a shifting landscape

IBC

Local TV woes, Hurricane Milton and an existential threat to journalism itself hung over NAB Show New York but FAST and fan-fuelled TV offer hope.

article here

NAB Show New York wrapped last week having attracted more than 12,000 attendees, around the same as 2023 and a couple thousand down on its pre-pandemic peak. The two-day event comes at a time when US broadcasters face serious pressures on their business model and an existential threat from a Trump presidency.

Were it not a presidential election year US TV stations would be in even more pain. Political ad spending on TV stations is expected to reach $3.94 billion, up 10% from the last presidential election year in 2020 boosting industry revenue, including gross national/local spot advertising, digital and retransmission fees 8.3% to over $40 billion.

“We believe the industry will become increasingly reliant on political advertising revenue in even years,” said Rose Oberman, Media & Entertainment director, S&P Global Ratings. “At the same time, many local TV broadcasters face refinancing upcoming debt maturities at higher interest rates in 2026 and beyond. As a result, we expect EBITDA will gradually decline and cash flow will weaken.”

S&P Global Ratings have already downgraded several local TV broadcasters, including Cox Media Group, Gray Media, Sinclair and E.W. Scripps giving them negative outlooks.

This matters at a show like NAB which is owned by Gray, CMG, Sinclair, Scripps and 8000 other members of National Association of Broadcasters. At its second event of the year, held on the east coast, the lobby group continued to argue that it’s battle versus streamers like YouTube TV will be lost if the government doesn’t step in.

The NAB is specifically calling on the Federal Communications Commission FCC to regulate OTT programming distributors like for like with traditional pay TV distributors like Charter Communications and Comcast.

The issue is important to station groups, which believe they can increase their retransmission consent revenue if they are able to negotiate directly with YouTube TV, Fubo, Hulu and other virtual multichannel video programming distributors (vMVPDs).

As reported by TV Technology the discussion has divided the broadcast industry between station owners, who want the vMVPDs reclassified, and broadcast networks owned by major media companies, which want to continue to handle deals with online providers as part of much larger distribution pacts.

Currently, station groups handle retrans negotiations with traditional multichannel video programming distributors (MVPDs) like Charter Communications and Comcast. But broadcast networks and major programmers like The Walt Disney Co. and Paramount Global handle retrans deals with virtual MVPDs because the streaming services are not classified as traditional pay TV operators (or MVPDs) under FCC rules.

“It is impossible to understand the demands local stations face without coming to grips with the sea change wrought by streaming,” NAB chief legal officer Rick Kaplan said in a recent letter to the FCC. “The Commission’s rules were designed for a different world, and if the agency is truly committed to ensuring local service to communities across the nation, then it must examine the interplay between local broadcasting, the dramatic rise of streaming and the unregulated Big Tech behemoths that have shattered the economics underpinning local journalism.”

The show also coincided with Hurricane Milton the coverage of which on local TV illustrated its vital importance for serving communities, CBS Stations President Jennifer Mitchell said at the NAB New York conference.

“Local news is so incredibly important and remains important,” she said. “Local journalists know how to cover [Hurricane Milton] better than anyone else. That will not change. It’s just, how will we distribute that information? How will viewers consume that information? It’s important for us to be in all of those different places and serve the consumer.”

While Mitchell was at NAB NY, Donald Trump was demanding that CBS News have its broadcast licence revoked for editing an election special interview with Kamala Harris.

This is not being dismissed as posturing since should Trump win the White House and implement right wing manifesto Project 2025 it would force the FCC into censoring all media companies and their licensed outlets for reporting negatively about the president.

FCC Chairwoman Jessica Rosenworcel released a statement saying that these threats against free speech are serious and should not be ignored. Rosenworcel wrote. “The First Amendment is a cornerstone of our democracy. The FCC does not and will not revoke licenses for broadcast stations simply because a political candidate disagrees with or dislikes content or coverage.”

CTV scale

In the show conference representatives of CBS News, NBCUniversal Local, and E.W. Scripps debated ‘Advanced Streaming and FAST Revenue Strategies for Local TV.’

There was optimism about the ability to leverage data with AI to deliver more targeted content and advertising. They also agreed that scale was needed to reach audiences on multiple platforms.

 Tom Sly, VP, Strategy at Scripps said, “We have some peers that are afraid to promote Connected TV, and they're afraid to talk about it…but we have to wake up and say that business and the audiences are transitioning and we’ve got to change with them and we’ve got to move fast.”

In another session focussing just on Tubi CEO Anjali Sud said the Fox-owned FAST service was exploring fan-driven content development as a unique area of user engagement and growth. “We just launched the first fan-fuelled studio in streaming which is designed to allow anyone to pitch an idea to be green-lit by Tubi and distributed on Tubi based on fan feedback. Fans are going to make the show. We're working with Color Creative to help and them succeed and we will learn about whether that's a model that can work. I'm interested in what are the scalable and efficient ways to bring in unique stories from unique storytellers.”

The AV convergence

With the broadcast market static to shrinking technology vendors are expanding their sales activities into adjacent sectors, notably from the corporate, government, higher education, and finance world. As IBC did this year, NAB is also making its exhibition more attractive to those buyers and said it had hosted executives from Bank of America, Best Buy, Dell, Ford, IBM, JP Morgan Chase, the U.S. Air Force, and U.S Military Academy among core audiences from ABC, CBS, NBC, FOX, HBO and Univision.

“Technologies like virtual production and inexpensive cine-style cameras make content creation much more easily available to people who aren't television broadcasters, but what's driven a real acceleration in the convergence of broadcast with AV is the ability to extend delivery of content directly,” says Ciaran Doran, CEO of cloud broadcast vendor M2A Media.

“Brands have woken up to the fact that they have been building a one-to-one direct relationship with their customers and that the next step is to communicate by live video and build an even tighter engagement with them.”

 

The Instigators: Henry Braham BSC on reworking the rules of how to shoot a blockbuster

RedShark News

Henry Braham BSC talks about shooting Apple TV’s movie The Instigators on RED in a truncated timeframe of a little over a month.

article here

Laid-back comedy caper The Instigators succeeds on its own terms in feeling unencumbered by the weight of a big studio movie despite the A-list charms of Matt Damon, Casey Affleck (who also co-wrote the script) and Hong Chau, a number of weighty character actors (Ving Rhames, Ron Perlman, Toby Jones) and being bankrolled by Apple.

Director Doug Liman is a master at flipping a paper thin plot into slickly entertaining escapade (see Mr. & Mrs. Smith, American Made) and has shot his last two movies in concert with cinematographer Henry Braham BSC.

Braham is renowned for his work in developing character driven performances stretching back to indie fayre like Stefan Schwartz’ Soft Top Hard Shoulder (1992) through Nanny McPhee (2005) to  studio blockbusters The Golden Compass (2007) and Guardians of the Galaxy Vol. 2 (2017).

“It sounds really obvious but Road House was about the heat and the water. Set in the Keys, shot in the Dominican Republic. It's a cowboy movie, an action movie, it’s all about the fights,” says Braham of the Jake Gyllenhaal feature directed by Liman which released earlier this year on Amazon Prime.

“The Instigators is the exact opposite. It's cold. It’s Boston. It’s a buddy movie about hapless guys for whom everything goes wrong. The common thing between them is Doug's approach to filmmaking.”

As part of his prep Braham scouted locations in Boston and New York. “[The production] had great collaboration from the city. We were allowed to run amok in Boston City Hall [the modern concrete behemoth that dominates downtown] and across the streets. It was important that we shot during winter time so that there was a sort of a greyness to it and nothing looked too pretty.”

That fits with the blue-collar post-industrial milieu of the film which to a degree recalls Michael Mann’s Chicago set heist movie Thief (1981) where the hero (a career best performance from James Caan) believes that stealing from the rich equates to social justice, provided no-one gets harmed.

Affleck’s screenplay for The Instigators also has downtrodden heroes sticking it ‘to the man’ – in this case corrupt politicians. Braham leans into the cloudy skies and gives the film a grey-blue hue.

“Obviously the story is fantastical but if you can ground it more in reality then you buy into the story better,” he says. “Although it's ridiculous there are all kinds of truths that come through the movie. So, photographically it's important that we ground it.”

He continues, “It wasn't important to me to show off the Boston locations. It was about telling the story of these two guys and the chaos around them.”

The interior of the mayor’s office was built at Broadway Stages in Brooklyn with a layout based on the real thing, duplicating Boston City Hall’s exposed concrete structure and high ceilings.

Braham’s camera is hardly ever static and often frames close-up to the actors and very close indeed with shots inside vehicles during the car chase [one of the most extensive ever filmed in downtown Boston which also filmed inside the I-93 expressway tunnel]. The cinematographer says his aim was to keep connected to the performances, not to over plan shots and to be intuitive to the moment.

“Hopefully the audience is not aware of the camera at all but just feels very connected to the story. A classic way of doing that is for the camera to stand back and be on longer lenses and to observe what's going on. There are brilliant examples of that in Doug's early movies, especially in his first Bourne picture (The Bourne Identity 2002).” 

Another example is In The Mood For Love (shot by Christopher Doyle for Wong Kar Wai) which Braham says is “beautifully observed and not shot conventionally at all.”

“Those are two ways of doing it but there's a new way of doing it which technology has recently allowed us to achieve. It's where we can shoot on large format cameras which are physically small and which enable you to get the camera close in and amongst the actors. 

“Because the negative format is large you can use wider lenses that don't look like wider lenses on screen. In every filmmaker's dream there's a thing called a ‘tight wide’ shot which is where you want to see everything but at the same time you want to be close and connected to what's going on. 

“In the old days we used to have a wide shot or a tight shot. But now, thanks to the advent of large format digital cameras, we can have our cake and eat it, which is a tight wide shot.

Braham has been developing this technique and related philosophy over a number of films with James Gunn including Suicide Squad (2021), and Guardians of the Galaxy Vol. 3 (2023) and The Flash for director Andy Muschietti.  All are in the Marvel or DC Universe and foreground character over VFX.

“It means you can be entirely intuitive with a handheld camera,” says Braham who likes to operate himself. “You can be very precise about the relationship of the camera to the actors. The actors aren't bound by marks on the floor. I can always keep the camera in the right relationship to them and they will always be in the right relationship to the other actors and the set.”

This approach frees actors, directors and camera to perform and respond more freely on set. “It's a really fun way of making a movie. It’s not consciously artful. That’s hard to explain but I think when you shoot something conventionally there's an artfulness to it which audiences are aware of. For lots of movies that's entirely appropriate. But in a movie like this, it’s not.”

He cites the 1971 classic The French Connection shot by Owen Roizman as revolutionary in breaking many of the established rules.

“They didn’t use wide, mid-shot to close-up. Their response to the story on location was intuitive and that was enabled by being able to use small cameras rolling a single strip of film and cameras which were among the first to be single-lens reflex – the operator could look through the lens and see exactly what the camera was seeing. Plus, it was one of the first films to take advantage of the  zoom lens. I believe where we are now with digital cine technology is the next genuine progression since that era of filmmaking.”

Most of the enjoyment in watching The Instigators comes from the repartee of the three leads. Their comic timing is often captured in camera rather than feeling manufactured in the edit and is testimony to Braham and Liman’s approach.

“Doug encouraged improvisation and because we have a shooting style that can evolve with the scene we can capture moments that feel spontaneous. That’s a lot harder when there's a team of people directly behind the camera worried about getting into territory that hasn't been discussed.

“In this case, most of the film is shot single camera and it is myself and my demon focus puller (A camera assistant Dermot Hickey) who has to react to what I’m doing which he often won’t know in advance. 

“Doug has got this antenna for things that don't feel truthful. If it's not working in one way, he'll try something completely different and that could just literally turn the whole scene on its head. To have the flexibility to do that is important for a filmmaker like that. We always ended up with something significantly better.” 

Shooting RED

The look of Road House may be entirely different but Braham used the same camera package of RED V-Raptor with Leitz M 0.8 glass. He’s been working with RED on all of his recent films and says he changes up as new sensors or features are released to always work with the latest model. 

“In camera technology there's been a steady progression of improvements but I've kept the same fundamental principles for every film I’ve shot over last four to five years.

“When RED developed their first VistaVision camera it was revolutionary because the moment you have the larger negative the lenses you can use change completely. When you get used to shooting VistaVision and then go back to 35mm it’s like going backwards to 16mm. The profound step forward in movie making technique is the move to large format. 

“Secondly, it’s putting that sensor in a camera the size of a Hasselblad. RED really nailed this. The V-Raptor is so light it enables you to do all sorts of things that are intimate or to put the camera in places which wouldn’t fit a larger one.

“In the days of film the cinematographer would take time to get to understand a new film stock when it came out. You had to get to know the material. It’s a bit like an artist testing the canvas or wood or texture on which they will paint. Your raw materials react differently. It's the same with photography. Each time something changes from my point of view, I really have to understand what that is. Once I’ve understood the tools then goes into the background. I’ll just know the photographic range I’m working with and what a particular camera is good at.”

In this endeavour Braham has been aided by the bespoke designs of David Freeth co-developer of the stabiliser system Stabileye.  “It’s incredibly expressive,” Braham says of the remotely-operated, miniature stabilised head. “It kind of allows the camera to dance with whatever's going on in front of it.” 

He has just shot Superman for James Gunn and says 90 per cent of the film was shot in this fashion.   “A few years ago we used to think a big, big movie like this had to be approached in a certain way but I don't think you need to now. Directors change their minds when they discover the freedom they get from this approach.”

The tight 36-day shoot would arguably not have been possible without the Braham-Freeth proprietary system that can emulate a dolly shot, a crane shot, a handheld shot, all with the same camera system. It meant that with a pre-lit the set designed for naturalism Liman could point the camera anywhere without need to wait between set-ups.

Big or small screen?

There has been some reported disagreement between Liman and Amazon over theatrical release of Road House. Liman seemed to suggest that an agreement by the streamer to give the film a cinema run was reneged on. The Instigators has also gone straight to AppleTV+ but the filmmaker seems to have known this in advance. 

“The way we shot Road House was intended for the big screen,” Braham says. “It was a very immersive kind of roller coaster ride. The fight scenes are specifically designed for IMAX and we colour timed those on a big screen so the idea is that the audience is literally part of the fight. 

“My approach is always to assume everything's going on a big screen. Because we shoot in a way I’ve described using the tight wide shot, it works whichever size screen. I’ll often see movies in the cinema but I’ll go back and watch them on TV over the years simply because I enjoy them. A good film well told will stand up to being viewed repeatedly.”

He worked with regular colourist Stefan Sonnenfeld of Company 3 to grade the film. “My process now is to shoot material during prep and to build a colour bible of how the movie is going to look. You can talk about these things and look at bits of reference but I found it most helpful to shoot some material and build the look so that everybody from design to costume can work toward the same goal. It’s like designing your own print stock. I love that it's like starting with a blank sheet of paper. Some people don't like that because they want some rules to work with. I'm more interested in there being no rules.”

 

Brian Sherwin: Grand master plays postproduction chess to manage episodic workflows

interview and copy written for Sohonet

article here

Take a selection of hit drama series over the past couple of decades – Salem, Detroit 1-8-7, Pachinko, Ugly Betty, The Haunting of Hill House, Panhandle, Yellowstone. Different studio, network or streamer (Warner Bros, Disney, Paramount, Sony Pictures, Netflix, TNT, AppleTV+, Roku). Different showrunner, director and genre. What they share in common is the post-production expertise of Brian Sherwin guiding them from start to finish as production manager, post coordinator or producer.

Hi Brian. Thanks for taking the time. Can you give us a brief resume about how you broke into the industry?

I moved out to LA right after graduating college and worked at E! Entertainment Television to kind of get my feet wet. I realized after a point that I really wanted to work in scripted television and that this was my passion. I left there as an associate producer and took a step down to be a post PA in scripted television because that's what I really wanted to do. Over the years, I've just worked my way up from post PA to post producer where I run the department on various shows. I'm hired by the show and work on that until completion. One of the most enjoyable aspects of my job is that every project, every team and every workflow is slightly different.

Presumably, the team that you work is dependent on the scope of the project. The post team on a show could just be myself, a s post supervisor with and a post production assistant, or if it's a bigger project, you may have a post supervisor, a post coordinator and post PA. It all depends on the size of the project and budget. The editorial team will vary in size accordingly.

‍How have you seen that working relationship change as more of it can now be managed in a distributed fashion?

Generally speaking, the process itself has remained the same, but technology has evolved and continues to change the location we are able to do it. In a situation like the pandemic, where everyone was forced to work remotely, you're forced to rapidly adopt remote technologies while using the same general process to move the show forward.

If you had a color correction session, for example, or a session where you're dropping in visual effects and you couldn’t actually go to the post facility, we would use Clearview Flex, the video collaboration software to be able to supervise that from home. The online editor might be working with the media in the post facility but by using Clearview Flex and the client is able to view the same screen online while being able to talk back and forth to approve shots in real time.

‍When you architect a workflow for a new show now how would you gauge the ratio of remote versus in-person?

It's kind of half and half. The starting point is dictated by whether the executive producer or showrunner really wants to be in the office or not. That’s the first thing you're trying to figure out.If the showrunner prefers to work in the office, then you find the space. If they say, they want to do some type of a hybrid situation, that's what we’ll do. Or it can be fully remote.

A lot of their decision making depends on where you’re shooting and where you are posting. If you're finishing final color correction and sound in another geographic location to the principal photography and you need to complete those while future episodes are being shot then you’ll need a very flexible workflow. For example, on Panhandle the crime comedy series for Sony Pictures, all of the finishing was completed in Vancouver in order to access tax breaks for that show. Editorial was managed here in LA. So, the workflow was completely remote.

For AppleTV+ drama Pachinko I was in Los Angeles and the post team was in New York. We had four editors on the show, three in New York and one in London. The executive producer was in New York where I worked for a short period of time.  I was flying back and forth for the final mixes.

What’s the biggest benefit of working with Clearview?

A hybrid workflow facilitated by Clearview has become a positive choice. When we do the final audio mix, if a person can’t physically be there in the room, they are able to fully participate using Clearview Flex. All the producers, studio executives and network executives are able to hear audio playback and give notes in real time. The biggest benefit is the ability to work in real time.

If we didn't have that option, you’d have to have the sound facility send the editor or assistant editor a file, which they would review and give notes and then pass those notes back to the facility. That to and from can prolong the process because you're adding a lot of extra steps.  Then adding a producer or executive would be another link in the chain.

‍How are you seeing AI assist in the post-production workflow?

Technologies are constantly evolving but the bones of the process remain the same. For example, VFX might use AI to track a shot frame by frame a little quicker. But as far as my role in managing the process and making sure we get from A to B to C to D, AI has not materially impacted that workflow as yet.

‍Scripted TV is suffering on a number of fronts. From your position on the inside what do you think the future holds?

I think you can take a positive or negative view on this depending on how you feel about it. A few years ago, it felt like the Golden Age of Television with maybe 600 shows being made a year. Following Covid and the strikes everything's contracted. Studios are now owned by large corporations. The content creation business of the major studios is a much smaller portion of the overall business of a corporation. If the whole corporation doesn't do as well then the entertainment division tends to suffer. That said, I detect things are slowly starting to pick up again. Everyone is trying to figure out what that new normal is going to be.

‍Can you give us a glimpse as to what it takes to post produce a major episodic show?

In pre-production I’m working very closely with the director, the director of photography, the producers and the studio or network to figure out technical specs such as what camera is going to be used and what the pipeline will be to get that material from location to editorial. You have the responsibility to make sure that it all runs smoothly and to budget. The footage also needs to be made available for the studio and network to view. Everyone needs to see it. So, there's a lot of pressure to get the footage pipe up very quickly. After all, there's a lot of money involved in these projects.

That's the beginning of the post production progresses. In addition, you’re working with the editors and assistant editors towards finishing and delivery. At the same time as they’re editing, you're shooting more shows while planning many moves ahead. It’s like chess. You're working with production to figure out what's going to be needed for the next episode being shot. For instance, are we going to shoot things practically or in use VFX? There's a lot happening in post during production - and that's before we even lock picture which is when our job really kicks into high gear!

‍Of all the successful shows you’ve been involved in what is your personal favorite?

I can't pick one stand out because each part of the process is different. For example, I may have worked with a cast in ADR which was phenomenal. There may be editors on a show with whom you have a particularly great experience. Other times, it’s the entire post team. It could be a show where like I love working with a specific colorist, visual effects house or sound facility.

Pachinko was an extremely difficult project but it was incredible just for the fact that a small percentage of it was in the English language and the rest was in Japanese and Korean. It was such a rewarding challenge. There's so many intricacies that make each project you're dealing with unique and memorable, in addition to the varied subject matter of the show.

 

Modelling intent: How evolution of AI will turbocharge creative capabilities

IBC

Framestore’s CTO Lincoln Wallen says the creative industries should acknowledge the art of math to craft new interactive AI-driven social experiences.

article here

The future of media and entertainment will be interactive, social, and fuelled by AI with companies versed in blending tech and art at the forefront of development, according to Framestore’s Chief Technology Officer (CTO), Lincoln Wallen.

“It’s going to be a slow burn but the technological basis of it has already been built,” he says. “Imagine experiences in which consumers can interact but are nevertheless driven by narrative visual storytelling. It’s a mixture of active lean forward and passive lean back. That is a media type that I expect to explode over this next period.”

Wallen joined Framestore and Company 3 Group (FC3) last summer from Improbable, a tech developer building platforms for multiplayer games and battlefield simulations which was valued at $3bn in 2022. He previously spent four years at Electronic Arts building its mobile games business before entering the film industry at DreamWorks, initially as Head of R&D from 2008, and then promoted to CTO in 2012.

That career journey has given Wallen an orbital perspective of future media in which technology and content, both IRL location-based and virtual, merge to create new forms of real-time communication and storytelling. 

“If you want to spot the metaverse as it emerges it’s going to come in interactive experiences in which large numbers of consumers come together,” he explains. “We do this today in social environments when out shopping, at sporting events, on the street, at festivals and conferences, but the digital world doesn’t properly support that sense of social engagement.

“The only markets in which people really come together digitally are video conferencing and video games but those experiences are designed for very specific numbers of players, not for crowds. Being able to interact with people meaningfully in a digital environment changes the game.”

Technology that facilitates simultaneous multi-player interactions at scale will underlie future media says Wallen, who believes this has been cracked by Improbable (of which he remains a Non-Executive Director). The next step is to evolve the user experience.

“VR really hasn’t left the laboratory in terms of the consumer experience. It still isolates the user. But the metaverse will come back as all truly disruptive ideas do,” he says.

An example of what he means is the work Improbable has done with Major League Baseball (MLB). Here baseball fans can gather in an MLB virtual ballpark and interact simultaneously in a single place, at a naturalistic scale. It claims to be the first service to host such large interactive sports experiences.

Another example includes fashion brands that might organise a virtual runway show inviting people to the physical catwalk and thousands more people to virtually inhabit the same space. Wallen believes this type of experience will eventually happen daily as physical interfaces like mixed reality goggles improve, but that the content itself won’t work unless crafted with storytelling expertise.

“The audiovisual world of linear media has built up an enormous bank of techniques creating adverts, TV shows or movies to grab a consumers’ attention and to stimulate emotions but those techniques are not used today within interactive environments,” he insists.

“Those working in linear media will call it ‘immersive’ as a way of trying to signal that this is something you’ve seen in VR experiences or in location-based entertainment. What we haven’t yet seen is that experience realised in a fully digital environment. It’s neither gaming nor linear but something in between. The potential of this middle ground is enormous.”

“The linearised model of film production is going to collapse into a much more iterative holistic model” Lincoln Wallen, Framestore

If Hollywood studios are too entrenched in their own structures to pivot fast enough, companies that are operating across market sectors will be able to synthesise these capabilities and exploit them. Framestore is one. Its work ranges from the VFX on Gladiator II to designing an interactive exhibit for the Science Museum.

“That isn’t to say that Framestore is suddenly going to become a content company,” Wallen stresses. “It means identifying those brands and IPs that are going to pioneer this new media and serve their existing customers in a different way.”

Framestore launched in 1986 at the birth of the computer graphics revolution. It remains headquartered in Soho where it has navigated and led the industry-wide transformation of post-production from an afterthought into an integral partner spanning the creative industries.

“The linearised model of film production is going to collapse into a much more iterative holistic model,” Wallen says. “Right now, the previz phase and the production phase feel separate. I think AI is going to bring the back end right up to the front entrance and make certain complex processes real-time.”

Bridging the uncanny valley

For Wallen, the key to the future of interactive media is the ability to create digital representations that talk, behave, respond and ‘think’ like a human. Such avatars could be our own online personas or the embodiment of a corporation. AI is already supercharging such developments in video games where AI-driven non-player characters (NPCs) are being introduced to interact in a more lifelike manner with game players.

“You are going to see NPCs having existential dialogue with game players,” Wallen says. “This is one of the most exciting things to me. Large language models (LLMs) have exposed a degree of structure in natural language that makes dialogue possible at a level that it hasn’t been in the past.

“When people look back on this era it is going to be the impact on language that will be seen as the most profound. Natural language will become a much more effective interface to [digital] systems and this will make them more accessible, which will be an enormous economic benefit.”

There is a problem though. The increasing sophistication of LLMs must be matched by sophistication in visual behaviour, emotion and characterisation. The uncanny valley is yet to be bridged.

“We don’t have a language of performance. There’s nothing to correlate spoken language and non-verbal gestures to make your NPC seem alive. Animators study human behaviour and know exactly how to produce that effect but we don’t have tools that do that automatically. You can’t take the animator out of the equation because they understand what human gestures mean, but the way they’re created right now is painstakingly frame by frame. We have to build that capability.”

Wallen believes that the creative industries should acknowledge the art of math to craft new interactive AI-driven social experiences.

The EU-funded research project PRESENT (Photoreal REaltime Sentient ENTity) of which Framestore is a partner aims to create virtual sentient agents that look and behave entirely naturalistically. The aims are to demonstrate emotional sensitivity, establish an engaging dialogue, add sense to the experience, and act as trustworthy guardians and guides in the interfaces for AR, VR, and more traditional forms of media.

“It’s only the high-end media industries that actually know how to produce digital performances, that that make people feel that NPCs are alive,” Wallen says.

Understanding how the world works

“There’s a very ambivalent relationship in the humanities between systematisation and what they think the humanities are all about, which is inspiration” Lincoln Wallen, Framestore

Wallen has a PhD in artificial intelligence from the University of Edinburgh, and subsequently taught at the Oxford University Computing Laboratory before moving into the media industry. He is also a non-exec board member of Finnish VR/MR headset maker Varjo, Pearson - the FTSE100 global education company, an advisor to bp and a 25-year member of the maths and AI strategic consultancy Smith Institute. Few people understand the intersection of science with the humanities as well as he and that means his view on AI as an artistic tool is not binary.

“All through my career the driver has been the interplay between modelling the world and activating it through computational systems,” he says. “Understanding how the world works is a passion.”

What strikes him about the current AI wave is the tension it has caused between inspiration and predictive calculation, but it’s a tension that is not new, and the two are not mutually exclusive.

“Making movies is a balance between algorithmic processes and artistic choice. That’s a very natural combination that AI is forcing to be recognised.”

He explains that neural networks, a mathematical construct designed to mimic human thought, has reached a level of maturity that demonstrates structure in natural language.

 

Stevie (Alba Baptista) will star as the protagonist of FLITE, Framestore’s experimental short film set in a futuristic London of 2053

“We now understand that there’s an awful lot of structure in writing. It’s not just a grammatical structure. Literary critics have argued this for centuries, but the humanities oscillate between seeing these patterns and then running away from them because it seems to be contradictory to creativity and choice. There’s a very ambivalent relationship in the humanities between systematisation and what they think the humanities are all about, which is inspiration.”

He argues that there is an art to mechanisation: designing-in specific outcomes that preserve human creative agency, with plenty of scope for machine learning technology.

“LLMs don’t manipulate intent,” he insists. “Humans create intent. They communicate and collaborate to organise their efforts to produce fantastic outputs, whether that’s a video game or a film. Without human dialogue, without the imagination that’s really framing the outcomes, you don’t get the products.”

We can use advanced AI techniques to try to realise that imagination, he says.

“Computational techniques are forcing recognition that there are structures, including within narrative visual storytelling.”

Language of intent

Wallen thinks the industry needs a standardised way for artists and filmmakers to communicate with each other in the age of AI, using AI tools. Current-generation AI tools, he says, are not intuitive and actually hinder artistic collaboration.

“GenAI tools work at a very low level. They don’t help you express what you’re trying to do. The only way to get the machine to respond is to give it instructions bit by bit of what you want to see but not why you want to see it.

“What we want is a language of intent,” he argues. “We want what the artist is trying to achieve made more directly expressible and reflected by the interfaces of these tools. If we had these languages we would be able to build technologies that really turbocharged people’s creative capabilities. That I think is what the end result of this AI wave will be but only if the creative industries start looking at their systemisation techniques and their content [Wallen uses the term folklore) and turning it into activatable language.”

This is something that Wallen is exploring at Framestore but it’s too vast a challenge for one company alone.

“I’d hesitate to say that this is a century of development but it’s also not easy. What I’m trying to do is stimulate academics and creatives to think about the problem. Even small areas of advancement in this area can lead to enormous benefits to the creative process. It is one of my personal motivations for joining Framestore.

“Whether it’s your digital proxy, virtual playmates, NPCs in games, a retail concierge, or part of the crowd in a virtual stadium doesn’t matter. There are so many applications of LLMs that are going to explode when coupled with character.”

Wallen completed a thesis on Mechanisation in Mathematical Art during the 1980s and suggests that the challenge is as old as the Scientific Revolution itself. He refers to early computers which worked by punching sequences into cards.

“By attending to the design of, and access to, the tools with which we ‘punch the cards’ today we can preserve the poets and even pursue ‘poetical science,’” he says.