Adobe unveils editing upgrades as Firefly labelled first GenAI video model “safe for commercial use”

IBC

article here

Adobe says it has worked closely with professional video creators to advance the Firefly Video Model with a particular emphasis on generative editing

Adobe has progressed its generative AI video product with further tools to enhance the editing process launching in beta.

First previewed earlier this year, the Firefly video model’s capabilities launching in beta this week are designed to help video editors fill gaps in a timeline and add new elements to existing footage.

“Video is hard,” says Alexandru Costin, Vice President, Generative AI and Sensei at Adobe. “We’ve been working on it for a while but we weren’t happy with the quality. Our research department has done wonders to handle 100 times more pixels, 100 times more data using thousands and thousands of GPUs to make sure we master the research. We’re now proud of the quality we’re delivering.”

A 17-year Adobe veteran, Costin is charged with building and integrating generative AI models into the firm’s creative tools, and building AI and ML data, training and inference technologies and infrastructure for Adobe Research. He has helped launch generative AI models for imaging, vectors and design, which are integrated into products accessible from Creative Cloud.

The company says some 13 billion images have been created using Firefly to date. Customers like toy maker Mattel are using it to refine packaging ideas. Drinks brand Gatorade just activated a marketing campaign which encourages customers to design their own virtual bottles using a version of Firefly on its website.

Now the focus is on video generation using text-to-video and image-to-video prompts. Adobe customers though want to use AI smarts to speed up and improve editing rather than for pure video generation.

“Video was a big ask from our customers since video is now a very prevalent medium for content creation,” Costin says. “The most use we get from a Firefly image is Generative Fill [in which users can add, remove, or modify images using simple text prompts inside Photoshop] because we’re serving an actual customer workflow. More than 70% of our use cases for Firefly are in editing versus pure creation. Generative editing is the most important thing our customers are telling us in terms of what they need.”

Generative Extend

Generative editing essentially means helping video creators extend and enhance the original camera footage they already have.

Costin explains: “Most video post-production is about assembling clips, making sure you match the soundtrack and the sounds with the actual clips. One big problem customers have is that sometimes they do not have the perfect shot and cannot match it up with the audio timeline.”

Generative Extend in Premiere Pro is a new tool in beta that allows users to extend any clip by several seconds to cover gaps in footage, smooth out transitions, or hold on shots longer. Not only is the video extended but so too is the audio track.

“We’re extending the ambient ‘room tone’ to smooth out audio edits. You can even extend sound effects that are cut off too early. It’s an amazing technology. We’ve already heard from customers that they’re super excited about this application.”

Generative Extend won’t create or extend spoken dialogue, so it’ll be muted. Music is also not supported due to potential copyright issues, but you can automatically lengthen and shorten tracks with the existing Remix tool.

Also available in beta are Firefly-powered Text-to-Video and Image-to-Video capabilities. The former includes generating video from text prompts, accessing a wide variety of camera controls such as angle, motion and zoom to finetune videos and referencing images to generate B-Roll that fills gaps in a timeline. With Image-to-Video, you can also utilise a reference image alongside a text prompt to create a complementary shot for existing content, such as a close-up, by uploading a single frame from your video. Or you could create new b-roll from existing still photography.

Costin reiterates the importance of Adobe’s “editing first” focus. “We want to make sure customers bring their own assets and use generative AI to continue editing those assets or derive new videos from images because we’ve heard this is the most important thing they need.”

Control is another important attribute that creators are asking for.

Within the Firefly video application users can already write a detailed prompt to which is now added a “first wave of control mechanisms” for calibration of the shot size, motion and camera angle

Controlled prompts

“Our customers are very picky. They want to be able to control the virtual camera and make sure that their Prompt is well understood. They want to make sure we can generate high-quality videos that they can actually use not only in ideation but also in production.”

Within the Firefly video application users can already write a detailed prompt to which is now added a “first wave of control mechanisms” for calibration of the shot size, motion and camera angle.

“Those are very important control points that will help video creators to generate new clips using image-to-video or text-to-video to basically direct their shots so they can tell the story they want. We have many more research capabilities in control to come but we’re very proud of this first step and we’re going to keep investing in it.”

Another generative editing use case is for overlays in which editors can add visual depth to existing footage by overlaying atmospheric elements like fire, smoke, dust particles and water inside Premiere Pro and After Effects.

“We’re also focusing our model to learn both imaginary worlds and the real world so that the quality [of output] of the imaginary worlds is as high as for the real world.”

Another generative editing use case is for overlays in which editors can add visual depth to existing footage by overlaying atmospheric elements like fire, smoke, dust particles and water inside Premiere Pro and After Effects

You can even change the original motion or intent of your shot in some cases. For example, if your clip has a specific action and you’re an editor who wishes to pitch a reshoot to a director, you can help to visualise how the update will help the story while maintaining the same look.

Or if production misses a key establishing shot you can generate an insert with camera motion, like landscapes, plants or animals.

Generative Extend has some limitations in beta. It is limited to 1920x1080 or 1280x720 resolutions; 16:9 aspect ratio; 12-30fps; 8 bit SDR and mono and stereo audio

“We’re rapidly innovating and expanding its capabilities for professional use cases with user feedback. We want to hear how it’s working or not working for you.”

Adobe advises that editors can use can unique identifiers known as ‘seeds’ to quickly iterate new variations without starting from scratch.

It suggests using as many words as possible to be specific about lighting, cinematography, colour grade, mood, and style. Users should avoid ambiguity in prompts by defining actions with specific verbs and adverbs. Using lots of descriptive adjectives is a plus as are use of temporal elements like time of day or weather. “Bring in camera movements as necessary,” Adobe advises. “Iterate!”

Content authenticity

“The Firefly model is only trained on Adobe stock data, and this is data we have rights to train on.” Alexandru Costin, Adobe

For all the focus on editing features, Adobe is insistent that its “responsible” approach to AI differentiates it from companies like OpenAI where few if any guardrails on copyright infringement are acknowledged.

It claims Firefly is “the first generative video model designed to be safe for commercial use” and says this is what its customers want more than anything.

“Our community has told us loud and clear that they needed first and foremost to make sure the model is usable commercially, is trained responsibly and designed to minimise human bias,” Costin says.

“The Firefly model is only trained on Adobe stock data, and this is data we have rights to train on. We don’t train on customer data and we don’t train on data we scrape from the internet. We only train on Adobe stock and public domain data which gives us the confidence and comfort to ensure our customers that we cannot infringe IP.”

It is why Adobe offers its largest [Enterprise] customers indemnification. “We offer them protection from any potential IP infringement.”

Costin also points to the Content Authenticity Initiative (CAI), a cross-industry community of major media and technology companies co-founded by Adobe in 2019 to promote a new kind of tamper-evident metadata. Leica, Nikon, Qualcomm. Microsoft, Pixelstream, SmartFrame, the BBC and even OpenAI are among the 2,500 members.

“We’re getting more and more companies to join the consortium. All the assets that are generated or edited with GenAI in our products are tagged with content credentials. We offer visibility in how content is created so consumers can make good decisions on what to trust on the internet.”

Plus, Content Credentials can be included on export from Adobe Premiere Pro after using Generative Extend.

“We’re also respecting principles of accountability and transparency. In terms of accountability, we have this feedback mechanism in Firefly where we ask for and act on customer feedback. This is what helps us design and improve the guard rails that minimise bias and harm and minimise the potential of defects for our model. We’re proud of the approach we took in building AI responsibly and we know this is a key differentiator that makes our models trusted and usable in real workflows.”

 

NAB Show New York: US broadcasters tackle challenges of a shifting landscape

IBC

Local TV woes, Hurricane Milton and an existential threat to journalism itself hung over NAB Show New York but FAST and fan-fuelled TV offer hope.

article here

NAB Show New York wrapped last week having attracted more than 12,000 attendees, around the same as 2023 and a couple thousand down on its pre-pandemic peak. The two-day event comes at a time when US broadcasters face serious pressures on their business model and an existential threat from a Trump presidency.

Were it not a presidential election year US TV stations would be in even more pain. Political ad spending on TV stations is expected to reach $3.94 billion, up 10% from the last presidential election year in 2020 boosting industry revenue, including gross national/local spot advertising, digital and retransmission fees 8.3% to over $40 billion.

“We believe the industry will become increasingly reliant on political advertising revenue in even years,” said Rose Oberman, Media & Entertainment director, S&P Global Ratings. “At the same time, many local TV broadcasters face refinancing upcoming debt maturities at higher interest rates in 2026 and beyond. As a result, we expect EBITDA will gradually decline and cash flow will weaken.”

S&P Global Ratings have already downgraded several local TV broadcasters, including Cox Media Group, Gray Media, Sinclair and E.W. Scripps giving them negative outlooks.

This matters at a show like NAB which is owned by Gray, CMG, Sinclair, Scripps and 8000 other members of National Association of Broadcasters. At its second event of the year, held on the east coast, the lobby group continued to argue that it’s battle versus streamers like YouTube TV will be lost if the government doesn’t step in.

The NAB is specifically calling on the Federal Communications Commission FCC to regulate OTT programming distributors like for like with traditional pay TV distributors like Charter Communications and Comcast.

The issue is important to station groups, which believe they can increase their retransmission consent revenue if they are able to negotiate directly with YouTube TV, Fubo, Hulu and other virtual multichannel video programming distributors (vMVPDs).

As reported by TV Technology the discussion has divided the broadcast industry between station owners, who want the vMVPDs reclassified, and broadcast networks owned by major media companies, which want to continue to handle deals with online providers as part of much larger distribution pacts.

Currently, station groups handle retrans negotiations with traditional multichannel video programming distributors (MVPDs) like Charter Communications and Comcast. But broadcast networks and major programmers like The Walt Disney Co. and Paramount Global handle retrans deals with virtual MVPDs because the streaming services are not classified as traditional pay TV operators (or MVPDs) under FCC rules.

“It is impossible to understand the demands local stations face without coming to grips with the sea change wrought by streaming,” NAB chief legal officer Rick Kaplan said in a recent letter to the FCC. “The Commission’s rules were designed for a different world, and if the agency is truly committed to ensuring local service to communities across the nation, then it must examine the interplay between local broadcasting, the dramatic rise of streaming and the unregulated Big Tech behemoths that have shattered the economics underpinning local journalism.”

The show also coincided with Hurricane Milton the coverage of which on local TV illustrated its vital importance for serving communities, CBS Stations President Jennifer Mitchell said at the NAB New York conference.

“Local news is so incredibly important and remains important,” she said. “Local journalists know how to cover [Hurricane Milton] better than anyone else. That will not change. It’s just, how will we distribute that information? How will viewers consume that information? It’s important for us to be in all of those different places and serve the consumer.”

While Mitchell was at NAB NY, Donald Trump was demanding that CBS News have its broadcast licence revoked for editing an election special interview with Kamala Harris.

This is not being dismissed as posturing since should Trump win the White House and implement right wing manifesto Project 2025 it would force the FCC into censoring all media companies and their licensed outlets for reporting negatively about the president.

FCC Chairwoman Jessica Rosenworcel released a statement saying that these threats against free speech are serious and should not be ignored. Rosenworcel wrote. “The First Amendment is a cornerstone of our democracy. The FCC does not and will not revoke licenses for broadcast stations simply because a political candidate disagrees with or dislikes content or coverage.”

CTV scale

In the show conference representatives of CBS News, NBCUniversal Local, and E.W. Scripps debated ‘Advanced Streaming and FAST Revenue Strategies for Local TV.’

There was optimism about the ability to leverage data with AI to deliver more targeted content and advertising. They also agreed that scale was needed to reach audiences on multiple platforms.

 Tom Sly, VP, Strategy at Scripps said, “We have some peers that are afraid to promote Connected TV, and they're afraid to talk about it…but we have to wake up and say that business and the audiences are transitioning and we’ve got to change with them and we’ve got to move fast.”

In another session focussing just on Tubi CEO Anjali Sud said the Fox-owned FAST service was exploring fan-driven content development as a unique area of user engagement and growth. “We just launched the first fan-fuelled studio in streaming which is designed to allow anyone to pitch an idea to be green-lit by Tubi and distributed on Tubi based on fan feedback. Fans are going to make the show. We're working with Color Creative to help and them succeed and we will learn about whether that's a model that can work. I'm interested in what are the scalable and efficient ways to bring in unique stories from unique storytellers.”

The AV convergence

With the broadcast market static to shrinking technology vendors are expanding their sales activities into adjacent sectors, notably from the corporate, government, higher education, and finance world. As IBC did this year, NAB is also making its exhibition more attractive to those buyers and said it had hosted executives from Bank of America, Best Buy, Dell, Ford, IBM, JP Morgan Chase, the U.S. Air Force, and U.S Military Academy among core audiences from ABC, CBS, NBC, FOX, HBO and Univision.

“Technologies like virtual production and inexpensive cine-style cameras make content creation much more easily available to people who aren't television broadcasters, but what's driven a real acceleration in the convergence of broadcast with AV is the ability to extend delivery of content directly,” says Ciaran Doran, CEO of cloud broadcast vendor M2A Media.

“Brands have woken up to the fact that they have been building a one-to-one direct relationship with their customers and that the next step is to communicate by live video and build an even tighter engagement with them.”

 

The Instigators: Henry Braham BSC on reworking the rules of how to shoot a blockbuster

RedShark News

Henry Braham BSC talks about shooting Apple TV’s movie The Instigators on RED in a truncated timeframe of a little over a month.

article here

Laid-back comedy caper The Instigators succeeds on its own terms in feeling unencumbered by the weight of a big studio movie despite the A-list charms of Matt Damon, Casey Affleck (who also co-wrote the script) and Hong Chau, a number of weighty character actors (Ving Rhames, Ron Perlman, Toby Jones) and being bankrolled by Apple.

Director Doug Liman is a master at flipping a paper thin plot into slickly entertaining escapade (see Mr. & Mrs. Smith, American Made) and has shot his last two movies in concert with cinematographer Henry Braham BSC.

Braham is renowned for his work in developing character driven performances stretching back to indie fayre like Stefan Schwartz’ Soft Top Hard Shoulder (1992) through Nanny McPhee (2005) to  studio blockbusters The Golden Compass (2007) and Guardians of the Galaxy Vol. 2 (2017).

“It sounds really obvious but Road House was about the heat and the water. Set in the Keys, shot in the Dominican Republic. It's a cowboy movie, an action movie, it’s all about the fights,” says Braham of the Jake Gyllenhaal feature directed by Liman which released earlier this year on Amazon Prime.

“The Instigators is the exact opposite. It's cold. It’s Boston. It’s a buddy movie about hapless guys for whom everything goes wrong. The common thing between them is Doug's approach to filmmaking.”

As part of his prep Braham scouted locations in Boston and New York. “[The production] had great collaboration from the city. We were allowed to run amok in Boston City Hall [the modern concrete behemoth that dominates downtown] and across the streets. It was important that we shot during winter time so that there was a sort of a greyness to it and nothing looked too pretty.”

That fits with the blue-collar post-industrial milieu of the film which to a degree recalls Michael Mann’s Chicago set heist movie Thief (1981) where the hero (a career best performance from James Caan) believes that stealing from the rich equates to social justice, provided no-one gets harmed.

Affleck’s screenplay for The Instigators also has downtrodden heroes sticking it ‘to the man’ – in this case corrupt politicians. Braham leans into the cloudy skies and gives the film a grey-blue hue.

“Obviously the story is fantastical but if you can ground it more in reality then you buy into the story better,” he says. “Although it's ridiculous there are all kinds of truths that come through the movie. So, photographically it's important that we ground it.”

He continues, “It wasn't important to me to show off the Boston locations. It was about telling the story of these two guys and the chaos around them.”

The interior of the mayor’s office was built at Broadway Stages in Brooklyn with a layout based on the real thing, duplicating Boston City Hall’s exposed concrete structure and high ceilings.

Braham’s camera is hardly ever static and often frames close-up to the actors and very close indeed with shots inside vehicles during the car chase [one of the most extensive ever filmed in downtown Boston which also filmed inside the I-93 expressway tunnel]. The cinematographer says his aim was to keep connected to the performances, not to over plan shots and to be intuitive to the moment.

“Hopefully the audience is not aware of the camera at all but just feels very connected to the story. A classic way of doing that is for the camera to stand back and be on longer lenses and to observe what's going on. There are brilliant examples of that in Doug's early movies, especially in his first Bourne picture (The Bourne Identity 2002).” 

Another example is In The Mood For Love (shot by Christopher Doyle for Wong Kar Wai) which Braham says is “beautifully observed and not shot conventionally at all.”

“Those are two ways of doing it but there's a new way of doing it which technology has recently allowed us to achieve. It's where we can shoot on large format cameras which are physically small and which enable you to get the camera close in and amongst the actors. 

“Because the negative format is large you can use wider lenses that don't look like wider lenses on screen. In every filmmaker's dream there's a thing called a ‘tight wide’ shot which is where you want to see everything but at the same time you want to be close and connected to what's going on. 

“In the old days we used to have a wide shot or a tight shot. But now, thanks to the advent of large format digital cameras, we can have our cake and eat it, which is a tight wide shot.

Braham has been developing this technique and related philosophy over a number of films with James Gunn including Suicide Squad (2021), and Guardians of the Galaxy Vol. 3 (2023) and The Flash for director Andy Muschietti.  All are in the Marvel or DC Universe and foreground character over VFX.

“It means you can be entirely intuitive with a handheld camera,” says Braham who likes to operate himself. “You can be very precise about the relationship of the camera to the actors. The actors aren't bound by marks on the floor. I can always keep the camera in the right relationship to them and they will always be in the right relationship to the other actors and the set.”

This approach frees actors, directors and camera to perform and respond more freely on set. “It's a really fun way of making a movie. It’s not consciously artful. That’s hard to explain but I think when you shoot something conventionally there's an artfulness to it which audiences are aware of. For lots of movies that's entirely appropriate. But in a movie like this, it’s not.”

He cites the 1971 classic The French Connection shot by Owen Roizman as revolutionary in breaking many of the established rules.

“They didn’t use wide, mid-shot to close-up. Their response to the story on location was intuitive and that was enabled by being able to use small cameras rolling a single strip of film and cameras which were among the first to be single-lens reflex – the operator could look through the lens and see exactly what the camera was seeing. Plus, it was one of the first films to take advantage of the  zoom lens. I believe where we are now with digital cine technology is the next genuine progression since that era of filmmaking.”

Most of the enjoyment in watching The Instigators comes from the repartee of the three leads. Their comic timing is often captured in camera rather than feeling manufactured in the edit and is testimony to Braham and Liman’s approach.

“Doug encouraged improvisation and because we have a shooting style that can evolve with the scene we can capture moments that feel spontaneous. That’s a lot harder when there's a team of people directly behind the camera worried about getting into territory that hasn't been discussed.

“In this case, most of the film is shot single camera and it is myself and my demon focus puller (A camera assistant Dermot Hickey) who has to react to what I’m doing which he often won’t know in advance. 

“Doug has got this antenna for things that don't feel truthful. If it's not working in one way, he'll try something completely different and that could just literally turn the whole scene on its head. To have the flexibility to do that is important for a filmmaker like that. We always ended up with something significantly better.” 

Shooting RED

The look of Road House may be entirely different but Braham used the same camera package of RED V-Raptor with Leitz M 0.8 glass. He’s been working with RED on all of his recent films and says he changes up as new sensors or features are released to always work with the latest model. 

“In camera technology there's been a steady progression of improvements but I've kept the same fundamental principles for every film I’ve shot over last four to five years.

“When RED developed their first VistaVision camera it was revolutionary because the moment you have the larger negative the lenses you can use change completely. When you get used to shooting VistaVision and then go back to 35mm it’s like going backwards to 16mm. The profound step forward in movie making technique is the move to large format. 

“Secondly, it’s putting that sensor in a camera the size of a Hasselblad. RED really nailed this. The V-Raptor is so light it enables you to do all sorts of things that are intimate or to put the camera in places which wouldn’t fit a larger one.

“In the days of film the cinematographer would take time to get to understand a new film stock when it came out. You had to get to know the material. It’s a bit like an artist testing the canvas or wood or texture on which they will paint. Your raw materials react differently. It's the same with photography. Each time something changes from my point of view, I really have to understand what that is. Once I’ve understood the tools then goes into the background. I’ll just know the photographic range I’m working with and what a particular camera is good at.”

In this endeavour Braham has been aided by the bespoke designs of David Freeth co-developer of the stabiliser system Stabileye.  “It’s incredibly expressive,” Braham says of the remotely-operated, miniature stabilised head. “It kind of allows the camera to dance with whatever's going on in front of it.” 

He has just shot Superman for James Gunn and says 90 per cent of the film was shot in this fashion.   “A few years ago we used to think a big, big movie like this had to be approached in a certain way but I don't think you need to now. Directors change their minds when they discover the freedom they get from this approach.”

The tight 36-day shoot would arguably not have been possible without the Braham-Freeth proprietary system that can emulate a dolly shot, a crane shot, a handheld shot, all with the same camera system. It meant that with a pre-lit the set designed for naturalism Liman could point the camera anywhere without need to wait between set-ups.

Big or small screen?

There has been some reported disagreement between Liman and Amazon over theatrical release of Road House. Liman seemed to suggest that an agreement by the streamer to give the film a cinema run was reneged on. The Instigators has also gone straight to AppleTV+ but the filmmaker seems to have known this in advance. 

“The way we shot Road House was intended for the big screen,” Braham says. “It was a very immersive kind of roller coaster ride. The fight scenes are specifically designed for IMAX and we colour timed those on a big screen so the idea is that the audience is literally part of the fight. 

“My approach is always to assume everything's going on a big screen. Because we shoot in a way I’ve described using the tight wide shot, it works whichever size screen. I’ll often see movies in the cinema but I’ll go back and watch them on TV over the years simply because I enjoy them. A good film well told will stand up to being viewed repeatedly.”

He worked with regular colourist Stefan Sonnenfeld of Company 3 to grade the film. “My process now is to shoot material during prep and to build a colour bible of how the movie is going to look. You can talk about these things and look at bits of reference but I found it most helpful to shoot some material and build the look so that everybody from design to costume can work toward the same goal. It’s like designing your own print stock. I love that it's like starting with a blank sheet of paper. Some people don't like that because they want some rules to work with. I'm more interested in there being no rules.”

 

Brian Sherwin: Grand master plays postproduction chess to manage episodic workflows

interview and copy written for Sohonet

article here

Take a selection of hit drama series over the past couple of decades – Salem, Detroit 1-8-7, Pachinko, Ugly Betty, The Haunting of Hill House, Panhandle, Yellowstone. Different studio, network or streamer (Warner Bros, Disney, Paramount, Sony Pictures, Netflix, TNT, AppleTV+, Roku). Different showrunner, director and genre. What they share in common is the post-production expertise of Brian Sherwin guiding them from start to finish as production manager, post coordinator or producer.

Hi Brian. Thanks for taking the time. Can you give us a brief resume about how you broke into the industry?

I moved out to LA right after graduating college and worked at E! Entertainment Television to kind of get my feet wet. I realized after a point that I really wanted to work in scripted television and that this was my passion. I left there as an associate producer and took a step down to be a post PA in scripted television because that's what I really wanted to do. Over the years, I've just worked my way up from post PA to post producer where I run the department on various shows. I'm hired by the show and work on that until completion. One of the most enjoyable aspects of my job is that every project, every team and every workflow is slightly different.

Presumably, the team that you work is dependent on the scope of the project. The post team on a show could just be myself, a s post supervisor with and a post production assistant, or if it's a bigger project, you may have a post supervisor, a post coordinator and post PA. It all depends on the size of the project and budget. The editorial team will vary in size accordingly.

‍How have you seen that working relationship change as more of it can now be managed in a distributed fashion?

Generally speaking, the process itself has remained the same, but technology has evolved and continues to change the location we are able to do it. In a situation like the pandemic, where everyone was forced to work remotely, you're forced to rapidly adopt remote technologies while using the same general process to move the show forward.

If you had a color correction session, for example, or a session where you're dropping in visual effects and you couldn’t actually go to the post facility, we would use Clearview Flex, the video collaboration software to be able to supervise that from home. The online editor might be working with the media in the post facility but by using Clearview Flex and the client is able to view the same screen online while being able to talk back and forth to approve shots in real time.

‍When you architect a workflow for a new show now how would you gauge the ratio of remote versus in-person?

It's kind of half and half. The starting point is dictated by whether the executive producer or showrunner really wants to be in the office or not. That’s the first thing you're trying to figure out.If the showrunner prefers to work in the office, then you find the space. If they say, they want to do some type of a hybrid situation, that's what we’ll do. Or it can be fully remote.

A lot of their decision making depends on where you’re shooting and where you are posting. If you're finishing final color correction and sound in another geographic location to the principal photography and you need to complete those while future episodes are being shot then you’ll need a very flexible workflow. For example, on Panhandle the crime comedy series for Sony Pictures, all of the finishing was completed in Vancouver in order to access tax breaks for that show. Editorial was managed here in LA. So, the workflow was completely remote.

For AppleTV+ drama Pachinko I was in Los Angeles and the post team was in New York. We had four editors on the show, three in New York and one in London. The executive producer was in New York where I worked for a short period of time.  I was flying back and forth for the final mixes.

What’s the biggest benefit of working with Clearview?

A hybrid workflow facilitated by Clearview has become a positive choice. When we do the final audio mix, if a person can’t physically be there in the room, they are able to fully participate using Clearview Flex. All the producers, studio executives and network executives are able to hear audio playback and give notes in real time. The biggest benefit is the ability to work in real time.

If we didn't have that option, you’d have to have the sound facility send the editor or assistant editor a file, which they would review and give notes and then pass those notes back to the facility. That to and from can prolong the process because you're adding a lot of extra steps.  Then adding a producer or executive would be another link in the chain.

‍How are you seeing AI assist in the post-production workflow?

Technologies are constantly evolving but the bones of the process remain the same. For example, VFX might use AI to track a shot frame by frame a little quicker. But as far as my role in managing the process and making sure we get from A to B to C to D, AI has not materially impacted that workflow as yet.

‍Scripted TV is suffering on a number of fronts. From your position on the inside what do you think the future holds?

I think you can take a positive or negative view on this depending on how you feel about it. A few years ago, it felt like the Golden Age of Television with maybe 600 shows being made a year. Following Covid and the strikes everything's contracted. Studios are now owned by large corporations. The content creation business of the major studios is a much smaller portion of the overall business of a corporation. If the whole corporation doesn't do as well then the entertainment division tends to suffer. That said, I detect things are slowly starting to pick up again. Everyone is trying to figure out what that new normal is going to be.

‍Can you give us a glimpse as to what it takes to post produce a major episodic show?

In pre-production I’m working very closely with the director, the director of photography, the producers and the studio or network to figure out technical specs such as what camera is going to be used and what the pipeline will be to get that material from location to editorial. You have the responsibility to make sure that it all runs smoothly and to budget. The footage also needs to be made available for the studio and network to view. Everyone needs to see it. So, there's a lot of pressure to get the footage pipe up very quickly. After all, there's a lot of money involved in these projects.

That's the beginning of the post production progresses. In addition, you’re working with the editors and assistant editors towards finishing and delivery. At the same time as they’re editing, you're shooting more shows while planning many moves ahead. It’s like chess. You're working with production to figure out what's going to be needed for the next episode being shot. For instance, are we going to shoot things practically or in use VFX? There's a lot happening in post during production - and that's before we even lock picture which is when our job really kicks into high gear!

‍Of all the successful shows you’ve been involved in what is your personal favorite?

I can't pick one stand out because each part of the process is different. For example, I may have worked with a cast in ADR which was phenomenal. There may be editors on a show with whom you have a particularly great experience. Other times, it’s the entire post team. It could be a show where like I love working with a specific colorist, visual effects house or sound facility.

Pachinko was an extremely difficult project but it was incredible just for the fact that a small percentage of it was in the English language and the rest was in Japanese and Korean. It was such a rewarding challenge. There's so many intricacies that make each project you're dealing with unique and memorable, in addition to the varied subject matter of the show.

 

Modelling intent: How evolution of AI will turbocharge creative capabilities

IBC

Framestore’s CTO Lincoln Wallen says the creative industries should acknowledge the art of math to craft new interactive AI-driven social experiences.

article here

The future of media and entertainment will be interactive, social, and fuelled by AI with companies versed in blending tech and art at the forefront of development, according to Framestore’s Chief Technology Officer (CTO), Lincoln Wallen.

“It’s going to be a slow burn but the technological basis of it has already been built,” he says. “Imagine experiences in which consumers can interact but are nevertheless driven by narrative visual storytelling. It’s a mixture of active lean forward and passive lean back. That is a media type that I expect to explode over this next period.”

Wallen joined Framestore and Company 3 Group (FC3) last summer from Improbable, a tech developer building platforms for multiplayer games and battlefield simulations which was valued at $3bn in 2022. He previously spent four years at Electronic Arts building its mobile games business before entering the film industry at DreamWorks, initially as Head of R&D from 2008, and then promoted to CTO in 2012.

That career journey has given Wallen an orbital perspective of future media in which technology and content, both IRL location-based and virtual, merge to create new forms of real-time communication and storytelling. 

“If you want to spot the metaverse as it emerges it’s going to come in interactive experiences in which large numbers of consumers come together,” he explains. “We do this today in social environments when out shopping, at sporting events, on the street, at festivals and conferences, but the digital world doesn’t properly support that sense of social engagement.

“The only markets in which people really come together digitally are video conferencing and video games but those experiences are designed for very specific numbers of players, not for crowds. Being able to interact with people meaningfully in a digital environment changes the game.”

Technology that facilitates simultaneous multi-player interactions at scale will underlie future media says Wallen, who believes this has been cracked by Improbable (of which he remains a Non-Executive Director). The next step is to evolve the user experience.

“VR really hasn’t left the laboratory in terms of the consumer experience. It still isolates the user. But the metaverse will come back as all truly disruptive ideas do,” he says.

An example of what he means is the work Improbable has done with Major League Baseball (MLB). Here baseball fans can gather in an MLB virtual ballpark and interact simultaneously in a single place, at a naturalistic scale. It claims to be the first service to host such large interactive sports experiences.

Another example includes fashion brands that might organise a virtual runway show inviting people to the physical catwalk and thousands more people to virtually inhabit the same space. Wallen believes this type of experience will eventually happen daily as physical interfaces like mixed reality goggles improve, but that the content itself won’t work unless crafted with storytelling expertise.

“The audiovisual world of linear media has built up an enormous bank of techniques creating adverts, TV shows or movies to grab a consumers’ attention and to stimulate emotions but those techniques are not used today within interactive environments,” he insists.

“Those working in linear media will call it ‘immersive’ as a way of trying to signal that this is something you’ve seen in VR experiences or in location-based entertainment. What we haven’t yet seen is that experience realised in a fully digital environment. It’s neither gaming nor linear but something in between. The potential of this middle ground is enormous.”

“The linearised model of film production is going to collapse into a much more iterative holistic model” Lincoln Wallen, Framestore

If Hollywood studios are too entrenched in their own structures to pivot fast enough, companies that are operating across market sectors will be able to synthesise these capabilities and exploit them. Framestore is one. Its work ranges from the VFX on Gladiator II to designing an interactive exhibit for the Science Museum.

“That isn’t to say that Framestore is suddenly going to become a content company,” Wallen stresses. “It means identifying those brands and IPs that are going to pioneer this new media and serve their existing customers in a different way.”

Framestore launched in 1986 at the birth of the computer graphics revolution. It remains headquartered in Soho where it has navigated and led the industry-wide transformation of post-production from an afterthought into an integral partner spanning the creative industries.

“The linearised model of film production is going to collapse into a much more iterative holistic model,” Wallen says. “Right now, the previz phase and the production phase feel separate. I think AI is going to bring the back end right up to the front entrance and make certain complex processes real-time.”

Bridging the uncanny valley

For Wallen, the key to the future of interactive media is the ability to create digital representations that talk, behave, respond and ‘think’ like a human. Such avatars could be our own online personas or the embodiment of a corporation. AI is already supercharging such developments in video games where AI-driven non-player characters (NPCs) are being introduced to interact in a more lifelike manner with game players.

“You are going to see NPCs having existential dialogue with game players,” Wallen says. “This is one of the most exciting things to me. Large language models (LLMs) have exposed a degree of structure in natural language that makes dialogue possible at a level that it hasn’t been in the past.

“When people look back on this era it is going to be the impact on language that will be seen as the most profound. Natural language will become a much more effective interface to [digital] systems and this will make them more accessible, which will be an enormous economic benefit.”

There is a problem though. The increasing sophistication of LLMs must be matched by sophistication in visual behaviour, emotion and characterisation. The uncanny valley is yet to be bridged.

“We don’t have a language of performance. There’s nothing to correlate spoken language and non-verbal gestures to make your NPC seem alive. Animators study human behaviour and know exactly how to produce that effect but we don’t have tools that do that automatically. You can’t take the animator out of the equation because they understand what human gestures mean, but the way they’re created right now is painstakingly frame by frame. We have to build that capability.”

Wallen believes that the creative industries should acknowledge the art of math to craft new interactive AI-driven social experiences.

The EU-funded research project PRESENT (Photoreal REaltime Sentient ENTity) of which Framestore is a partner aims to create virtual sentient agents that look and behave entirely naturalistically. The aims are to demonstrate emotional sensitivity, establish an engaging dialogue, add sense to the experience, and act as trustworthy guardians and guides in the interfaces for AR, VR, and more traditional forms of media.

“It’s only the high-end media industries that actually know how to produce digital performances, that that make people feel that NPCs are alive,” Wallen says.

Understanding how the world works

“There’s a very ambivalent relationship in the humanities between systematisation and what they think the humanities are all about, which is inspiration” Lincoln Wallen, Framestore

Wallen has a PhD in artificial intelligence from the University of Edinburgh, and subsequently taught at the Oxford University Computing Laboratory before moving into the media industry. He is also a non-exec board member of Finnish VR/MR headset maker Varjo, Pearson - the FTSE100 global education company, an advisor to bp and a 25-year member of the maths and AI strategic consultancy Smith Institute. Few people understand the intersection of science with the humanities as well as he and that means his view on AI as an artistic tool is not binary.

“All through my career the driver has been the interplay between modelling the world and activating it through computational systems,” he says. “Understanding how the world works is a passion.”

What strikes him about the current AI wave is the tension it has caused between inspiration and predictive calculation, but it’s a tension that is not new, and the two are not mutually exclusive.

“Making movies is a balance between algorithmic processes and artistic choice. That’s a very natural combination that AI is forcing to be recognised.”

He explains that neural networks, a mathematical construct designed to mimic human thought, has reached a level of maturity that demonstrates structure in natural language.

 

Stevie (Alba Baptista) will star as the protagonist of FLITE, Framestore’s experimental short film set in a futuristic London of 2053

“We now understand that there’s an awful lot of structure in writing. It’s not just a grammatical structure. Literary critics have argued this for centuries, but the humanities oscillate between seeing these patterns and then running away from them because it seems to be contradictory to creativity and choice. There’s a very ambivalent relationship in the humanities between systematisation and what they think the humanities are all about, which is inspiration.”

He argues that there is an art to mechanisation: designing-in specific outcomes that preserve human creative agency, with plenty of scope for machine learning technology.

“LLMs don’t manipulate intent,” he insists. “Humans create intent. They communicate and collaborate to organise their efforts to produce fantastic outputs, whether that’s a video game or a film. Without human dialogue, without the imagination that’s really framing the outcomes, you don’t get the products.”

We can use advanced AI techniques to try to realise that imagination, he says.

“Computational techniques are forcing recognition that there are structures, including within narrative visual storytelling.”

Language of intent

Wallen thinks the industry needs a standardised way for artists and filmmakers to communicate with each other in the age of AI, using AI tools. Current-generation AI tools, he says, are not intuitive and actually hinder artistic collaboration.

“GenAI tools work at a very low level. They don’t help you express what you’re trying to do. The only way to get the machine to respond is to give it instructions bit by bit of what you want to see but not why you want to see it.

“What we want is a language of intent,” he argues. “We want what the artist is trying to achieve made more directly expressible and reflected by the interfaces of these tools. If we had these languages we would be able to build technologies that really turbocharged people’s creative capabilities. That I think is what the end result of this AI wave will be but only if the creative industries start looking at their systemisation techniques and their content [Wallen uses the term folklore) and turning it into activatable language.”

This is something that Wallen is exploring at Framestore but it’s too vast a challenge for one company alone.

“I’d hesitate to say that this is a century of development but it’s also not easy. What I’m trying to do is stimulate academics and creatives to think about the problem. Even small areas of advancement in this area can lead to enormous benefits to the creative process. It is one of my personal motivations for joining Framestore.

“Whether it’s your digital proxy, virtual playmates, NPCs in games, a retail concierge, or part of the crowd in a virtual stadium doesn’t matter. There are so many applications of LLMs that are going to explode when coupled with character.”

Wallen completed a thesis on Mechanisation in Mathematical Art during the 1980s and suggests that the challenge is as old as the Scientific Revolution itself. He refers to early computers which worked by punching sequences into cards.

“By attending to the design of, and access to, the tools with which we ‘punch the cards’ today we can preserve the poets and even pursue ‘poetical science,’” he says.

 

Glenn Garland ACE and Joel Salazar use ClearView Flex to cut indie feature for director Lorena Villarreal

interview and copy written for Sohonet

article here

With credits that include Rob Zombie's Halloween and King of California, the series Preacher, The Vampire Diaries, and Rian Johnson's Poker Face among many others, Glenn Garland ACE is an established Hollywood editor who has also achieved wider renowned for his podcast ‘Editors on Editing’ for American Cinema Editors.

Garland’s longtime assistant and now editor in his own right is Joel Salazar (Red Camaro). Together they had worked on projects including Silencio (2018) the sophomore feature of Mexican filmmaker Lorena Villarreal. Their current project is Villarreal’s follow-up, the Spanish language drama with the working title La Vida Es, which Salazar is editing and Garland is executive producing.

It also marks the first time that have worked with ClearView Flex, our video collaboration software. “We're on it all day. It's very robust, and frankly I’m not sure we could have made this movie without it,” says Salazar. “And believe me I have tried all the alternatives.”

‍Using ClearView Flex for Remote Editing

Salazar was on location in Baja California, Mexico for the shoot. Among other duties he helped get feedback on a number of single shot ‘oners’ filmed for the drama. Now back at his edit bay in Los Angeles he is working on the director’s cut while Villarreal is remote.

“Lorena is often busy in another state or in Mexico which is where ClearView comes into its own,” he says “We connect over Google Meet to talk while using Sohonet to make cuts on the fly. She loves it too and because it’s browser-based she will use ClearView tools to annotate on screen.”

‍Challenges of Remote Collaboration and How ClearView Flex Solved Them

“I remember a time when everyone said that editing could never be done remotely. Then the pandemic happened and I feel like it sort of forced everyone to solutions. I worked on shows in the early parts of the pandemic where we used the studio’s corporate Zoom which had a lot of delay in the stream. That was very frustrating. Other solutions require third party software and can be very clunky but Sohonet has really streamlined the whole workflow.

”Garland adds, “We definitely needed a remote editing tool for this particular project because the director was not going to be able to sit with Joel in the room. That was one issue and another was that Lorena is in many different locations, not always with the best internet connection. So, we needed a system that was powerful enough to work no matter where she was.

“You know what? ClearView has been a dream. There is no latency, which is obviously fantastic. Without dropouts, without stuttering or any of the other irritations that you get with some of the other systems.”

Salazar’s Use of AI for Voiceover Narration in the Editing Process

Intriguingly, Salazar has used the AI text-to-speech tool from ElevenLabs to generate a voiceover narration as a placeholder before ADR.

“What’s great is that I can switch from the Avid to the AI app and show Lorena the browser of the AI voice generator on ClearView,” he says. “The AI is just a template,and it will take five or six or seven times for it to kick out one that she likes but because we’re using ClearView she is able to tell me what she likes or doesn’t like inthe voiceover and I can tweak the AI and in seconds we have a refined version.”

Garland and Salazar’s Ongoing Partnership with Director Lorena Villarreal

Garland explains how the project came about. “It was such a great experience working with Lorena on Silencio. She is really talented. We kept in touch and when this project came up, I thought it was such a beautiful script that I wanted to be a part of it and help her make it.

“Joel is very talented in his own right and I thought this would be a perfect film for him to cut. I’m supervising editor as well as executive producer so I’ve been watching dailies and giving notes on the cuts that Lorena and Joel have done.

“Given our experience on this project, I’d definitely use ClearView again. It has worked brilliantly. I feel like it's an extremely useful tool for any time you're doing remote editing because it's so robust. It just works.”

La Vida Es is being prepped for screenings at next year’s festival circuit.

 

The bright future for floating solar tech

IEC

The potential for floating solar photovoltaic panels and farms is tremendous, despite initial deployment costs. The IEC is preparing the standards for it to be used safely and efficiently – and in all weather conditions.

article here

Floating solar photovoltaic (FPV) panels – sometimes called floatovoltaics – is a relatively new renewable energy option, but one with huge potential. According to the IEA, in 2023, solar PV alone accounted for three-quarters of renewable capacity additions worldwide. Yet most of the solar panels installed so far lie on land, which pose crucial issues for land use particularly in countries where land is at a premium. This includes island states, for instance, but also countries with high population density where land is a constraint.

It’s one of the reasons why the deployment of solar technologies which float on water are predicted to increase rapidly over the next few years. “With 70% of the world covered with water, research and development of FPV on ocean platforms opens a new era of solar energy with the advancement of robust floating structures,” an international research team states in a review of the field published in April.

The scientists conclude that FPV systems outperform land-based solar PV systems under similar conditions but warn that offshore weather and harsh ocean currents may pose serious challenges to FPV structures. “Therefore, research and development efforts addressing this issue are crucial,” they say.

Shading the water

FPV modules are solar PV panels mounted on raft-like structures that float on a body of water such as drinking water reservoirs, quarry lakes, irrigation canals, hydropower or agriculture reservoirs, industrial ponds and near-coastal areas.

While one of the obvious advantages of floating solar PV tech is that it avoids taking up land space. It can also allow for power generation to be sited much closer to areas where demand for electricity is high. This makes the technology an attractive option for countries with high population density and competing uses for available land.

Floating sun-powered farms also solve another problem plaguing conventional land-based solar PV modules: inefficiency when the panels become too hot. FPV panels generate extra energy because of the cooling effect of the water they hover over. The proximity to water of floating solar modules is estimated to increase their electricity production by up to 15%.

Stationary floating panels also double as shades for the water body, which reduces the evaporation of water. This is an added advantage in areas where water is becoming scarcer. (For more information on the preservation of water read: when cities run dry: tackling water scarcity).

Asia leads the way

The technology is particularly well suited to countries in Asia where land is scarce but there are many hydroelectric dams connected to the electricity grid. The world’s first floating solar plant was built in Japan. The country’s inland lakes and reservoirs are now home to 73 of the world's 100 largest floating solar plants. A floating solar plant in East China generates almost 78 000 megawatts (MW), enough to power 21 000 homes. A South Korean plant delivers 102,5 MW, capable of powering 35 000 homes. Island state Singapore has constructed a solar farm with 13 000 FPV panels in the Strait of Johor with the ability to produce up to 5 MW – sufficient energy to power 1 400 residential flats all year-round. A project at Sirindhorn Dam in Thailand is estimated to help reduce carbon emissions by 0,546 tons per 1 000 kilowatt per hour (kWh) produced. 

Covering just 10% of all man-made reservoirs in the world with FPV panels would result in an installed capacity of 20 Terawatts (TW) – 20 times more than the entire global solar PV capacity today, according to an analysis by the Solar Energy Research Institute of Singapore (Seris).

Reliability and financial concerns

As it stands however, less than 1% of the world's solar installations are floating, according to the Centre for Renewable Energy Systems Technology at the UK’s Loughborough University. This is partly due to technical and financial constraints. Saltwater causes corrosion while positioning panels at the right angle is tricky and expensive on a floating platform.

Although panels clip together and are then pushed out onto the water, they require an anchoring system, which helps to keep the pontoon stable. The deeper the body of water, the higher the cost of the anchor. Also influencing deployment costs are water level variations, characteristics of soil/bedrock and the type of floats used to support the PV modules.

For example, the reported anchoring costs for the Anhui project in China are relatively low at around 10 USD/kW. It is in shallow water and has benefited from the local manufacturing facilities and labour force. But for a similar scheme in Japan the anchoring price is substantially higher.

Researchers also point to the lack of supporting policies and development roadmaps by governments that could hinder the technology’s growth and viability. Installations of floating modules on freshwater bodies may also face opposition if they compete with other leisure activities, such as angling. There are further concerns that large-scale plants may harm marine ecosystems by blocking sunlight. The risk of disruption or even destruction due to volatile weather is also discouraging investment. A typhoon damaged Japan’s largest FPV plant in 2019, for instance.

Where standards can help

As is the case with other emerging technologies, standards can help to bring the cost of FPV down. They can also set benchmarks for the construction of solar PV plants, ensuring they are able to withstand severe weather conditions and do not pose problems for the environment. The IEC is working on a new technical specification (TS) due to be published at the end of 2024, which establishes the guidelines and recommendations for the design of FPV plants. The plan is to later expand the TS into a full standard. Issues addressed include how to implement electrical earthing in an installation over water and carry out insulation resistance measurements, how to implement proper mooring and anchoring of the modules, selecting the suitable cables and connectors, as well as cable routing and management, location of inverters and transformers (e.g. over water or on land) – right down to issues such as the cleaning of bird droppings!

Standards are required because of the technology challenges that need to be addressed but also because basic estimates of the potential for floating solar are overwhelming. One study found that solar panels floating on just 1% of Africa's hydropower reservoirs could double the continent's hydropower capacity and increase electricity generation from dams by 58%. This is precisely because FPV panels stop water from evaporating. According to nature.com, if 10% of the world’s hydropower reservoirs were covered with FPV it could generate as much electricity as all the world’s operating fossil fuel power plants combined. The market is expected to grow 43% a year over the next ten years, reaching USD 24,5 bn (Euro 22,2 bn) by 2031. Standards will enable the tech to truly prosper alongside other renewable energies as we aim to meet our net zero targets.