Grading and Audio Post in the Cloud

IBC

https://www.ibc.org/trends/grading-and-audio-post-in-the-cloud/6121.article

As workflows from editing to VFX gradually shift online, two aspects of the finishing process remain outliers. Audio mixing and colour grading will be the last parts of post to shift entirely to the cloud – if at all – because of the highly exacting nature of the work. 

“The movement to cloud is not a fad,” says Brandon Heaslip of dailies and mastering software developer Colorfront. “A lot of steps need to be worked out and production still needs to understand more, but it is happening.” 

As it stands today, grading tools like DaVinci Resolve can share projects, media, camera metadata and raw debayer data, project and colour space preferences—pretty much any part of a grade— between editors and colourists, making remote grading a possibility. 

The links between colourists and their teams could be as simple as exporting a project, with supporting Colour Decision Lists, and using the software’s media management tools to share and distribute files to an attached cloud service ready for download and remote work.  

It could be more collaborative via a VPN, hosting multiple users within a project by utilising the grading tool’s software to map and manage project databases to users. At the timeline level, features like colour trace (in Resolve) allow colourists to copy grades from timelines to keep them up to date, and lightbox will review changes and advise on ungraded clips to keep the project moving along.  

There’s a reason that post-facilities remain an important aspect of production. With QC and approvals, they have the required playback and screening setups to show a project as intended and the in-house staff to ensure calibration and colour accuracy.  

“It isn’t difficult to hand off a grade to a colourist on the other side of the Atlantic, and then receive a completed project of beautifully graded images as long as the correct preferences for colour space and colour management are followed at both ends and reference monitors are consistent. Small inconsistencies or changes can have knock on effects,” says Craig Heffernan, Blackmagic Design’s EMEA technical sales director. 

“Whilst network services, cloud storage and online collaborative tools allow teams and clients to work remotely, there is the need to have correctly calibrated and accurate monitoring to truly recreate a grade as the colourist has designed it or to avoid issues in error checking or sign off - especially with HDR formats.” 

That said, it could depend on the final destination for the content. If the production is bound for an online service like YouTube or Vimeo, for a corporate or commercial client, then it could be appropriate to review on that exact platform; as the intended audience would see it.  

“Final colour pass on a great many kinds of content is absolutely possible from a work-from-home situation by using a variety of different working/viewing platforms,” says Fergus McCall, head of colour, The Mill NY. 

These range from point-to-point locked bandwidth real-time streaming with adjusted monitoring at both ends, to using third party (but secure) cloud-based solutions with clients making judgements looking at the content on a browser on their phone.  

“I wouldn’t recommend the latter,” says McCall, “but I’ve also graded sessions where creatives who are sitting in the room with me looking at a $40K professionally calibrated display have elected to make their final decisions looking at a QuickTime on their phone.” 

What it is categorically not possible to do, he insists, is to make final decisions on a long form feature project that first and foremost is being colour corrected for theatrical release.  

“The same also applies to HDR, Wide Color Gamut and some extended resolution deliverables. For these, a phone or tablet isn’t going to cut it.” 

The monitoring gap is closing. Heffernan points to development of affordable but very high-quality OLED TVs with HDR format support up to Dolby Vision standard. “These can be a suitable alternative to professional reference monitors with full DCI P3 colour accuracy, depending on the intended release platforms,” he says. 

Remote collaboration Replicating face to face interaction between client (DoP) and colorist especially during the fine tuning of a grade requires synchronous sub-frame accurate streams. 

Collaboration tools like Sohonet’s ClearView Flex, Evercast or Streambox Chroma can be used to invite colleagues to a live stream session. You could perform a live edit or review and approval in real-time with over the shoulder instruction like ‘back up two frames’, ‘cut this’, ‘tweak that’, as near as you would in a suite. 

Again, the catch is that ‘critical review’ quality output is not yet possible from the cloud.  

Sohonet says it is working to increase the colour depth (10-bit), chroma (4:2:2) and colour space (Rec 2020 and HDR) “to provide an acceptable ‘critical review’ alternative” that can be delivered over high capacity home internet speeds.  Streambox has developed a virtual encoder – Spectra – claimed capable of colour-accurate remote review and collaboration sessions over the cloud with virtual Avid Media Composer or Adobe Premier machines. Future iterations of Spectra are in the works to support the same capability for virtual versions of DaVinci Resolve, Autodesk Flame and FilmLight’s Baselight. 

Blackmagic has integrated Resolve Studio 16 with Frame.io to enable feedback and approvals. Heffernan explains that by signing into a Frame.io account, Resolve can render and export directly from the deliver page to Frame.io where others can review shots or full sequences, checking edits, grades, or sound design, and adding comments or suggestions back to the creative team.  

“Frame.io markers are then automatically updated in Resolve allowing clients to directly advise on changes down to the frame, and rapidly speeding up approvals and decisions between teams and clients,” Heffernan says. 

Non-verbal cues Light Iron, the U.S. finishing facility which services a number of feature film and commercials cinematographers, managed to set up its senior colourists from home over the last few months with systems good enough to finish a 4K series with HDR and SDR deliverables and a feature film. 

Its colourists set up physical BaseLight and Resolve panels at home and accessing the high-res media residing on premises at Light Iron’s East and West Coast facilities.  

The biggest hurdle was not so much technical but not being able to pick up on the kind of non-verbal cues you get while sitting in a room with someone. They couldn’t read body language, gauge reactions or look directly at the client to see if they were understanding one another properly. 

“Personally, I like having all my team around me,” reports Steven Bodner, supervising colourist, Light Iron NYC. “I find it a lot easier to collaborate and get things done with my regular team in adjacent rooms. Right now, everything is a lot slower simply because every interaction requires a videoconference of phone call. I find it hard to see how anything I am doing now from home will be permanent.” 

Cloud grading proof of concept  
An experimental production of a short film produced in the cloud at a recent Hollywood Professional Association (HPA) event was able to show how raw master files could be linked to Nuke, Avid or Resolve for colour correction in the cloud almost as soon as the take had been filmed. 

For the demo, a cloud-based integration between Colorfront and Frame.io made it possible to stream camera originals to Amazon’s AWS cloud and provide ACES compliant colour and sound synched files for dailies and editorial. 

The proof of concept also demonstrated that, when all the media is stored in the cloud postproduction workflows, could be allocated to the best creative located anywhere.  

“We had colourists in Dublin, London and Hollywood and all could have worked on remote colour correction for us,” explains Joachim Zell, VP, technology at LA facility EFILM. 

They chose to work with Encore in Hollywood. Its colorist in LA graded the rushes live and using ClearView Flex to interact with the DoP miles away in Palm Springs. 

“There were some [facilities] who said they couldn’t do it in time, [so] we found another vendor and were able to redirect the data virtually anywhere in the world and still meet our deadline,” Zell says. 

Security remains essential. Zell pointed out that for a million-dollar blockbuster production, they wouldn’t be so blasé “about giving access [to the media] that easily to people they’ve never before.”  

Jack Wenzinger a solutions architect for AWS working on the project, stressed, “Because global access is phenomenal you need great security to go with it.”  

Learning curve If there’s a shred of silver lining possible to take from the last three months it’s that the industry managed to shift remarkably successfully to remote workflows. The Mill, for instance, set up five colour suites in five different locations in NYC to deliver quality work.  

“It’s been a steep learning curve, a challenge for us and our vendors and not without its teething problems but it’s a situation that has clearly made us rethink the traditional working practices of colour correction,” says McCall.  “Ultimately, having access to a trusted theatre or colour room (and trusted colourist) will still be the most failsafe means of unified creative collaboration and locked down colour science.” 

Remote Audio Mixing 
The same functionality with remote user access, network sharing and collaboration offered for edit and grade on the picture side, can be used for sound design and mixing. A sound designer will need to create, compose and build sound elements to replace those not captured on set, or to enhance a scene with elements not available when shooting e.g. foley, music or ADR. 

“It’s typical that audio post may not even be handled by the same facility as picture elements, so connectivity and management of media and assets between the two is critical,” says Heffernan. “These audio files are likely to come from other external teams or companies as well, such as a score uploaded from a composer. Having the ability to quickly download and incorporate new audio media into working projects means the process keeps moving, and can be dynamic.”   

Matt Skilton, senior dubbing mixer at Envy has been working from his home studio for four years. He will typically perform the entire premix there using ProTools and speakers professionally installed by HHB. 

“With directors and editors busy on multiple projects they don’t tend to have time for the premix so commuting into London to spend several days alone in a mixing theatre didn’t make much sense for me on a personal level,” he says. “I know how to get a programme to broadcast quality sound from being premixed at home knowing that on the final mix day I can still tweak it.” 

It is the final mix which has proved most challenging during coronavirus and the one area which will continue to be problematic in any remote scenario. 

“The one thing we can’t replicate is the final mix day where I have a room full of people including director, editor and executive and we thrash out what everyone’s opinions are. At this point we are talking about subtleties in finessing music cues and it’s something that can really only be done live.” 

Sending notes with files to the multiple decision makers at this stage “is painful for everyone,” Skilton says. 

Envy CTO Daniel Sasson has simulated the experience as best as possible by adapting Zoom to give key creatives a shared voice over session but sound quality depends on individual home set-ups.  

Remote premixing for standard stereo TV work or even 5.1 mixes is not only possible but increasingly the norm. As you move up through scales of surround sound mixing and into specialist areas such as delivery for Dolby Atmos or 3D sound spaces for cinematic quality audio, the less likely you are to be working remotely.  

“For Dolby Atmos there is no replicating that big theatre feel,” Skilton says. “Drama, for example, is more layered with music stems, composition, sound design and could be mixed in so many different ways.” 

For these projects a dedicated facility with multiple speaker monitoring and mixing desks set up for precise, positional mixing is necessary, according to Heffernan. 

The biggest issue remains final mix collaboration. “There’s nothing quite like being face to face when everyone is hearing the same thing at the same time,” says Skilton. “It’s impossible to do via phone or skype. Headphones might help but that introduces its own critical elements which may also distort the experience.”