Volumetric video: now you can try it for yourself!

RedShark News

Prince thought the idea “demonic” but holographic proxies for pop stars are closer to the bone than ever. In a claimed world first, a single by Brazilian chanteuse Laura Rizzotto has been released, accompanied by an AR ‘holographic’ app. Fans can see her performing the track in virtual three-dimensionality anywhere they point their phone.

https://www.redsharknews.com/vr_and_ar/item/6821-volumetric-video-now-you-can-try-it-for-yourself

An augmented reality app in and of itself is not groundbreaking, but “the holographic performance we are premiering with Laura Rizzotto absolutely is,” according to its makers, the LA-based volumetric capture studio Metastage, quoted in Forbes. 

Rizzotto’s performance was filmed with 106 video cameras - 53 RGB cameras and 53 infrareds – shooting 30fps, although the rig has 60fps capability. The data was passed through the Microsoft Mixed Reality Capture software to output a 3D asset that Metastage claims maintains the integrity and fidelity of the original performance.

Anything ‘first’ is guaranteed publicity so this could be dismissed as a publicity gimmick for a singer-songwriter who is not exactly Beyonce-famous.

On the other hand, the dystopian scenario of Charlie Brooker’s recent Black Mirror episode, in which Miley Cyrus’ singer Ashley O was replaced with a hologram, is but a hop, skip and a step from coming to pass.

We’ve had holographic resuscitations from Tupac to Roy Orbison, completely (hollow) graphic pop-stars like Japanese pop idol Hatsune Miku and fictional social influencers like Lil Miquela (1.6 million Instagram followers and counting), but the concept of an actual artist being able to multiply their presence and therefore their revenue by materialising at any venue in the world (simultaneously) must be music to the ears of record company execs.

Cold Play might find it a useful way of cutting down their carbon footprint on tour. The Rolling Stones might like to consider it as a way of crystallising their own carbon imprint before extinction.

After all, one of the business goals of Metastage is to create a digital archive of holographic assets that can be preserved and enjoyed (played back) forever.

There’s more, though.

“For the first time ever, users can tailor the artist’s performance to their own taste and comfort,” says Carolina Rizzotto, co-producer of her sister’s Metastage Experience. “You can take Laura and choose where she’ll perform and even change her capture size. You can make her twice her normal size...or you can even make her the size of an ant and have her perform in your own hand.”

Twice her normal size – that’s eerily reminiscent of the Black Mirror episode too.

For this to be streamed on a mobile device, with or without 5G, the file size needs to be manageably small and so it is with Metastage claiming the Microsoft process can compress the 10 gigabytes of data it captures per second down to 10 megabits a minute of volumetric capture. Arguably that’s ok on a phone where HD rather than 4K video, provides a good enough experience.

Metastage’s app is developed in partnership with Magnopus (the company behind immersive experiences for Pixar’s Coco and Blade Runner: 2049). It suggests that end-to-end production costs around $10k per second – not excessive compared to high-end commercials - but will rise depending on the photorealism required.

Unlike traditional pop promos, performers can’t yet rely on an edit of the volumetric capture to mask any failures in performance or dancing skill. The capture zone itself is confined to an 8-foot-diameter circular area. Rizzotto had to get the entire choreography right in full, uninterrupted takes.

“There is no editing to the (volumetric) performance, as it has to be in one take only, which makes it even rawer,” according to Carolina Rizzoto.

Carolina, incidentally, previously produced a short educational film called ‘The Future is Here - How Sci-fi Becomes Reality’, starring George Takei (Sulu).

The most important take away from this fun development of holography is the idea that we might all be able to own 3D avatars of ourselves in the emerging three-dimensional internet. Operating a hologram is only a step removed from the 2D online presence we now portray of ourselves on social media.

The chief executive of Metastage, Christina Heller, talks of the ability for volumetric capture to overcome the ‘uncanny valley’ of the inauthenticity of existing animated human techniques.

“When you watch something that was captured volumetrically, you know that it was captured live. It wasn’t reanimated,” she said in an interview with XR for Business. “It wasn’t puppeteered. This is something that you can watch with the same awe that you would a live performance happening right in front of you.”

The issue – whether you are a pop star, an actor (alive or dead), or plain Joe Public - is who owns our digital selves? In a world in which deep fakes are in many cases indistinguishable from the real thing, who protects our avatars on the 3D internet?

Behind the scenes: The Irishman

IBC

Martin Scorsese returns to organised crime in an epic feature which could last a mini-series and is all about time.

https://www.ibc.org/create-and-produce/behind-the-scenes-the-irishman/5248.article

“It is an essential part of what the movie is about,” says Rodrigo Prieto, ASC. “How time affects us as human beings and how we relate to the story of our lives.”

Specifically, The Irishman is recollected by Frank Sheeran (Robert De Niro), World War II veteran turned trade union enforcer and mob associate in a tale spanning six decades and a movie time of 209 minutes.

“One of the first things Scorsese and I talked about was memory and how we could portray that,” Prieto tells IBC365. “Scorsese mentioned home movies as a reference but we didn’t want to film as a home movie so I started exploring amateur stills photography as a way to convey the passage of decades.”

While Super 8 and 16mm would have been the logical choice to achieve this look, the filmmakers didn’t want to make it handheld or too grainy, plus they had to account for the decision to use extensive VFX to de-age De Niro and some of the other principal actors.

“We decided we had to shoot digital for the scenes that required de-ageing and use negative the rest of the time,” he says. “I talked with VFX Supervisor Pablo Helman and he assured me that they would be able to match the look of film with digital.”

Colour of nostalgiaPrieto spent considerable time researching, consulting with colour scientists and testing film stock to emulate the colours of old home photography.

“I was born in the sixties [1965] and my memories of the 1950s are evoked by the slide photography of my parents which was on Kodachrome,” he says. “I felt the look of Kodak’s Ektachrome was more appropriate for the 1960s.”

“The crux of the story happens primarily in the 1970s so I decided to base the look here on a film developing technique first made by Technicolor in Italy. It’s a process in which you skip the bleaching of the print and keep some of the silver on it to create an image with more contrast and less colour.”

Prieto worked with digital workflow developer Codex and facility Harbor Picture Company to design a lookup table to map the colours shot on 35mm to digital. A second step was adding in grain to match the film’s emulsion. In the process, he concluded that Red Helium cameras shooting 8K offered the best means of emulating the analogue look.The editorial idea is that The Irishman begins with the nostalgic saturated colours of Kodachrome (“the reds pop”) and as the film unfolds it transitions (“to more of a blue-green palette”), gradually more colour is taken away from the image to signify the draining of hope.

Hydra camAbout half of The Irishman is shot digitally and half on ARRI film cameras, with the digital scenes requiring the design of a new camera rig comprising two Alexa Minis on either side of the Red to feed data into the VFX pipeline.

All three cameras for each angle had to move in unison and have their shutter’s synchronised. The rig had to be lightweight enough for all the cameras to fit on one head. None of that would have been possible using film cameras with magazines that needed loading and unloading.

The ‘hydra’ rig also had to be fitted to different heads – a crane, remote head or fluid head. “We wanted to be sure Scorsese was not limited in any way in the design of shots,” Prieto says. “We failed the first time. After some trial and error, new motors and more lightweight materials we got there.”

Scorsese prefers to shoot dialogue scenes from two camera angles simultaneously – one over the shoulder of the actor and vice-versa – which gave Prieto the challenge of hiding cameras that were facing each other in shot.

“If I were to do this again, I’d look for a rig where the cameras are lighter and as small as possible,” says Prieto. “Sometimes we’d have a third angle which meant nine cameras rolling at the same time – which means a much larger crew including a focus puller for each camera. It’s a big investment.”

Instead of helmets or physical tracking markers, the actor’s faces were painted with infrared make-up which the two witness cameras could ‘see’. Additional volumetric information about each scene came from a Lidar scan.

“VFX had all this information to compute the position and intensity of light and even the reflectivity off the sets to apply to the facial de-ageing. They didn’t have to rebuild the light for each scene, it was kind of an automatic process.”

Having lensed Scorsese’s previous two films The Wolf of Wall Street and Silence, the cinematographer has come to a good understanding of what the director wants.

“He writes notes on the script or diagrams sometimes also drawings about the compositions he has in mind,” Prieto explains. “He will give a sense of the energy, the mood or editorial motivation for certain parts of the movie. Within that, I can propose where the camera might be, what focal length to select, and how to light the shot.”

Blast from the pastIt’s clear that The Irishman is as much a reminiscence for the director as it is for his central character. “He would talk about the Villa di Roma restaurant [where the movie opens] and [Italian seafood restaurant] Umberto’s Clam House as the type of places where men like these would meet and how he had vivid memories of these places growing up.”

At one point in the movie, Sheeran sets fire to a row of yellow Chicago cabs in a clear nod to Scorsese’s Taxi Driver. The casting of Joe Pesci and Harvey Keitel also unites the director with his earlier work, but stylistically the camera work and editing is more restrained than Casino or Goodfellas.

“This isn’t based on any limitations in the hydra rig,” Prieto wants to emphasise. “Scorsese constantly wanted to tell the story from Frank’s perspective. Frank is not a complicated man. He paints houses [murders] in a very methodological way so when we are with Frank the camera movement reflects this with simple pans that we might show repeated later. Scorsese used the term ‘clockwork’ to describe Frank’s routine. Even the cars are shown in perfect profile and sometimes the camera is just static.”

One of those shots is a close-up held on De Niro the moment he boards a small plane en route to perform a murder that even for Frank Sheeran might be above and beyond the call of duty.

“One of the privileges of a cinematographer is being right there a few feet from the energy of an actor giving their first performance. It’s more than a front-row seat in a theatre. You are participating in the moment. You are feeling it as you operate the camera. Watching through the eyepiece of a film camera is superior to watching on a calibrated monitor because everything – the costume, the production design, the virtuosity of the acting – is seen first through that lens and you are responsible for capturing it.

“In this case, De Niro doesn’t do much at all, outwardly. Frank is forced to go on the airplane. He knows what he is going to do and it’s horrendous for him but he also can’t show any nerves. He is swallowing all these emotions and you can feel this churning up inside. He is baring his soul in this moment. It was incredible to witness. Hopefully, that translates to screen.”

Quantel gave broadcast a quantum leap into the era of digital creative technology

IBC

Given the march of technology, breakthroughs are inevitable sooner or later but it’s the sooner part of the equation that can propel certain inventors and companies ahead of the pack. Some can even shift entire industries forward.

https://www.ibc.org/create-and-produce/industry-innovators-quantel/5245.article

Such was the case at Quantel, the British brand which gave broadcast a quantum leap into the era of digital creative technology.

During the 1980s and 1990s, Quantel was in its heyday, riding high on the nascent postproduction industry it helped catalyse. Tools like Harry and Henry were must-have items for broadcast graphics and finishing houses on both sides of the Atlantic, with facilities trading on their ownership of one.

It arguably kickstarted the CGI revolution in 1986 when Paintbox, its best-loved machine, was used by director Steve Barron to make the video for Dire Straits Money For Nothing.

David Hockney, no less, enthused over Paintbox’s possibilities for “drawing with light on glass” in a richness of colour “that not even paint can give”.

CodebreakersPaul Kellar, Quantel’s research director, likens the firm’s formation to the code breakers of the Enigma machine during World War II.

“Both were capable of lateral thinking, using imagination to find new ways of doing things,” Kellar tells IBC365. “Both were also a disparate collection of people. A coming together of minds. Richard Taylor was a fabulous engineer and leader. Others, like myself, had no management or leadership skills but were quite good at dreaming stuff up.”

Quantel was formed in 1973 by Anthony Stalley and John Coffey with the backing of Micro Consultants, a Newbury-based signal processing manufacturer set up a few years earlier by Peter Michael (knighted in 1989 and who passed in 2009) and Bob Graves.

Originally to be known as Digit-Tel; the name was rejected by Companies House as being too close to US firm DEC necessitating a rethink. Peter Owen, who was its first employee, recalls that ‘Quantel’ was coined by his wife Rhiannon over breakfast “during an attempt at explaining the nature of quantised television signals.”

Its first products were digital time base correctors sold under the IVC (International Video Corporation) name in the US to compete with the then dominant Ampex Corporation.

When Coffey and Stalley left in 1975 to launch two new companies, Questech and GML, Quantel became the responsibility of Richard Taylor as managing director, and Paul Kellar as head of research, with Peter Michael as chair.

Taylor’s background was in cognitive systems and image analysis; Kellar was a Cambridge educated electrical scientist who had previously worked for Rolls Royce.

“From the beginning, the sales emphasis was to be on export,” says Owen. “This policy avoided the influence of and over-dependence on the BBC and ITV in the UK.”

1976 was a year of unprecedented export opportunity in North America; Montreal hosted the Olympics whilst the US faced presidential elections and celebrated the bicentennial anniversary of the day of Independence - all major television events, stimulating equipment purchases including new technology such as framestore synchronisers.

“Broadcasters were screaming out for some way of improving coverage,” Owen says. “We had to make a technology leap from time-based correctors to framestores.”

Luckily Quantel had one up its sleeve. Kellar had devised Intellect, the first framestore-based image analysis system which could accept and display live video, while at the same time allowing fast random access from a read/write computer port, a feature which was to become a key technology for future Quantel products.

“Once you have random access you can do anything with it,” Owen says. “You had the ability to make a quarter picture in picture and put a cool border on a live signal.”

Kellar and Taylor combined the Intellect with video interface mechanisms to build the DFS 3000 Framestore Synchroniser used by the Olympics broadcaster in 1976 to showcase a PiP inset of the flaming torch while the rest of the picture featured runners entering the stadium.

“Having made the leap to picture manipulation we knew there was potential in creative uses of technology,” Owen says.

At a time when this capability was regarded as requiring a mega-machine and huge rack space, Quantel’s manufacture of a complete framestore system in only 8.75 inches was both a bold step and important engineering advance.

“Because we found a way to make a framestore that was smaller and cheaper we could use several,” Kellar says.

The idea of having more than one framestore in a machine was considered “ludicrous” but the company’s next development relied on multiples of them.

IBC 1976 Paintbox prototypeTaylor identified the challenge of creating a system which would paint a line ‘as if a camera had looked at a real painted line’. The solution was to blend a bell-shaped brush of colour with the existing underlying image, a digital printing process that was repeated rapidly, overlapping previous stamps, as the brush was moved by the artist.

Quantel’s first public demonstration of digital painting, on a blank screen and onto stored images, took place at IBC 1976. R&D work continued to improve the quality based around a fundamental architecture which included pressure-controlled painting and multiple framestores for creating graphics in a way which did not destroy the original image.

It was the refinement of a pressure-sensitive pen which proved the killer app.

This innovation, disruptive at a time of keyboard ubiquity, effectively put Paintbox into the hands of artists and graphic designers. When it debuted in 1981 it benefitted from a menu and palette system designed by illustrator Martin Holbrook which was so intuitive it was able to be understood by non-technicians.

“The technology always aimed for realtime,” Owen says. “With that speed and with those tools it suddenly opened up the possibilities for creatives and broadcasters.”

Sales were made via broadcaster design departments, not engineering departments.

“It was about taking hardware for granted in a way that an artist could use and love it,” Kellar says. “We didn’t spend a lot on marketing in those days. We didn’t need to.”

Paintbox was expensive. “People quipped they could buy a lot of pencils for 70,000 dollars,” says Kellar. “It was good enough, useful enough - and we sold hundreds of them.”

Quantel’s next generation of live effects machine, known as Mirage, solved the problem of manipulating a live 2D television image into a 2D representation of a real 3D shape (the now-familiar ‘page-turn’ was created for the first time on this machine).

Taking advantage of the latest disc drive developments, it found that they could gang four parallel transfer drives of 470MB each to store 75 secs of 4:2:2 video and that with time code control of a D1 VTR it could be used for longer from editing.

The first commercial incarnation of this idea was launched in 1985 as ‘Harry’. This product fundamentally changed the post industry by bringing the tools of graphics, compositing, image resizing and editing to a pen and tablet workstation.

The concept was evolved with Henry the vision for which was expressed internally as ‘Harry with a sense of history’. Kellar says, “As long as any previous operation could be changed without starting again, then a serial processing engine could give unlimited flexibility. Effects, Paintbox and all the other processes could be included without having to build everything as a separate engine. New processes could even be added during the life of the machine.”

Henry’s ten-year life in the market place was proof of how advanced this concept was.

“It was exciting because the company was growing but when you have growth, it has to be managed all the way from R&D to manufacturing,” Owen recalls. “We were shipping and exporting a huge amount of kit which needed huge investment in automated assembly lines and pick and place machine which were pioneering at the time.”

This was a time of almost fabled extravagance when tools for post dominated headlines at tech trade shows. Digital telecine machines, for example, cost the best part of a million dollars.

“We had something special,” Owen says. “It was a very flat company in a way – everybody knew everybody else on first name terms. It meant that everyone from UI design to engineering to sales felt able to give each other feedback.”

It couldn’t last of course. Owen left in 2000, taking up a senior role within IBC. Kellar resigned in 2006 and has devoted much of his time since with teams at Bletchley Park helping to reconstruct an operational replica of the famous code-breaking machine Turing Welchman Bombe.

Today, Quantel has almost ceased as a brand having merged with Snell to form Snell Advanced Media, subsequently acquired by the Belden Group and currently for sale as part of Belden’s divestiture of another stalwart brand, Grass Valley.

Kellar traces the seeds of decline to the firing of Taylor in 2005 by LDC, the venture capitalist group which took control of the company in 2000.

“They took the view that that company was too dependent on him. Their solution was to fire him. From then on the company was doomed.”

He adds, “We knew perfectly well that sooner or later machines would get cheaper and more software-based but there’s no question that, at that time, Quantel products needed purpose-built hardware. Making the transition to the desktop was not easy and one that has taken the industry more than 15 years [from 2000] to actually deliver.”

Quantel was ahead of the game. Its legacy lies in multiple patents (Kellar alone holds 33 and was recently granted Honorary Membership by SMPTE). The company was lauded with numerous Queen’s awards in recognition of export and innovation and it claimed eight Technical Emmies including for Dylan, a proprietary disc technology named after a character in The Magic Roundabout.

“We were the first to bring tools to graphics and illustrators which in turn opened up compositing for video,” says Owen. “We opened it up to those who were the least interested in what was in the machine bay. It was creative technology.”

Let the first hi-tech Games begin

Broadcast

The countdown to the Tokyo 2020 Olympic and Paralympic Games is on the last lap. With eight months to go until the Opening Ceremony of the Olympic Games, host broadcaster Olympic Broadcasting Services (OBS) is all set to score a number of firsts by producing more content than ever before and taking the Games further into the digital era. 

p12-15 https://edition.pagesuite-professional.co.uk/html5/reader/production/default.aspx?pubname=&edid=bb7eceaa-835f-44e4-98c9-4fc563f140cc

OBS will break its own records by producing 75 live feeds, more than 9,500 hours of live coverage and 10,000 short-form clips for the Olympics plus approximately 1,300 hours for the Paralympics. The combined events are expected to be the most watched Games in history across all platforms – broadcast, digital and mobile.

Underpinning it all will be a cloud-based infrastructure, by all accounts the largest ever built for broadcast, which is set to transform Olympics coverage next year and far beyond.

Since 2001 the Olympic organisation managed to grow its global audience at successive games with a formula built on a massive hosted outside broadcast.  This centred on a huge temporary media centre built in components, tested, and shipped to the host city and assembled.

This included galleries for production of the official broadcast feed and space for rights holding broadcasters to book and build their own custom studios and edit suites. Fleets of OB facilities would be freighted to the venues with signals contributed by satellite to the centre outside of which was a temporary satellite farm for distribution of finished feeds. OBS would store as much material as they could on servers within the media centre for rights holders to plunder and produce their own packages.

The main downside was cost, which still required broadcasters to send hundreds of staff to operate many miles from home over many weeks, and the limited ability to manage and distribute the multiple camera coverage.

From 2020, this resource intensive, creatively restrictive OB operation is being replaced by one that is ideally more fleet of foot but lighter in footprint.

The IBC is still the operational hub but one linked to the OBS Cloud, a data centre comprised of Intel servers with network connectivity managed by Chinese e-commerce giant Alibaba.

According to OBS, cloud broadcasting “will accelerate the digital transformation of traditional TV broadcasting.” The solution, it says, “moves labour-based production efforts into the cloud for processing, improves the efficiency of broadcasting while reducing the cost and enables content creation innovation.”

That’s quite a list but it matches the promises made for the wider transformation of sports coverage from on-site equipment and personnel to virtualised (software-based) remote produced workflows.

Olympic rights holders will be able to make cost savings by keeping elements of production at home, accessing live feeds from OBS Cloud.

“You’re going to see a lot more done from Europe, especially with the number of feeds coming back,” Dave Schafer, SVP content and production, Discovery told a SportsPro event earlier this year. “Our footprint on site will probably be more content-based, to go out and tell those stories.”

For the 2018 Winter Olympics, Discovery still fielded 1000 people in PyeongChang but had a 1200 team back in Europe, a ratio it believes was a tipping point in the company’s transformation.

The switch to a more economically and environmentally sustainable broadcast infrastructure in 2020 will only be the start. The IBC is still built with 350 containers shipped to Tokyo albeit with some of its panel modules and cables recovered from the previous Games’ operation.

There will still be OB trucks at venues around Tokyo but there will also be a 5G network laid by telco NTT DoCoMo for contributing video as a replacement to fibre and satellite in an extension of workflows first trialled in PyeongChang. This will principally be used for 8K to VR live streams.

2020 lays the groundwork for a new broadcast model which will see more and more workflows relocated to the cloud over successive Games. Alibaba’s commitment to the IOC runs through 2028 and not coincidentally includes the Winter Games Beijing 2022.

Data centric

Cloud-based workflows are also allowing OBS to introduce more data-heavy applications to enhance presentation. Chief among them is a sports AI platform developed by Intel for use in the run up to and during the Games.

Instead of using wearable sensors, the 3D Athlete Tracking Technology (3DAT) uses information from multiple standard cameras which provide different angles of the athletes as they train, processed in Alibaba’s cloud.

“It’s a first of its technology using AI and computer vision to enhance the viewing experience,” explains OBS CEO Yiannis Exarchos. “3DAT extracts the form and motion of athletes from multiple 4K cameras and leverages advanced pose estimation algorithms to transform that into insight for broadcast overlay.”

It will for example, help viewers better understand the different phases of a 100m race.                         

Millennial reach

The move to the cloud is essential for the IOC and its media partners since both want to reach more people, younger audiences especially, across more channels by gaining access to virtually unlimited material from the Games.

Discovery is taking this further than most. It will dovetail the OBS Cloud with its own ambitious remote production build out across Europe in what will be the first serious test of the value of the Euros1.3 billion it spent acquiring pan-European Olympic rights until 2024.

Eurosport Cube, an AR-based analysis tool which debuted in PyeongChang, will return. The graphics give the appearance of putting presenters inside the sport and is a key to making niche sports like softball, sport climbing and surfing relevant to audiences.

Via SVOD Eurosport Player, it intends to offer “every minute” of the Games with live coverage of every event supplemented with personalised and tailor-made content.

It will also publish to a range of social media including on Twitter with live streams of the opening and closing ceremonies and highlights throughout the event in return for in-stream sponsorships sold by Discovery, Eurosport and Twitter (Twitter has a separate Tokyo 2020 deal with NBCUniversal which includes live streaming of bespoke NBCU-produced 20-minute show).

Eurosport’s Olympics clips in the UK will be subject to a three-hour delay due to the terms of Discovery’s existing sublicense agreement with the BBC.

While Discovery and the Olympic movement desire to extend their reach online, TV is still the biggest platform. Of the nearly two thirds of Europe’s entire 700 million population Discovery claim watched some part of its Winter coverage the bulk of 4.5 billion video views were on TV.

Jean-Briac Perette, president and chief executive of Discovery said, “While there is so much talk about digital, the reality is that the primary driver and still the king in terms of delivering this audience, is traditional television.”

Resolution shifts up a gear

Tokyo will be the first Games covered in UHD 4K High Dynamic Range (HDR) which should enhance the viewing experience by bringing a greater level of detail, sharpness and colour even to HD sets.

8K is likely to play a larger but still niche role in the 2020 Games. OBS will produce live and on-demand coverage in 360 8K VR of track and field, boxing, beach volleyball, gymnastics and the opening/closing ceremonies in an extension of trials at the PyeongChang winter games. It’s not clear which rights holder, if any since free to air broadcasters dominate, will have the 5G capacity to distribute this stream in full 8K.

State broadcaster NHK has recorded key events in its 8K format stretching back to London 2012 and plans to produce and broadcast 8K coverage of the 2020 Games over its domestic Super-Hivision channel.

While many analysts are sceptical that 8K broadcast content will ever take off outside of Japan, Italian pubcaster Rai aims to become the first European broadcaster to do so when it kicks off a UHD 4K and 8K service, starting with the Olympics.

Japanese vendors have led the field in developing 8K broadcast technology and will take the opportunity to showcase kit in Tokyo. Panasonic has timed commercial rollout of its 8K Organic Sensor technology at the Games. Among its advantages is a Global Shutter that will make for sharper images of fast-moving objects.  

Esports, AI and outer space

Esports will inch closer to becoming an accepted Olympic sport at Paris 2024 with the Intel World Open hosted at Zepp DiverCity, Tokyo just ahead of next year’s Games. Players will compete in Street Fighter V and Rocket League – although for an un-Olympic commercial prize pool of U$500k. The tournament format mimics the Games with national qualifying rounds feeding into national team qualifying rounds, the seven best of which plus Japan will compete in the finals.

The Games will go stratospheric when a pair of Japanese astronauts broadcast messages to cheer on the Olympic torchbearers direct from the International Space Station (ISS). In addition, a specially commissioned satellite will launch from the ISS next spring to orbit the earth with robotic versions of the Games’ mascots.

Even the event’s official theme music is generated by AI. Composer Kevin Doucette feed thousands of melodies into Intel’s AI software and used a selection of the tunes processed by the algorithm to form the basis of the anthem.

John Toll, ASC, on Rooting Harriet in Nature’s Light

Studio Daily

The heroism of abolitionist Harriet Tubman in the context of pre-Civil War adversity is so powerful that it needs little adornment. The filmmakers chronicling her life in a new biopic, currently in theaters, sought as authentic a treatment as possible.

https://www.studiodaily.com/2019/11/john-toll-asc-roots-harriet-natures-light/

Harriet follows Tubman on her 1849 escape from slavery and subsequent missions to free dozens of slaves through the Underground Railroad. This extraordinary yet seemingly powerless woman is one of the few — even to this day — to have led an armed expedition in U.S. history, and she remains a source of national courage in the fight for emancipation.

Kasi Lemmons (Talk to Me, Eve’s Bayou) directs from the screenplay she co-wrote with Gregory Allen Howard. Cynthia Erivo, Tony Award-winner for The Color Purple, takes the lead with support from Leslie Odom Jr. and Joe Alwyn.

Toll, who won back-to-back Oscars for Legends of the Fall and Braveheart and earned a third nomination for The Thin Red Line, has an unrivaled track record in rendering period drama to the screen. “I didn’t need to go out of the way to create a period look,” he says. “Once you put actors in costume and in the right environment, you pretty much know what it would have looked like at the time. The aim was to use all the natural elements of the location without filtration or augmentation, which would risk distracting from the story by appearing too modern.”

Harriet was filmed entirely on location in Virginia during an uncharacteristically wet fall of 2018, which required the cast and crew to work through grueling conditions — many sequences were shot in the woods with the actors and filmmakers having to contend with rain, wind, mud and insects.

“My preference was to go digital for this film, for practical reasons, but also because I don’t subscribe to the idea that telling a period story necessitates film,” Toll says.

The practical reasons included working to a low budget and a tight 35-day schedule for a shoot that required a significant number of night exteriors. Settings also included fields, swamps, and forests. “We had a lot of scenes in wooded areas. That’s an environment that can easily suck up the light, so I needed a camera with high sensitivity.”

Toll enlisted Panavision in New York for his camera package, which included the Sony Venice. He tested Panavision lenses including the H series and Artiste before selecting a set of Panavision Sphero 65 lenses 24–180, which lend the digital image a softer, classical look.

“Shooting digital allowed us a certain level of efficiency. We didn’t have the luxury of coming back the next day, so we had to hit it first time out. And we didn’t have to sacrifice anything in terms of overall look with this combination of camera and lenses.”

The production was rooted in extensive research. It helped that the Civil War era marked the advent of photography with daguerreotypes of the 1840s, which provided evidence for costume and set design. There are even daguerreotypes of Tubman herself, although they were taken after she had led slaves out of bondage.

Properties on the Berkeley Plantation, a National Historic Landmark, served as both the Brodess Plantation — where Tubman was enslaved — and the home of New York Governor William Seward, in the film. “We shot practical exteriors and interiors of the farmhouse which contains fewer and smaller windows than modern buildings,” Toll explains. “Even at midday the interiors were quite dark.”

The restriction of budget and time was almost an attribute, Toll adds. “We were all determined to make this film.”

The next generation of immersive sees the light

IBC

In Steven Spielberg’s movie Ready Player One there’s a shot of actor Ty Sheridan putting on virtual reality (VR) headgear which transitions imperceptibly from real to virtual cameras as the shot moves to an extreme close-up. In Gemini Man, Will Smith’s digital double is among the most realistic yet created for the screen.

https://www.ibc.org/create-and-produce/interview-paul-debevec-google/5226.article

Both instances made use of a Light Stage facial scanning system at Google and is just one of a number of breakthrough applications led by Paul Debevec, a senior scientist working in the company’s immersive computing wing.

A pioneer in image-based rendering who directed experimental short The Camponile in 1997 using photorealistic animation techniques adopted by the makers of The Matrix two years later, Debevec was named one of the top 100 innovators in the world aged under 35 by MIT in 2002. He’s been working with Google since 2015 as well as being an adjunct professor at the USC Institute for Creative Technologies in Los Angeles.

IBC365 caught up with Debevec at the VIEW Conference for visual effects in Turin where he presented Google’s latest efforts to capture and process light fields for a more realistic sense of presence in VR.

“Filming in 360-degrees only captures one perspective on how different materials react to light,” he says. Light fields can give you an extremely high-quality sense of presence by producing motion parallax and extremely realistic textures and lighting.

“We need to replicate how the world reacts to you as you move your head around and there are clues to this with how light bounces off surfaces in different ways.”

VR at a crossroadsIt is not, however, a great time to be in consumer VR. The BBC has just disbanded the team it created to make VR content, Disney and Sky-backed VR tech venture Jaunt was recently sold to Verizon and Google has halted sales of its Daydream View smartphone headsets.

Debevec believes VR is still “on the incline” but admits it was hyped out of proportion.

“So over-hyped that [Google] pulled me and my group out of our environment at the University. For a moment it looked like VR had potential as a new and interesting media and that it would become a platform that, if you were not on it, you would miss the boat. That kind of mindset gets a big tech company to throw people and resources at something.”

He says the main concentration in the tech industry now is on augmented reality (AR) but flags that it’s another instance “where the VPs and execs see it both as an opportunity with great potential and a risk that they’d miss the boat if they don’t get involved.”

There is a quality problem with VR which Debevec is trying to solve.

“Users are presented with a stereo view in any direction. If your head moves, the whole image comes with you. In effect, your whole perceptual system is attached to the world and that causes nausea.”

He says: “If you want to create a great virtual experience that takes advantage of 6 degrees of freedom (6 DoF), we need to record not just two panoramas but an entire volume of space that is able to be explored interactively as you move your head around.”

Light field is the answer. It’s a means of capturing the intensity and direction of light emanating from a scene and using that information to recreate not only the volume of the space but subtle light changes, shadows and reflections.

A very brief history of light fieldThe idea goes as far back as motion picture’s founding father Eadweard Muybridge who, in 1872, recorded subjects moving sequentially in still images.

A hundred years later, another array of cameras was used to take images of a subject simultaneously, combined into a time-slice and used to create synthetic camera movements.

Deployed first on film in Wing Commander and Lost in Space then, ironically, on The Matrix, virtual camera techniques have become increasingly sophisticated.

“Light field rendering allows us to synthesise new views of the scene anywhere within the spherical volume by sampling and interpolating the rays of light recorded by the cameras on the rig,” he says.

Under Debevec’s direction, Google has built a number of light field camera arrays. These include a modified Odyssey Jump called Oddity which consists of 16 GoPros revolving in an arc and triggered to take photographs synchronously.

“Absolutely the key concept of light field rendering is that once you record all the rays of light coming into that sphere (scene) you can use the pixel values and the RGB values of each image to create images from different perspectives and views where you never actually had a camera,” he explains.

“By sampling or interpolating information from the hundreds of recorded images, you can synthetically create camera moves moving up and down forward and back – every view you might want to view in a VR headset with 6 DoF.”

Test shoots included one aboard NASA’s Discovery command module at the Smithsonian Institute’s Air and Space Museum.

Google focused on static scenes first, partly so it could work with relatively inexpensive camera rigs and also to perfect techniques required to create the best image quality.

When light field camera maker Lytro folded last year with Google in pole position to acquire its assets, it was Debevec who decided not to pursue development.

Rather than camera arrays, Lytro had built single body video cameras with dozens of micro-lenses including a cinema camera that was the size of a small car.

“That should be in a museum,” Debevec says. “The main drawback of Lytro’s system was that its spatial resolution was decimated by the lens array,” Debevec says. “If they had an 11-megapixel sensor the output resolution would only shoot 1k x 1k images.”

Light field video experimentsWhen Google turned to video, they retained the camera array arrangement but needed even higher quality machine learning algorithms to generate interpolations.

This is what Google’s computer vision experts have advanced with a machine learning process it calls DeepView.

“DeepView gives quite high quality viewing interpolations using an ML technique,” he explains. “It’s not depth maps plus geometry but volume with RGB alpha output.”

In a first test, it modified the Oddity rig into one called Iliad using 16 GoPros to generate 100 depth points of RGB alpha. With this data, they were able to generate synthetic camera moves around such ephemeral elements as smoke and fire, as well as recreating realistic reflections and specular light formations.

“It’s not completely artefact free but it blew our minds,” Debevec says.

Its latest light field camera array is its largest yet. The Sentinel comprises 47 x 4K action sports cameras capable of capturing a 120 x 90-degree field of view.

One application is as an aid for postproduction effects including camera stabilisation, foreground object removal, synthetic depth of field, and deep compositing.

“Traditional compositing is based around layering RGBA images to visually integrate elements into the same scene, and often requires manual artist intervention to achieve realism especially with volumetric effects such as smoke or splashing water,” he says. “If we use DeepView and a light field camera array to generate multiplane images it offers new creative capabilities that would otherwise be very challenging and time-intensive to achieve.”

At its offices in Playa Vista, Google has also built a larger volumetric light stage capable of scanning the whole human body, not just the face. It’s one of a number of such capture stages springing up around the world. Hammerhead VR operates one based on Microsoft technology in London. Paramount and Intel have built one in LA covering 10,000 sq ft, the world’s biggest, ringed with 100 8K cameras.

At Google, experiments continue with DeepView including the recording light fields of Google staff performing various simple movements, then using machine learning to render them into entirely new scenes, complete with detailed illuminations that match the new environment.

There are problems, though, in building the technology out to capture larger volumes.

“We wish we could take you all around a room in a light field but we’d have to move the camera to different parts of the room then find a way of linking images captured from each position. Just managing the amount of data is still daunting at this point. We’d have to ask machine learning to step in and help us.”

He is sceptical of holographic displays although believes the technology will advance.

“Any solution to this needs to have an extremely high pixel density,” Debevec says. “We may have hit the limit of human vision for conventional displays, so is there enough market to create 1000 pixel per inch (PPI) displays let alone 5000 and 10,000 PPI displays that will allow you to use the pixel surplus to output arrays of light in omnidirections?”

Editorially too, Debevec thinks there’s a lot of learning to do for VR to become as compelling an experience as cinema.

“We need to figure out how to tell stories in immersive filmmaking. Once you fill a user’s whole field of view so that they can see everything all the time and you take away the directed ability to zoom in on close-ups, you are giving them a lot of extraneous information.

“It would be like reading a novel where between any line there would be a whole paragraph describing what is going on in the rest of the scene. You would lose the story, become confused. The danger for immersive media is that it doesn’t focus the user’s attention.”