Super Resolution: Why more pixels makes creative sense

 IBC

The industry isn’t stopping at 8K. All bets are off for in a million Megapixel-plus future where massive digital screens, VR and lightfields take centre stage.

https://www.ibc.org/trends/super-resolution-why-more-pixels-makes-creative-sense/6367.article

Ultra High Definition is far from the last word in TV resolution. Though not yet widespread, the industry is going beyond 8K UHD and entering the era of “Super Resolution” where there are no limits to what can be achieved.  

Common broadcast systems may not reach let alone exceed 8K any time soon but Super Resolution technologies capable of new creative options and visual experiences will eventually consign even 4K imaging to a blur. 

“The moment when the resolution ceases to matter and we can cover 12K or 16K resolution per eye for VR (virtual reality) - which requires 36K or 48K respectively - we are getting somewhere,” says Jan Weigner, Cinegy co-Founder & CTO. “Then there is volumetric video. 12K just gets us warmed up.” 

Why 8K is the new norm  
Currently 8K only accounts for a minor share of sales in both the B2B and consumer display markets. Figures from analyst Futuresource Consulting reveal 8K represented less than 0.1% of global B2B flat panel display volumes and about 0.1% of global TV volumes in 2019. 

The high-resolution specs of the recently announced Blackmagic URSA Mini Pro 12K and 8K internal raw on the Canon EOS R5 mirrorless camera are certainly headline grabbers, but for most users these acquisition formats will only have situational use and will not be the go-to recording format.  

“The high frame rate 4K modes on both cameras are arguably a bigger draw for a greater number of end-users,” says Futuresource analyst Chris Evans. 

Productions shooting in 8K are outliers but they are increasing in number. Among them is David Fincher’s latest feature Mank which is being lensed in black and white for a 4K Netflix release. Fincher and cinematographer Erik Messerschmidt are making a habit of shooting 8K, having done so for season 2 of Mindhunter. 

“I would prefer the optics to be the bottleneck in relation to the image and not the sharpness of the sensor or its resolution,” Messerschmidt says. “The highest resolution sensor is best for me because I can make a more measured choice in terms of what I am trying to give the audience visually. It has always led to a better image, in my opinion, when I’ve shot on a higher resolution sensor.” 

He adds: “I look forward to the 12K capture workflow.”  

Even broadcast facility Envy says it has started to deliver in 8K. “The amount of data involved in a project can approach 1PB, and it doesn’t make sense to start pushing an amount of data like this to third party cloud providers in order to provide short term services,” says technical operations director, Jai Cave. “To that end, we envisage finishing post remaining an on-prem service, with offline being remote where it offers an advantage to clients.” 

Oversampling for VFX  
Oversampling higher resolutions for post pan and scan or VFX happened with 4K over HD and acquiring 6K for lower spec deliverables. Blackmagic Design’s stated reasons for a 12K camera release is consistent with this trend. 

“Once you cross the threshold into using oversampling and/or downsampling for a production, by adding further resolution you are only amplifying your received benefits,” Evans says. “Compared against a 6K image sensor, a 12K sensor is only giving you a higher ratio of oversampling and/or down sampling to 4K at the cost of a heavier data footprint and a more resource intensive workflow for your image pipeline.”

The overhead of extra resolution is always a careful consideration when it comes to VFX. “Larger files with more data, sometimes means less iterations because of rendering time which could affect the creative depending on the type of project,” says Gavin Wellsman, creative director & VFX supervisor, The Mill NYC. “Having the choice is always a bonus as long as the pipeline from production through to post can support it. 

“When we went from SD to HD then to Ultra HD the jumps in standards always seemed excessive but once you have a way to enjoy these larger formats it quickly becomes the norm and you wonder how you could ever view a picture with less detail.  

Wellsman adds, “If 8K becomes the standard over the next few years, having the ability to reformat and zoom in to enjoy extra detail from a 12K image might not be as excessive as it sounds.” 

The ‘K’ race 
The arrival of a number of 6K camera options such as the Sony Venice, Blackmagic Pocket Cinema Camera 6K and upcoming RED Komodo underline that 6K currently occupies a sweet spot for downsampling to 4K since it has a lighter file size relative to 8K.  

For this reason, camcorders with resolutions greater than 4K but less than 8K are forecast by Futuresource to grow at a quicker rate than pure 8K products and account for nearly a third of the global pro camcorder market by 2024. 8K products, by contrast, will only claim 5% volume share in that time. 

“With the push to larger sensor formats, whether full frame or 65mm in future, the increase in sensor size given the same photosite size will lead to higher resolution sensors,” says Sam Measure, Technical Consultant at CVP. “With 12K on the horizon, we’re also seeing a backend that’s able to handle the increase in data throughput, from more efficient compression technologies to faster and cheaper storage and compute.” 

There will always be a group of users interested in acquiring at the highest fidelity available at the point of capture. However, there are only a finite number of photosites that can fit within the physical dimensions of an image sensor and decreasing the size of photosites to accommodate more of them comes with trade-offs in other aspects of sensor performance such as low-light capture. 

“As a result, Futuresource highlights a divergence in the large sensor camera market. Evans says,  “Increasingly we will see the release of products that have been developed to either prioritise resolution in excess of 4K or not.” 

The Canon C300 MIII, for example, trades the pursuit of greater pixel count for innovation in dynamic range. The cinema camera is Canon’s first to introduce Dual Gain Output technology on the image sensor, which enables it to deliver a claimed 16+ stops of dynamic range in 4K. 

“The release of more cameras that can record 6K, 8K, 12K and so on is inevitable,” says Evans.  

Why do we need Super Resolution? 
The need for an 8K display, let alone anything higher, is typically debunked as being a waste of time since humans can’t physically resolve the visual detail. 

This is being countered by arguments that the most common method for determining whether resolution is visible to humans—termed simple acuity—only tells part of the story.

“Generally, simple acuity is measured via the Snellen eye chart which determines the ability to see distinct black‐on‐white horizontal and vertical high‐contrast elemental blocks,” explains Chris Chinnock, executive director, 8K Association. “However, human vision is far more complex than a simple acuity measurement. Research suggests that 8K images are engaging other senses that can be difficult, if not impossible, to measure but are real nonetheless.” 

He observes that some stars in the night sky are far too small to be seen according to simple acuity theory - but we can see them nonetheless.  

There appear to be two other factors at play: Vernier acuity and the brain’s ability to “fill in the gaps.” According to Chinnock, Vernier, or hyperacuity, refers to the ability to discern slight misalignments between lines—an ability that’s impossible using simple acuity descriptions of human vision.  

“Hyperacuity means we can perceive fine details even at fairly long viewing distances,” he says. “Such results are now motivating investigators to undertake additional studies, which will be very helpful in convincing 8K skeptics to at least have an open mind on the benefits of higher resolution displays.” 

Hyperrealism 
Samsung even suggests that higher resolution displays are able to provide smoother gradients and improve sharpness to the point where objects seem even more realistic than in reality.  

“When we are looking at an object in the real world, we are placing it in focus, while not concentrating deeply on the rest of the surroundings,” writes YungKyung Park, associate professor in Color Design at Ewha Womans University Seoul for Samsung. “However, when we are looking at the same object using the [8K] display, we can focus on every point of the screen, allowing us to perceive substantially more information about this object.” 

Such ‘hyperrealism’ is achieved, she explains, “when we are able to capture and comprehend even the subtlest lighting and shading effects and the display is able to give us abundant information about the object.” 

Almost all 8K content is being created in High Dynamic Range. Any 8K deliverable would be accompanied by immersive audio. Fame rate is another vector that enhances realism by reducing motion blur.  

“You’ve got simple and hyperacuity coming together in the brain to create an image that dimensionalises the image in ways that we’re not fully aware of,” Chinnock contends. “So, whether you have an 8K image or a 100K image, the higher fidelity and lack of artefacts reinforces what the brain is able to do. The brain is not having to work as hard to recreate the image.” 

Since the human eye consists of 100 million rods and cones, perhaps the ideal resolution for a camera is 100 Megapixels. 500 Megapixel cameras exist and are in use albeit in security not broadcast. 

Super Resolution applications 
In just over a month, BT Sport plans to introduce the world’s first live regular 8K sports broadcasts for production of the 2020-21 English Premier League. 

“There are benefits for live productions where you have the potential to operate within a window from a static, high resolution wide camera and be able to place the cameras where you may not necessarily be able to put operators,” says Measure. “For productions centred around wildlife, where capturing behaviour on film could be a once-in-a-lifetime experience, super resolution offers a great benefit being able to either re-frame or build digital pans and zooms. Also, for higher resolution volumetric capture for more realistic, detailed background scans and digital people, whether that be for VFX pipelines or even analysis in a separate industry.”  

In 2018, filmmaker Martin Lisius claimed the first 16K film. He used a pair of 50MP Canon EOS 5DS cameras shooting 12 000 images which he then laboriously stitched into a 4-minute time-lapse video (of extreme weather conditions) at a resolution of 15,985 x 5,792. 

“Mega high-resolution projects are being worked on for concert venues and event venues that need to wow people to justify the ticket prices, or more simply their existence,” Weigner says. “With 8K becoming home cinema that increases the stakes. The LED walls modules to build 16K exist.” 

Last year, Sony installed a 16K screen into the front of a cosmetics store in Yokohama, south of Tokyo. “When you get to this resolution it delivers almost a quasi-virtual reality experience as your eyes perceive there to be depth to the content, respected pundit David Mercer told the BBC.  

Measure thinks low pixel pitch, high resolution LED panels may take over from projection as the cinema screens of the future.  

“Depending on viewing distance, low pixel pitch LED panels can demand extremely high resolution, and that’s not necessarily just for delivery,” he says. “We’re also seeing this in use now for things like virtual production using LED screens as immersive backgrounds. The virtual set can of course just be a render, but it can also be a scan from a stitched super resolution, volumetric capture array.”  

“If you take 4K content and scale it across a large canvas it won’t look so good,” agrees Chinnock. “That’s why super high-resolution matters.” 

It’s also the target for a new music and entertainment venue proposed for Stratford, London by the Madison Square Garden group. The 90-ft high MSG Sphere will house the “largest and highest resolution LED screen in the world” - understood to be 32K. 

Another, costing $1.2bn, is being built by MSG at The Venetian in Las Vegas. 

“More and more often we are bidding on projects with super giant screens whether at a stadium screen or a multi-screen installation in a shopping mall,” reports Wellsman. “Most of the time the content is either fully CG or design based and we retrofit a current piece of TVC content and render it at the larger resolution. As larger format cameras are more readily available and 8K becomes the norm there will be a desire to show live action content on these giant screens with various degrees of visual effects.” 

Outside of niche fixed installations consultant and analyst Benjamin Schwarz expects millimetre-thin screens, foldable screens and bezel-less screens to enable whole wall screens with a resolution well above 8K. “Such setups will appear first in places like high-end stores but eventually reach people’s homes,” he says. “A fibre to the home or 6G connection would deliver streams at resolutions above 8K.” 

Given that current UHD standards stop at 8K and because initial applications of super resolutions are in digital signage and venues like theme parks, standards for super-resolution might be driven by the ProAV market and bodies like InfoComm.  

Slightly longer term and the arrival of 5G, then 6G, would finally bring Virtual Reality to fruition. 

“16K for VR is not a whole lot,” says Weigner, whose company has developed a codec called Daniel2 claimed capable of manipulating 16K data. “Give me the pixels, the more the better, and the applications will follow.” 

Even longer term and we can start thinking about lightfield capture and holographic display or genuine 3D TV. The ability to change focus dynamically in post and pull keys from depth maps is raises exciting possibilities but whether you capture with a multi-camera array or some kind of huge sensor with a micro lens array you are talking hundreds of millions of pixels. 

“8K is just a number,” says Weigner. “So is 12K. 16K will come for sure. And higher.”