Virtually untethered: V-Nova replicates game-like freedom for XR video

IBC

article here

Data compression specialist V-Nova is offering a solution to deliver pre-recorded volumetric content to XR headsets with ‘Hollywood-grade CGI, lifelike parallax, occlusions, reflections, and textures, at full display frame rate.’

The technology combines V-Nova’s point-cloud compression with technology from Belgium company Parallaxter which it acquired a year ago.

CEO Guido Meardi tells IBC365, “We have invented a technology that uses the same pipeline used today to produce CG and VFX content to generate and stream XR content with six degrees of freedom. We allow the user to be inside as if it’s a video game.”

Meardi is pitching volumetric production as a new era in music and XR entertainment. Fans don’t just watch—they’re part of the scene, creating an unparalleled connection to the content, he says.

That’s in contrast to most current video content made for VR or XR displays. “Stereoscopic 3D works in the cinema because your head is still and the 3D can only be right for a certain head position. But in a VR environment when you are immersed in the content and you can move your head around, you need the content to alter according to parallax (change of perspective) without lag just as we see the real world. The XR entertainment market is blocked if it cannot get around this issue.”

V-Nova’s point-cloud compression technology compressed hundreds of gigabytes of data to transform a movie into a manageable asset, ready for distribution, real-time decoding and consumption on VR platforms. It is built on V-Nova’s previous standards LCEVC MPEG-5 and SMPTE VC-6. 

PresenZ enables existing 3D assets to be turned into a volumetric experience by rendering the images from a cube of points of views rather than a single camera point.   

Users can experience 6DoF (Six Degrees of Freedom) meaning they can move within the virtual scene “as though physically present,” he says.

“The biggest benefit of 6DoF technology is that, unlike standard 360-degree VR movies, it realistically responds to the position and orientation of the VR headset, so you can get closer to objects and characters,” Meardi says. “Users can explore all angles and look around and behind objects with a more natural feeling, eliminating the motion sickness commonly associated with VR.”

He claims, “The possibilities opened by V-Nova PresenZ not only promise to rapidly unlock a latent multi-billion-dollar market among existing XR users, but also set the stage for a new XR use case with mass-market appeal. If you don't use our technology, even just a single volumetric image is several gigabytes of data.”

V-Nova Studios, its content production arm, has just debuted a music promo in the format, available via VR platform StreamVR. Weightless, features Albanian X-Factor winner Arilena Ara performing her new song. It’s believed to be the first time a music track has been released first in this format.

“We can put an audience in front of a star to experience the choreography of the performance,” says Meardi. “She's is less than half a meter from the viewer. The detail is incredible down to the pores of her skin and eyelashes.”

The song also features in Sharkarma, an upcoming shark-themed cinematic 6DoF production by V-Nova Studios.

V-Nova is working with NBCU and DreamWorks to turn IP like How to Train Your Dragon and Abominable into future content using its tech.

“It’s much easier to do if it is a 100 per cent CG library item. In that case it is very inexpensive. You re-render the asset in our format and you’re done. If the content is part CG part live action like Iron Man or Spider-man then you would you need to use AI tech to reconstruct the live action elements in 3D.”

Commercials for luxury brands are another possible revenue generator. “If you are Louis Vuitton or Cartier, Ferrari or Lamborghini you don't want to show people a video game-y looking product that may be detrimental to your brand. You want to do it at the highest level of quality or not at all.   With this technology, we can do everything that would you do for normal high-end CG productions but in immersive XR.”

Apple is building out its spatial computing content shop with new episodes, films, series, and concerts captured in Apple Immersive Video set to debut later this year, with more coming early next year. Its first scripted short film, Submerged written and directed by Academy Award-winning filmmaker Edward Berger, is already available. Apple teamed with The Weeknd to launch the artist’s new album and has new music series Concert for One, launching beginning with a special set from British singer-songwriter RAYE.

Meanwhile, Meta is developing Hyperscape, its format for delivering photorealistic digital replicas of spaces from the physical world into Quest headsets using cloud-based processing.

Meardi says V-Nova’s tech merges the Apple and the Meta technologies.

None of this is, however, a live experience and Meardi doesn’t think that this is possible given the limitations and cost of existing camera arrays and without the computer processing power on the scale of a NASA moon landing.

“A single volumetric image is several GB of data—we make it possible to compress this data massively to stream super high quality video at 90 fps motion,” he says. “You need to playback at this display frame rate because you can be really close to the character or person and at 60fps you would see a mismatch and be nauseas. Fundamentally you need not just supreme video quality and photoreal lighting and fur and physics simulation but 6K 90fps. There’s no way real-time engines can do that.”

Consequently, the assets need to be pre-rendered. Meardi says one frame of a typical Hollywood VFX movie would take three to seven hours to render.  Of course, you’d only need to do it once but you still end up with a massive data set. V-Nova’s technology make it possible to compress and stream 25Mbps using MPEG-5 LCEVC (a standard which is based on V-Nova algorithms).

“For live capture you would need to improve massively the quality of authoring tools and translation of the asset.”

The number of cameras would also have to be reduced but AI can play a role in interpolating gaps in the image puzzle.

Other Hollywood studios are reportedly interested. The company is even talking with “a rapper who likes to smoke” (which we guess is Snoop Dogg) about an XR experience in which the rapper’s psychedelic imaginings are also depicted. “You can see what it looks like when he’s high,” Meardi muses.

“What about transforming a star into a werewolf or rejuvenating ABBA or other aged stars? I would love to talk to Michael Jackson’s family. Our technology allows you to see Michael Jackson moonwalk in front of you, on the moon.”