VR gigs point to the future of live music

IBC

Although audiences continue to be largely locked out of attending music venues they can still experience live performances using virtual reality technology.

https://www.ibc.org/trends/vr-gigs-point-to-the-future-of-live-music/7627.article

One of the leaders in this space, MelodyVR, has teamed up with Live Nation to deliver 360° virtual reality gigs from London’s iconic O2 Academy Brixton.

“We want the end user to be the director of their own live music video by being able to jump between camera positions,” says Matthew Williams, technical production manager, MelodyVR. “Bringing fans together with the artists they love using immersive video has never been as important as now when they are prevented from attending in person.”The shows feature 90-minute performances from Tom Grennan, Frank Carter & The Rattlesnakes, Burner Boy, Fontaines D.C., Kaiser Chiefs and Blossoms, with fans able to choose from multiple camera angles – including onstage – viewable through the MelodyVR app on smartphones and VR headsets.

Immersive setup

Five 360º virtual reality camera rigs designed and built inhouse using multiple Sony 4K sensors were positioned at front of house, side of stage and on stage with the drummer and keyboards (or sax, or bass guitar, act depending) and the lead singer at the O2 Academy. A director’s cut offers a sixth viewer-selectable feed.

When the signal leaves each camera it is routed over an extensive network comprising multiple pieces of Blackmagic Design equipment. Banks of more than 30 Teranex Minis convert HDMI, SDI and fibre with all of the 4K feeds managed by three Smart Videohub 12G 40x40 which act as the main matrix for the production.

Each of the 4K lens feeds is routed to PCs housing DeckLink (4K and 8K) cards for stitching, playback and encoding of the high-resolution 360° video. The stitching software is from Swedish developer Voysys.

Williams explains: “The stitched 360-video is composed of a 16x9 equirectangular image with a top and bottom added. The viewer is effectively putting their head inside a sphere of 3D video.”

Stitching is processed in just 400ms and the feed switched via an ATEM Constellation 8K, which also generates four multiviews for MelodyVR’s production team both on-site and remotely.

vMix is the live video production software solution with TouchDesigner for graphics overlay and VFX animation. The live streams are fed from vMix into the cloud where MelodyVR has its own custom cloud distribution network. It is able to stream at multiple bitrates to ticketed end users accessing the MelodyVR app or web player.

All feeds from the live shows including individual lenses and equirectangular outputs are recorded in Blackmagic Design HyperDeck Studio 12G in ProRes HQ.

“This gives us the greatest bitrate and the most scope for grading in future,” Williams says. “Should we need to go back and restitch anything we can go direct to source and do that.”

Just like a crowd-attended concert, there are IMAG screens displaying live 2D content from the show captured from a variety of Blackmagic PCC 6K, Micro Studio Cameras, URSA Mini Pro 4.6K and PTZ robotic cameras where they can’t get operators on stage.

“We run multiple dress rehearsals prior to going live and multiple internal tests for different types of lighting and LED screen. We are constantly putting cameras in positions we’ve never tried before.”“360 immersive video is not something users have had much experience of before and where they have it’s not necessarily been the most reliable or enjoyable experience with audio out of sync and or some technical issue,” Williams says. “It is really important for us to focus on the reliability of the stream.

Remote workflows

Social distancing restrictions meant that this set-up was tougher than usual. The producers had to slash the number of crew they would typically have had on site.

“Most of what we call our ‘war room’ – our operations room – is remote,” Williams says. “All our stitching is remote. We just have a core team on site. We’ve had to strip it back and multi-skill to be as remote as possible. Our rack systems are set up in a way that socially distancing is possible. All our operators are two metres apart. We have clear segregation on site so that artists never have to come into contact with crew. We’re using QR codes at entrances. We log everyone, we track and trace. PPE is compulsory.”

He continues: “Remote workflows have put a lot more pressure on our internet lines. We have three lines into the Academy; one satellite link and two hardwired from different suppliers so if there are any issues we have instant back up.

“We have multiple operators working remote on these shows, some as far away as the US but others in Hastings and the rest of the UK.”

Virtual stage invasion

With Frank Carter and his team, MelodyVR and Innovate Live devised a ‘stage invasion’ in which select fans had the opportunity to virtually appear on stage with the artist.

“Imagine a Zoom call with 150 fans in a huge multiview grid displayed on LED screens at the back of the stage,” says Williams. “There were people at home on webcam. We could see them dancing, with their pets, drinking, playing guitar, having fun.

“Every other track we were able to choose someone to have a ‘hot seat’. The idea was to have someone who could pop up in the middle of the screen and have a one-to-one conversation with Frank. They could request songs. So, it was a massive Zoom call on stage. This even included Frank Carter’s mum on screen at one point.

“We had to configure the audio delays between the users at home and Frank so they could interact properly and get everything in sync. It took a lot of trial and error but we managed to pull it off for the show.”

Each of the six ‘jump spots’ or viewer-selectable camera positions is streamed at 4K. MelodyVR is developing tiling encoding so it just streams the portion of the stream the viewer is looking at.

“We want to get the lowest bitrate at the highest quality user experience so those users on the lowest internet bandwidth have access to 4K and beyond content,” Williams says. “We’re trying to push the boundaries of what is possible with immersive video.”

It is exploring depth mapping and six degrees of freedom (6DoF) - where the user can virtually walk around the stage during a live performance.

Elements of the workflows will enable them to produce 8K VR down the line. “When our camera systems do step up to 8K resolution we have 8K-compatible hardware already there.”

Reviews have been decent for the shows that took place before Christmas. “Although it will never live up to the raw energy of seeing the band in the flesh wrote one “the set was an excellent placeholder until we can all get back in the pit again”.

MelodyVR takes on Spotify

MelodyVR previously produced a three-day festival in July which delivered 130,000 streams of live music from Alexandra Palace. It also notched up over 100,000 live streams during a virtual reality broadcast of Good Morning America’s Summer Concert Series in September.

The five-year old AIM-listed company has a market value of £74 million and claims to be the only licensed VR music platform. It also claims to hold the world’s largest library of immersive and interactive music experiences (some 1800 tracks) from the likes of Post Malone, Kelly Clarkson, Fall Out Boy and The Chainsmokers and has partnerships with O2 Telefónica and Facebook.

That has now been boosted by the $70 million acquisition last August of Rhapsody International, the US company that owns the original pioneering digital music brand Napster. Rhapsody registered 10.8 billion streams in 2019 with revenues of $113 million on the back of an ad-free model that charges users $9.99 per month for its service. It has 3 million users worldwide and 90 million licensed tracks in its catalogue.

It’s a move that puts MelodyVR into more direct competition with the likes of Spotify and Apple iTunes.

“MelodyVR’s acquisition of Napster will result in the development of the first ever music entertainment platform which combines immersive visual content and music streaming,” said Anthony Matchett, CEO and executive chairman, MelodyVR, in a statement. “Our purchase of Napster, one of the music industry’s original disruptors, is born out of our wish to deliver the world’s foremost music experience, available seamlessly across audio and visual media and in turn presenting a truly next-generation music service.”

As with sports, live-streaming performances in VR effectively eliminates capacity restrictions for any concert or venue, enabling an unlimited number of fans to purchase VR tickets to an otherwise sold-out show. VR could prove a complementary revenue stream to the live experience when gig-goers are allowed to mosh once more.