Preserving History in Volumetric VR

NAB

In any other hands a virtual reality experience of the Holocaust might feel questionable but not when used to preserve a witness account of the atrocity for future generations.

https://amplify.nabshow.com/articles/preserving-history-in-volumetric-vr/

In Munich until November, German production company UFA and the Fraunhofer Heinrich Hertz Institute (HHI) are presenting the world’s first volumetric contemporary witness interview of the Holocaust.

The memories are those of 89-year-old Ernst Grube who was 13 when he and his mother were deported to Theresienstadt concentration camp in Nazi occupied Czechoslovakia.

Grube was recorded in the Volucap studio in Potsdam, “transforming his life story into a walkable movie,” says Joachim Kosack, CEO of UFA and UFA Serial Drama.

The three-minute proof of concept has been distilled from 80 minutes of volumetric interview footage recorded in 2019.

“With volumetric video, we have created a unique opportunity to preserve contemporary witness interviews with Holocaust survivors,” says Oliver Schreer at Fraunhofer HHI. “Using these interviews, we can provide a realistic experience for future generations. Our high performance and versatile volumetric video capture offers enormous potential for education and our culture of remembrance. This is more important than ever, especially in times when anti-Semitic tendencies are emerging again.”

Grube and also a young student interviewing him were recorded at the Volucap stage using 32 (16 pairs) of 20 megapixel (4K and 5K) cameras. From this 1.6TB of raw data, a depth map and meshed point cloud was derived as part of a complex video processing process.

There are six episodes planned. The first three-minute one was troubled with considerable delay caused by Covid-19.

“In such experiences, it is possible to get close to the ‘volumetric’ persons and to look into their faces,” says Fraunhofer’s Ralf Schäfer. “In such cases it might be desirable to have direct eye contact with these persons. Therefore, we will develop technologies to manipulate volumetric video [to deliver interactive gaze correction].”

The exhibition is a collaboration between the Munich Documentation Centre for the History of National Socialism, the Jewish Museum of Hohenems and the Flossenbuerg Concentration Camp Memorial.

It’s the latest in a line of volumetric experiments by the Institute. In February 2017, a proof-of-concept production was performed with a limited capture set-up for full body acquisition reaching a 200 degrees field of view. It was showcased for the first time at NAB Show 2017.

Later that year, Fraunhofer made its proof-of-concept production in 360 degrees. Called “An entire life,” the VR experience was a journey through the history of movie production coinciding with the 100th anniversary of UFA.

Other tests featured boxing trainer Uli Wegner and actress Hanna Juergens.