Inside COVID19 is a 360-degree stereoscopic Virtual Reality (VR) documentary produced by Gary Yost, one of the pioneers in software design, now a passionate documentary film-maker.
The film was created in partnership with Director Adam Loften, the Oculus team at Facebook Reality Labs and the Chaos Group along with generous donations of time and talent from 3D artist Andy Murdock, composer/sound designer James Jackson and medical VR engineer Whitney Lai from Surgical Theater. This is the first production to visualise molecular processes in 360-degree VR. The small band of creators at WisdomVR make some of the most inspiring visuals seen in this beautiful, immersive genre.
Take a deep breath and read on.
Inside COVID19 documents the voyage through COVID19 alongside an American doctor, Dr. Josiah Child, as the virus spreads across the world. Viewers go with Dr. Child as he prepares emergency departments in five hospitals for the pandemic. But they also witness his personal struggle as the virus attacks his own body as well as his family. Powerful stuff. This spectacular journey delves deeply into the molecular replication process of SARS-CoV-2 through VR. Inside COVID19 also uses breathtaking, vivid, animated processes discussed below.
This is a documentary that doesn’t take sides. It shows everything as it is, and sometimes it’s really, really hard. People need to be asking very tough questions to the science, and this production brings out a lot of answers. Along the way, the immersive experience is total. This medium can and should be also used as entertainment, no doubt, but it cries out to be used in the documentary genre. Not just for the voyeur element, but for the experiential and educational opportunities.
WisdomVR’s decision to use this genre, places the ‘viewer’ right inside the situation. It reminds us that this is beyond any national politics. Inside COVID19 delivers a vital perspective on how far we’ve come and frames the road ahead to meet and overcome this existential challenge.
“Many people might not know someone who has been deeply affected by COVID-19, but this VR documentary brings you into my experience of what it was like as a frontline medical worker to suddenly become seriously ill,” said Dr. Child. “The virus takes advantage of our instinct to love, appreciate and care for each other. It takes advantage of what is best in humankind and exploits it as a weakness. And so we turn against each other. Hopefully, my experience will help others cut through the confusion and chaos.”
WisdomVR founder and Inside COVID19 producer Gary Yost describes the project as the big story that needed to be told. Yost left software development for 3ds Max as a frustrated film maker. His customers were seen doing amazing projects and he felt the creative pull. In his own words, he “wanted to make real films about real people.”
When Yost was much younger, while recovering from a fairly major illness, he wrote and asked the CEO of Atari to employ him in any which way he could. Right at that moment, Atari needed a software tester, as a part of the Atari Institute for Educational Action Research. This was the seminal act that launched his software design career.
In 1987, while working with Tom Hudson on 3D animation software for Atari, he convinced Tektronix to design and make a pair of liquid-crystal shutter glasses which were named StereoTek for the AtariST microcomputer. This was the only consumer product they’ve ever made, costing around US$150, which displayed 15fps video. This allowed Yost’s original Atari software development team to create stereoscopic animation and display it in a headset. Yost would spend his time going to stereoscopic imaging conferences, which were filled with mostly defence-related people. He would be the only ‘entertainment-consumer’ market guy in the whole room.
After leaving the Atari world in 1988 and starting up the Autodesk/3D Studio/3ds Max development team, Yost said that he always “wanted to stay close to stereoscopic production and film-making.” By the time he sold the 3ds Max source code and IP to Autodesk at the end of the century, he was looking for a change. “I began making short films but always with an eye towards stereoscopic filmmaking,” Yost added.
When the first Omni and GoPro 360 filmmaking rigs came out, he took notice. The Google Jump and Yi Halo platform caught his eye, but he couldn’t afford the US$40,000 price-point. But then after meeting with the CEO of Shenzhen-based Z-Cam, Yost became the development tester for the V1 Virtual Reality camera.
The live action component of Inside COVID19 was shot on the Z-Cam V1. This critter has 10 cameras in a tightly packed radial array of lenses, set around the fuselage of a black cast-aluminum cylinder. Quite the spectacle.
Each lens has a 190-degree field of view, and the resulting images are stitched with MistikaVR stitching software from Spain-based SGO. “You can use Nuke’s CaraVR, but SGO has the most accessible and generally-useful stitching software,” adds Yost. “360 stitching is very tricky in stereo, because you are deriving both eye-views from the one dataset. Each of the 10 separate camera outputs, including the zenith of the shot, needs to be perfectly assembled into the 3D sphere to be blended with an optical flow algorithm and MistikaVR does a really good job. Sometimes, it must be said, I have to touch up the zeniths a bit and do rig removal in AfterEffects, but it’s impressive.”
On the shoot, the crew would set up the shot, with the camera on a monopod. Any batteries and gear below, would have to be painted out, which Yost would do with BorisFX’s Mocha Pro. “They have a really great tool for rig removal, which does lighting interpolation as well, for when lighting in the scene changes,” he says. “You can create a patch and the Mocha Pro will interpolate the lighting to match the area surrounding the patch as the lighting changes in the scene.”
“I’m trying to lean hard into this medium to push it forward, into the documentary-making space and there aren’t many high-quality examples of this yet.”
Staying true to his drive into the documentary production space, he got together with an experienced film director, Adam Loften. They were funded by Oculus to create a series of short films under the auspices of the nonprofit WisdomVR when the pandemic hit. Exploring the truth and science of the pandemic in VR seemed as natural a decision as drawing the next breath. And then a friend introduced Gary and Adam to Dr. Josiah Child.
While watching the scenes in Inside COVID19, I found myself in one of Dr. Josiah Child’s hospital wards with him describing the situation, then I’m sitting in the lounge with the Child family. I’m then in the front passenger seat of his car with Josiah in the backseat with an oxygen tank, clearly suffering. His wife is driving us to hospital, and I have many questions in my head. I want to know where they are, what’s out front, how is the patient? By being ‘in the scene’, I can clarify these points by turning my head, but I’m not being distracted. And they are ‘my camera-moves’. My questions are being answered as they arise because I’m in charge of my view.
“Within this VR presentation, we refer to each shot as a chapter. In each of these chapters we work with Josiah, to derive his sharing of the authentic truth about the challenges he had to overcome,” says Yost. “We gave each chapter a specific theme and while capturing these scenes we’re very conscious of what Josiah was saying and how it fit into the story. We chose our locations very carefully so they provided important context for what Josiah was saying, so even if the viewer is looking around the scene, every element allows the story to go deeper.” Gary tells me WisdomVR’s goal is to teach this technique to others, as this will accelerate the drive for further VR documentary creation.
Yost’s original inspiration for creating these VR productions was Waves of Grace, about the 2013 Ebola crisis in West Africa and the first VR 360 film that brought the true power of the genre to clarity for him, and Clouds Over Sidra, a 2015 virtual reality film about the Syrian refugee crisis. Both these films were created by Gabo Arora and Chris Milk in partnership with the United Nations and Samsung. “Gabo showed us what to do, and his work is what convinced me that this is the medium I need to work in,” says Yost.
“As soon as I heard about the Center of Disease Control’s early visualisation of the virion using 3ds Max, I thought, ‘well, if they can do it, I can do it, and Andy is just down the street, let’s get on with this!,” Yost continues. But first up, Yost and Murdock needed to find out how membrane fusion happens, which allows the virus to enter our cells. This was the start of their crash course in molecular biology.
They spoke to everyone available, and homed in on the work of David Goodsell at the Scripps Research Institute who had so clearly illustrated the coronavirus and how it replicated within our epithelials cells. “To visualise such complex activity, you need lots of compute power and laser-focused guidance to visualise the real action,” he adds. “But rendering with such detail, at 49-megapixels per rendered Master frame, there was no way their machines could handle the job.”
Creating a digital SARS-CoV-2 virion, rigging the spikes and animating the virus was a crucial part of the documentary. Using concept illustrations, Yost reached out to one of his long-time 3ds Max contacts and found Andy Murdock, founder of animation studio LotsOfRobots, living only just down the street. The virus model was modelled entirely in 3ds Max with TyFlow used to create the surface proteins, including the famous spikes on a geosphere. The Chaos Group’s V-Ray renderer’s ‘Fur’ shader was used to provide the lush visual complexity of a biological surface.
Andy Murdock was making little Coronavirus monster characters and playing with stereo versions even before he spoke with Gary Yost about the project. He was having particular fun with TyFlow rigs . “It’s a wonderful set of tools,” he said. “It’s not just a particle system, it’s a whole new set of updated modifiers and caching tools. TyFlow is an absolute joy to work with. I just want to make a bouncy virus that could attach to other objects.”
“There are some things you can’t do in stereo 360 that are a big part of a regular 2D project,” explains Murdock. “In stereo everything needs to be in scene as a 3D model. Tricks and fakes are only for distant backgrounds that are too far from the camera to have a stereo effect. Also, extra detail is needed on the models, every single little mistake shows up really well in the Oculus. We could pretty much focus our detailing efforts on what was right in front of the viewer. The environment behind the viewer is nice to have just in case someone wants to look around, but it’s not really important to the story.”
The deep cellular environment is moving like a liquid ripple to give the feeling that everything was moving fast, alive with activity. Andy created a 30-frame loop for the background cell wall elements and put the virus elements matted on top of that. “You could not really tell it was looping unless you watched it carefully,” he explains. “Those 30 frame loop elements were several layers, and took about an hour per frame for some, so this was the only way we could setup and render such a rich environment.”
Murdock used V-Ray ‘Fur’ to cover the skins with short fat hairs that looked more like little capsule shapes. “Sending this to Chaos Cloud over the internet was lightning fast. All the mesh is created at render time with no need to mesh it out into the Chaos Scene file,” he says. “Meshing out this scene if I used particles instead of ‘Fur’ would have generated a massive file size that would not have worked.”
Luckily, they had friends with benefits. Frank DeLise from NVIDIA sent Andy two Titan RTX cards, and they were gifted almost unlimited rendering horsepower from Phil Miller and Vlados at Chaos Group. They were finally able to render out almost eight-minutes of 7K by 7K stereoscopic 360-degree frames on the Chaos Cloud, with incredibly intricate complexity.
Yost calls himself a super nerd. A technologist. “I’m really good at knowing what software, cameras and other tools can be made to do,” he says. “And, with this very small team I was able to tell one of the most important stories of my lifetime. The fear of the pandemic is eclipsed by the excitement of being able to tell this story. And, what’s more, the size of the team who put 3D Studio together so many years ago is exactly the same size as the one that created the Inside COVID19 immersive documentary.”
The Inside COVID19 production was nominated in 2021 for an Emmy Award for Outstanding Interactive Program.