Using the Light Field camera and lots of SAUCE to make content smarter.
Interview by Paul Hellard
SAUCE (Smart Assets for re-Use in Creative Environments) is a three-year European Union funded Research and Innovation project through the European Union’s Horizon 2020 Research and Innovation Programme. There will be a dedicated track at the annual FMX 2019 event in Stuttgart, coming up soon (April 30 – May 3). The theme this year at FMX is Bridging the Gap, where the sessions will highlight worldviews and workflows that inspire exchange in the fields of animation, effects, games and immersive media.
I spoke to Volker Helzle, Head of Research and Development at Animationsinstitut at Filmakademie Baden-Wuerttemberg about SAUCE and the Light Field technology.
“SAUCE is a set of professional tools and techniques for making content ‘smarter’,” says Helzle. “These tools are fully adaptive in an unprecedented manner. Adaptive to context (which facilitates reuse), to purpose (among or within industries), to the user (improving the viewing experience), and to the production environment, so that it is ‘future-proof’.”
This wide-reaching approach is based on research into light field technology; automated classification and tagging using deep learning and semantic labelling to describe and draw inferences; and the development of tools for automated asset transformation, smart animation, storage and retrieval.
THE CAMERA
The Light Field camera rig SAUCE had access to was designed and crafted at Saarland University under the lead of Prof. Dr. Thorsten Herfet. “It consists of 64 FLIR industrial cameras with a global shutter, and it has a 1920×1200 resolution on a 1/1.2-inch sensor. Each camera is equipped with a 12.5mm prime lens which is equivalent to a 65mm full frame. The framerate at maximum resolution is at 41fps. Custom electronics even allows triggering at 10µs precision.”
“Each camera is connected to an NUC6i5SYK drive, which has a Core i5, 8GB RAM and 256GB SSD. This system produces 507GB of un-debayered RAM data per minute, which is cached by the NUC’s SSD drive and later transferred to a dedicated server.”
Helzle believes that production using a Light Field camera should be designed to make use of all the new possibilities introduced by it. “Of course, a Light Field camera could be used just like a traditional camera but this does not put the technical overhead into perspective,” he says. “Light Field cameras should for now be seen as special cameras for special scenarios rather than a potential replacement for traditional ones.”
“The first tests we did with the rig were elements of fire and smoke,” explains Volker Helzle. “This test shoot was performed in our studio at Filmakademie. The captured light field is stored as 64 individual videostreams corresponding to the 64 cameras used in the rig. By intelligently blending and interpolating these camera views, physically impossible lens configurations can be simulated based on real (not rendered) footage. The data is released under creative commons and is on the project website.”
UNFOLDING
An ambitious test production for SAUCES, featured the talented cellist Isabel Gehweiler, named Unfolding.
Filmakademie Baden-Württemberg developed a script in close collaboration with the Director of Photography Matthias Bolliger and prepared a complete previsualization for the final trailer.
In the Unfolding production, a lens with an aperture smaller than f0.1 was simulated in post-production through the captured Light Field along with animated tilt-shift effects, which made completely new creative decisions possible.
“One topic of our research is the creation of 3D geometry based on Light Field data,” he adds. “This geometry then could serve as an asset for a variety of 3D software packages. One of the key points here is that the cameras will not directly produce any 3D data.” As the SAUCES crew are in a very early stage of the project, they are working with Matlab code from their partners at TCD in Dublin to create depth maps. Those are then used with CaraVR and Ocula in Nuke for compositing.
NO GREEN SCREEN
Looking at potential benefits Light Field data can provide, the most useful would be to generate high quality, dynamic (moving) depth maps or even geometric reconstructions out of a 5D Light Field. This would not only help to overcome the need of colour keying but it would also unlock possibilities that could lead to a workflow to freely edit the captured scenery including its geometric information, lighting and surface characteristics after the actual recording.
SAUCE is a collaboration between eight partners, each deep into production and education within the active creative fields. They are: Foundry, Double Negative, Filmakademie Baden-Württemberg and Animationsinstitut, BRNO University of Technology, Trinity College Dublin, Universität des Saarlandes, IKinema and Disney Research. Another major partner is Universitat Pompeu Fabra, based in Barcelona.
PAPERS TO DEVOUR
Herfet Thorsten, Lange Tobias, Hariharan Harini Priyadarshini: “Enabling Multiview and Light Field-Video for Veridical Visual Experiences”, Keynote at the 4th IEEE International Conference on Computer and Communications, Chengdu, China, 2018
Ryan S. Overbeck, Daniel Erickson, Daniel Evangelakos, Matt Pharr, Paul Debevec.
“A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality”, SIGGRAPH Asia 2018
The first results of this production and a detailed behind-the-scenes presentation will be held at FMX2019, Light Fields for Movie Productions (Friday, May 03, Bertha-Benz-Saal, 15:15 – 16:15). In fact, at FMX there will be a full track dedicated to the topic of Light Fields.
Related links:
VFX SCIENCE 