Fyusion has announced a new kind of light field imaging tech that allows users to create 3D photo-realistic views of complex real-world scenes and products. The new system can be used to generate imagery, merely by using an app on their phone. Fyusion has presented the new tech at SIGGRAPH in Los Angeles but I had a chance to read their paper early, and chat with the CTO and Co-Founder of the venture, Stefan Holzer.
“Fyusion’s software is already in-market in a number of commercial applications, including automotive and fashion, where it is helping retailers increase and enhance engagement with online shoppers,” he said.
Backed by prestigious Silicon Valley investors and powerful global businesses, Fyusion has raised more than $50 million to digitize the real world using light fields and 3D computer vision and automatically extract valuable information from it, in an easy to use manner and accessible on any computing device. In fact the Fyusion App is downloadable for Android and Apple phones and tablets, and I’ve been experimenting and uploading results of my own experiments in their very active community forum. What’s more, the results can be exported as mp4, or even gifs that can be triggered to activate continuously to demonstrate the products.
During the experimentation and creation of the rendered visual samples created for Fyusion app, the team found many interesting barriers and then set about working out this new view synthesis problem. “Ideally, we could simply sample the scene’s light field and interpolate the relevant captured images to render new views,” says Holzer. “Such light field sampling strategies are particularly appealing because they pose the problem of image-based rendering (IBR) in a signal processing framework where we can directly reason about the density and pattern of sampled views required for any given scene.” The required view sampling rate increases linearly with the reciprocal of the closest scene depth.
Retailers have been using 3D imaging as part of their digital sales and marketing programs for some time now. The goal is to provide consumers with a vivid and realistic view of a product, so they feel like they’re experiencing it before buying. Earlier this year, Fyusion received a major investment from Cox Automotive, a leading digital wholesale marketplace for used vehicles, which is using Fyusion’s software to display 3D images of cars on its websites. And then last month, Fyusion announced an investment from Itochu, one of the largest Japanese e-commerce fashion and apparel companies, which is using Fyusion to show images of models wearing outfits on its brands’ retail sites. Fyusion’s tech effortlessly handles notoriously difficult scenarios, including fine-grained textures like grass and foliage, transparent surfaces and reflections.
3D imaging isn’t new but making the technology accessible to the masses has proven elusive. Traditional 3D imaging methods rely on costly laser-scanning hardware and manual studio touch-ups; others eschew 3D entirely in favor of simple photo stitching. Fyusion’s technology is groundbreaking because it is low cost and produces the highest quality results. It’s also conceptually simple. First, Fyusion uses a deep network to promote each source view to a layered representation of the scene, advancing recent work on the multiplane image (MPI) representation. Fyusion then synthesizes novel views by blending renderings from adjacent layered representations. The result is a 4,000x decrease in the number of images needed to produce a 3D image, making it easy for anyone to create high-quality 3D images using only a smartphone.
“The most compelling virtual experiences completely immerse the viewer in a scene, and a hallmark of such experiences is the ability to view the scene from a close interactive distance,” says Pantelis Kalogiros, co-founder and VP of Web at Fyusion. “This is currently possible with synthetically rendered scenes, but a high level of image ‘intimacy’ has been very difficult to achieve for virtual experiences of real-world scenes. When creating the 3D image from real-world complex scenes, it is critical to maintaining the accuracy and photographic fidelity of the digitized 3D scene as the camera viewpoint is changed. Our new technology is the best in the world at this.”
Members of the Fyusion R&D staff will give a presentation based on the technology on Monday, July 29, at 10:45am PT at SIGGRAPH2019.