Building a character in a point-cloud became an entirely new photographic and VFX challenge for the Blade Runner 2049 crew.
“The Overall VFX Supervisor John [Nelson] came to us fairly early for the project,” says Paul Lambert, the DNEG studio’s VFX Supervisor for Blade Runner 2049. “At the time, there were no concepts for the Joi character at all. But we entered into testing the ideas for her straight away.” There were two teams, aptly named TeamLA and the Joi Division. The DNEG crew used some footage they’d tracked already of another actor, and iterated some initial looks, but everything was rejected by Villeneuve, mainly because he felt everything was too much like an effect.
In these tests, first Joi was made of wireframes, there were particles, then there was smoke. “Then she was made of glass, y’know,” says Lambert, “every single effect we showed him in this initial program of effect types, he’d seen before as heavily effects-based personas, or what a CG character could be. We went through the entire gamut. John Nelson came to us months and months before the shoot and based on those initial tests, even though we hadn’t cracked it, we were awarded the work because it was a big job.”
Denis wanted the audience to believe Joi was a real girl. It couldn’t have an overall feeling of being an effect; she couldn’t always look transparent because it would take you out of the movie. “There are times when she’s crying so you can get emotionally involved in the narrative and the last thing you want to see is a visual effect,” Lambert reiterates. ”That was the defining goal for Joi.”
By the time DNEG was onto the shoot, they knew what not to do but they’d still not cracked what Joi was going to be. “We spent six months in Budapest shooting and we would meet every weekend to talk about Joi and at that time we would be doing continuous tests,” Lambert continues. “We had footage of Ana de Armas to work on in Budapest, but the tests prior to that were of somebody else in completely different lighting and it was a different look entirely. That effect is all about the lighting setup in the movie because Roger (Deakins) lit it in a very specific way. As soon as we had Ana as Joi, that was the first time we could complete our testing.”
We shot Joi (Ana de Armas) is a particular way because we had to capture her performance from multiple angles. For a 3D digital performance, you would shoot a leading lady with Mo-Cap dots all over her, so then you could do movement tracking but, that was never going to happen on this movie.”
In the end, DNEG used a whole bunch of witness cameras, GoPros in fact, for inside the apartment. “We couldn’t fit our bigger production cameras in there because it was such a small space,” explains Lambert. “The cameras were always set up differently for each shot. They would record the scene from various angles which gave us a lot of information in order to track the various performances. We still didn’t know what Joi was going to eventually be and we were still testing, and it was only in the last week of shooting when Roger and John showed Denis our test of the idea of Joi as a ‘volume’. We came up with the idea that she would be slightly transparent, but then rather than just a 2D effect where you may start to see a bit of the background behind, Joi is a hollow vessel.”
Basically, Joi is covered in this skin and a dress and has hair and such but inside she’s completely hollow. So if you make her slightly transparent, you can see her front, but can also see her back shell as well, before also seeing the background. This added a complexity which you only saw really when she moved, or when there was a bright light or when the bright light hit her back shell. There are times there is an effect and there are times she looks completely ‘there’, playing as a normal character, and this was exactly how Denis wanted Joi to be. That’s how DNEG played it.
“At no time did we have to ramp up the effect for the effect’s sake,” says Lambert. “It’s just not that kind of a movie. It’s all about the narrative and not to have any gratuitous kind of visual effects saying ‘hey, look at me’. It just wasn’t that kind of movie at all. That was the guiding principle for the whole movie as well. We wanted the whole visual to be as realistic as possible. Once he saw the final test of Joi, Denis called us and said simply, ‘That’s it. Thank you John. Thank you team.’ And this was the last week of the shoot. The timing was perfect.”
“As well as rendering out the pixels in a shot within NUKE, you can also render out the points,” Adrien Saint Girons, CG Supervisor at Framestore explains further about Joi’s revolutionary hollow shell effect. “The points are the coordinates of where the pixel is in 3D space. In NUKE, you are able to rebuild the points used in the 3D render, and recolourise them using the render. “It’s like having a point-cloud in NUKE that you can rerender from another camera,” Saint Girons says. “It’s for times when a complete mirror of a render camera doesn’t cover enough information. We needed a few cameras to render from those used to recreate the deep data. In effect, using a unique point-cloud version of Joi in NUKE was required to exploit this effect correctly.”
Framestore built the digital version of Ana de Armas and this was used to drive the glitches. When she was in K’s room, or out on the rain with him, also stressed out during the final moments of the flight in the spinner. A clean background plate was taken to use as a reference for the lights behind, shining through her intermittently. But it wasn’t as simple as that.
“We tracked the digital version of Ana de Armas against the plates of the actor,” says Adrien. “If she‘s in front of a bright light you can see it through her. We always had a reverse side view of her as well. The way we did this was by creating a mirror camera, which was mirrored around her, matching the tracking camera and her CG double was rendered from that angle using the deep data in NUKE. Then the compositing team was able to rerender her back through the original camera.”
The glitching moment for Joi was generated by voxelising the CG double. Depending on how extreme the emotion is in the scene, the voxels are large and almost random, recreating this 3D pixelating effect which was comped together with the back-facing camera.
Overall VFX Supervisor for Blade Runner 2049 John Nelson developed from being a cameraman many years ago. He says, if he knows he can shoot something, he’ll shoot it. “If it’s better in 2D then I’ll shoot it like that. I’ll go to 3D as a last resort, but we knew there was going to be a lot of very elaborate 3D in this movie,” he says. “Early on when we were doing concepts and storyboards, we’d break down these big sequences and make an effort to make everything close to camera to try to make it real. Build it close and keep reality close. If we could shoot it practical, we would try to shoot it.”
“Denis Villeneuve knows what he wants. He’s making the film in his mind as he’s shooting it,” Nelson says. “Denis and Roger [Deakins] has this kind of Vulcan ‘mind-lock’. Working with a director and DOP like these two guys makes my job even easier because instead of searching for what they want, you just have to hit that mark. I felt the same way about Jim Cameron in Terminator 2 and Ridley Scott in Gladiator. Those directors know what they want and Denis is like that.
In Blade Runner 2049, John Nelson broke down around 1,400 shots, and there are about 1,200 in the final movie. “There’s a lot of visual effects in this production, no matter what you try to look at it,” he quipped. “Scale is one of the features in 2049, it’s really, really big, and so even if you build a Trash Mesa site that is as big as a football field, if you’re not careful it’ll still look pretty small when you photograph it. There was a real art to capturing the immensity of a set.”
DNEG Vancouver was given 291 shots; Framestore in Montreal, and their London art department had 281 shots; UPP Prague won 188; BUF Montreal and Paris were awarded 105; RodeoFX got 64; Atomic Fiction, 33; MPC in Montreal also won 33 shots, which included the digital Rachael.
“Farming out particular sequences for ‘Blade Runner 2049’ to VFX Studios was pretty easy really,” says Nelson. “I was casting on their strengths.”
Some of the most complex and difficult sequences the crew had to figure out and dedicate themselves to was the 3D shell character of Joi, but there was also the Merge of Joi and Mariette, then also the recreation of Rachael. MPC was awarded this particularly difficult job and this will be described in depth in a future article by those involved.
Related links:
Blade Runner official site
VFX SCIENCE 