Vue for Pirates of the Caribbean II

The Matte department at Industrial Light & Magic (ILM), the SFX company behind the Star Wars hexalogy, the Indiana Jones trilogy, the Harry Potter sequels and much more, was comissioned to create over 60 VFX shots for the epic second installment in the Pirates of the Caribbean saga.

Susumu Yukuhiro, digital matte supervisor for Pirates of the Caribbean: Dead Man's Chest was kind enough to answer some of our questions:

About Susumu

E-on: Can you tell us a bit about yourself?

Susumu Yukuhiro: I always wanted to be a matte artist and to paint environments. I started as an apprentice at Matteworld Digital, and joined ILM’s Matte department in 1999. The first project I worked on here was Star Wars Episode 1. At that time we were already mixing 2D and 3D stuff for our environments. I am now a digital matte supervisor at ILM, and I most recently worked on Pirates of the Caribbean 2.

When did you see Vue for the first time? Can you let us know what was the selection process for Vue ?

SY: The VFX supervisor John Knoll first came to me asking if there were any solutions for Cannibal Island to be done in complete CG instead of traditional 2D Matte paintings or miniature. We wondered how we’d do it since creating organic environments has always been difficult. We were looking at all sorts of options over the Internet and we tested Vue (trial version for a few weeks). We were pretty happy with the results that came out of it. It was the closest to what we wanted to achieve, so we added Vue to our pipeline and got going!

Was it complicated for you to add Vue to your pipeline?

SY: Not really. It was great in that a lot of the team started using Vue and within a week, they were putting out work.

On the Workflow

Could you explain the few steps that lead from the concept art to the final shot? What was the validation process for each shot?

SY: First we receive a shot from the studio, from which we kind of have to do a sketch phase. It can take about a week or less, then we send this back to Gore Verbinsky THE director. He’s the one approving all the shots, and giving us the directions and updates he wants.

If this is 2D painting, the next step is to do the final painting and send that to the compositing department. There the compositors blend everything together.

Or if the shot requires 3D work (for instance if the camera moves into the scene) we do the 3D matte painting. Then we start splitting layers from the painting and project that onto rough geometry so that if there is a camera motion the animation keeps the parallax and the “3Dness”. Then we have to render the scene and do the compositing stuff ourselves.

If we do 2D paintings only and project that onto the cards it can sometimes look too flat, so doing it all in 3D is really adding a sense of “volume” to the matte. This gives us a lot more freedom as well. For instance, if we need let’s say more 3 dimensional trees, we can render trees only out of Vue with the camera motion so that we still get the “3Dness” we’re after, then we finally composite all together and it’s done.

What was Vue’s place in the production pipeline? Did you start using Vue at the beginning of production, or did you start with rough sketches and then re-create everything in Vue?

SY: Actually, it was kind of both ways. Sometimes we started to draw really quick sketches and then render something out of Vue. But Vue also does actually really quick rendering so it was easy and simple to come up with rock, trees, mountains and all that kind of stuff [without the preliminary sketches]. So yes, we did do both actually.

How did you add the smoke and clouds to the flythrough scene (just before discovering the cannibal bridge)?

SY: Smoke and clouds were painted, and then projected onto flat cards, and all were layered to give the image a sense of Volume. For middle and background they were only flat cards. There were like 4 or 5 layers in the foreground, and sometimes more. Since we wanted to give a sense of dimensionality, we cheated with a lot of cards with cloud textures to give that a volumetric look. Back then we did not have any solid Volumetric cloud solution that was photoreal enough. So that’s the method we’ve always used and it always looks fine.

How did you match the local vegetation inside of Vue? Did you use stock Vue trees or create your own species?

It was kind of both. The flickering was still a problem with Vue 5, and for some shots, John (Knoll) wanted to have some wind reacting trees, so it was kind of difficult to achieve that right out of Vue. For most of the shots it was ok but for one really big shot where we couldn’t accept any flicker, what we did was get a static plant mesh and bring it into our proprietary tool called Zeno, then used RenderMan to render the full sequence. The new plant anti-flickering technology in Vue 6 is something we’re looking into seriously.

How did you match the lighting between the original shots and the 3D stuff created in Vue? SY: What helped us a lot was being able to render out with separate passes (like diffuse pass and ambient occlusion) in Vue, as you get a lot of control afterwards. We never render as a single pass and use it straight. All of the renders were done in several passes and adjusted in Photoshop for shadows touch ups and all that, or just adjusting the color to blend that in the original plate. Sometimes the scene was rendered in pieces and later composited. How did you match the camera moves? SY: Camera movement matching was done outside of Vue. This was the most challenging aspect actually. We tried to not do that much because of the flickering in Vue 5. There was one particular shot where we really wanted wind animated trees, so we’ve brought in the camera move, but the camera was pretty static so we didn’t have to worry about flickering. But that’s something I’m hoping to be able to test in Vue 6.

Original Shot Matte Painting from the shot

What was the creation process for textures and materials? (photo based, procedural?)

SY: We had thousands of great pictures from the Caribbean. So one thing we always do is to go for the real stuff first, so we started with photographs. If we couldn’t find what we were after, then we had to go for another method. What was beautiful is that when we wanted specific mountains textures or tree edges, instead of having to extract that from photographs we just had to render some tree edges right out of Vue, and also use the alpha channel as well to cut out that part using the same alpha and that technique worked really well. Some artists even used Vue renders to create some general textures for future use.

On the Benefits of Using Vue

What did Vue bring to your team in comparison with the first episode?

SY: Between episode 1 and 2, the environments are really different. For instance, episode 1 did not have any “Cannibal Island-ish” landscape, so the comparison between the two can be difficult.

But the thing we really wanted to achieve was to give the second episode some extra details like wind reacting trees, and something like that was always really difficult to do unless we have the good toolset to do it. So with Vue, we could simply put in some trees, and add some wind and we got really nice motion. Something like that alone I think was a huge benefit between the 2 episodes.

And then also of course using Ecosystem to spread millions of trees on simple geometry and just render it is really a quick method to come up with really nice photoreal mountains and so. So that really added to the Cannibal Island look, it was just perfect.

Did you save any time by using Vue, over having to do the same job the traditional way?

SY: Let’s say for instance you do your painting in a traditional way. You have to paint a mountain using a specific lighting for one shot, and in another shot, you might see the exact same mountain but under a different angle and different lighting condition. Then you have to start all over again.

We did not have to do that because we could just re-render it in Vue. Vue saved us at least a week or 2 per shot depending on the shot, so this was a huge time saver. Now we could have gone with a miniature, but the problem with that kind of work is that creating the miniature asset takes a while, and then you have to take pictures of these and you still have to find the perfect angle, otherwise you have to shoot it again; with Vue we could apply little changes and just re-render and that’s all.

What were the render quality settings you used in Vue?

SY: What we did is to crank up Vue’s render quality to the highest level possible. It was not a luxury to be able to have a renderfarm to render everything.

One of the reasons we pushed the settings to the maximum was that we wanted to get rid of the flickering as much as we could, and we almost succeeded with that. I can truly see the difference between rendering something at medium level versus really heavy duty render setups but to get a graving look always take a while to render and we were dealing with billions and billions of polygons so that’s really understandable.

You always can add a lot of touch up on top of everything but we wanted to minimize that to spare some time afterwards, so if we could afford pushing everything to max to get a great quality render to begin with, we always wanted to try that. But still, we had to be realistic with rendering vs time.

How many computers were used to render the Vue scenes?

SY: We were lucky to be able to use a renderfarm of 20 machines, and that was really helpful.

Are there any feature you wouldn't be without? What's your favorite part of Vue?

SY: Ecosystems. Without that we probably wouldn’t have picked up the software. That was alone a huge force for us in order to do what we wanted. And the wind effects were very helpful as well.

How long did you work on Pirates 2? How long did the project take from concept to finished art?

SY: We worked on Pirates 2 for 10 months.

What's Cooking at ILM?

What are the next movies where we’ll be able to spot some Vue stuff?

SY: None that we can talk about right now.

Finally, what do you expect from Vue 6, did you start using it in production? (Pirates 3?)

SY: I’m really looking forward to using the new Clouds in Vue 6 and seeing how we can vitalize that specific tool. The new plant anti-flickering is also something we’ll be looking into seriously. Unfortunately, I’m not able to talk about our use of Vue for upcoming releases, so…