My intent for the following experiment was to find a way to produce decent shadows on flat objects in After Effects. But first a word about Afx itself.
Working in After Effects can be a soul suck. Sometimes it will read your mind and everything falls into place in perfect logic, other times its a star crossed monkey aids affair. This isn’t necessarily a complaint about the program itself; Afx is a wonderfully deep program that allows you to achieve a great many number of things, a great many number of ways. The problem is that its a hellishly deep program that allows you to achieve a great many number of things, a hellish number of ways.
You can throw some lights around in Afx, and if directed correctly at your composition, you can produce some nice gradients across the flat shapes. Objects will also get darker as they recede away from the light, and brighter as they come towards it. This is all very well and neat, and has been used quite a bit in Afx composited animations already. I wanted to differentiate the look of the animation a bit by creating some faux shadows, not to approximate a 3 dimensional look, but to achieve a semi-stylized feel.
- Shadow goals:
- Shade a simple geometric object
- Create a unique look, that’s pleasing
- Textured, instead of flat or smooth shaded
- Renders at a fair pace; this effect will be added to lots of things and when other effects are piled in, it’ll just be murder on the computer
- Flexible, to be easily changed and key framed to get the right look
The first attempt was a quick gestural test. My thought was: in order to get a shadow on an object, first I needed the base layer (the original blue dot we’ll call “bDot”), and a shadow layer on top of it (named “shadLay”), which we’ll use the multiply blend mode on. So the stacking order so far is shadLay on top, bDot on bottom. In order to create the shadLay, I created a silhouette of the original bDot object in a pre-composed layer, taking the hue saturation down to zero and hue lightness to max. Basically this creates a white dot on a black background. 
As you can see in following picture 
(I’ve taken the transparency of shadLay down a little so you can see through it), when shadLay is set to multiply mode, everywhere that will be white on it will not affect bDot, and everywhere that is black will darken bDot. The problem however, is that shadLay will apply its effect across the entire screen. We need to mask it so that it only effects bDot. shadLay needs to track matte bDot, but in Afx, the layer being tracked needs to be on top, but I needed shadLay to be on top so that it can apply its effect to bDot. The solution, duplicate bDot again, place it at the very top, and have shadLay alpha track matte it. 
Which gives us this 
. All we need to do is blur shadLay, and we’ll get soft edge. You can vary the opacity of shadLay to vary the contrast between light and dark, drag it around to change where the direction the light and shadow move.
That seems like a lot of effort, but it made sense at the time in my brain. It was relatively easy to do but time consuming. Test 2 was very similar to above (except with a luma matte), and I discovered if I blurred bDot2, I could get a nice “reflected shadow”, which just made things appear more pleasing. 
Still, thats a whole lotta layers. Snooping around the Afx effects menus, I found the Channel Effects. Cue the heavenly music. Using the Channel Combiner Effects, I could just direct shadLay to use bDot’s alpha. 
Horrah! No need to noodle around with track matte anymore, whose stacking order requirement on the timeline is a real pain to organization anyway!
Still, precomposing can get pretty messy itself. If I needed to precompose each layer that needed shading, I would end up with a bunch of junk in the project folder. It’d be nicer if I could I could just duplicate a layer and throw it to the side. Test 4 you say? I am but your humble servant.
Deleting the shadLay forever, I brought back bDot2. This time I added effects to this layer, taking the hue saturation to zero (making it black and white), and the hue lightness to max (making it white) 
, then adding blur to give it that soft edge 
. Next add the Channel Combiner effect, and set it “From: Straight to premultiplied” and check “Solid Alpha”. The premultiplying business tells the RGB channels to multiply out the alpha now instead of when rendering, and solid alpha means just that. This results in the color black filling in everywhere that bDot2 was transparent. 
Combined, the Hue/Saturation and Channel Combiner effect effectively recreates what the precomposed layer did. But since its just a collection of effects, we can save it as a preset and throw it on any duplicated layer we need quickly. Add one more Channel Combiner effect using the same settings as Test 3 
, set bDot2 to multiply mode and voila, Test 4 is complete.
At this point I was pretty pleased with my clever little self. Some nice shading done, and it only took a few hours to figure out. Then a horrible though struck me, doesn’t Photoshop have a inner shadow filter? If Photoshop has something, more than likely there’s an equivalent in Afx. Oh F-word.
Here it is, in all its filthy glory, using just one Layer Style setting, the rather appropriately named “Inner Shadow”. Doh. Some experimenting with however, shows that its difficult to get a nice texture going on it. Ha ha, take that, easy way! A callous victory, but one nonetheless!
Truly though, I was ready to settle on this method. It was supremely easier and simpler to do. I even threw in a bevel Layer Style to make it look even better. Afx saw me too please however, and threw a monkey wrench into the whole thing. As it turns out, due to the way Afx handles Layer Styles, it just wouldn’t work for me. Afx renders Effects first, than transformations, than Layer Styles. So for instance, the blue dot in 3D space, Afx will render all its effects first, then move it wherever it needs to be, and this is all peachy. Sweet sweet lemons. But since it renders Layer Styles last, even if I move the object further into 3d space behind something, it will still appear to render on top. Lemons, friggin lemons!
Oh wells, back to the old way. Its not the most elegant, but it works in most cases. I’ll look around to see if there’s a way to get Layer Styles to render first before transformations, and do an animation test to see how everything works together. Expect that Wednesday (to)night, and a follow up on how satisfied I am with this method. Hopefully I’m misinformed about something obvious and it’ll all work out nicely.
On Thursday night, I’ll write up a post about how I think Guillermo del Toro differs from George Lucas on creature effects, and why I think del Toro’s approach is superior.
Labels: development
vs. Layer Styles (the easy way out) 
. In both images, the scene is rendered with 3d layers and blue sphere should be behind the yellow. Sadly it fails in the much simpler Layer Styles example which renders at a speedy 7 frames a second on my computer, compared to 1 frames per second with the more complicated fancy shading method. A benefit of the fancy method though, is that Effects can be easily turned off with a layer switch, enabling quick previews. We’ll call it a net gain :)
. Using our super duper fancy shading method, we can achieve this result 
, which I feel is more pleasing than the original. Look-wise, we can consider all that shading experimentation thus far a success.
, duplicated and stacked correctly on the timeline 
, and 7 effects with minor changes applied to each one 
. The multistage process is long and prone to clerical errors, which I’m extra susceptible to. Being able to save collections of Effects as presets 
helps quite a bit, but it still takes some noodlin’ about. I’m looking at Afx scripting right now, and hopefully I’ll find a way to automate the process a bit more. Ideally, I’d like to highlight a layer, hit a “fancy shadowfy” button and let the machine take care of the rest, but we’ll come back to that another day.
Before I leave you, a quick video from my next round of tests, light blooming! Its not quite developed yet, but its already looking pretty neat, and adds a nice subtle effect that makes the scene look more delicious and dramatic than the original 
. I stole the hdr image from petervanallen on Flickr and carelessly flipped the image horizontally as you can tell by the backwards lettering on the boat. I pointed it out so I don’t want any guff about that okay?!
