The wonderful world of Shadows!

Something Dark this way comes...

So let me begin by stating my intentions with this blog post. Shadows are a fascinating area of computer graphics and to this day remain predominant on game developers' list of attributes to perfect. Playing through several games in my lifetime has given me a clear and concise window into the power of shadows and lighting within a game environment and how important these elements are in the development of an all round great game on any console. It is important to keep in mind here that shadow rendering is limited to the capabilities of the hardware in question. However, while this is an important matter of consideration, it is not the be all and end all as game designers and graphic engineers continue to push the boundaries of consoles in the way they choose to implement dynamic shadow mapping and lighting systems within their games. 

Several notable games that have done shadows right include Killzone 3, InFamous 2, God of War 3 and LittleBigPlanet 2. I will be sampling methods of shadow rendering from a couple of these big titles and describing why they have been so successful in their endeavors. It is most important to  keep in mind that lighting and shadows make a huge difference in the quality of a game and stand to be pillars of game design that developers strive to perfect. 

A lot of the information I'm going to present is taken from a Siggraph Talk in 2012. The slides which accompany the talk can be found here.

Shadow Science & Effect

What is a shadow? In the simplest of terms, shadows are dark projections that appear when light is being blocked by an object, thereby preventing another object to receive the light. In many instances several shadows can be naturally present in a scene or environment. When shadows overlap, they typically form a union and two objects seem to be projected as one due to the overlap artifact. Moving an object closer to the light source causes the object's shadow to be projected larger due to the fact that the object is now blocking more light than before. An important property of shadows is that they rely on the transparency/translucency of the object blocking the light. Simple logic then dictates that the more transparent an object is, the more light passes through it and the fainter the shadow that is projected. Hence, the darkness of a shadow is also directly related to the brightness of the light in the environment. 

Although these facts are very fundamental, I've always felt that it is important to have a foundation of knowledge before moving on to more complex topics further in the blog. 

The importance of shadows in video games is monumental in comprehending the spatial relationships of objects in the game world. Think about a character on the ground as a pose to the same character jumping in the air. His shadow changes based solely on his position relative to the ground and the light source. If we perceive the light source to be, for example, the Sun, then we can say that the character gets minimally closer to the Sun (light source) when he jumps provided the Sun is directly above the character. This causes his shadow to not only seemingly detach from the model but to grow slightly bigger and perhaps get skewed a little further. In some cases, the hardest shadows are those caused by objects placed steadily on a flat plane. In retrospect, a shadow may even be unnoticeable should the light source be directly over the character's head. At least until the character or gets close/farther away from the light source. 

With this mentioned, let us think about the effect of shadows in games. A clear indication of dynamic movement can be created through the use of shadows. This is an important consideration when talking about color palettes. Should a character's boots or clothing happen to be the same color as the ground they are standing upon (a very poor choice in palette on the part of the artist), then it is very possible that physics isn't clear enough without shadows. This is a major problem in our game right now, there is no definitive way of demonstrating our character jumping and it more or less looks like he is crouching and extending his legs without moving. 

Shadows give us a good understanding of the distance the player is from not only the light source but other objects in the environment. Say a boulder is rolling towards the player, it is more likely that the player will see it coming if it has a moving shadow being projected below it. This is especially true of dynamic flying objects in scenes. Often spells or smaller weapons such as daggers and shurikens would be more noticeable from certain perspectives where the shadows are evident. Sometimes dynamic lighting in combination with detailed shadows have the potential to create tension or atmosphere within a game. A simple shadow of blood splatter or a creature running through a ventilation shaft can quickly and easily intensify the mood. For a great example, let's take a look at the following video walkthrough of Dead Space. Here, Isaac walks into a loud room with violently shifting shadows. The lighting along with the horrific sounds are a great example of how shadows can affect a game's atmosphere effectively. 

WARNING: The following video contains graphic violence, profanity and mature themes, viewer discretion is advised. 

*Feel free to skip to 3:18 to see the loud room. 

Shady Games

Now let's move on to the real important and fun stuff, shadows and how they're rendered in games. To begin, let us understand what game developers have to contend with on a daily basis. At 30 frames per second, a frame is allowed roughly 33 ms to render on screen. Splitting that time quotient we see that 13 ms are used to render the geometry and shaders while the post processing takes around 10 ms, leaving 10 ms left for lighting and shadow calculations. The creators of Killzone 3 divide these 10 ms up by dedicating 5 ms on shadow map rendering and 3 ms on physically correct light calculations. This leaves 2 ms for any kind of shadow filtering they choose to implement within the game. Like many games, Killzone 3 uses shadow techniques such as percentage closer filtering, variance shadow maps and cascaded shadow maps. A few special games make use of specific tricks to render their shadows. Take for example Media Molecule and their popular LittleBigPlanet franchise.

These guys make use of all kinds of techniques that seem like complete giberish to me...for now. The creators of LittleBigPlanet decided to take a unique approach to shadows and voxelize their scenes every frame to compute the irradiance volume and use raycasting to render the volumetric shadows and ambient occlusion. This stuff is quite complex, yet, they managed to do it all using PS3 hardware at that! 

Killzone 3, on the other hand remained true to standard shadow rendering practices, managing to run at 720p and 30FPS on the PS3. 

Making use of a deferred renderer and a cascaded shadow map renderer, they were able to have 10 or more shadow casting lights on screen. In addition, they reused single shadow map which allowed their transparent objects to have no shadows. Using SPUs meant the number of objects on screen was generally not a problem with 5000 objects rendered per frame. Keeping it simple, they determined whether each object had a big enough contribution to the shadow (based on the bounding sphere size) and if the size was below a certain threshold, they wouldn't render it into the shadow map, thereby preventing visual glitches. Local lights half the shadow map resolution based on the distance of the light and shadows are faded out fairly aggressively based on either the size or distance of the light. Depth bounds are used, as in most PS3 games, to check during light rendering. 

Besides all this interesting stuff, the team's usage of pre-rendered sunlight shadows is probably the most impressive. They store a number of these pre-rendered shadows in one of the lightmap channels for each object and end up mixing the pre-rendered shadows with real-time ones. Pre-rendered shadows also enable them to quickly skip the pixels that are completely in shade during the sunlight rendering. 

Games such as InFamous 2 use very similar methods for shadow rendering since the shadow mapping techniques used in Killzone 3 are fairly basic. Let's take a look at InFamous' specialized lighting setup for cutscenes. 

The engine designers of InFamous 2, made use of perspective shadow maps aimed at tight boxes around the head for head closeups. Characters and enemies in cutscenes typically use a special 4 light rig: one light casts detailed shadows while 3 lights are non shadowed. The entire setup is rendered as a fullscreen pass and uses depth bounds and stencil tests to limit the effect on the character or enemy. Interiors use another special setup where only the 4 closest spotlights cast shadows. This causes shadows on characters to be much more precise than the ones in the background of the scene. 

Breakthroughs in shadow mapping and lighting can be seen in games like God of War 3 where the Ready At Dawn engine aims for a lot of lights per frame with a high amount of shadows lights. Since most of the lights are static, they implemented a very efficient offline shadow culling process. Games like Far Cry 3 use custom shadow rendering solutions which map well to large open worlds. They use cascaded shadow maps for closer distances and blend to the long range shadows after 40 metres. The interesting part in all this is that the entire world is split into 64 x 64 metre tiles around the player as 144 tiles are loaded at the same time on the console. Long range shadows are stored in a single top-down 768 x 768 shadow map texture with 32 x 32 tiles associated with each sector which stores the sunlight occlusion height. 

Imperfect Shadows

Before I leave to take refuge in my cave...because clearly I AM the night. I want to touch on an interesting concept I stumbled upon while researching shadow mapping for this blog post. I'm talking about the concept of imperfect shadow maps and I feel that it is probably more relevant to game developers in our program, considering it uses clever methods for interactive computations of illumination. 

The Last of Us made exceptional use of imperfect shadow mapping by computing indirect illumination in large and fully dynamic scenes based on approximate visibility queries. Since the high-frequency nature of direct lighting require accurate visibility, indirect illumination takes care of this as it consists of mostly smooth gradations which tend to mask errors due to incorrect visibility. Imperfect shadow maps exploit this by approximating visibility for indirect illumination. Low resolution shadow maps are rendered from a crude point-based representation of the scene. These, in conjunction with with a global illumination algorithm based on virtual lights, enables indirect illumination of dynamic scenes at real-time frame rates. At the end of the day (ahem), imperfect shadow maps are a valid approximation of visibility, which makes the simulation of global illumination considerably faster than using accurate visibility. 

This concept, along with thorough examples, is portrayed within the following video.

Conclusion


Most of this information touches on advanced topics that we may or may not cover as part of the course syllabus in Intermediate Graphics, but it is still fascinating to learn about all these techniques and how award-winning titles use methods we learn on a day-to-day basis. The world of shadows, including shadow mapping and ambient occlusion, is broad and spectacular. It is very interesting and is a world that is constantly expanding as more and more games explore the various avenues and manage to continually push the limits of our gaming consoles to bring us bold and beautiful graphical achievements using complex algorithms and clever tricks to captivate our minds leaving us wondering...how on EARTH do they do that? AND the game manages to run smoothly? SORCERY!