shadow mapping like a pro

[lank]

a brief recipe for the creation of custom shadow maps that will blow away anything produced in loathing.

ingredients:

1 x photoshop or similar
1 x 3D program capable of applying displacement maps to meshes (e.g. bryce, or any respectable program)
1 x mesh complete with displacement map, light sources and models in loathing

you may also need spite and malice, old tools that can extract myth model/geometry data and export to a format called meshwork, which is available for mac os x and classic. the latest release of meshwork is from 2003, so compatibility with newer versions of mac os x is not guaranteed by me.

spite and malice (available on the tain) require mac os classic to run, which on newer macs without the classic emulator or on any other platform requires sheepshaver. similarly, classic emulation may allow you to run meshwork for classic if it doesn't work on your version of os x.

you will have to do the tedious work of exporting and converting all the models before you really begin. and don't ask about shadows on animated models - i'm afraid i don't have that worked out.

method:

in loathing, make sure that the "range" values under the displacement maps menu exceed the maximum height and depth of any vertices on your mesh, otherwise you may see nasty results (the tops of hills may become the deepest pits, that sort of thing) when you export. keep track of these numbers, because you'll want them later.

once your displacement map exports without any truncation, also export the media displacement (if you have any media).

now at this stage there are some different paths you can take. is your map indoors or outdoors? does it have media or not?

dealing with media can be tricky, but it doesn't take too much effort, so let's cover that first.

back in loathing, as well as the media displacement, export the media mask. in photoshop, use the media mask to make transparent everything that is not media in the media displacement map. take those pixels and overlay them on the terrain displacement map and then you'll have something you can safely use in your 3D program as if you didn't have media at all. this technique may also show you if you don't have your media and mesh heights matching up at the water's edge - fix that now and get back to this stage.

now you're ready for the 3D work. you know the dimensions of your map and you know the range values from when you exported the displacement map. divide the length and width of your map by 8 to get its size in myth world units and subtract the lower mesh range from the higher to get its total height. create a mesh to which you can apply the displacement map and give it these dimensions so that it exactly resembles the map in myth.

a point of subtlety here is the resolution of the mesh itself. if you set the mesh resolution too high you will see distinct stepping in your render. this is bad. if, however, you set it too low, you will lose detail. also bad. the ideal is clearly to have the mesh grid as close myth as possible, and then subdivide it as desired if necessary. the mechanics of this depend on the program you use. for instance in bryce it is impossible to create anything other than a square terrain, so use your best judgement.

i speak of subdividing above because although the mesh has only a certain resolution, there are in fact 8 pixels per mesh cell in the final colour and shadow maps. so if you need to, you can fake the effect of geometry in myth by actually putting it into this mesh. this results in an effect similar to bump mapping when you finally get it into myth (but be careful not to overdo it).

so. now that you have your basic mesh set up and sized correctly in your 3D program, you'll need to import and possibly rescale and rotate any/all models, then sit them just so on the mesh, so they cast the correct shadows. i'll note now that some models may require you to re-render without them in place, in cases where they extend past their own feet (e.g. caves and some statues). now make sure everything has uniform texture - grey, of about 50-70% brightness, and turn down the ambient light levels.

and now we come back to an earlier choice: are you working outdoors or indoors? if the former, is it day or night or somewhere in between?

again because it's quicker, let's start outdoors in the daylight. this is simple: place a parallel light (or sun light) at an azimuth of 45°, measured counter clockwise from the right hand side - in other words, exactly in the top-right corner - at a height that reflects the time of day (but probably not at an altitude of much more than about 45-50°). if you like you can have whatever light effects you like - soft edges, whatever. for low light angles soft edges look really nice, but even at high angles it's nice if the edge isn't hard like really old-fashioned raytracers (unless that's the effect you want).

for night time, guess what? unless it's meant to be a cloudy night with a new moon, you do exactly the same - but you'll do post-processing differently, so pay attention then.

indoors, however, it's a different matter. for this i'm going to assume the level is a cave or deep inside a castle with no celestial light (though it's not difficult to add such effects - i'll leave it in your hands to figure out such embellishments).

presumably you have placed torches, camp fires and/or other local light sources on your map. for obvious reasons, you want to make sure that the light sources you place in your 3D program match up with those you placed in loathing. (alternatively you might want to work the other way around, which may be easier...)

if you've already placed your light sources, hope you've got enough. even if you don't, when i discuss the alternative above you'll be saved. go back to loathing and bring up the scenery palette, then go to the tool options and make sure it shows on the map view only the object you have selected. take screenshots of the map view for all light sources and then use photoshop to turn the little blue dots into white dots on a black background. use this as a texture on your 3D mesh, rendered so that the dots are all visible, and use them as guides to place your light sources (which i'll generally assume to be spherical lights).

when placing your lights, their height above the ground is important. suppose you have a torch on a tall stand next to a wall. the wall is going to have a bright spot on it that corresponds to the height of the torch. if you get this very far wrong, the results in game will look unsatisfactory. if you don't believe me, try it yourself. the height of the torch (as well as its overall brightness) affects how big and bright a spot it casts on the floor, too.

if you have the option, choose a 1/r^2 falloff for the light sources, though other falloff profiles may suit your purposes better. a common feature of local extended light sources (like candle or torch flames) is soft edges. although it takes longer to render, i strongly recommend it (but again, choose the strength of the effect, if you use it at all, to suit the effect you wish to create in your level). the actual brightness of your lights is also important - candles don't cast much light, torches cast a lot more. a dimly lit area can create a very scary mood.

now if you're working in the opposite direction, you can place your lights until the lighting is right and then put torches, etc. in the brightest spots when you get it into loathing (or if you need it to be exact, do the reverse of the above process, rendering dots where objects are placed and importing that as a temporary colour map, appropriately rescaled and formatted). this way is certainly easier, but the above works well for retrofitting shadow maps to levels you don't want to alter too much.

at last, render your map from the top down. if you don't have an exact top down renderer (bryce annoys me that way), set the zoom as long as you can and move the camera up so far that there's only a small border around the mesh - this reduces errors from parallax. ideally you should have your border set to a very distinct colour from the mesh, making it easy to crop, and also render at a size substantially larger than the colour map's dimensions, so that the most possible detail is retained.

a note about brightness. while it may suit your purposes to have areas that are "blown out" white or completely underexposed, as a general rule you probably don't want such when you render (the exception being places "between the walls" in underground/indoor maps, where black is best).

once you've got your render done, crop the edges and convert to greyscale. you don't need that colour information any more. it's up to you (and it may not make much difference) whether you resize before or after the next stage, which is the most important in getting the shadows to look right.

photoshop's "levels" tool is your best friend here. by using this to change the brightest and darkest parts of your rendered shadow map and the weighting in between, you can compare the greys you produce to those from a standard loathing shadow map (and admire the difference between them at this stage!) to create the best feel, be it harsh desert sun, gloomy pre-dawn light or maybe the bright full moon. indoors shadows from the terrain and models you've placed will create gloomy corners and nice places for mahir to hide.

at this stage you will probably do a lot of tweaking, importing and re-importing into loathing to make sure it looks right. run riot and see what you like.

if you're working on an outside sunlit map this will probably suffice, but anything else will require you to go into your mesh in fear and modify any or all of these values: light/dark fraction/colour (there are four separate values in that), transition point and global tint fraction/colour.

usually it suffices to leave the light/dark colours alone. these affect what colours the light and shadow become at the extremes. if you've ever imported a completely flat white colour map into loathing and added shadows (if you haven't, do it now) you'll see what it does. if you add a tall hill and recalculate the shadows you'll see more.

light/dark fraction determine the opacity with which the light/dark colours are applied. notice the dark fraction is near 1 and the light fraction is near 0. 1 means opaque colour, 0 means transparent.

next, transition point tells you what shade of grey (from 0 = black to 1 = white) at which the tint/shade of the shadow map will change. for night and underground maps it's perfectly ok to put this near or even over 1. i've never had cause to put the transition point below 0, but be creative - see what effects you can create.

next global tint. the global tint colour is applied over top of the collections of any units, scenery and models whose collection references are checked to use global tint. again, the global tint fraction is how transparent or opaque it is. for darker than average maps a global tint colour of black (or nearly black blue or orange, if you're in moonlight or fire light respectively) applied with a fraction of about 0.3 gives things the right "brightness" relative to the mesh itself in loathing and in game. nothing is worse than units which look like they're in daylight when the map is dark night.

at this stage you should also make sure that any models you use have global tinting turned on. in many cases, especially for bungie's models, this is not the case, and daylit models are as bad as daylit units and scenery.

back to units, especially if you are indoors, they may be walking through areas of both light and dark - but the global tint is fixed. 0.3 is a good intermediate value for the maps i have done, but you see what looks best on your levels.

if you've followed this tutorial up until now, you may be wondering what's left to do. rightly so - aside from minor adjustments to suit your aesthetics, you've covered all the most important details about making custom shadow maps and those that determine how a shadow map will look.

i will add, though, that if you are conscious of file size, you can use just the fear values discussed above to turn a daytime map into night. this can be useful if you are revisiting a level in a solo campaign or if you merely wish to add variety to a series of netmaps.

i hope you've enjoyed this tutorial, and i look forward to seeing your results!