Graphics Runner

New Prey 2 E3 Trailer

2011-06-04T20:56:00.002-04:00

Bethesda released a new trailer for Prey 2. I'm not gonna lie, it looks awesome.

Prey 2 Concept and Screenshot

2011-04-07T15:10:00.001-04:00

Bethesda posted a screenshot and concept piece from Prey 2 today: http://bethblog.com/index.php/2011/04/07/sneak-peek-at-prey-2/

Instant Radiosity using Optix and Deferred Rendering

2011-03-28T02:58:00.009-04:00

This comes a little later than I wanted, I hadn’t factored in Crysis 2 taking up as much of my time as it did last week :-)

I’ve been using Nvidia’s Optix raytracing API for quite some time, and decided that a good introduction to Optix and what it can do for you would be using it in an Instant Radiosity demo. The demo is fairly large so I won’t cover all of it here, as that would be entirely too long of a post, but just the main parts.

Instant Radiosity

Instant Radiosity is a global illumination algorithm that approximates the diffuse radiance of a scene by placing many virtual point lights that act as indirect light. The algorithm is fairly simple: for each light in the scene you cast N photon rays into the scene. At each intersection the photon either bounces and another ray is cast or, through Russian Roulette, is killed of. At each of the intersections you create a Virtual Point Light (VPL) that has the same radiance value as the photon. Once you have these VPLs you render them as you would any other light source.

One optimization that the demo makes is to divide the scene into a regular grid. For each grid voxel, we find all the VPLs in the voxel and merge them together to form a new VPL that represents the merged VPLs. Any voxels that don’t contain any VPLs are skipped. This dramatically reduces the number of VPLs that we need to render, and trades off indirect accuracy for speed. The following couple of shots demonstrate this idea. The image on the left shows the VPLs as calculated from our Optix program. The image on the right shows the merged VPLs.

Optix

Optix is Nvidia’s ray tracing API that runs on Nvidia GPUs (on G80 and up). Giving an overview to optix could take many blog posts so I won’t go that in depth here. There are a couple of SIGGRAPH presentations that give a good overview:

http://nvidia.fullviewmedia.com/siggraph2010/04-dev-austin-robison.html
http://graphics.cs.williams.edu/papers/OptiXSIGGRAPH10/

To create an optix program you essentially need two things: a ray generation program and a material program ( essentially a shader ) that gets called when a ray intersects geometry. The ray generation program does exactly as it sounds, it generates rays. The program is called for each pixel of your program’s dimensions. Rays cast by your ray generation program will traverse the scene for intersections, once a ray intersects geometry it will call its material program. The material program is responsible for say shading in a classic ray tracer, or any other computation you want to perform. In our case we’ll use it to create our Virtual Point Lights. So lets get down to business.
Here we have the ray generation program that will cast rays from a light. In the case of our cornell box room, we have an area light at the ceiling and we need to cast photons from this light.

RT_PROGRAM void instant_radiosity()
{
     //get our random seed
     uint2 seed = seed_buffer[ launch_index ];

     //create a random photon direction
     float2 raySeed = make_float2( ( (float)launch_index.x + rnd( seed.x ) ) / (float)launch_dim.x,
                                   ( (float)launch_index.y + rnd( seed.y ) ) / (float)launch_dim.y );

     float3 origin = Light.position;
     float3 direction = generateHemisphereLightPhoton( raySeed, Light.direction );

     //create our ray
     optix::Ray ray(origin, direction, radiance_ray_type, scene_epsilon );

     //create our ray data packet and launch a ray
     PerRayData_radiance prd;
     prd.radiance = Light.color * Light.intensity * IndirectIntensity;
     prd.bounce = 0;
     prd.seed = seed;
     prd.index = ( launch_index.y * launch_dim.x + launch_index.x ) * MaxBounces;
     rtTrace( top_object, ray, prd );
}

So here we cast a randomly oriented ray from a hemisphere oriented about the direction of the light. Once we have our ray, we setup a ray data packet that will collect data as this ray traverses the scene. To cast the ray we make a call to rtTrace, providing the ray and its data packet.

Next we have our material program. This program is called when a ray hits the closest piece of geometry from the light. And it is responsible for updating the ray data packet, placing a VPL, and deciding to cast another ray recursively if we’re under the maximum number of bounces.

RT_PROGRAM void closest_hit_radiosity()
{
     //convert the geometry's normal to world space
     //RT_OBJECT_TO_WORLD is an Optix provided transformation
     float3 world_shading_normal   = normalize( rtTransformNormal( RT_OBJECT_TO_WORLD, shading_normal ) );
     float3 world_geometric_normal = normalize( rtTransformNormal( RT_OBJECT_TO_WORLD, geometric_normal ) );
     float3 ffnormal     = faceforward( world_shading_normal, -ray.direction, world_geometric_normal );

     //calculate the hitpoint of the ray
     float3 hit_point = ray.origin + t_hit * ray.direction;

     //sample the texture for the geometry
     float3 Kd = norm_rgb( tex2D( diffuseTex, texcoord.x, texcoord.y ) );
     Kd = pow3f( Kd, 2.2f ); //convert to linear space
     Kd *= make_float3( diffuseColor ); //multiply the diffuse material color
     prd_radiance.radiance = Kd * prd_radiance.radiance; //calculate the ray's new radiance value

     // We hit a diffuse surface; record hit if it has bounced at least once
     if( prd_radiance.bounce >= 0 ) {
          //offset the light a bit from the hit point
          float3 lightPos = ray.origin + ( t_hit - 0.1f ) * ray.direction;
          VirtualPointLight& vpl = output_vpls[ prd_radiance.index + prd_radiance.bounce ];
          vpl.position = lightPos;

          //the light's intensity is divided equally among the photons. Each photon starts out with an intensity
          //equal to the light. So here we must divide by the number of photons cast from the light.
          vpl.radiance = prd_radiance.radiance * 1.0f / ( launch_dim.x * launch_dim.y );
     }

     //if we're less than the max number of bounces shoot another ray
     //we could also implement Russion Roulette here so that we would have a less biased solution
     prd_radiance.bounce++;
     if ( prd_radiance.bounce >= MaxBounces )
          return;

     //here we "rotate" the seeds in order to have a little more variance
     prd_radiance.seed.x = prd_radiance.seed.x ^ prd_radiance.bounce;
     prd_radiance.seed.y = prd_radiance.seed.y ^ prd_radiance.bounce;
     float2 seed_direction = make_float2( ( (float)launch_index.x + rnd( prd_radiance.seed.x ) ) / (float)launch_dim.x,
                                          ( (float)launch_index.y + rnd( prd_radiance.seed.y ) ) / (float)launch_dim.y );

     //generate a new ray in the hemisphere oriented to the surface
     float3 new_ray_dir = generateHemisphereLightPhoton( seed_direction, ffnormal );

     //cast a new ray into the scene
     optix::Ray new_ray( hit_point, new_ray_dir, radiance_ray_type, scene_epsilon );
     rtTrace(top_object, new_ray, prd_radiance);
}

With both of these programs created we need to launch our optix program in order to generate the VPLs. When we’re done running the optix program, we gather all the VPLs into a grid, merging lights that are in the same voxel. Once the VPLs are merged, we add them to the deferred renderer.

//run our optix program
mContext->launch( 0, SqrtNumVPLs, SqrtNumVPLs );

//get a pointer to the GPU buffer of virtual point lights.
VirtualPointLight* lights = static_cast< VirtualPointLight* >( mContext["output_vpls"]->getBuffer()->map() );

//the following block merges the scattered vpls into a structured grid of vpls
//this helps dramatically reduce the number of vpls we need in the scene
if( mMergeVPLs )
{
     //Here we traverse over the VPLs and we merge all the lights that are in a cell
     for( int i = 0; i < TotalVPLs; ++i ) 
     {
          optix::Aabb node = mBoundingBox;

          //start with the root cell and recursively traverse the grid to find the cell this vpl belongs to
          int index = 0;
          if( FindCellIndex( mBoundingBox, -1, mVoxelExtent, lights[ i ].position, index ) )
          {
               //make sure we found a valid cell
               assert( index >= mFirstLeafIndex );

               //subtract the first leaf index to find the zero based index of the vpl
               index -= mFirstLeafIndex;
               float3& light = mVPLs[ index ];
               light += lights[ i ].radiance;
          }
     }

     //once the VPLs have been merged, add them to the renderer as indirect lights
     int numLights = 0;
     int lastIndex = -1;
     for( int i = 0; i < mVPLs.size(); ++i ) 
     {
          const float3& vpl = mVPLs[i];
          if( dot( vpl, vpl ) <= 0.0f )
               continue;

          numLights++;
          float3 radiance = vpl;
          D3DXVECTOR3 pos = *(D3DXVECTOR3*)&mVoxels[i].center();

          Light light =    {    LIGHT_POINT,                                                //type
                                GetColorValue(radiance.x, radiance.y, radiance.z, 1.0f),    //diffuse
                                pos,                                                        //pos
                                Vector3Zero,                                                //direction
                                1.0f                                                        //intensity
                           };

          renderer->AddIndirectLight( light );

          //also add as a light source so we can visualize the VPLs
          LightSource lightSource;
          lightSource.light = light;
          lightSource.Model = mLightModel;
          renderer->AddLightSource( lightSource );
     }
}

Now for some eye candy. The first set are your typical cornell box + dragon. In the Instant Radiosity shot you can see the light bleeding from the green and red walls onto the floor, the dragon and the box.

Direct lighting:

Direct Lighting + Indirect VPLs:

Direct Lighting:

Direct Lighting + Indirect VPLs:

The next set is from the sponza scene. Here too you can notice the red bounced light from the draperies onto the floor and in the ambient lighting in the shadows.

Direct Lighting:

Direct Lighting + Indirect VPLs:

Direct Lighting:

Direct Lighting + Indirect VPLs:

Direct Lighting:

Direct Lighting + Indirect VPLs:

Notes

To Build the demo you’ll need boost 1.43 or later. To run the demo you’ll need at least an Nvidia 8800 series or later ( anything Computer 1.0 compliant ).

Files of interest are in the Demo project: OptixEntity.cpp and InstantRadiosity.cu.

Controls:
Show VPLs : L
Toggle GI : I
Toggle Merge VPLs : M

Download:

Sorry for requiring two download links but skydrive limits file sizes to 50MB
OptixInstantRadiosity Part 1 - Code
OptixInstantRadiosity Part 2 - Assets

Instant Radiosity with Optix

2011-03-19T16:29:00.002-04:00

I've been working on a new sample for the past few days, Instant Radiosity using Optix and DirectX. I should have a writeup and sample in the coming week.

Here’s a few shots with the obligatory cornell and sponza scenes.

Direct Lighting

Direct + Indirect VPLs

Direct Lighting

Direct + Indirect VPLs

Prey 2 Teaser Trailer

2011-03-15T20:11:00.002-04:00

Human Head has been keeping me busy since I've been working there and I can finally say why: Prey 2. The game was announced on Monday :) Here's the Prey 2 teaser trailer:

Xbox 360 Games - E3 2011 - Prey 2

http://www.g4tv.com/videos/51782/Prey-2-Debut-Trailer---Exclusive-Premiere/

Animating Water Using Flow Maps

2010-08-03T04:23:00.004-04:00

Last week I attended SIGGRAPH 2010, and among the many good presentations, Valve game a talk on the simple water shader they implemented for Left For Dead 2 and Portal 2. So on the plane ride back from LA, I whipped up this little sample from what I could remember of the talk. Edit: You can find the talk here: http://advances.realtimerendering.com/s2010/index.html

The standard technique for animated water is scrolling normal maps, as I’ve previously written about. The problem with this is that it looks unnatural as water does not uniformly move in one direction. So Valve came up with the idea of using flow maps ( based on a flow viz paper from the mid 90s ). The basic idea of flow maps is that you create a 2D texture that you will map to your water. And this map will contain the flow directions that you want the water to flow, with each pixel in the flow map representing a flow vector. This allows you to have varying velocity ( based on length of the flow vector ), and varying flow directions ( based on the color of the flow vector ). You then use this flow map to alter the texture coordinates of the normal maps instead of scrolling them. Lets get to work :)

The Flow Map

First we need to create a flow map. Here’s what I came up with in a couple of minutes in Photoshop. This flow map was designed around the column with dragon scene as with the previous scene. Note, this flow map is greatly exaggerated to demonstrate the effect.
Using The Flow Map

Now we need to use the flow map to alter the water normal maps. We do this by taking the texture coordinate of the current water pixel and offset it using the flow vector from the flow map based on a time offset. We then render the water as we did in the previous water sample. But there’s a problem with this, after awhile the texture coordinates will become so distorted that the normal maps will be stretched and will have nasty filtering artifacts. So to solve this we limit the amount of distortion of the texture coordinates by resetting the time offset. This solves the over-distortion, but now the water will reset every X seconds. So we introduce another layer, that is offset from the first by half a time cycle. This will ensure that while one layer is fading out and beginning to reset, the next layer is fading to where the last layer was. Here’s a diagram to visualize this phase-in phase-out of the 2 layers.

The graph illustrates that during a cycle time from 0 to 1, we want the layer to be fully interpolated at the mid-point in the cycle, and fully un-interpolated at 0 and 1. Lets see the code:

//get and uncompress the flow vector for this pixel
float2 flowmap = tex2D( FlowMapS, tex0 ).rg * 2.0f - 1.0f;

float phase0 = FlowMapOffset0;
float phase1 = FlowMapOffset1;

// Sample normal map.
float3 normalT0 = tex2D(WaveMapS0, ( tex0 * TexScale ) + flowmap * phase0 );
float3 normalT1 = tex2D(WaveMapS1, ( tex0 * TexScale ) + flowmap * phase1 );

float flowLerp = ( abs( HalfCycle - FlowMapOffset0 ) / HalfCycle );
float3 offset = lerp( normalT0, normalT1, flowLerp );

In the code above, HalfCycle would be .5 if our cycle was from 0 to 1. We can see here that we unwrap the flow vector (as it is stored in [0,1] and we need it in [-1,1]), fetch the normals using the flow vector and then lerp between the two normals based on the cycle time. This however will lead to a subtle pulsing affect, which I couldn’t really notice when the water was rendered, but I included the fix for completeness. To fix this pulsing effect, we perturb the flow cycle at each pixel using a noise map.

//get and uncompress the flow vector for this pixel
float2 flowmap = tex2D( FlowMapS, tex0 ).rg * 2.0f - 1.0f;
float cycleOffset = tex2D( NoiseMapS, tex0 ).r;

float phase0 = cycleOffset * .5f + FlowMapOffset0;
float phase1 = cycleOffset * .5f + FlowMapOffset1;

// Sample normal map.
float3 normalT0 = tex2D(WaveMapS0, ( tex0 * TexScale ) + flowmap * phase0 );
float3 normalT1 = tex2D(WaveMapS1, ( tex0 * TexScale ) + flowmap * phase1 );

float flowLerp = ( abs( HalfCycle - FlowMapOffset0 ) / HalfCycle );
float3 offset = lerp( normalT0, normalT1, flowLerp );

And that’s pretty much it. I’ll update the post/source when the slides are posted from SIGGRAPH in case I left anything out. Video time!

Source/Demo:

Volume Rendering 202: Shadows and Translucency

2010-05-06T13:00:00.008-04:00

Finally, here is the last sample on volume rendering. It’s only taking me a year to get around to finishing it. Is anyone even visiting this page anymore? I’d better post this for my sanity anyhow.
Last time I left you with some basic optimizations, one being a pseudo-empty space skipping. But as I noted, the volumes needed to be sorted in order for it to work completely. We sort the sub-volumes back to front with respect to distance to the camera. This insures that we have a smooth framerate no matter what angle the camera is at. A speedup we can do here is to only sort the volumes if the camera has moved 45 degrees since we last sorted.
So now our subvolumes are sorted w.r.t. the camera. But we have alpha blending artifacts because depending on the view, the pixels of the subvolumes are not drawn in the correct order. What we can do to fix this is to draw a depth only pass, and ensure that we only draw pixels that will contribute to the final image.

Left: no depth prepass. Right: depth prepass

Translucency

The first sample includes an approximated translucency. It is far from realistic, but it gives fairly good results. The idea is very similar to depth mapping, compare the current pixels depth to that of the depth map, and either use this value to look up into a texture or perform an exponential falloff in the shader (the sample does the latter).

Shadows

There isn’t much to say here. The sample below uses variance shadow mapping.

Well, there it is. Anticlimactic wasn't it?

Ground control to Major Tom

2010-05-05T22:15:00.004-04:00

Wow, it’s been over a year since the last post on volume rendering! I must sound like a broken record. Anyhow, I’ve had time to fix a couple of bugs with the last installment in the past couple of weeks and it should be coming online pretty soon.

So why have I been absent lately? Last spring I was recruited to work on American Sign Language teaching software for Purdue University. The project ranged from database implementation, to layered skeletal animation with additive blending support and facial animation, to creating a language and compiler for ASL scripts (Antlr was amazing for this). Also, our paper was accepted at SIGGRAPH in the Education section.

On top of that I accepted a job at Human Head Interactive in January as a tech programmer ( these ramblings actually paid off :) ). I’m really excited to be working with some smart and talented people. We have some cool rendering tech – thanks to our lead graphics programmer – and pretty slick game play ideas.

Also, sorry to anyone who has commented on a post and it hasn't been posted, I've been spammed by bots for awhile now.

Water for your monies?

2010-04-10T01:38:00.012-04:00

I got an email a couple of weeks ago from someone ( Maximinus ) who actually put the water shader from the water game component to good use. Here's a description of the game on the xbox indie marketplace:

Missile Escape for Xbox Indies is simple : go flying, evade many
missiles and unlock new fighters along the way ! Warning : Fighter
pilot spirit required.

Particle Spectrum Animation

2009-08-27T02:45:00.003-04:00

Really cool video I found last night. Uses Particular and Adobe After Effects.

Water in Your Browser

2009-07-30T18:04:00.003-04:00

Recently I’ve been playing around with O3D. If you don’t know, O3D is Google’s new browser graphics API. It enables you to develop 3d interactive applications that run inside a browser window (and quite easily mind you). In fact it rivals XNA on getting an app up and running quickly.

And on that note, I’ve ported the water sample to O3D (minus the camera animation). Besides a bug I encountered with the sample content converter (and promptly fixed by one of the o3d developers), it was a relatively painless conversion. All that was required was to setup a scene in max, apply materials, export to collada and convert to the o3d format. Setting up the render targets also took minimal effort :). The shaders, for the most part, remained untouched.

Click the picture to have a go!

If you’re interested in the max file or source you can get both here:

Non-photorealistic SSGI

2009-06-19T00:58:00.009-04:00

I recently got back from my vacation to the Mediterranean and finally have had time to post this video.

For one of my final projects last semester I implemented a deferred renderer with SSAO and simplified global illumination (SSGI). But I wanted to have a dream-like result, so I over emphasized the color bleeding and used the light accumulation buffer plus emissive color of objects; and lots of blurring. The result looks pretty neat I think, not at all accurate lighting but cool none the less. This one was done in C++/DirectX instead of the usual XNA fare.

Large HD video on youtube

Volume Rendering Update, DirectX blog

2009-04-23T03:05:00.003-04:00

Wow, I can't believe it's almost May already. I also can't believe my last post was back in February! I've had the next installment in the volume rendering series basically done since January. I just have to put some finishing touches on it and write the post. I've just been very busy finishing my last semester, research, searching for a job, and considering graduate school.

I've been working on a deferred renderer and delving into screen space ambient occlusion and global illumination for one of my final projects. I'll probably post some images of the final result when it's done.

Ysaneya has a very interesting post on deferred rendering and instant radiosity.

Also, if you haven't noticed, the DirectX team has a new blog: http://blogs.msdn.com/DirectX/

Volume Rendering 201: Optimizations

2009-02-04T22:42:00.007-05:00

The discussion on hand this time is optimizing the performance of the volume renderer. I’ll cover a few of these optimizations and provide a rough implementation of one of them.

Cache Efficiency and Memory Access

Picture from Real-time Volume Graphics.

Currently we load our volume data into a linear layout in memory. However, the ray that is cast through the volume is not likely to access neighboring voxels as we traverse it through the volume when the data is in a linear layout. But we can improve the cache efficiency by converting the layout to a block-based manor through swizzling. With the data in a block format, the GPU is more likely to cache neighboring voxels as we walk through the volume, which will lead to an increase in memory performance.

Empty-Space Leaping

In the previous samples we ray-casted against the entire bounding volume of the data set, even if we were just sampling samples with zero alpha along the way. But we can skip these samples all together, and only render parts of the volume that have a non-zero alpha in the transfer function. More on this in a bit.

Occlusion Culling

If we render the volume in a block-based fashion as above, and sort the blocks from front to back, we can use occlusion queries to determine which blocks are completely occluded by blocks in front of them. There are quite a few tutorials on occlusion queries on the net, including this one at ziggyware.

Deferred Shading

We can also boost performance by deferring the shading calculations. Instead shading every voxel during the ray-casting, we can just output the depth and color information into off-screen buffers. Then we render a full-screen quad and use the depth information to calculate normals in screen space and then continue to calculate the shading. Calculating normals this way also has the advantage of being smoother and have less artifacts that computing the gradients of the volume and storing them in a 3D texture. We also save memory this way since we don’t have to save the normals, only the isovalues in the 3D texture.

Image Downscaling

If the data we are rendering is low frequency (e.g. volumetric fog), we can render the volume into an off-screen buffer that is half the size of the window. Then we can up-scale this image during a final pass. This method is also included in the sample.

Implementing Empty-Space Leaping

To implement empty-space leaping we need to subdivide the volume into smaller volumes and we also need to determine if these smaller volumes have an opacity greater than zero. To subdivide the volume we follow an approach very similar to quadtree or octree construction. We start out with an original volume from [0, 0, 0] to [1, 1, 1]. The volume is then recursively subdivided until the volume width is say .1 (so we basically divide the volume along each dimension by 10). Here’s how we do that:

private void RecursiveVolumeBuild(Cube C)
{
     //stop when the current cube is 1/10 of the original volume
     if (C.Width <= 0.1f)
     {
          //add the min/max vertex to the list       
          Vector3 min = new Vector3(C.X, C.Y, C.Z);       
          Vector3 max = new Vector3(C.X + C.Width, C.Y + C.Height, C.Z + C.Depth);       
          Vector3 scale = new Vector3(mWidth, mHeight, mDepth);       
          //additively sample the transfer function and check if there are any       
         //samples that are greater than zero       
         float opacity = SampleVolume3DWithTransfer(min * scale, max * scale);       
         if(opacity > 0.0f)       
         {                          
               BoundingBox box = new BoundingBox(min, max);         
               //add the corners of the bounding box      
               Vector3[] corners = box.GetCorners();         
               for (int i = 0; i < 8; i++)         
               {           
                    VertexPositionColor v;           
                    v.Position = corners[i];           
                    v.Color = Color.Blue;           
                    mVertices.Add(v);         
               }       
          }     
          return;
     }

     float newWidth = C.Width / 2f;
     float newHeight = C.Height / 2f;
     float newDepth = C.Depth / 2f;

     ///  SubGrid        r  c  d
     ///  Front:
     ///  Top-Left    :  0   0   0
     ///  Top-Right   :  0   1   0
     ///  Bottom-Left :  1   0   0
     ///  Bottom-Right:  1   1   0
     ///  Back:
     ///  Top-Left    :  0   0   1
     ///  Top-Right   :  0   1   1
     ///  Bottom-Left :  1   0   1
     ///  Bottom-Right:  1   1   1
     for (float r = 0; r < 2; r++)
     {
          for (float c = 0; c < 2; c++)          
          {
                for (float d = 0; d < 2; d++)       
                {         
                     Cube cube = new Cube(C.Left + c * (newWidth),         
                                          C.Top + r * (newHeight),     
                                          C.Front + d * (newDepth),         
                                          newWidth,         
                                          newHeight,        
                                          newDepth);         
                                          RecursiveVolumeBuild(cube);            
                }         
          }
     }
}

To determine whether a sub-volume contains any samples that have opacity, we simply loop over the volume and additively sample the transfer function:

private float SampleVolume3DWithTransfer(Vector3 min, Vector3 max)
{
     float result = 0.0f;

     for (int x = (int)min.X; x <= (int)max.X; x++)
     {
          for (int y = (int)min.Y; y <= (int)max.Y; y++)       
          {         
               for (int z = (int)min.Z; z <= (int)max.Z; z++)         
               {           
                    //sample the volume to get the iso value           
                    //it was stored [0, 1] so we need to scale to [0, 255]          
                    int isovalue = (int)(sampleVolume(x, y, z) * 255.0f);           
                    //accumulate the opacity from the transfer function          
                    result += mTransferFunc[isovalue].W * 255.0f;         
               }      
          }     
     }

     return result;
}

Depending on the transfer function (a lot of zero opacity samples), this method can increase our performance by 50%.

Problems

Now, a problem that this method introduces is overdraw. You can see the effects of this when rotating the camera to view the back of the bear; here the frame rate drops considerably. To remedy this the sub-volumes need to be sorted front to back by their distance to the camera each time the view changes. I’ve left this as an exercise for the reader. The new demo implements empty space leaping and downscaling. And when rendering the teddy bear volume frame rates on my 8800GT are at about 190 FPS. Compare this to the the last demo from 102 at 30 FPS. All at a resolution of 800x600. Pretty good results! Next time I’ll be introducing soft shadows and translucent materials.

References: Real-time Volume Graphics

Volume Rendering 102: Transfer Functions

2009-01-20T18:59:00.003-05:00

Last time I introduced the concept of volume ray-casting. And to follow that up, today I'll talk about transfer functions and shading.

Mummy (top) and Male (bottom) volumes colored with a transfer function (left) and shaded (right).

A transfer function is used to assign RGB and alpha values for every voxel in the volume. A 1D transfer function maps one RGBA value for every isovalue [0, 255]. Multi-dimensional transfer functions allow multiple RGBA values to be mapped to a single isovalue. These are however out of the scope of this tutorial, so I will just focus on 1D transfer functions.

The transfer function is used to "view" a certain part of the volume. As with the second set of pictures above, there is a skin layer/material and there is a skull layer/material. A transfer function could be designed to just look at the skin, the skull, or both (as is pictured). There are a few different ways of creating transfer functions. One is two manually define the transfer functions by specifying the RGBA values for the isovalues (what we will be doing), and another is through visual controls and widgets. Manually defining the transfer function is a lot like guess work and takes a bit of time, but is the easiest way to get your feet wet. Visually designing transfer functions is the easiest way to get good results quickly as it happens at run-time. But this method is quite complex to implement (at least for a tutorial).

Creating the transfer function:

To create a transfer function, we want to define the RGBA values for certain isovalues (control points or control knots) and then interpolate between these values to produce a smooth transition between layers/materials. Our transfer function will result in a 1D texture with a width of 256.

First we have the TransferControlPoint class. This class takes an RGB color or alpha value for a specific isovalue.

public class TransferControlPoint
{
    public Vector4 Color;
    public int IsoValue;

    /// <summary>
    /// Constructor for color control points.
    /// Takes rgb color components that specify the color at the supplied isovalue.
    /// </summary>
    /// <param name="x"></param>
    /// <param name="y"></param>
    /// <param name="z"></param>
    /// <param name="isovalue"></param>
    public TransferControlPoint(float r, float g, float b, int isovalue)
    {
        Color.X = r;
        Color.Y = g;
        Color.Z = b;
        Color.W = 1.0f;
        IsoValue = isovalue;
    }

    /// <summary>
    /// Constructor for alpha control points.
    /// Takes an alpha that specifies the aplpha at the supplied isovalue.
    /// </summary>
    /// <param name="alpha"></param>
    /// <param name="isovalue"></param>
    public TransferControlPoint(float alpha, int isovalue)
    {
        Color.X = 0.0f;
        Color.Y = 0.0f;
        Color.Z = 0.0f;
        Color.W = alpha;
        IsoValue = isovalue;
    }
}

This class will represent the control points that we will interpolate. I've added two lists to the Volume class, mAlphaKnots and mColorKnots. These will be the list of transfer control points that we will setup and interpolate to produce the transfer function. To produce the result for the Male dataset above, here are the transfer control points that we will define:

mesh.ColorKnots = new List<TransferControlPoint> {
                        new TransferControlPoint(.91f, .7f, .61f, 0),
                        new TransferControlPoint(.91f, .7f, .61f, 80),
                        new TransferControlPoint(1.0f, 1.0f, .85f, 82),
                        new TransferControlPoint(1.0f, 1.0f, .85f, 256)
                        };

mesh.AlphaKnots = new List<TransferControlPoint> {
                        new TransferControlPoint(0.0f, 0),
                        new TransferControlPoint(0.0f, 40),
                        new TransferControlPoint(0.2f, 60),
                        new TransferControlPoint(0.05f, 63),
                        new TransferControlPoint(0.0f, 80),
                        new TransferControlPoint(0.9f, 82),
                        new TransferControlPoint(1f, 256)
                        };

You need to specify at least two control points at isovalues 0 and 256 for both alpha and color. Also the control points need to be ordered (low to high) by the isovalue. So the first entry in the list should always be the RGB/alpha value for the zero isovalue, and the last entry should always be the RGB/alpha value for the 256 isovalue. The above list of control points produce the following transfer function after interpolation:

So we have defined a range of color for the skin and a longer range of color for the skull/bone.

But how do we interpolate between the control points? We will fit a cubic spline to the control points to produce a nice smooth interpolation between the knots. For a more in-depth discussion on cubic splines refer to my camera animation tutorial.

Here is a simple graph representation of the spline that is fit to the control points.

Using the transfer function:

So how do we put this transfer function to use? First we set it to a 1D texture and upload it to the graphics card. Then in the shader, we simply take the isovalue sampled from the 3d volume texture and use that to index the transfer function texture.

value = tex3Dlod(VolumeS, pos);     

src = tex1Dlod(TransferS, value.a);

Now we have the color and opacity of the current sample. Next, we have to shade it. While this is simply diffuse shading I will go over how to calculate gradients (aka normals) for the 3D volume.

Calculating Gradients:

The method we will use to calculate the gradients is the central differences scheme. This takes the last and next samples of the current sample to calculate the gradient/normal. This can be performed at run-time in the shader, but as it requires 6 extra texture fetches from the 3D texture, it is quite slow. So we will calculate the gradients and place them in the RGB components of our volume texture and move the isovalue to the alpha channel. This way we only need one volume texture for the data set instead of two: one for the gradients and one for the isovalues.

Calculating the gradients is pretty simple. We just loop through all the samples and find the difference between the next and previous sample to calculate the gradients:

/// <summary>
/// Generates gradients using a central differences scheme.
/// </summary>
/// <param name="sampleSize">The size/radius of the sample to take.</param>
private void generateGradients(int sampleSize)
{
    int n = sampleSize;
    Vector3 normal = Vector3.Zero;
    Vector3 s1, s2;

    int index = 0;
    for (int z = 0; z < mDepth; z++)
    {
        for (int y = 0; y < mHeight; y++)
        {
            for (int x = 0; x < mWidth; x++)
            {
                s1.X = sampleVolume(x - n, y, z);
                s2.X = sampleVolume(x + n, y, z);
                s1.Y = sampleVolume(x, y - n, z);
                s2.Y = sampleVolume(x, y + n, z);
                s1.Z = sampleVolume(x, y, z - n);
                s2.Z = sampleVolume(x, y, z + n);

                mGradients[index++] = Vector3.Normalize(s2 - s1);
                if (float.IsNaN(mGradients[index - 1].X))
                    mGradients[index - 1] = Vector3.Zero;
            }
        }
    }
}

Next we will filter the gradients to smooth them out and prevent any high irregularities. We achieve this by a simple NxNxN cube filter. A cube filter simply averages the surrounding N^3 - 1 samples. Here's the code for the gradient filtering:

/// <summary>
/// Applies an NxNxN filter to the gradients.
/// Should be an odd number of samples. 3 used by default.
/// </summary>
/// <param name="n"></param>
private void filterNxNxN(int n)
{
    int index = 0;
    for (int z = 0; z < mDepth; z++)
    {
        for (int y = 0; y < mHeight; y++)
        {
            for (int x = 0; x < mWidth; x++)
            {
                mGradients[index++] = sampleNxNxN(x, y, z, n);
            }
        }
    }
}

/// <summary>
/// Samples the sub-volume graident volume and returns the average.
/// Should be an odd number of samples.
/// </summary>
/// <param name="x"></param>
/// <param name="y"></param>
/// <param name="z"></param>
/// <param name="n"></param>
/// <returns></returns>
private Vector3 sampleNxNxN(int x, int y, int z, int n)
{
    n = (n - 1) / 2;

    Vector3 average = Vector3.Zero;
    int num = 0;

    for (int k = z - n; k <= z + n; k++)
    {
        for (int j = y - n; j <= y + n; j++)
        {
            for (int i = x - n; i <= x + n; i++)
            {
                if (isInBounds(i, j, k))
                {
                    average += sampleGradients(i, j, k);
                    num++;
                }
            }
        }
    }

    average /= (float)num;
    if (average.X != 0.0f && average.Y != 0.0f && average.Z != 0.0f)
        average.Normalize();

    return average;
}

This is a really simple and slow way of filtering the gradients. A better way is to use a seperable 3D Gaussian kernal to filter the gradients.

Now that we have the gradients we just fill the xyz components of the 3D volume texture and put the isovalue in the alpha channel:

//transform the data to HalfVector4
HalfVector4[] gradients = new HalfVector4[mGradients.Length];
for (int i = 0; i < mGradients.Length; i++)
{
    gradients[i] = new HalfVector4(mGradients[i].X, mGradients[i].Y, mGradients[i].Z, mScalars[i].ToVector4().W);
}

mVolume.SetData<HalfVector4>(gradients);
mEffect.Parameters["Volume"].SetValue(mVolume);

And that's pretty much it. Creating a good transfer function can take a little time, but this simple 1D transfer function can still produce pretty good results. Here's a few captures of what this demo can produce:

A teapot with just transfer function:

The teapot with transfer function and shaded:

Now a CT scan of a male head. Just transfer function:

Now with shading:

And a mummy:

*The CT Male data set was converted from PVM format to RAW format with the V3 library. The original data set can be found here: http://www9.informatik.uni-erlangen.de/External/vollib/

Volume Rendering 101

2009-01-15T22:58:00.004-05:00

Pictures above from: http://www.cs.utah.edu/~jmk/simian/

There is quite a bit of documentation and papers on volume rendering. But there aren't many good tutorials on the subject (that I have seen). So this tutorial will try to teach the basics of volume rendering, more specifically volume ray-casting (or volume ray marching).

What is volume ray-casting you ask? You didn't? Oh, well I'll tell you anyway. Volume rendering is a method for directly displaying a 3D scalar field without first fitting an intermediate representation to the data, such as triangles. How do we render a volume without geometry? There are two traditional ways of rendering a volume: slice-based rendering and volume ray-casting. This tutorial will be focusing on volume ray-casting. There are many advantages over slice-based rendering that ray-casting provides; such as empty space skipping, projection independence, simple to implement, and single pass.

Volume ray-casting (also called ray marching) is exactly how it sounds. [edit: volume ray-casting is not the same as ray-casting ala Doom or Nick's tutorials] Rays are cast through the volume and is sample along equally spaced intervals. As the ray is marched through the volume scalar values are mapped to optical properties through the
use of a transfer function which results in an RGBA color value that includes the corresponding emission and absorption coefficients for the current sample point. This color is then composited by using front-to-back or back-to-front alpha blending.

This tutorial will focus specifically on how to intersect a ray with the volume and march it through the volume. In another tutorial I will focus on transfer functions and shading.

First we need to know how to read in the data. The data is simply scalar values (usually integers or floats) stored as slices [x, y, z], where x = width, y = height, and z = depth. Each slice is x units wide and y units high, and the total number of slices is equal to z. A common format for the data is to be stored in 8-bit or 16-bit RAW format. Once we have the data, we need to load it into a volume texture. Here's how we do the whole process:

//create the scalar volume texture
mVolume = new Texture3D(Game.GraphicsDevice, mWidth, mHeight, mDepth, 0,
                        TextureUsage.Linear, SurfaceFormat.Single);

private void loadRAWFile8(FileStream file)
{
    BinaryReader reader = new BinaryReader(file);

    byte[] buffer = new byte[mWidth * mHeight * mDepth];
    int size = sizeof(byte);

    reader.Read(buffer, 0, size * buffer.Length);

    reader.Close();

    //scale the scalar values to [0, 1]
    mScalars = new float[buffer.Length];
    for (int i = 0; i < buffer.Length; i++)
    {
        mScalars[i] = (float)buffer[i] / byte.MaxValue;
    }

    mVolume.SetData(mScalars);
    mEffect.Parameters["Volume"].SetValue(mVolume);
}

In order to render this texture we fit a bounding box or cube, that is from [0,0,0] to [1,1,1] to the volume. And we render the cube and sample the volume texture to render the volume. But we also need a way to find the ray that starts at the eye/camera and intersects the cube.

We could always calculate the intersection of the ray from the eye to the current pixel position with the cube by performing a ray-cube intersection in the shader. But a better and faster way to do this is to render the positions of the front and back facing triangles of the cube to textures. This easily gives us the starting and end positions of the ray, and in the shader we simply sample the textures to find the sampling ray.

Here's what the textures look like (Front, Back, Ray Direction):

And here's the code to render the front and back positions:

//draw front faces
//draw the pixel positions to the texture
Game.GraphicsDevice.SetRenderTarget(0, mFront);
Game.GraphicsDevice.Clear(Color.Black);

base.DrawCustomEffect();

Game.GraphicsDevice.SetRenderTarget(0, null);
//draw back faces
//draw the pixel positions to the texture
Game.GraphicsDevice.SetRenderTarget(0, mBack);
Game.GraphicsDevice.Clear(Color.Black);
Game.GraphicsDevice.RenderState.CullMode = CullMode.CullCounterClockwiseFace;

base.DrawCustomEffect();

Game.GraphicsDevice.SetRenderTarget(0, null);
Game.GraphicsDevice.RenderState.CullMode = CullMode.CullClockwiseFace;

Now, to perform the actual ray-casting of the volume, we render the front faces of the cube. In the shader we sample the front and back position textures to find the direction (back - front) and starting position (front) of the ray that will sample the volume. The volume is then iteratively sampled by advancing the current sampling position along the ray at equidistant steps. And we use front-to-back compositing to accumulate the pixel color.

float4 RayCastSimplePS(VertexShaderOutput input) : COLOR0
{
    //calculate projective texture coordinates
    //used to project the front and back position textures onto the cube
    float2 texC = input.pos.xy /= input.pos.w;
    texC.x =  0.5f*texC.x + 0.5f;
    texC.y = -0.5f*texC.y + 0.5f;
 
    float3 front = tex2D(FrontS, texC).xyz;
    float3 back = tex2D(BackS, texC).xyz;
 
    float3 dir = normalize(back - front);
    float4 pos = float4(front, 0);
 
    float4 dst = float4(0, 0, 0, 0);
    float4 src = 0;
 
    float value = 0;
 
    float3 Step = dir * StepSize;
 
    for(int i = 0; i < Iterations; i++)
    {
        pos.w = 0;
        value = tex3Dlod(VolumeS, pos).r;
             
        src = (float4)value;
        src.a *= .5f; //reduce the alpha to have a more transparent result 
         
        //Front to back blending
        // dst.rgb = dst.rgb + (1 - dst.a) * src.a * src.rgb
        // dst.a   = dst.a   + (1 - dst.a) * src.a     
        src.rgb *= src.a;
        dst = (1.0f - dst.a)*src + dst;     
     
        //break from the loop when alpha gets high enough
        if(dst.a >= .95f)
            break; 
     
        //advance the current position
        pos.xyz += Step;
     
        //break if the position is greater than <1, 1, 1>
        if(pos.x > 1.0f  pos.y > 1.0f  pos.z > 1.0f)
            break;
    }
 
    return dst;
}

And here's the result when sampling a foot, teapot with a lobster inside, engine, bonsai tree, ct scan of an aneurysm, skull, and a teddy bear:

So, not very colorful but pretty cool. When we get into transfer functions we will start shading the volumes. The volumes used here can be found at volvis.org

Notes:

Refer to the scene setup region in VolumeRayCasting.cs, Volume.cs, and RayCasting.fx for relevant implementation details.

Also, a Shader Model 3.0 card (Nvidia 6600GT or higher) is needed to run the sample.

Water Game Component

2008-11-13T17:32:00.019-05:00

I had some requests awhile back for a more independent water effect that I had provided in my camera animation tutorials. And just the other day I remembered that I had totally forgot about this (doh!). So here is a DrawableGameComponent for the water effect. Have a look here to see it in action.

To setup the component we need to fill out a WaterOptions object that will be passed to the water component.

WaterOptions options = new WaterOptions();
options.Width = 257;
options.Height = 257;
options.CellSpacing = 0.5f;
options.WaveMapAsset0 = "Textures/wave0";
options.WaveMapAsset1 = "Textures/wave1";
options.WaveMapVelocity0 = new Vector2(0.01f, 0.03f);
options.WaveMapVelocity1 = new Vector2(-0.01f, 0.03f);
options.WaveMapScale = 2.5f;
options.WaterColor = new Vector4(0.5f, 0.79f, 0.75f, 1.0f);
options.SunColor = new Vector4(1.0f, 0.8f, 0.4f, 1.0f);
options.SunDirection = new Vector3(2.6f, -1.0f, -1.5f);
options.SunFactor = 1.5f;
options.SunPower = 250.0f;

mWaterMesh = new Water(this);
mWaterMesh.Options = options;
mWaterMesh.EffectAsset = "Shaders/Water";
mWaterMesh.World = Matrix.CreateTranslation(Vector3.UnitY * 2.0f);
mWaterMesh.RenderObjects = DrawObjects;

So here will fill out various options such as width and height, cell spacing, the normal map asset names, etc. We then create the water component, and assign it the options object. We then provide the filename of the Water.fx shader, the water's position and we then assign its RenderObjects delegate a function that will be used to draw the objects in your scene.

The component tries to be relatively independent of how your represent your game objects. All that it asks for is that you provide a function that takes a reflection matrix. This function should go through the objects that you want to be reflected/refracted and combine the reflection matrix with the object's world matrix.

Here's an example of what your DrawObjects() function might look like.

private void DrawObjects(Matrix reflMatrix)
{
   foreach (DrawableGameComponent mesh in Components)
{
  Matrix oldWorld = mesh.World;
  mesh.World = oldWorld * reflMatrix;

  mesh.Draw(mGameTime);

  mesh.World = oldWorld;
}
}

mWaterMesh.RenderObjects is the delegate that has the signature of: public delegate void RenderObjects(Matrix reflectionMatrix);

Basically this function should just go through your game objects and render them.

Lastly, before you draw your objects in the scene, you need to send to the water component the ViewProjection matrix and the camera's position by using WaterMesh.SetCamera(). And you need to call WaterMesh.UpdateWaterMaps() to update the reflection and refraction maps. After this, you can clear your framebuffer and draw your objects. For how this effect looks you can take a look at my camera animation tutorials.

Water GameComponent:

using System;
using System.Collections.Generic;
using System.Text;

using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Content;

namespace WaterSample
{
//delegate that the water component to call to render the objects in the scene
public delegate void RenderObjects(Matrix reflectionMatrix);

/// <summary>
/// Options that must be passed to the water component before Initialization
/// </summary>
public class WaterOptions
{
//width and height must be of the form 2^n + 1
public int Width = 257;
public int Height = 257;
public float CellSpacing = .5f;

public float WaveMapScale = 1.0f;

public int RenderTargetSize = 512;

//offsets for the texcoords of the wave maps updated every frame
public Vector2 WaveMapOffset0;
public Vector2 WaveMapOffset1;

//the direction to offset the texcoords of the wave maps
public Vector2 WaveMapVelocity0;
public Vector2 WaveMapVelocity1;

//asset names for the normal/wave maps
public string WaveMapAsset0;
public string WaveMapAsset1;

public Vector4 WaterColor;
public Vector4 SunColor;
public Vector3 SunDirection;
public float SunFactor;
public float SunPower;
}

/// <summary>
/// Drawable game component for water rendering. Renders the scene to reflection and refraction
/// maps that are projected onto the water plane and are distorted based on two scrolling normal
/// maps.
/// </summary>
public class Water : DrawableGameComponent
{
#region Fields
private RenderObjects mDrawFunc;

//vertex and index buffers for the water plane
private VertexBuffer mVertexBuffer;
private IndexBuffer mIndexBuffer;
private VertexDeclaration mDecl;

//water shader
private Effect mEffect;
private string mEffectAsset;

//camera properties
private Vector3 mViewPos;
private Matrix mViewProj;
private Matrix mWorld;

//maps to render the refraction/reflection to
private RenderTarget2D mRefractionMap;
private RenderTarget2D mReflectionMap;

//scrolling normal maps that we will use as a
//a normal for the water plane in the shader
private Texture mWaveMap0;
private Texture mWaveMap1;

//user specified options to configure the water object
private WaterOptions mOptions;

//tells the water object if it needs to update the refraction
//map itself or not. Since refraction just needs the scene drawn
//regularly, we can:
// --Draw the objects we want refracted
// --Resolve the back buffer and send it to the water
// --Skip computing the refraction map in the water object
private bool mGrabRefractionFromFB = false;

private int mNumVertices;
private int mNumTris;
#endregion

#region Properties

public RenderObjects RenderObjects
{
set { mDrawFunc = value; }
}

/// <summary>
/// Name of the asset for the Effect.
/// </summary>
public string EffectAsset
{
get { return mEffectAsset; }
set { mEffectAsset = value; }
}

/// <summary>
/// The render target that the refraction is rendered to.
/// </summary>
public RenderTarget2D RefractionMap
{
get { return mRefractionMap; }
set { mRefractionMap = value; }
}

/// <summary>
/// The render target that the reflection is rendered to.
/// </summary>
public RenderTarget2D ReflectionMap
{
get { return mReflectionMap; }
set { mReflectionMap = value; }
}

/// <summary>
/// Options to configure the water. Must be set before
/// the water is initialized. Should be set immediately
/// following the instantiation of the object.
/// </summary>
public WaterOptions Options
{
get { return mOptions; }
set { mOptions = value; }
}

/// <summary>
/// The world matrix of the water.
/// </summary>
public Matrix World
{
get { return mWorld; }
set { mWorld = value; }
}

#endregion

public Water(Game game) : base(game)
{

}

public override void Initialize()
{
base.Initialize();

//build the water mesh
mNumVertices = mOptions.Width * mOptions.Height;
mNumTris = (mOptions.Width - 1) * (mOptions.Height - 1) * 2;
VertexPositionTexture[] vertices = new VertexPositionTexture[mNumVertices];

Vector3[] verts;
int[] indices;

GenTriGrid(mOptions.Height, mOptions.Width, mOptions.CellSpacing, mOptions.CellSpacing,
          Vector3.Zero, out verts, out indices);

//copy the verts into our PositionTextured array
for (int i = 0; i < mOptions.Width; ++i)
{
  for (int j = 0; j < mOptions.Height; ++j)
  {
      int index = i * mOptions.Width + j;
      vertices[index].Position = verts[index];
      vertices[index].TextureCoordinate = new Vector2((float)j / mOptions.Width, (float)i / mOptions.Height);
  }
}

mVertexBuffer = new VertexBuffer(Game.GraphicsDevice,
                               VertexPositionTexture.SizeInBytes * mOptions.Width * mOptions.Height,
                               BufferUsage.WriteOnly);
mVertexBuffer.SetData(vertices);

mIndexBuffer = new IndexBuffer(Game.GraphicsDevice, typeof(int), indices.Length, BufferUsage.WriteOnly);
mIndexBuffer.SetData(indices);

mDecl = new VertexDeclaration(Game.GraphicsDevice, VertexPositionTexture.VertexElements);
}

protected override void  LoadContent()
{
base.LoadContent();

mWaveMap0 = Game.Content.Load<Texture2D>(mOptions.WaveMapAsset0);
mWaveMap1 = Game.Content.Load<Texture2D>(mOptions.WaveMapAsset1);

PresentationParameters pp = Game.GraphicsDevice.PresentationParameters;
SurfaceFormat format = pp.BackBufferFormat;
MultiSampleType msType = pp.MultiSampleType;
int msQuality = pp.MultiSampleQuality;

mRefractionMap = new RenderTarget2D(Game.GraphicsDevice, mOptions.RenderTargetSize, mOptions.RenderTargetSize,
                                  1, format, msType, msQuality);
mReflectionMap = new RenderTarget2D(Game.GraphicsDevice, mOptions.RenderTargetSize, mOptions.RenderTargetSize,
                                  1, format, msType, msQuality);

mEffect = Game.Content.Load<Effect>(mEffectAsset);

//set the parameters that shouldn't change.
//Some of these might need to change every once in awhile,
//move them to updateEffectParams if you need that functionality.
if (mEffect != null)
{
  mEffect.Parameters["WaveMap0"].SetValue(mWaveMap0);
  mEffect.Parameters["WaveMap1"].SetValue(mWaveMap1);

  mEffect.Parameters["TexScale"].SetValue(mOptions.WaveMapScale);

  mEffect.Parameters["WaterColor"].SetValue(mOptions.WaterColor);
  mEffect.Parameters["SunColor"].SetValue(mOptions.SunColor);
  mEffect.Parameters["SunDirection"].SetValue(mOptions.SunDirection);
  mEffect.Parameters["SunFactor"].SetValue(mOptions.SunFactor);
  mEffect.Parameters["SunPower"].SetValue(mOptions.SunPower);

  mEffect.Parameters["World"].SetValue(mWorld);
}
}

public override void Update(GameTime gameTime)
{
float timeDelta = (float)gameTime.ElapsedGameTime.TotalSeconds;

mOptions.WaveMapOffset0 += mOptions.WaveMapVelocity0 * timeDelta;
mOptions.WaveMapOffset1 += mOptions.WaveMapVelocity1 * timeDelta;

if (mOptions.WaveMapOffset0.X >= 1.0f  mOptions.WaveMapOffset0.X <= -1.0f)
  mOptions.WaveMapOffset0.X = 0.0f;
if (mOptions.WaveMapOffset1.X >= 1.0f  mOptions.WaveMapOffset1.X <= -1.0f)
  mOptions.WaveMapOffset1.X = 0.0f;
if (mOptions.WaveMapOffset0.Y >= 1.0f  mOptions.WaveMapOffset0.Y <= -1.0f)
  mOptions.WaveMapOffset0.Y = 0.0f;
if (mOptions.WaveMapOffset1.Y >= 1.0f  mOptions.WaveMapOffset1.Y <= -1.0f)
  mOptions.WaveMapOffset1.Y = 0.0f;
}

public override void Draw(GameTime gameTime)
{
UpdateEffectParams();

Game.GraphicsDevice.Indices = mIndexBuffer;
Game.GraphicsDevice.Vertices[0].SetSource(mVertexBuffer, 0, VertexPositionTexture.SizeInBytes);
Game.GraphicsDevice.VertexDeclaration = mDecl;

mEffect.Begin(SaveStateMode.None);

foreach (EffectPass pass in mEffect.CurrentTechnique.Passes)
{
  pass.Begin();
  Game.GraphicsDevice.DrawIndexedPrimitives(PrimitiveType.TriangleList, 0, 0, mNumVertices, 0, mNumTris);
  pass.End();
}

mEffect.End();
}

/// <summary>
/// Set the ViewProjection matrix and position of the Camera.
/// </summary>
/// <param name="viewProj"></param>
/// <param name="pos"></param>
public void SetCamera(Matrix viewProj, Vector3 pos)
{
mViewProj = viewProj;
mViewPos = pos;
}

/// <summary>
/// Updates the reflection and refraction maps. Called
/// on update.
/// </summary>
/// <param name="gameTime"></param>
public void UpdateWaterMaps(GameTime gameTime)
{
/*------------------------------------------------------------------------------------------
* Render to the Reflection Map
*/
//clip objects below the water line, and render the scene upside down
GraphicsDevice.RenderState.CullMode = CullMode.CullClockwiseFace;

GraphicsDevice.SetRenderTarget(0, mReflectionMap);
GraphicsDevice.Clear(ClearOptions.Target  ClearOptions.DepthBuffer, mOptions.WaterColor, 1.0f, 0);

//reflection plane in local space
Vector4 waterPlaneL = new Vector4(0.0f, -1.0f, 0.0f, 0.0f);

Matrix wInvTrans = Matrix.Invert(mWorld);
wInvTrans = Matrix.Transpose(wInvTrans);

//reflection plane in world space
Vector4 waterPlaneW = Vector4.Transform(waterPlaneL, wInvTrans);

Matrix wvpInvTrans = Matrix.Invert(mWorld * mViewProj);
wvpInvTrans = Matrix.Transpose(wvpInvTrans);

//reflection plane in homogeneous space
Vector4 waterPlaneH = Vector4.Transform(waterPlaneL, wvpInvTrans);

GraphicsDevice.ClipPlanes[0].IsEnabled = true;
GraphicsDevice.ClipPlanes[0].Plane = new Plane(waterPlaneH);

Matrix reflectionMatrix = Matrix.CreateReflection(new Plane(waterPlaneW));

if (mDrawFunc != null)
  mDrawFunc(reflectionMatrix);

GraphicsDevice.RenderState.CullMode = CullMode.CullCounterClockwiseFace;

GraphicsDevice.SetRenderTarget(0, null);


/*------------------------------------------------------------------------------------------
* Render to the Refraction Map
*/

//if the application is going to send us the refraction map
//exit early. The refraction map must be given to the water component
//before it renders
if (mGrabRefractionFromFB)
{
  GraphicsDevice.ClipPlanes[0].IsEnabled = false;
  return;
}

//update the refraction map, clip objects above the water line
//so we don't get artifacts
GraphicsDevice.SetRenderTarget(0, mRefractionMap);
GraphicsDevice.Clear(ClearOptions.Target  ClearOptions.DepthBuffer, mOptions.WaterColor, 1.0f, 1);

//reflection plane in local space
waterPlaneL.W = 2.5f;

//if we're below the water line, don't perform clipping.
//this allows us to see the distorted objects from under the water
if (mViewPos.Y < mWorld.Translation.Y)
{
  GraphicsDevice.ClipPlanes[0].IsEnabled = false;
}

if (mDrawFunc != null)
  mDrawFunc(Matrix.Identity);

GraphicsDevice.ClipPlanes[0].IsEnabled = false;

GraphicsDevice.SetRenderTarget(0, null);
}

/// <summary>
/// Updates effect parameters related to the water shader
/// </summary>
private void UpdateEffectParams()
{
//update the reflection and refraction textures
mEffect.Parameters["ReflectMap"].SetValue(mReflectionMap.GetTexture());
mEffect.Parameters["RefractMap"].SetValue(mRefractionMap.GetTexture());

//normal map offsets
mEffect.Parameters["WaveMapOffset0"].SetValue(mOptions.WaveMapOffset0);
mEffect.Parameters["WaveMapOffset1"].SetValue(mOptions.WaveMapOffset1);

mEffect.Parameters["WorldViewProj"].SetValue(mWorld * mViewProj);

mEffect.Parameters["EyePos"].SetValue(mViewPos);
}

/// <summary>
/// Generates a grid of vertices to use for the water plane.
/// </summary>
/// <param name="numVertRows">Number of rows. Must be 2^n + 1. Ex. 129, 257, 513.</param>
/// <param name="numVertCols">Number of columns. Must be 2^n + 1. Ex. 129, 257, 513.</param>
/// <param name="dx">Cell spacing in the x dimension.</param>
/// <param name="dz">Cell spacing in the y dimension.</param>
/// <param name="center">Center of the plane.</param>
/// <param name="verts">Outputs the constructed vertices for the plane.</param>
/// <param name="indices">Outpus the constructed triangle indices for the plane.</param>
private void GenTriGrid(int numVertRows, int numVertCols, float dx, float dz,
                  Vector3 center, out Vector3[] verts, out int[] indices)
{
int numVertices = numVertRows * numVertCols;
int numCellRows = numVertRows - 1;
int numCellCols = numVertCols - 1;

int mNumTris = numCellRows * numCellCols * 2;

float width = (float)numCellCols * dx;
float depth = (float)numCellRows * dz;

//===========================================
// Build vertices.

// We first build the grid geometry centered about the origin and on
// the xz-plane, row-by-row and in a top-down fashion.  We then translate
// the grid vertices so that they are centered about the specified
// parameter 'center'.

//verts.resize(numVertices);
verts = new Vector3[numVertices];

// Offsets to translate grid from quadrant 4 to center of
// coordinate system.
float xOffset = -width * 0.5f;
float zOffset = depth * 0.5f;

int k = 0;
for (float i = 0; i < numVertRows; ++i)
{
  for (float j = 0; j < numVertCols; ++j)
  {
      // Negate the depth coordinate to put in quadrant four.
      // Then offset to center about coordinate system.
      verts[k] = new Vector3(0, 0, 0);
      verts[k].X = j * dx + xOffset;
      verts[k].Z = -i * dz + zOffset;
      verts[k].Y = 0.0f;

      Matrix translation = Matrix.CreateTranslation(center);
      verts[k] = Vector3.Transform(verts[k], translation);

      ++k; // Next vertex
  }
}

//===========================================
// Build indices.

//indices.resize(mNumTris * 3);
indices = new int[mNumTris * 3];

// Generate indices for each quad.
k = 0;
for (int i = 0; i < numCellRows; ++i)
{
  for (int j = 0; j < numCellCols; ++j)
  {
      indices[k] = i * numVertCols + j;
      indices[k + 1] = i * numVertCols + j + 1;
      indices[k + 2] = (i + 1) * numVertCols + j;

      indices[k + 3] = (i + 1) * numVertCols + j;
      indices[k + 4] = i * numVertCols + j + 1;
      indices[k + 5] = (i + 1) * numVertCols + j + 1;

      // next quad
      k += 6;
  }
}
}
}
}

Water.fx shader:

//Water effect shader that uses reflection and refraction maps projected onto the water.
//These maps are distorted based on the two scrolling normal maps.

float4x4 World;
float4x4 WorldViewProj;

float4  WaterColor;
float3    SunDirection;
float4  SunColor;
float    SunFactor; //the intensity of the sun specular term.
float   SunPower; //how shiny we want the sun specular term on the water to be.
float3  EyePos;

// Texture coordinate offset vectors for scrolling
// normal maps.
float2  WaveMapOffset0;
float2  WaveMapOffset1;

// Two normal maps and the reflection/refraction maps
texture WaveMap0;
texture WaveMap1;
texture ReflectMap;
texture RefractMap;

//scale used on the wave maps
float TexScale;

static const float      R0 = 0.02037f;

sampler WaveMapS0 = sampler_state
{
Texture = <WaveMap0>;
MinFilter = LINEAR;
MagFilter = LINEAR;
MipFilter = LINEAR;
AddressU  = WRAP;
AddressV  = WRAP;
};

sampler WaveMapS1 = sampler_state
{
Texture = <WaveMap1>;
MinFilter = LINEAR;
MagFilter = LINEAR;
MipFilter = LINEAR;
AddressU  = WRAP;
AddressV  = WRAP;
};

sampler ReflectMapS = sampler_state
{
Texture = <ReflectMap>;
MinFilter = LINEAR;
MagFilter = LINEAR;
MipFilter = LINEAR;
AddressU  = CLAMP;
AddressV  = CLAMP;
};

sampler RefractMapS = sampler_state
{
Texture = <RefractMap>;
MinFilter = LINEAR;
MagFilter = LINEAR;
MipFilter = LINEAR;
AddressU  = CLAMP;
AddressV  = CLAMP;
};

struct OutputVS
{
float4 posH            : POSITION0;
float3 toEyeW        : TEXCOORD0;
float2 tex0            : TEXCOORD1;
float2 tex1            : TEXCOORD2;
float4 projTexC        : TEXCOORD3;
float4 pos            : TEXCOORD4;
};

OutputVS WaterVS( float3 posL    : POSITION0,
         float2 texC   : TEXCOORD0)
{
// Zero out our output.
OutputVS outVS = (OutputVS)0;

// Transform vertex position to world space.
float3 posW  = mul(float4(posL, 1.0f), World).xyz;
outVS.pos.xyz = posW;
outVS.pos.w = 1.0f;

// Compute the unit vector from the vertex to the eye.
outVS.toEyeW = posW - EyePos;

// Transform to homogeneous clip space.
outVS.posH = mul(float4(posL, 1.0f), WorldViewProj);

// Scroll texture coordinates.
outVS.tex0 = (texC * TexScale) + WaveMapOffset0;
outVS.tex1 = (texC * TexScale) + WaveMapOffset1;

// Generate projective texture coordinates from camera's perspective.
outVS.projTexC = outVS.posH;

// Done--return the output.
return outVS;
}

float4 WaterPS( float3 toEyeW        : TEXCOORD0,
       float2 tex0            : TEXCOORD1,
       float2 tex1            : TEXCOORD2,
       float4 projTexC        : TEXCOORD3,
       float4 pos            : TEXCOORD4) : COLOR
{
projTexC.xyz /= projTexC.w;
projTexC.x =  0.5f*projTexC.x + 0.5f;
projTexC.y = -0.5f*projTexC.y + 0.5f;
projTexC.z = .1f / projTexC.z;

toEyeW    = normalize(toEyeW);
SunDirection = normalize(SunDirection);

// Light vector is opposite the direction of the light.
float3 lightVecW = -SunDirection;

// Sample normal map.
float3 normalT0 = tex2D(WaveMapS0, tex0);
float3 normalT1 = tex2D(WaveMapS1, tex1);

//unroll the normals retrieved from the normalmaps
normalT0.yz = normalT0.zy;
normalT1.yz = normalT1.zy;

normalT0 = 2.0f*normalT0 - 1.0f;
normalT1 = 2.0f*normalT1 - 1.0f;

float3 normalT = normalize(0.5f*(normalT0 + normalT1));
float3 n1 = float3(0,1,0); //we'll just use the y unit vector for spec reflection.

//get the reflection vector from the eye
float3 R = normalize(reflect(toEyeW,normalT));

float4 finalColor;
finalColor.a = 1;

//compute the fresnel term to blend reflection and refraction maps
float ang = saturate(dot(-toEyeW,n1));
float f = R0 + (1.0f-R0) * pow(1.0f-ang,5.0);

//also blend based on distance
f = min(1.0f, f + 0.007f * EyePos.y);

//compute the reflection from sunlight, hacked in color, should be a variable
float sunFactor = SunFactor;
float sunPower = SunPower;

if(EyePos.y < pos.y)
{
sunFactor = 7.0f; //these could also be sent to the shader
sunPower = 55.0f;
}
float3 sunlight = sunFactor * pow(saturate(dot(R, lightVecW)), sunPower) * SunColor;

float4 refl = tex2D(ReflectMapS, projTexC.xy + projTexC.z * normalT.xz);
float4 refr = tex2D(RefractMapS, projTexC.xy - projTexC.z * normalT.xz);

//only use the refraction map if we're under water
if(EyePos.y < pos.y)
f = 0.0f;

//interpolate the reflection and refraction maps based on the fresnel term and add the sunlight
finalColor.rgb = WaterColor * lerp( refr, refl, f) + sunlight;

return finalColor;
}

technique WaterTech
{
pass Pass1
{
// Specify the vertex and pixel shader associated with this pass.
vertexShader = compile vs_2_0 WaterVS();
pixelShader  = compile ps_2_0 WaterPS();

CullMode = None;
}
}

Edit: A demo of the water component is now available.

Scientific Visualization

2008-11-07T01:20:00.003-05:00

This semester I've been taking a class in scientific visualization. It's pretty interesting and it covers a lot of techniques such as color visualization, human vision and color perception, contours, isosurfacing, volume rendering, flow visualization such as stream lines and stream surfaces and texture based methods.

There are good and bad parts to this course. The bad part is that it is a fairly new graduate course and has no prerequisites. The professor is just trying to build up interest in the course. This equates to us not actually coding the different algorithms, but using a visualization framework, VTK, and c++/java/python/tcl to implement the techniques. The good part is that this semester is one of the busiest I've had, so minimizing my work is a good thing :)

Now on to some pictures.

Contours, Heightmaps:

Isosurfacing:

Volume Rendering:

Glyphs, Stream lines, stream surfaces:

Interviews, research, and future posts

2008-10-07T15:38:00.007-04:00

I don't have a new sample/tutorial today. I have been very busy with interviews, research, and school as of late.

As this is my last year of school, I've been interviewing with various companies and trying to get an on-line portfolio together. I recently had a great time in Portland meeting with Intel.

Last spring I was involved in the research of using non-pinhole impostors for reflections and refractions. We submitted to Eurographics (specifically EGSR), but unfortunately didn't get accepted. So this semester we are looking to work on the short comings that some of the reviewers noted and resubmit to I3D. So until I get that out of the way there probably won't be any new samples/tutorials for a couple of weeks.

As far as for future posts, I've been working on rain as a particle system and as a post process (see Tatarchuck's AMD/ATi paper on Rain). Besides this I've been wanting to have a series of samples/tutorials on different lighting methods. We all know Gouraud and [Blinn-]Phong shading, but I wanted to cover other methods such as Cook-Torrance, Oren-Nayar, Ward lighting and others. I also might do a tutorial on Depth Impostors that would build off of the Billboard Impostor tutorial I wrote earlier in the year.

For now, here's a couple of teaser images that we submitted to EGSR. You're looking through a glass bunny's ear. You can see that with regular depth impostors, you are missing a significant amount of data for the teapot lid.

Planar Pinhole Camera Depth Impostor

Non-Pinhole Camera Depth Impostor

Dual-Paraboloid Variance Shadow Mapping

2008-07-18T10:31:00.004-04:00

Edit: Added the video that I recently made

I have to say, I really like variance shadow mapping. It's such a simple(ingenious) technique to implement, but it provides such nice looking results. I haven't had the need to implement the technique before, but I'm glad I did. Last post we implemented dual-paraboloid shadow mapping. And those of you with a PS 3.0 graphics card were able to have semi-soft shadows with percentage closer filtering. But now when we get rid of the PCF filter, and replace it with variance shadow mapping, we can fit all the code inside the PS 2.0 standard. Anyway, on to the code.

Variance Shadow Mapping Paper + Demo

Building the shadow maps:

Variance shadow mapping is really simple to implement. First thing we need to change is to create either a RG32F or RG16F surface format for our front and rear shadow maps (instead of R32F/R16F). This allows us to store the depth of the pixel in the red channel and the squared depth of the pixel in the green channel. So our new pixel shader for building the depth/shadow maps is this:

return float4(z, z * z, 0, 0, 1);

Blurring the shadow maps:

Variance shadow mapping improves upon standard shadow mapping by storing a distribution of depths at each pixel (z * z, and z) instead of the single depth (as with standard shadow mapping). And because it stores a distribution of depth, we can blur the shadow maps. This would produce some funky/incorrect results if we were just doing standard shadow mapping with a PCF filter.

So, after we have created our depth maps, we will blur them with a separable Gaussian blur. This will perform two passes on each shadow map; the first will perform a horizontal blur and the second will perform a vertical blur. There is a wealth of information on the internet on how to do this so I won't explicitly cover this. Here's what our front shadow map looks like after being blurred:

Variance shadow mapping:

We build our texture coordinates exactly the same as the previous method of shadow mapping. But the depth comparison is a little different. You can refer to the VSM paper for an in-depth discussion, but here is the gist of it. Since we filtered our shadow maps with a Gaussian blur, we need to recover the moments over that filter region. The moments are simple the depth and squared depth we stored in the texture. From these we can build the mean depth and the variance at the pixel. And as such the variance can be interpreted as a quantitative measure of the width of a distribution (Donelly/Lauritzen). This measure places a bound on the distribution and can be represented by Chebychev's inequality.

float depth;
float mydepth;
float2 moments;
if(alpha >= 0.5f)
{
    moments = tex2D(ShadowFrontS, P0.xy).xy;
    depth = moments.x;
    mydepth = P0.z;
}
else
{
    moments = tex2D(ShadowBackS, P1.xy).xy;
    depth = moments.x;
    mydepth = P1.z;
}

float lit_factor = (mydepth <= moments[0]);
 
float E_x2 = moments.y;
float Ex_2 = moments.x * moments.x;
float variance = min(max(E_x2 - Ex_2, 0.0) + SHADOW_EPSILON, 1.0);
float m_d = (moments.x - mydepth);
float p = variance / (variance + m_d * m_d); //Chebychev's inequality

texColor.xyz *= max(lit_factor, p + .2f); //lighten the shadow just a bit (with the + .2f)

return texColor;

5x5 Guassian Blur

9x9 Guassian Blur

And there you go. Nice looking dual-paraboloid soft shadows thanks to variance shadow mapping.

As before, your card needs to support either RG16F or RG32F formats (sorry again Charles :) ). You can refer to the VSM paper and demo on how to map 2 floats to a single ARGB32 pixel if your card doesn't support the floating point surface formats.

Dual-Paraboloid Shadow Maps

2008-07-17T12:34:00.005-04:00

Last time I introduced using dual-paraboloid environment mapping for reflections. Well now we're going to apply the same process to shadows. So if you haven't looked at my previous post, read it over before going on.

Creating the depth/shadow maps is exactly the same as when we created the reflection maps with one exception. Instead of outputting color in the pixel shader, we output the depth of the 3d pixel, like so:

return depth.x / depth.y;

Where depth.x is the depth of the pixel and depth.y is the w component. And here is the resulting depth/shadow map for the front hemisphere.

Now, to map the shadows the process is also very similar to how we generated the reflections. We follow a similar process in the pixel shader:

Generate the texture coordinates for the front and rear paraboloids
Generate the depth of the pixel
Test to see if the pixel is in shadow

We generate the texture coordinates exactly as when we generated the reflection texture coordinates. To generate the depth of the pixel we take the length of the vector from the vertex to the origin of the paraboloid (0, 0, 0) and divide by the light attenuation. Also to check which hemisphere we are in, we calculate an alpha that is the Z value of the transformed vertex and offset by .5f;

float L = length(pos);
float3 P0 = pos / L;

float alpha = .5f + pos.z / LightAttenuation;
//generate texture coords for the front hemisphere
P0.z = P0.z + 1;
P0.x = P0.x / P0.z;
P0.y = P0.y / P0.z;
P0.z = L / LightAttenuation;

P0.x = .5f * P0.x + .5f;
P0.y = -.5f * P0.y + .5f;

float3 P1 = pos / L;
//generate texture coords for the rear hemisphere
P1.z = 1 - P1.z;
P1.x = P1.x / P1.z;
P1.y = P1.y / P1.z;
P1.z = L / LightAttenuation;

P1.x = .5f * P1.x + .5f;
P1.y = -.5f * P1.y + .5f;

Now that we have generated our texture coordinates we need to test the depth of the pixel against the depth in the shadow map. To do this we index either the front or rear shadow map with the texture coordinates we generated to get the depth and compare this to our depth. If the depth is less than our depth, then the pixel is in shadow.

float depth;
float mydepth;
if(alpha >= 0.5f)
{
    depth = tex2D(ShadowFrontS, P0.xy).x;
    mydepth = P0.z;
}
else
{
    depth = tex2D(ShadowBackS, P1.xy).x;
    mydepth = P1.z;
}

//lighten the shadow just a bit so it isn't completely black
if((depth + SHADOW_EPSILON) < mydepth)
    texColor.xyz *= 0.3f;

return texColor;

And that's it. Now we have dual-paraboloid shadow mapping. If you have a pixel shader 3.0 graphics card, then the shadow also has a percentage closer filter applied to it. You also may notice seams in the shadows. This is because the splitting plane of the paraboloids is the x-axis (since the paraboloids look down the +/- z-axis). This is one of the problems of using paraboloid mapping for shadows. One has to be careful where they place the split plane to avoid this situation. Pixels that are in the center of either hemisphere suffer little distortion. But this is just a tutorial so I didn't worry too much about it.

Also you're graphics card must be able to support R32F or R16F surface formats to run the demo out of the box (sorry Charles ;) ). Otherwise, you must use the ARGB32 format and pack the depth values in all 4 channels. Here is some code to pack/unpack to/from an ARGB32 surface format. You pass the depth value to the pack method when you render to the shadow maps, and you pass the float4 color to the unpack method when you fetch from the shadow maps. I decided not to implement this so the code wouldn't become complicated by something that doesn't add to the tutorial.

//pack the depth in a 32-bit rgba color
float4 mapDepthToARGB32(const float value)
{
    const float4 bitSh = float4(256.0 * 256.0 * 256.0, 256.0 * 256.0, 256.0, 1.0);
    const float4 mask = float4(0.0, 1.0 / 256.0, 1.0 / 256.0, 1.0 / 256.0);
    float4 res = frac(value * bitSh);
    res -= res.xxyz * mask;
    return res;
}

//unpack the depth from a 32-bit rgba color
float getDepthFromARGB32(const float4 value)
{
    const float4 bitSh = float4(1.0 / (256.0 * 256.0 * 256.0), 1.0 / (256.0 * 256.0), 1.0 / 256.0, 1.0);
    return(dot(value, bitSh));
}

Next time I'll introduce using variance shadow mapping with our dual-paraboloid shadow mapping to give nice soft shadows that we can still use with pixel shader 2.0 cards.

Dual-Paraboloid Reflections

2008-07-16T11:29:00.010-04:00

I recently had to investigate dual-paraboloid reflections at work for an unnamed console. What are these you ask? Great question! :) Lets start with some background.

The standard way of calculating reflections is to use an environment map, more specifically a cube map. In my last tutorial on reflections, this basic type of reflection mapping was used to compare against billboard impostor reflections. Now, cubemaps are great for static scenes, and are relatively low cost to perform a texture fetch on in a shader. However if you have a dynamic scene you have to update all 6 sides of the cubemap (this is not technically true, aggressive culling and other optimizations can guarantee at most 5 sides). Holy crap, now we have to render our scene 6 times!

This is where dual-paraboloid reflections come in. They are a view-independent method of rendering reflections just like cubemaps. Except you only have to update 2 textures, not 6! The downside is that you are going to lose quality for speed, but unless you have to have high-quality reflections, paraboloid reflections will probably provide sufficient results.

Reference articles:

View-Independent Environment Maps

Shadow Mapping for Hemispherical and Omnidirectional Light Sources

Dual Paraboloid Mapping in the Vertex Shader

In the interest of keeping this post from getting too long, I won't go into great detail on the mathematical process. I suggest you refer to the first and third papers for an in-depth discussion on the details.

Now lets move on to what exactly paraboloid mapping is. Lets look at what a paraboloid is.

The basic idea is that for any origin O, we can divide the scene into two hemispheres, front and rear. For each of these hemispheres there exists a paraboloid surface that will focus any ray traveling in the direction of O into the direction of the the hemisphere. Here is a 2d picture demonstrating the idea:

A brief math overview:

What we need to find is the intersection point where the incident ray intersects the paraboloid surface. To do this we need to know the incident ray and the reflected ray. Now because the paraboloid reflects rays in the same direction, it is easy to compute the reflection vector: it's the forward direction of the hemisphere! So the front hemisphere's reflection vector will always be <0, 0, 1> and the rear hemisphere's reflection vector will always be <0, 0, -1>. Easy! And the incident ray is calculated the same as with environment mapping by reflecting the ray from the pixel position to the eye across the normal of the 3D pixel.

Now all we have to do is find the normal of the intersection which we will use to map our vertices into paraboloid space. To find the normal, we add the incident and reflected vectors and divide the x and y components by the z value.

Generating the Paraboloid maps:

What we are basically going to do is, in the vertex shader, place each vertex ourselves that has been distorted by the paraboloid. First we need to transform the vertex by the view matrix of the paraboloid 'camera'. We don't apply the projection matrix since we're going to place the point ourselves

output.Position = mul(input.Position, WorldViewProj);

Next we need to find the vector from the the vertex to the origin of the paraboloid, which is simply:

float L = length( output.Position.xyz );
output.Position = output.Position / L;

Now we need to find the x and y coordinates of the point where the incident ray intersects the paraboloid surface.

output.Position.z = output.Position.z + 1;
output.Position.x = output.Position.x / output.Position.z;
output.Position.y = output.Position.y / output.Position.z;

Finally we set the z value as the distance from the vertex to the origin of the paraboloid, scaled and biased by the near and far planes of the paraboloid 'camera'.

output.Position.z = (L - NearPlane) / (FarPlane - NearPlane);
output.Position.w = 1;

And the only thing we need to add in the pixel shader is to make sure and clip vertices that are behind the viewpoint using the intrinsic clip() function of HLSL.

Reflections with paraboloid maps:

In the reflection pixel shader we will: generate the reflection vector the same way as cube mapping, generate texture coordinates for both the front and rear paraboloids' textures, and blend the samples taken from the textures.

The texture coordinates are generated exactly as how we generated them before in the generation step. We also scale and bias them to correctly index a D3D texture. And then we take a sample from each map and pick the sample with the greater color value:

// calculate the front paraboloid map texture coordinates
float2 front;
front.x = R.x / (R.z + 1);
front.y = R.y / (R.z + 1);
front.x = .5f * front.x + .5f; //bias and scale to correctly sample a d3d texture
front.y = -.5f * front.y + .5f;

// calculate the back paraboloid map texture coordinates
float2 back;
back.x = R.x / (1 - R.z);
back.y = R.y / (1 - R.z);
back.x = .5f * back.x + .5f; //bias and scale to correctly sample a d3d texture
back.y = -.5f * back.y + .5f;

float4 forward = tex2D( FrontTex, front );    // sample the front paraboloid map
float4 backward = tex2D( BackTex, back );    // sample the back paraboloid map

float4 finalColor = max(forward, backward);

Optimizations:

If you align the paraboloid 'camera' such that it is always facing down the +/- z axis, you don't need to transform the vertices by the view matrix of the camera. You only need to do a simple translation of the vertex by the camera position.

Conclusion:

As you can see, paraboloid maps give pretty good results. The won't give you the quality of cubemaps, but they are faster to update and require less memory. And in the console world, requiring less is almost reason enough to pick this method over cubemaps.

One drawback of paraboloid maps is that the environment geometry has to be sufficiently tessellated or will we will have noticeable artifacts on our reflector. Another drawback is that on spherical objects we will see seems. However with objects that are reasonably complex (such as the Stanford bunny or dragon) and are not simple shapes, the seams will not be as noticeable.

Next time I will present dual-paraboloid mapping for use with real-time omnidirectional shadow mapping of point lights.

Terrain and Atmospheric Scattering Source

2008-06-20T13:24:00.004-04:00

I figured I would release my source for the terrain rendering demo. There aren't that many terrain and atmospheric scattering examples on the net, so I figured I might as well post mine for anyone that would like to see it.

It's still a work in progress (that I paused about a year ago), so it's kind of immature. But it is the same code that produced the images I posted awhile back. You'll probably need at least a 6 series Nvidia graphics card (or ATi equivalent). For the terrain, the demo uses a simple quad tree with frustum culling, so it's not the greatest performer for anything above 512x512 heigtmaps. Also, it is in Managed DirectX, but it shouldn't be too hard to port to XNA.

Anyway, enough of that. If you didn't know, the Creature Creator for Spore has been released, and it's an absolute blast to mess around with. The possibilities on what you can create are pretty much endless. Here's a couple that I made:

Post Process Framework Sample

2008-06-03T11:28:00.019-04:00

Today I'm posting a post processing framework sample. For those that don't know, post processing is any manipulation of the frame buffer after the scene has been rendered (but usually before the UI), hence the word "post" in the name. This process could be a simple blurring, or it could be a motion blur or depth of field technique. But what if you have many post processes? For instance, maybe we have bloom, motion blur, heat haze, depth of field, and other post processes that we need to apply after we render our scene. Connecting all these post processes together can get a little hairy, and that's what this sample helps us do.

So lets talk about how we're going to structure our framework. At the very bottom of the hierarchy, we have a PostProcessComponent (component from here on out). And what this class represents, is a single atomic process, that is, it cannot be broken up into smaller pieces/objects. Each component has an input and an output in the form of textures. A component is also able to update the backbuffer. It is the simplest object in the hierarchy and is combined with other components to form a PostProcessEffect (effect from here on out).

An effect contains a chain of components to implement a single post process such as bloom or motion blur. An effect also handles output from one component to input to another. In this way the components can be independent of one another, and the effect handles linking all the components that it owns. Because of this, each component is very simple. Also, like components, effects have inputs and outputs in the form of textures. Effects can also be enabled/disabled dynamically at runtime.

Next we have the PostProcessManager(aka manager). The manager is comprised of a chain of effects, and it handles linking the output of one effect to the input of the next effect. And just like with components, this enables each effect to be independent of the next. The manager also takes care of linking the backbuffer and depth buffer to components that need it.

Because each component is not dependent on other components, each one is very simple in its implementation. This not only helps in understanding the code, but also in robustness and tracking down any possible errors in a component. Another nice feature about this framework is that you do not need to create a separate class deriving from PostProcessEffect for each effect. In this way, you can dynamically create your effects at run time. Most of the "magic" occurs in the LoadContent() function inside PostProcessManager. So lets have a look at it:

public void LoadContent()
{
   #region Create common textures to be used by the effects
   PresentationParameters pp = mGraphicsDevice.PresentationParameters;
   int width = pp.BackBufferWidth;
   int height = pp.BackBufferHeight;

   SurfaceFormat format = pp.BackBufferFormat;

   resolveTexture = new ResolveTexture2D(mGraphicsDevice, width, height, 1, format);
   #endregion

   int i = 0;
   foreach (PostProcessEffect effect in effects)
   {
       effect.LoadContent();

       int j = 0;
       //if a component requires a backbuffer, add their function to the event handler
       foreach (PostProcessComponent component in effect.Components)
       {
           //if the component updates/modifies the "scene texture"
           //find all the components who need an up to date scene texture
           if (component.UpdatesSceneTexture)
           {
               int k = 0;
               foreach (PostProcessEffect e in effects)
               {
                   int l = 0;
                   foreach (PostProcessComponent c in e.Components)
                   {
                       //skip previous components and ourself
                       if (k < i)
                       {
                           continue;
                       }
                       else if (k == i && l <= j)
                       {
                           l++;
                           continue;
                       }
                       else if (c.RequiresSceneTexture)
                       {
                           component.OnUpdateSceneTexture += new UpdateSceneTextureEventHandler(c.UpdateSceneTexture);
                       }

                       l++;
                   }

                   k++;
               }
           }

           //add the compontent's UpdateBackBuffer method to the event handler
           if (component.RequiresBackbuffer
               (effect == effects[0] && component == effect.Components[0]))
               OnBackBufferResolve += new BackBufferResolveEventHandler(component.UpdateBackbuffer);

           if (component.RequiresDepthBuffer)
               OnDepthBufferResolve += new DepthBufferResolveEventHandler(component.UpdateDepthBuffer);

           j++;
       } //components foreach

       i++;
   } //effects foreach

   if (effects.Count > 0)
   {
       //ensure the last component renders to the backbuffer
       effects[effects.Count - 1].IsFinal = true;

       if (OnDepthBufferResolve != null)
       {
           depthBuffer = new BuildZBufferComponent(mContent, mGraphicsDevice);
           depthBuffer.Camera = camera;
           depthBuffer.Models = models;
           depthBuffer.LoadContent();
       }
   }
}

Here we first setup the resolve texture that will resolve the backbuffer every frame. Then we search through the components, looking for ones that update the backbuffer(scene texture). If a component does this, then we need to find all the components after it that need the most recent version of the backbuffer and assign its update function to the OnUpdateSceneTexture event. Next, we search through all the effects, looking for components that need the backbuffer and depth buffer and attach its backbuffer update function to the OnBackBufferResolve and OnDepthBufferResolve events.

Below is what a sample post process effect chain could look like:

Here we have three post process effects: bloom, motion blur, and depth of field. Each contains its own components that perform operations on the texture given to it. Some components need to blend with the backbuffer, and therefore need to update the backbuffer in turn so that other components are able to have the most recent copy of the backbuffer (components linked with red lines).

Here is an example of how we could setup the above effect chain:

PostProcessEffect bloom = new PostProcessEffect(Content, Game.GraphicsDevice);
bloom.AddComponent(new BrightPassComponent(Content, Game.GraphicsDevice));
bloom.AddComponent(new GaussBlurComponent(Content, Game.GraphicsDevice));
bloom.AddComponent(new BloomCompositeComponent(Content, Game.GraphicsDevice));

PostProcessEffect motionblur = new PostProcessEffect(Content, Game.GraphicsDevice);
motionblur.AddComponent(new MotionVelocityComponent(Content, Game.GraphicsDevice));
motionblur.AddComponent(new MotionBlurHighComponent(Content, Game.GraphicsDevice));

PostProcessEffect depthOfField = new PostProcessEffect(Content, Game.GraphicsDevice);
depthOfField.AddComponent(new DownFilterComponent(Content, Game.GraphicsDevice));
depthOfField.AddComponent(new PoissonBlurComponent(Content, Game.GraphicsDevice));
depthOfField.AddComponent(new DepthOfFieldComponent(Content, Game.GraphicsDevice));

postManager.AddEffect(bloom);
postManager.AddEffect(motionblur);
postManager.AddEffect(depthOfField);

In the sample, I have included 3 post processes: bloom, depth of field, and a distortion process. I've documented the classes and functions pretty well, so these combined with this write up should be enough to help you understand how everything works. Should you have any questions, just post in the comments.

Homework for the reader:

In this sample, I create a new render target fore every component that needs one. This is bad. In my own implementation that I use for my own projects, I create a render target pool. The render target pool stores and fetches render targets. When a componet requests a render target, the pool will check to see if one of the specified dimensions and format has been stored already. If it has it will return it, if it hasn't it will return a new render target. Now we won't create a new render target for every component. The pool allows us to reuse render targets which helps save gpu memory and increase performance. So if we have 5 effects with 3 components each, we might only create 5 render targets (depending on how many share similar formats and dimensions) instead of 15.

Edit (6/6/2008):Added support for cards that do not support the R32F (Single) SurfaceFormat (hopefully :) ). Removed in-progress auto focusing code in depth-of-field effect.

Edit (6/5/2008): Added some new functionality, namely being able to specify the states and color channel modulation of the spritebatch.

New XNA site and other happenings

2008-05-20T16:49:00.003-04:00

If you didn't know there's a new version of the creators site up, go check it out.

I've been pretty busy lately. This summer I have an internship with a local game developer, Gabriel Entertainment. They have me working with a Wii dev kit to help bring one of their PC games to the Wii. It's pretty cool stuff.

I'm also working on a remake of the Missile Game 3D flash game in XNA in my free time. I'll be posting the post process manager that I'm using for this project once I port it from XNA 1.0 to 2.0. So stay tuned for this. I'll probably include with this motion blur and depth-of-field post processes.

Camera Animation, Part II

2008-05-03T12:45:00.013-04:00

Camera Animation with a cubic spline:

So last time I lied a little bit about our camera path. I showed this picture:

to describe the path that we wanted. But with linear interpolation, what we really ended up with was something like:

But today, I'll introduce cubic spline interpolation, and this will produce a truly smooth curve. Most of the code is the same, the only thing that has changed is that I have added a cubic option to the build function and have changed the animation to be reliant on time.

Building the natural cubic spline is more math intensive than with linear interpolation, so to start out here's a few links on cubic splines:

//a point on a cubic spline is affected by every other point
//so we need to build the cubic equations for each control point
//by looking at the entire curve. This is what calculateCubicSpline does
Vector3[] looks = new Vector3[mKeyFrames.Count];
Vector3[] positions = new Vector3[mKeyFrames.Count];
Vector3[] ups = new Vector3[mKeyFrames.Count];

for (int i = 0; i < mKeyFrames.Count; i++)
{
    looks[i] = mKeyFrames[i].Look;
    positions[i] = mKeyFrames[i].Position;
    ups[i] = mKeyFrames[i].Up;
}

Cubic[] pos_cubic = calculateCubicSpline(mKeyFrames.Count - 1, positions);
Cubic[] look_cubic = calculateCubicSpline(mKeyFrames.Count - 1, looks);
Cubic[] up_cubic = calculateCubicSpline(mKeyFrames.Count - 1, ups);

for (int i = 0; i < mKeyFrames.Count - 1; i++)
{
    for (int j = 0; j < mPathSteps; j++)
    {
        float k = (float)j / (float)(mPathSteps - 1);

        Vector3 center = pos_cubic[i].GetPointOnSpline(k);
        Vector3 up = up_cubic[i].GetPointOnSpline(k);
        Vector3 look = look_cubic[i].GetPointOnSpline(k);

        Vector3 r = Vector3.Cross(up, look);
        Vector3 u = Vector3.Cross(look, r * -1f);

        Camera cam = new Camera();
        cam.SetLens(mFOV, mAspect, mNearZ, mFarZ);
        cam.Place(center, look, u);

        cam.BuildView();

        mCameraPath.Add(cam);
    }
}

So the first thing we do is build the cubic polynomials for each control point. And because each point controls the shape of the spline, we have to consider each point when building the polynomials. In this way, it is very different from linear interpolation in that we have to know information about the whole curve to perform a cubic interpolation rather than just the current and next control point as with linear interpolation. You might be thinking that "well Vector3 already contains a SmoothStep function that performs cubic interpolation". And you're right it does, but just interpolating the two points without knowledge of the whole curve still produces segmented(i.e. not smooth) animation. Once we have the polynomials for each control point, we can perform a cubic interpolation by just looking at the current and next control point, in the same fashion that we did with linear interpolation.

This is what the calculateCubicSpline function does, it builds polynomials for each control point, successively building off of the last control point's polynomial. Let's look at it:

Vector3[] gamma = new Vector3[n + 1];
Vector3[] delta = new Vector3[n + 1];
Vector3[] D = new Vector3[n + 1];
int i;
/* We need to solve the equation
 * taken from: http://mathworld.wolfram.com/CubicSpline.html
   [2 1       ] [D[0]]   [3(v[1] - v[0])  ]
   : 1 4 1    : :D[1]:   :3(v[2] - v[0])  :
   :  1 4 1   : :  . : = :       .        :
   :  .....   : :  . :   :       .        :
   :     1 4 1: :  . :   :3(v[n] - v[n-2]):
   [       1 2] [D[n]]   [3(v[n] - v[n-1])]

   by converting the matrix to upper triangular.
   The D[i] are the derivatives at the control points.
 */

//this builds the coefficients of the left matrix
gamma[0] = Vector3.Zero;
gamma[0].X = 1.0f / 2.0f;
gamma[0].Y = 1.0f / 2.0f;
gamma[0].Z = 1.0f / 2.0f;
for (i = 1; i < n; i++)
{
    gamma[i] = Vector3.One / ((4 * Vector3.One) - gamma[i - 1]);
}
gamma[n] = Vector3.One / ((2 * Vector3.One) - gamma[n - 1]);

delta[0] = 3 * (v[1] - v[0]) * gamma[0];
for (i = 1; i < n; i++)
{
    delta[i] = (3 * (v[i + 1] - v[i - 1]) - delta[i - 1]) * gamma[i];
}
delta[n] = (3 * (v[n] - v[n - 1]) - delta[n - 1]) * gamma[n];

D[n] = delta[n];
for (i = n - 1; i >= 0; i--)
{
    D[i] = delta[i] - gamma[i] * D[i + 1];
}

// now compute the coefficients of the cubics
Cubic[] C = new Cubic[n];
for (i = 0; i < n; i++)
{
    C[i] = new Cubic(v[i], D[i], 3 * (v[i + 1] - v[i]) - 2 * D[i] - D[i + 1],
             2 * (v[i] - v[i + 1]) + D[i] + D[i + 1]);
}
return C;

Once we find the cubic equation for each control point (key frame), we can use GetPointOnSpline() uses the cubic equation to calculate the intermediate point along the cubic spline.

public Vector3 GetPointOnSpline(float s)
{
    return (((d * s) + c) * s + b) * s + a;
}

And that's it. Now we have a smooth animation of our camera. There are many more ways to calculate the curve, including: hermite curves, b-splines, bezier curves, catmull-rom splines, NURBS(Nonuniform Rational B-Splines) and others. Microsoft also has a sample on creating a camera path that appears to be using hermite curves. But they hide all the details from you, and it's just more interesting to know the math behind it isn't it ;) ?

Camera Animation, Part I

2008-05-01T14:13:00.008-04:00

Due to the video being such high frequency, the blogger, youtube video compressors always make the video look like crap. Original can be found here:

So today I'll be giving a tutorial/sample on cut-scene animation, more specifically camera animation. This will be a 2 part tutorial, and this one will focus on linear interpolation of camera key frames.

So what are key frames? Key frames are the control points of an animation. They are the main/key points that you specifically define where the camera should be, and everything else in between is interpolated. You can also think of them as an outline of what the animation should look like. Here's a picture to help describe what I'm talking about.

Here, the green line is the path we want to follow, the orange dots are our key frames, and the red dots are the intermediate cameras that we want to generate.

So how do we create these intermediate cameras?

for (int i = 0; i < mKeyFrames.Count - 1; i++)
{
    for (int j = 0; j < mPathSteps; j++)
    {
        //We could alternatively use the Vector3.lerp function
        //but I wanted to show the math behind performing the linear interpolation

        Vector3 diff = mKeyFrames[i + 1].Position - mKeyFrames[i].Position;
        diff = Vector3.Multiply(diff, (float)j / (float)(mPathSteps - 1));
        Vector3 center = mKeyFrames[i].Position + diff;

        diff = mKeyFrames[i + 1].Look - mKeyFrames[i].Look;
        diff = Vector3.Multiply(diff, (float)j / (float)(mPathSteps - 1));
        Vector3 look = mKeyFrames[i].Look + diff;

        diff = (mKeyFrames[i + 1].Up - mKeyFrames[i].Up);
        diff = Vector3.Multiply(diff, (float)j / (float)(mPathSteps - 1));
        Vector3 up = mKeyFrames[i].Up + diff;

        Vector3 r = Vector3.Cross(up, look);
        Vector3 u = Vector3.Cross(look, r * -1f);

        Camera cam = new Camera();
        cam.SetLens(mFOV, mAspect, mNearZ, mFarZ);
        cam.Place(center, look, u);

        cam.BuildView();

        mCameraPath.Add(cam);
    }
}

Pretty self explanatory, a basic linear interpolation. XNA is handy in that we can cut out all of that code and just use Vector3.Lerp().

The included demo has a pre defined path that is shown in the video. But you can also create your own path at run-time. Here are the controls:

1 - Default camera view
2 - Play the camera animation
3 - Rewind the camera animation
4 - Pause the camera animation
C - Clear the current key frames
K - Add the current camera to the key frames list
B - Build the camera path from the current key frames
O - Save the current key frames to an XML file

When you start the demo, the predefined path is already loaded, all you have to do is hit '2', and it will play. If you want to make your own path first clear the key frames by pressing 'C', and then navigate to where you want to be along your path and press 'K' to add your key frames. Once you're happy with your path, press 'B' to build the camera animation, and then you can play it by pressing '2'. Normally, I would use winForms or a GUI such as neoforce for the controls, but this adds a lot of clutter.

So there's probably one thing that you might notice about the camera animation, and that is that it's not very smooth. Next time I will introduce cubic interpolation to smooth out the animation.

Reflections with Billboard Impostors

2008-04-17T12:00:00.030-04:00

Last time I mentioned that I would post a tutorial on generating reflections using billboard impostors, well here it is. Amidst the craziness of school right now, I've found time (or rather worked on this since it's more fun) to write up this tutorial. So lets get started.

Some prerequisites for this tutorial:

Shader Model 3.0 compliant graphics card
Experience with RenderTargets
Experience with Custom Content Processors (not too important, I just won't go over these specifically)

Anyways, Shawn Hargreaves posted about using environment maps for reflections a few weeks ago. But we're going to take it one step further and use impostors as well as the environment map for reflections. The main problem with environment mapped refections is that everything is assumed to be infinitely far away, and thus objects appear distorted and don't intersect with geometry correctly. Using billboarded impostors tries to alleviate some of this problem by taking the reflected rays and intersecting them with billboards of the scene geometry.

So here's a couple of images to demonstrate the limitations of environment mapping.

Environment mapping:

As you can see the sphere looks like it's floating above the quad when it is actually intersecting it. Also, the dragon's reflection is too far away in the reflection.

Impostors:

Now with the impostors, the reflection of the dragon is much more accurate, and the sphere now correctly intersects the floor quad.

The details:
First we want to render each diffuse object from the reflector's point of view. So we first get the bounding box corners of the impostor geometry and setup the camera to be positioned at the center of the reflector, and looking at the center of the diffuse geometry.

// Get the 8 corners of the bounding box and transform by the world matrix
Vector3[] corners = new Vector3[8];
Vector3.Transform(mBoundBox.GetCorners(), ref mWorld, corners);

// Get the transformed center of the mesh
// alternatively, we could use mBoundingSphere.Center as sometimes it works better
Vector3 meshCenter = Center;

// The quad is a special case. Since it's so much bigger than the reflector
// we have to make sure that our camera is looking straight at the quad and
// not it's center or we'll get a distorted view of the quad.
if (mMeshType == MeshType.Quad)
{
   meshCenter.X = reflectorCenter.X;
   meshCenter.Z = reflectorCenter.Z;
}

// Construct a camera to render our impostor
Camera impostorCam = new Camera(camera);

// Set the camera to be at the center of the reflector and to look at the diffuse mesh
impostorCam.LookAt(reflectorCenter, meshCenter);
impostorCam.BuildView();

Next we want to project these corners to screen space and find the bounding box that fits around these projected corners:

// Now we project the vertices to screen space, so we can find the AABB of the screen
// space vertices
Vector3[] screenVerts = new Vector3[8];
for (int i = 0; i < 8; i++)
{
   screenVerts[i] = mGraphicsDevice.Viewport.Project(corners[i],
                           impostorCam.Projection, impostorCam.View, mWorld);
}

// compute the screen space AABB
Vector3 min, max;
ComputeBoundingBoxFromPoints(screenVerts, out min, out max);

// construct the quad that will represent our diffuse mesh
Vector3[] screenQuadVerts = new Vector3[4];
screenQuadVerts[0] = new Vector3(min.X, min.Y, min.Z);
screenQuadVerts[1] = new Vector3(max.X, min.Y, min.Z);
screenQuadVerts[2] = new Vector3(max.X, max.Y, min.Z);
screenQuadVerts[3] = new Vector3(min.X, max.Y, min.Z);

Now we want to unproject these screen space vertices into world space so that we can form a 3D impostor quad of our geometry. We will use this quad later in the reflection shader for a reflective object.

We also render our diffuse object to a RenderTarget that will be the texture for our impostor quad. To do this, we setup an Orthographic Projection so that we don't have any of the perspective distortion that comes with a regular perspective projection matrix. We clear the scene with zero alpha, and we render the impostor with full alpha so that we only capture the diffuse object and nothing else.

//now unproject the screen space quad and save the
//vertices for when we render the impostor quad
for (int i = 0; i < 4; i++)
{
   mImpostorVerts[i] = mGraphicsDevice.Viewport.Unproject(screenQuadVerts[i],
                           impostorCam.Projection, impostorCam.View, mWorld);
}

//compute the center of the quad
mImpostorCenter = Vector3.Zero;
mImpostorCenter = mImpostorVerts[0] + mImpostorVerts[1] + mImpostorVerts[2] + mImpostorVerts[3];
mImpostorCenter *= .25f;

// calculate the width and height of the imposter's vertices
float width = (mImpostorVerts[1] - mImpostorVerts[0]).Length() * 1.2f;
float height = (mImpostorVerts[3] - mImpostorVerts[0]).Length() * 1.2f;

// We construct an Orthographic projection to get rid of the projection distortion
// which we don't want for our impostor texture
impostorCam.Projection = Matrix.CreateOrthographic(width, height, .1f, 100);
impostorCam.BuildView();

//save the WorldViewProjection matrix so we can use it in the shader
mWorldViewProj = impostorCam.ViewProj;

mGraphicsDevice.SetRenderTarget(0, mRenderTarget);
mGraphicsDevice.Clear(ClearOptions.Target, new Color(1, 0, 0, 0), 1.0f, 1);

Draw(impostorCam);

mGraphicsDevice.SetRenderTarget(0, null);


// finally, compute the normal for the impostor quad, and push the vertices
// back to the center of the diffuse mesh
Vector3 trans = meshCenter - mImpostorCenter;
for (int i = 0; i < 4; ++i)
{
   mImpostorVerts[i] += trans;
}

mNormal = impostorCam.Look;

Here's our resulting impostor image:

Now that we have constructed our impostor quad, it's time to render our reflective object. Here's the pixel shader code that we will use to do this:

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
   input.NormalW = normalize(input.NormalW);
  
   float3 fromEyeW = normalize(input.PosW - EyePos);
  
   //find the relflected ray by reflecting the fromEyeW vector across the normal
   float3 reflectedRay = reflect(fromEyeW, input.NormalW);
  
   float4 finalColor = texCUBE(EnvTex, reflectedRay);
  
   if(UseImpostors)
   {
       for(int i = 0; i < 2; i++)
       {
           //take the dot product of the reflected ray and the normal of the impostor
           //quad to see how orthogonal the ray is to the quad
           //if a = 0, then the ray is parallel to the quad
           //if a = 1, then it is orthogonal to the quad
           float a = dot(-reflectedRay, Impostors[i].Normal);

           //if less than this, the ray is nearly or entirely parallel to the plane
           if(a > 0.001f)
           {                   
               //we construct the vector from a point on the quad to the pixel position   
               float3 vec = Impostors[i].Vertex - input.PosW;
              
               //the signed distance from the pixel position to the quad is given by the negative
               //dot product of the quad normal and vec
               float b = -dot(vec, Impostors[i].Normal);

               //divide b by a to get our distance from the world pixel position
               float r = b / a;

               if(r >= 0)
               {
                   //Using the Ray equation: P(t) = S + tV. Where S is the orgin, V is the direction,
                   //and t is the distance along V
                   //We find the intersection by starting at the origin (PosW), and walk to the end
                   //of the ray by multiplying the reflectedRay by r - the distance to the quad
                   //and adding it to the orgin
                   float3 intersection = input.PosW + r * reflectedRay;

                   float2 texC;
                  
                   //project the intersection point with the WVP matrix used to render the quad
                   float4 projIntersect = mul(float4(intersection, 1.0), Impostors[i].WVP);

                   //perform the perspective divide and transform to NDC coordinates [0, 1]
                   texC = projIntersect.xy / projIntersect.w * .5 + .5;

                   //make sure the intersection is in the bounds of the image [0, 1]
                   if((texC.x <= 1 && texC.y <= 1) &&
                      (texC.x >= 0 && texC.y >= 0))
                   {
                       float4 color = tex2D(ImpostorSamplers[i], float2(texC.x, 1-texC.y));
                      
                       //blend based on the alpha of the sampled color
                       finalColor = lerp(finalColor, color, color.a);
                   }
               }
           }
       }
   }
  
   finalColor.rgb *= MaterialColor;
   return finalColor;
}

So, we iterate over each impostor in the Impostors array. We find the reflected ray as with environment mapping, and we use this ray to see if it intersects any of the impostor quads in our scene. If we find an intersection, we get the color from the impostor texture and blend with the environment map color based on the alpha component of the color from the impostor texture (the alpha will be zero for any part of the texture that isn't part of the diffuse object).

That's pretty much all there is to it. Not too complicated, and still very fast. The demo runs pretty fast: ~400 FPS with 16x MSAA at 1024x768 on my 8800GT.

Limitations:
One of the problems with billboard impostors is that there is no motion parallax. Also, the intersection of complex objects is not possible. This is where depth impostors and non-pinhole impostors come in. Depth impostors allow correct intersections and motion parallax. And non-pinhole impostors add on to depth impostors by allowing you to see almost the entire diffuse object from one viewpoint (whereas depth impostors suffer from occlusion errors because it can only capture the front face of any object).

Paper Submitted to Eurographics

2008-04-08T14:00:00.005-04:00

Well we finally got our paper submitted to Eurographics Symposium on Rendering, with 5 minutes to spare before deadline :). I think I had about 3 hours of sleep in the last few days. Hopefully we'll get accepted. The title of our paper was Non-Pinhole Impostors.

So now I have to implement my own malloc for Operating Systems, and a databases project due in the next week. I'm going to try to work on the Billboard Impostors tutorial that I mentioned last post also. Busy times.

Reflections with Depth Impostors

2008-03-31T23:13:00.008-04:00

Since the beginning of the semester I've been researching reflections using Occlusion Camera Depth Impostors. The occlusion camera had been developed in the same graphics lab a few years ago, and my task was to apply it to depth impostors. The idea of using depth impostors is that, you can render a diffuse object to a buffer, and then use the color and depth information to correctly intersect the depth map with reflected rays from a reflective surface. Now the problem here is that we can only see so much from the viewpoint of the reflector. Meaning that some rays would actually intersect the real diffuse object but won't intersect our depth map because we don't have enough information.

What the occlusion camera brings to the table is the ability to store more depth information from the diffuse object. It does this by distorting rays such that we can sample the top and bottom of the diffuse object, where if we were to use a regular camera we wouldn't capture these areas. Yes, kind of vague and hard to visualize without pictures, but those will come in due time. It's a really cool topic of research that I will discuss in more depth once we submit our paper to the Eurographics Symposium.

But for now I'll leave you with some pictures of some early work that I did as a prerequisite for the research. Also, in the coming days (read- after our paper deadline) I will post a tutorial on using billboard impostors for reflections in XNA.

A reflected bunny with correct intersections

3rd order and 2nd order reflections showing the impostor normals

2nd order reflections

Volumetric Clouds

2008-03-25T02:56:00.016-04:00

Last summer I did some experiments with an unusual way of rendering volumetric clouds. The usual way to render volumetric clouds is with a bunch of quads/voxels/slices. But this approach is much simpler, and at least for the exterior, gives really good results.

The problem came when rendering multiple clouds in a field. Due to the way the cloud is rendered, overlapping clouds would produce artifacts, and short of rendering each cloud by itself I couldn't figure out how to eliminate the artifacts. But then again, I didn't spend too much time on them either.

The general approach is as follows:

Draw your scene as usual
Then render a low poly cloud mesh - this could be created in Max/Maya with a group of spheres/ellipsoids in the shape of a cloud
Copy the backbuffer to a render target, and clear the alpha channel
Render the clouds with full alpha and your lighting of choice (I only implemented simple phong shading, a better lighting algorithm would have helped). We'll call this the cloud map.
Blur the cloud map using a Gaussian filter, or another if you like.
After blurring we need to distort the blurred cloud map. To do this, we place a quad at the center of each cloud and billboard each vertex to the camera (make sure the billboard covers the entire cloud from any angle).
Now we render this quad to distort the cloud map. In the pixel shader for the quad we use the projected position as the texture coordinates and shift the texture coordinates based on the angles to the X and Y axis. We then sample a 2-channel fractal/noise texture with these texture coordinates to obtain our distortion offset. Next, we sample the blurred cloud map using the texture coordinates distorted by the distortion offset and the distance from the quad to the camera:
float4 distortedColor = tex2D(BlurredCloudSampler, texC + offset/ dist);
Optionally, after we have distorted our cloud map, we can perform a radial blur for a softer look.
Finally we merge our render target with the back buffer.

A visual representation of the process:

A really simple algorithm that produces a pretty good result. And as you can see, it is essentially a post processing technique. However, it seems that this approach is more suitable to large volcanic or nuclear plumes rather than a dense field of many cumulus clouds (e.g. a large cloud model with many mega particles, as can be seen in the author's images).

My experiment:

Author's images:

There's an article on this technique in Shader X5 and you can find the slides here. If you're thinking of trying this technique out, you really don't need the book. The slides are more than you need to implement it. The only detail is that you don't actually use a fractal cube like the slides say, you really just use a billboarded quad.

Fluid Dynamics

2008-03-22T13:15:00.010-04:00

Fluid dynamics has always been an interest of mine. I've just never found the time to research any of the techniques. Hopefully sometime this summer I will have time...

Anyways, the work by Ron Fedkiw is just amazing. Take a look at his website to see for your self.

http://physbam.stanford.edu/~fedkiw/

More Software Rendering...

2008-03-20T19:49:00.005-04:00

Quick update. Just found a video I had made of the software renderer.

Not perfect, but pretty good I'd say.

Technorati Profile

Terrain Rendering and Atmospheric Scattering

2008-03-20T12:30:00.018-04:00

Terrain has always been an interest of mine, ever since I loaded my first 8-bit raw heightmap. And last summer I decided to dive into Atmospheric Scattering after reading Ysaneya's developer journal over on gamedev for quite some time.

So I read ATi's paper and looked at their demo and set out to work. Now, I'm not math genious (having only taken calculus and linear algebra), but implementing atmospheric scattering is not for the faint of heart. However, after about a month I had a pretty good working implementation based on the Hoffman and Preetham paper.

Then I started on the water rendering component. There were two articles in Shader X3/4 that were of great help when it came to getting the color just right.

After a couple of months I had a pretty good looking demo. The terrain wasn't anything special, it was just broken down into a quadtree, so I was only able to render 2048x2048 terrain. In order to do this I made the water be at half the height of the terrain and culled the non-visible areas when i was above/below the water, so I was effectively only rendering half of the terrain when I was above/below the water line.

Anyways, on to the pictures:
Scene details:

1024x1024 terrain with multi-texturing and aerial perspective
Sky dome with sun and skylight scattering
2048x2048 water plane(size not number of vertices)
Bloom post processing
Written in c# and managed directx

And a video of it in action:

The water has realistic coastal coloring, soft edges when it intersects the terrain, under water fogging, and depth fogging.

This summer I plan to rewrite the whole application in c++. I also want to extend the terrain rendering with geomipmapping, fix the huge sun, and add volumetric clouds. I had tried implementing volumetric clouds using mega particles, but didn't turn out too well in a dense cloud field. Worked really well for volcanic or nuclear plumes though. More on this in a later post.

Software Rendering

2008-03-19T15:09:00.016-04:00

About a year ago, I took a class on software rasterization. It really helped me understand everything that DirectX/Opengl does underneath the hood. We started out by rasterizing 2D lines and images and then moved onto a full 3D renderer all done in software (i.e. on the cpu). It was a pretty cool class. The focus of the class was mainly on implementation rather than performance, so it isn't as fast as other software renderers.

I wrote the rasterizer using c#, and the Tao opengl framework (only for sending the pixel information to the graphics card with glDrawPixels()). I posted this on image of the day at gamedev: http://www.gamedev.net/community/forums/topic.asp?topic_id=446142

Here's what I had accomplished by the end of the semester:

Gouraud shading
Phong shading
Blinn and Phong specular reflection
Directional and Point lights
Perspective correct texture mapping
Normal Mapping
Parallax Mapping
Projective Texturing
Shadow Mapping
Environment Mapping - for skybox and distant reflections
Distance Fog
Depth of Field
Bilinear Filtering
Camera Interpolation, for movie creation

environment mapped reflections:

perspective correct texture mapping:

depth of field:

distance based fog:

parallax mapping:

shadow mapping with percentage closer filtering:

Yes... another blog

2008-03-19T14:53:00.003-04:00

Why am I starting this blog you ask? Well, after reading other graphics related blogs for some time now, I figured it would be a good place for me to document not only current work but previous work.

I've been developing 3D demos/applications for a couple of years now. It all started when I took a class on game development for the Sony line of mobile phones. After that I started working with DirectX and haven't stopped since.

My interests lie mainly in terrain and atmosphere effects, but more recently I've become interested in post processing techniques as well. This semester I began researching reflections with my professor. We're developing a method to produce accurate reflections of objects using non-pinhole camera depth impostors. This will allow us to have reflected rays intersect the depth map that would not be able to using a regular depth map impostor (more on this to come).

The first few posts will most likely be historical. I'll mainly be posting these previous few projects as sort of documentation for them.