Tuesday, June 3, 2008

Post Process Framework Sample


Today I'm posting a post processing framework sample. For those that don't know, post processing is any manipulation of the frame buffer after the scene has been rendered (but usually before the UI), hence the word "post" in the name. This process could be a simple blurring, or it could be a motion blur or depth of field technique. But what if you have many post processes? For instance, maybe we have bloom, motion blur, heat haze, depth of field, and other post processes that we need to apply after we render our scene. Connecting all these post processes together can get a little hairy, and that's what this sample helps us do.

So lets talk about how we're going to structure our framework. At the very bottom of the hierarchy, we have a PostProcessComponent (component from here on out). And what this class represents, is a single atomic process, that is, it cannot be broken up into smaller pieces/objects. Each component has an input and an output in the form of textures. A component is also able to update the backbuffer. It is the simplest object in the hierarchy and is combined with other components to form a PostProcessEffect (effect from here on out).

An effect contains a chain of components to implement a single post process such as bloom or motion blur. An effect also handles output from one component to input to another. In this way the components can be independent of one another, and the effect handles linking all the components that it owns. Because of this, each component is very simple. Also, like components, effects have inputs and outputs in the form of textures. Effects can also be enabled/disabled dynamically at runtime.

Next we have the PostProcessManager(aka manager). The manager is comprised of a chain of effects, and it handles linking the output of one effect to the input of the next effect. And just like with components, this enables each effect to be independent of the next. The manager also takes care of linking the backbuffer and depth buffer to components that need it.

Because each component is not dependent on other components, each one is very simple in its implementation. This not only helps in understanding the code, but also in robustness and tracking down any possible errors in a component. Another nice feature about this framework is that you do not need to create a separate class deriving from PostProcessEffect for each effect. In this way, you can dynamically create your effects at run time. Most of the "magic" occurs in the LoadContent() function inside PostProcessManager. So lets have a look at it:

public void LoadContent()
#region Create common textures to be used by the effects
PresentationParameters pp = mGraphicsDevice.PresentationParameters;
int width = pp.BackBufferWidth;
int height = pp.BackBufferHeight;

SurfaceFormat format = pp.BackBufferFormat;

resolveTexture = new ResolveTexture2D(mGraphicsDevice, width, height, 1, format);

i = 0;
foreach (PostProcessEffect effect in effects)

int j = 0;
//if a component requires a backbuffer, add their function to the event handler
foreach (PostProcessComponent component in effect.Components)
//if the component updates/modifies the "scene texture"
//find all the components who need an up to date scene texture
if (component.UpdatesSceneTexture)
int k = 0;
foreach (PostProcessEffect e in effects)
int l = 0;
foreach (PostProcessComponent c in e.Components)
//skip previous components and ourself
if (k < i)
else if (k == i && l <= j)
else if (c.RequiresSceneTexture)
component.OnUpdateSceneTexture += new UpdateSceneTextureEventHandler(c.UpdateSceneTexture);



//add the compontent's UpdateBackBuffer method to the event handler
if (component.RequiresBackbuffer
(effect == effects[0] && component == effect.Components[0]))
OnBackBufferResolve += new BackBufferResolveEventHandler(component.UpdateBackbuffer);

if (component.RequiresDepthBuffer)
OnDepthBufferResolve += new DepthBufferResolveEventHandler(component.UpdateDepthBuffer);

} //components foreach

} //effects foreach

if (effects.Count > 0)
//ensure the last component renders to the backbuffer
effects[effects.Count - 1].IsFinal = true;

if (OnDepthBufferResolve != null)
depthBuffer = new BuildZBufferComponent(mContent, mGraphicsDevice);
depthBuffer.Camera = camera;
depthBuffer.Models = models;

Here we first setup the resolve texture that will resolve the backbuffer every frame. Then we search through the components, looking for ones that update the backbuffer(scene texture). If a component does this, then we need to find all the components after it that need the most recent version of the backbuffer and assign its update function to the OnUpdateSceneTexture event. Next, we search through all the effects, looking for components that need the backbuffer and depth buffer and attach its backbuffer update function to the OnBackBufferResolve and OnDepthBufferResolve events.

Below is what a sample post process effect chain could look like:


Here we have three post process effects: bloom, motion blur, and depth of field. Each contains its own components that perform operations on the texture given to it. Some components need to blend with the backbuffer, and therefore need to update the backbuffer in turn so that other components are able to have the most recent copy of the backbuffer (components linked with red lines).

Here is an example of how we could setup the above effect chain:

PostProcessEffect bloom = new PostProcessEffect(Content, Game.GraphicsDevice);
bloom.AddComponent(new BrightPassComponent(Content, Game.GraphicsDevice));
bloom.AddComponent(new GaussBlurComponent(Content, Game.GraphicsDevice));
bloom.AddComponent(new BloomCompositeComponent(Content, Game.GraphicsDevice));

PostProcessEffect motionblur = new PostProcessEffect(Content, Game.GraphicsDevice);
motionblur.AddComponent(new MotionVelocityComponent(Content, Game.GraphicsDevice));
motionblur.AddComponent(new MotionBlurHighComponent(Content, Game.GraphicsDevice));

PostProcessEffect depthOfField = new PostProcessEffect(Content, Game.GraphicsDevice);
depthOfField.AddComponent(new DownFilterComponent(Content, Game.GraphicsDevice));
depthOfField.AddComponent(new PoissonBlurComponent(Content, Game.GraphicsDevice));
depthOfField.AddComponent(new DepthOfFieldComponent(Content, Game.GraphicsDevice));


In the sample, I have included 3 post processes: bloom, depth of field, and a distortion process. I've documented the classes and functions pretty well, so these combined with this write up should be enough to help you understand how everything works. Should you have any questions, just post in the comments.

Homework for the reader:

In this sample, I create a new render target fore every component that needs one. This is bad. In my own implementation that I use for my own projects, I create a render target pool. The render target pool stores and fetches render targets. When a componet requests a render target, the pool will check to see if one of the specified dimensions and format has been stored already. If it has it will return it, if it hasn't it will return a new render target. Now we won't create a new render target for every component. The pool allows us to reuse render targets which helps save gpu memory and increase performance. So if we have 5 effects with 3 components each, we might only create 5 render targets (depending on how many share similar formats and dimensions) instead of 15.

Edit (6/6/2008):Added support for cards that do not support the R32F (Single) SurfaceFormat (hopefully :) ). Removed in-progress auto focusing code in depth-of-field effect.

Edit (6/5/2008): Added some new functionality, namely being able to specify the states and color channel modulation of the spritebatch.


Charles Humphrey said...

Not got time to read the post, but looks awesome mate. Can't wait to have a play with it. Nice one.

Kyle Hayward said...

Hey thanks. Hope you enjoy playing around with it.

I updated the source with some new functionality and a couple of fixes.

Charles Humphrey said...

Sorry for not looking more into this (and creating a huge note), but I get an unexpected error when the Blur LoadConetnt sets the outputRT

This line:
outputRT = new RenderTarget2D(graphics, width, height, 0, backBuffer.Format);

in PoissonBlurComponent.cs

Exception message reads:
"An unexpected error has occurred."

Stack looks like this:
" at Microsoft.Xna.Framework.Graphics.RenderTarget.CreateRenderTarget(GraphicsDevice graphicsDevice, Int32 width, Int32 height, Int32 numberLevels, SurfaceFormat format, MultiSampleType multiSampleType, Int32 multiSampleQuality, RenderTargetUsage usage, Boolean isTexture2D, _D3DSURFACE_DESC* pDesc)
at Microsoft.Xna.Framework.Graphics.RenderTarget..ctor(GraphicsDevice graphicsDevice, Int32 width, Int32 height, Int32 numberLevels, SurfaceFormat format, MultiSampleType multiSampleType, Int32 multiSampleQuality, RenderTargetUsage usage, Boolean isTexture2D)
at Microsoft.Xna.Framework.Graphics.RenderTarget2D..ctor(GraphicsDevice graphicsDevice, Int32 width, Int32 height, Int32 numberLevels, SurfaceFormat format)
at PostProcessingSample.PoissonBlurComponent.LoadContent() in C:\Documents and Settings\CH2\Desktop\PostProcessing\PostProcessingSample\PostProcessingSample\PostProcessing\Components\PoissonBlurComponent.cs:line 51
at PostProcessingSample.PostProcessEffect.LoadContent() in C:\Documents and Settings\CH2\Desktop\PostProcessing\PostProcessingSample\PostProcessingSample\PostProcessing\PostProcessEffect.cs:line 93
at PostProcessingSample.PostProcessManager.LoadContent() in C:\Documents and Settings\CH2\Desktop\PostProcessing\PostProcessingSample\PostProcessingSample\PostProcessing\PostProcessManager.cs:line 187
at PostProcessingSample.PostProcessingSample.LoadContent() in C:\Documents and Settings\CH2\Desktop\PostProcessing\PostProcessingSample\PostProcessingSample\PostProcessingSample.cs:line 123
at Microsoft.Xna.Framework.Game.Initialize()
at PostProcessingSample.PostProcessingSample.Initialize() in C:\Documents and Settings\CH2\Desktop\PostProcessing\PostProcessingSample\PostProcessingSample\PostProcessingSample.cs:line 106
at Microsoft.Xna.Framework.Game.Run()
at PostProcessingSample.Program.Main(String[] args) in C:\Documents and Settings\CH2\Desktop\PostProcessing\PostProcessingSample\PostProcessingSample\Program.cs:line 14"

Any ideas, or is it my crappy GCard :P

Kyle Hayward said...

Have you tried not adding the depth of field effect? Maybe your graphics card can't handle that many render targets? I'm using the default back buffer format so that shouldn't be the problem. I've only tested on a 7900 and an 8800.

One thing that I notice, is that there is a 0 for the number of mipmap levels in the render target creation. That should be a 1.

I'll put up a fix for that, and we'll see if that's it.

--Edit: just tested on a 6800 with the latest version and it works.

Charles Humphrey said...

Well, that fixed that issue...BUT. The next error I get is "The device does not support creating a render target of the given format.
Parameter name: format"

on this line:
outputRT = new RenderTarget2D(graphics, width, height, 1, SurfaceFormat.Single);

in the LoadContent of BuildZBufferComponent.cs

Guess my card sux, so will remove that PP and see if it works.

Yes it did. Shame was that PP effect I was really after...oh well I will try a 360 build and see if that works.

Kyle Hayward said...

Well all the buildZBuffer component does is save the depth to a texture. So you could use your own method of doing that.

:) Maybe I'll update it to save the depth to the default back buffer format instead of SurfaceFormat.Single.

Does your card support VS/PS 2.0?

--EDIT: I (hopefully) added support for cards (by storing the depth across the RGBA channels) that do not support the R32F(Single) SurfaceFormat.

diwidog said...

Hey Thanks, I want you to keep it nice and simple just like you have.

Charles Humphrey said...

Sweet, will get a copy and give it a go.

Yes my poop GC does support SM2.0 :P

Charles Humphrey said...

ooooooohhhhhhhhhh! How nice is that Depth of Field effect!

Thanks for the fixes Kyle, good work mate.

Kyle Hayward said...

Hey, no problem! I'm glad it's working for ya.

Glad you like the sample diwidog. :)

javacoder said...

why do user need to set nearclip and farclip in depth of field?? nearclip and farclip will already be there in the engine, right ?

Kyle Hayward said...

You need to set your near and far focus planes so that the blur happens before/after these, and so that you can dynamically modify these at run-time. If you used the near/far clip planes used by the projection matrix, everything would be in focus, since geometry closer than near is culled, and geometry farther than far is culled.