3
\$\begingroup\$

Right now I am not using any depth information in my engine, but I just found out that it would be neat to be able to render the silhouette of important game objects that are "behind" something with a shader, though I am quite at a loss how to do this in a 2D scene with little to no depth information. I will propably have to use depth information for something like that, but I am not quite sure how.

I guess the abstract concept would be:

zn = depth of pixel that should be rendered zc = depth of pixel at current position in some sort of buffer if(zn < zc) render pixel with a "silhouette" color else render pixel with it's intended color

Is this a proof of concept I can follow in my next few workhours? How can I use and access stuff like the Z buffer of the current framebuffer in the shader?

As an example, I most recently saw this in Titan Souls:

left half is obscured, right half is rendered normally

Also remember that I am thinking 2D only, so application of the painter's algorithm might or might not be desirable.

EDIT: So right now, from comments and such, I got the following proof of concept:

activate stencil mechanism to set stencil render obscuing objects deactivate stencil render scene, using painter's algorithm or depth checks to obscure activate stencil mechanism to check weather or not something got rendered within the stencil area render important objects again, using a different fragment shader to render pixels within the stencil as a dark shadow

amirite?

\$\endgroup\$
15
  • \$\begingroup\$ Wanting to sample THE depth buffer (not A depth buffer) usually indicates a design problem. Additionally, sampling THE depth buffer is impossible. While it is possible that early-z will calculate every depth value before the pixel shader is invoked, the values are not written until after the pixel shader completes, making them defacto unsampleable. \$\endgroup\$ Commented Apr 6, 2015 at 3:36
  • \$\begingroup\$ There would probably be work-arounds to the implications but, in the end, you cannot bind a buffer as both a render target(or depth buffer) and a shader resource, at the same time. You can copy the previous frame's depth-buffer into a shader resource but that, specifically, is the design flaw indicator. \$\endgroup\$ Commented Apr 6, 2015 at 3:41
  • \$\begingroup\$ If this indeed does not work, how is that very common effect created then? Even in 3D games, if the view to important objects is obscured, you can most often see a shadow / an outlining of the object on the obscuring object. \$\endgroup\$ Commented Apr 6, 2015 at 10:32
  • \$\begingroup\$ I would have to guess that they used the stencil buffer. It is extremely configurable; it can count the number of times a pixel is written, the number of times a pixel fails the depth test (gets clipped), etc., etc. For that image, draw the player sprite first, using depth as usual, but also writing 1's to the stencil anywhere a pixel gets colored. Then, render as usual with the stencil disabled. Finally, render a full-screen quad with alpha-blending active and the stencil set to only render where there are 1's. Tint the existing color any way you'd like. \$\endgroup\$ Commented Apr 6, 2015 at 11:07
  • \$\begingroup\$ Sorry I wrote this in DirectX; I missed the tags. The concept will be the same, although you will need to say potayto instead of my potahto. \$\endgroup\$ Commented Apr 6, 2015 at 11:10

1 Answer 1

Reset to default
1
\$\begingroup\$

Using the stencil buffer:

When writing:

dsDesc.StencilEnable = true; dsDesc.StencilReadMask = D3D11_DEFAULT_STENCIL_READ_MASK; dsDesc.StencilWriteMask = D3D11_DEFAULT_STENCIL_WRITE_MASK; dsDesc.FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS; //Always Pass dsDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_INCR; //Increment dsDesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP; //Doesn't matter dsDesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_INCR; //Increment

When reading:

dsDesc.StencilEnable = true; dsDesc.StencilReadMask = D3D11_DEFAULT_STENCIL_READ_MASK; dsDesc.StencilWriteMask = D3D11_DEFAULT_STENCIL_WRITE_MASK; //Or 0 to disable all writes dsDesc.FrontFace.StencilFunc = D3D11_COMPARISON_GREATER; //Only greather than REF dsDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_KEEP; //No writes dsDesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP; //No writes dsDesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP; //No writes

Render the scene as usual with the stencil buffer active, incrementing.

g_d3dContext->OMSetDepthStencilState(stencilWriteState, 0x00); Draw(...);

Then, render a full-screen quad, using the stencil buffer as a reference to apply the pixel shader effect only to pixels with stencil(>1).

g_d3dContext->OMSetDepthStencilState(stencilReadState, 0x01);

If you know ahead of time, which "important objects" are behind others:
Render them separately, and first, with the stencil active. Then disable the stencil for the remainder of the (occluding) objects. Finally, render the full-screen quad to apply the shader to pixels with stencil(>0).

g_d3dContext->OMSetDepthStencilState(stencilReadState, 0x00);

Example usage:
example stencil usage

\$\endgroup\$
4
  • \$\begingroup\$ OP seems to be using OpenGL, see the glsl and jogl tags. \$\endgroup\$ Commented Apr 6, 2015 at 9:40
  • \$\begingroup\$ @Eejin, I missed that. Any chance of you translating for me? \$\endgroup\$ Commented Apr 6, 2015 at 10:20
  • \$\begingroup\$ As a sprite is a rectangle, can I also affect the stencil buffer with some kind of alpha test so that I can only affect the stencil buffer on pixels that were actually rendered, instead of having a 0 alpha value. \$\endgroup\$ Commented Apr 8, 2015 at 14:44
  • \$\begingroup\$ glEnable(GL_ALPHA_TEST); glAlphaFunc(GL_GREATER, 0.05); //reject any pixels with alpha less than 0.05 \$\endgroup\$ Commented Apr 8, 2015 at 22:47

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.