Unwanted darkening at polygon edges when using normal maps with SSAO

Question

I'm using a SSAO alghorithm called "SAO", seen here: Scalable Ambient Obscurance.

I'm currently having issues with "smooth" normals, for example when using normal/height maps.

To give an example, here is the SAO and the normals for a chair rendered with "raw" normals, i.e not using the accompanied normal map:

sao1

normal1

Looks pretty OK overall.

now.. when I apply the chair's normal map:

sao2

normal2

What could be wrong? Is there anything obvious wrong with the normals?

I'm using the SAO shader, with only very minor changes. The normal/position textures are in world space, and then converted to view space. I know this is not optimal, but for the sake of the alghorithm it should not matter.

#ifndef SSAO_PIXEL_HLSL #define SSAO_PIXEL_HLSL #include "Constants.h" #include "Common.hlsl" static const float gNumSamples = 11.0; static const float gRadius = 0.2; static const float gRadius2 = gRadius * gRadius; static const float gProjScale = 500.0; static const float gNumSpiralTurns = 7; static const float gBias = 0.01; static const float gIntensity = 1.0; cbuffer SSAOCBuffer : register(CBUFFER_REGISTER_PIXEL) { float4x4 gViewProjMatrix; float4x4 gProjMatrix; float4x4 gViewMatrix; float2 gScreenSize; }; Texture2D gPositionTexture : register(TEXTURE_REGISTER_POSITION); Texture2D gNormalTexture : register(TEXTURE_REGISTER_NORMAL); SamplerState gPointSampler : register(SAMPLER_REGISTER_POINT); float3 reconstructNormal(float3 positionWorldSpace) { return normalize(cross(ddx(positionWorldSpace), ddy(positionWorldSpace))); } /** Read the camera - space position of the point at screen - space pixel ssP + unitOffset * ssR.Assumes length(unitOffset) == 1 */ float3 getOffsetPosition(int2 ssC, float2 unitOffset, float ssR) { // Derivation: // mipLevel = floor(log(ssR / MAX_OFFSET)); // TODO: mip levels int mipLevel = 0; //TODO: clamp((int)floor(log2(ssR)) - LOG_MAX_OFFSET, 0, MAX_MIP_LEVEL); int2 ssP = int2(ssR*unitOffset) + ssC; float3 P = gPositionTexture[ssP].xyz; P = mul(gViewMatrix, float4(P, 1.0)).xyz; return P; } float2 tapLocation(int sampleNumber, float spinAngle, out float ssR) { // Radius relative to ssR float alpha = float(sampleNumber + 0.5) * (1.0 / gNumSamples); float angle = alpha * (gNumSpiralTurns * 6.28) + spinAngle; ssR = alpha; return float2(cos(angle), sin(angle)); } float sampleAO(uint2 screenSpacePos, float3 originPos, float3 normal, float ssDiskRadius, int tapIndex, float randomPatternRotationAngle) { float ssR; float2 unitOffset = tapLocation(tapIndex, randomPatternRotationAngle, ssR); ssR *= ssDiskRadius; // The occluding point in camera space float3 Q = getOffsetPosition(screenSpacePos, unitOffset, ssR); float3 v = Q - originPos; float vv = dot(v, v); float vn = dot(v, normal); const float epsilon = 0.01; float f = max(gRadius2 - vv, 0.0); return f * f * f * max((vn - gBias) / (epsilon + vv), 0.0); } float4 ps_main(float4 position : SV_Position) : SV_Target0 { uint2 screenSpacePos = (uint2)position.xy; float3 originPos = gPositionTexture[screenSpacePos].xyz; originPos = mul(gViewMatrix, float4(originPos, 1.0)).xyz; float3 normal = gNormalTexture[screenSpacePos].xyz;//reconstructNormal(originPos); normal = mul(gViewMatrix, float4(normal, 0.0)).xyz; // Hash function used in the HPG12 AlchemyAO paper float randomPatternRotationAngle = (3 * screenSpacePos.x ^ screenSpacePos.y + screenSpacePos.x * screenSpacePos.y) * 10; float ssDiskRadius = -gProjScale * gRadius / originPos.z; float ao = 0.0; for (int i = 0; i < gNumSamples; i++) { ao += sampleAO(screenSpacePos, originPos, normal, ssDiskRadius, i, randomPatternRotationAngle); } float temp = gRadius2 * gRadius; ao /= temp * temp; float A = max(0.0, 1.0 - ao * gIntensity * (5.0 / gNumSamples)); return A; } #endif

EDIT: update, using reconstructNormals() to reconstruct normals - for simpler models it looks okay (the chair) but the more advanced models like the alien looks really bad/blocky. Any way to fix this?

In general you should use geometry normal and not normalmap for SSAO. — JarkkoL
– JarkkoL, Commented Oct 14, 2014 at 23:19
Why is that? I havn't found any sources on that. Also, the model importer I use, assimp (assimp.sourceforge.net), has an option to calculate "smooth" normals, with the same result as using a normal map. Are these not considered geometry normals? — KaiserJohaan
– KaiserJohaan, Commented Oct 15, 2014 at 7:46
No, smooth normals are calculated by averaging triangle normals attached to a vertex. What you need is the actual triangle normals, which in your shader is calculated in the reconstructNormal() function. You need the actual geometry normals because SSAO is calculated using depth samples in depth buffer and their relation to given pixel. Using wrong normal will result wrong weighting for the samples, e.g. think of a depth sample on the same geometry plane as the pixel whose SSAO you are calculating. — JarkkoL
– JarkkoL, Commented Oct 15, 2014 at 16:21
reconstructed normals looks really jagged/blocky on some advanced models, edited first post. Is it really viable to use that? — KaiserJohaan
– KaiserJohaan, Commented Oct 16, 2014 at 17:10
Looks wrong that convex shape (head) has self-occlusion along the polygon edges. Could it be that your cross product in reconstructNormal() is wrong way around? — JarkkoL
– JarkkoL, Commented Oct 16, 2014 at 23:13

Paul · Accepted Answer · 2024-04-13 13:12:05Z

This is something of a cold case, but I thought I'd take a crack at it.

My answer to the original question "Is there anything obvious wrong with the normals?" is that there does appear to be something wrong, although it's not particularly obvious.

To begin, I wanted to emulate the normal colouring shown in the screenshots. At first I thought it matched the standard Unity normal colouring, but it doesn't. One difference, shown by the cube at the back right of the screen is that blue represents normals pointing towards the camera. In Unity, this is represented as negative in the blue channel and so clipped to 0, that is, black. The other difference, shown by the colouring of the floor, is that the normals are in world space. If it were in camera space, the floor would be apparently tipped towards the camera and mix some blue with the green.

This project contains a modified post processing debug layer that displays the normals in world space with blue towards us for comparison purposes.

Included in this project is the scene NormalTest. This scene shows normal directions by reading them from the pixel under the mouse pointer and then projecting a line in the direction of the normal. This helps visualize the surfaces.

The seat

Focussing on the seat of the chair, broadly speaking it should be a flat surface pointing up and therefore should be a flat green colour. However, the normals have been smoothed. If I make a squashed cube and smooth the normals (the SmoothedNormals scene), this is the colouring I get:

A smoothed and flattened cube: it is full green in the middle, tinting red to the right and blue to the front.

The seat of the original chair without normal mapping is roughly the same. It is full green in the middle, tinting red to the right and blue to the front.

But with normal mapping, in the middle it already appears tinted towards the blue, and then it tints further from there. This doesn't appear to be correct.

For comparison, here is an asset store chair: https://assetstore.unity.com/packages/3d/props/furniture/wooden-chairs-2-variations-125757. Without normals it shows a similar pattern.

With the normal map applied, you can see changes to colours in the detail, but on average the colours remain the same over the surface.

The legs

Focussing now on the legs, again broadly speaking, they should be cylinders coloured blue (because they should be facing you) and tinted red to the right.

Again, this is roughly what we see in the chair without normal mapping.

With normal mapping applied, it appears as if the normals have been twisted to the right.

The back

Finally, looking at the back rest, this appears to be curved and so on the left hand side, I expect the normals to be rotated towards the right.

This is what we see in the unmapped version above.

However, in the normal mapped version, this tilt appears to have gone.

Hypothesis

This suggests to me that either the normal maps have been generated incorrectly, encoded incorrectly, decoded incorrectly, or applied incorrectly.

Excluding the shader

To eliminate the possibility of a shader error I have taken the code provided and patched it into the Unity post process SSAO effect. It appears to work correctly without demonstrating the issues noted. In this project I have included a chair and an alien model from the asset store as test files. Both of these appear to render a decent looking SSAO effect, with and without normal maps. You can use the SSAO scene to test this.

https://assetstore.unity.com/packages/3d/characters/creatures/alien-character-20838

Reconstructing the normals

There was some discussion in the original post about using a function to reconstruct the normals from the 3D position of each pixel. I have included an additional option to apply this function. In this case, the normal map is not used. I managed to get reasonable results from this function, but there was a bug in the code. I needed to swap the parameters of the cross product or I got bad results like those shown in the original post.

float3 reconstructNormal(float3 positionWorldSpace) { return normalize(cross(ddy(positionWorldSpace), ddx(positionWorldSpace))); }

Conclusions

I suspect the normal maps are the cause of the original issue. The maps should not be making such large scale modifications to the normals. I would recommend checking known good normal maps and models with the engine to see if the issue was in the creation or application of the normal maps.

Using reconstructed normals, the cross product needs reversing to give a good result.

Wiley Rush · Accepted Answer · 2023-06-13 15:25:00Z

I'm not the expert to answer this question, but perhaps:

Points on your detailed normal map are skewing some of your offset samples back into the geometry?

The details on your normal map contain dark spots/crevices, which are being read by the shader as corners?

Stack Exchange Network

Unwanted darkening at polygon edges when using normal maps with SSAO

2 Answers 2

You must log in to answer this question.

Hot Network Questions

Unwanted darkening at polygon edges when using normal maps with SSAO

2 Answers 2

You must log in to answer this question.

Related

Hot Network Questions