The Goal
I have a game in Unity which utilizes pixel art. I'm not following the standard rules of pixel art, however. I'm ok with sprites having different sized pixels, pixels rotating, and pixels not aligning with the pixel grid. An example (not my video): https://www.youtube.com/shorts/FCJWPYqV0TI. I'd also like the player to be able to run the game at whatever resolution they want while maintaining the same vertical camera size, i.e. with a wider resolution you'd see more of the world horizontally, but not vertically. Also, specific to my game, is the player's ability to zoom the camera in and out.
Solution 1: A Standard Setup
The first thing I tried, since I wasn't following the usual pixel art practices, was to just use a standard 2D game setup. I placed my sprites in the scene, along with a camera. I set the filter mode on my sprites to point (no filter), set compression to none, and turned off all forms of AA. I set my camera to follow the player in a smooth motion, no pixel perfect camera here since that's not the effect I'm after. The results:
Sprite shimmering. This post is a fantastic read on why this happens: What makes scaling pixel art different than other images?. You'll also have jagged edges when rotating sprites.
So how do we solve this? Usually the answer is bilinear filtering. That gets rid of these high contrast positions between pixels. You'll also want to pad your sprite by 1 pixel, so that the edges can blend with transparency. Of course, for a sprite of this size, the results are less than ideal (I also didn't add the padding in this example, resulting in harsh edges):
Solution 2: Bilinear Filtering Shader
So the next solution was to make a shader that took advantage of bilinear filtering, but only applied it where needed. There is a great video on this subject here: https://www.youtube.com/watch?v=d6tp43wZqps. Here is the shader code:
void PixelHD_float(UnityTexture2D Texture, float2 UV, out float4 Color) { float2 boxSize = clamp(fwidth(UV) * Texture.texelSize.zw, 1e-5, 1); float2 texel = UV * Texture.texelSize.zw - boxSize * 0.5; float2 offset = smoothstep(1 - boxSize, 1, frac(texel)); float2 uv = (floor(texel) + 0.5 + offset) * Texture.texelSize.xy; Color = Texture.SampleGrad(Texture.samplerstate, uv, ddx(UV), ddy(UV)); } Basically what is happening is we are getting the average color inside a box the size of a single pixel on screen. With this we just set our Sprites to use bilinear filtering, and make a material that uses the shader. Then slap that shader onto the sprite renderers. The results are very promising:
The shimmering is getting subtle enough that you can't really make it out in the gif! Not perfect, but we're getting there.
Solution 3: My Custom Shader
After weeks and weeks of research, trial and error, and also realizing I wanted to replace colors, I ended up making this shader myself:
#ifndef PIXELHD_INCLUDED #define PIXELHD_INCLUDED bool UVInBounds(float2 UV) { return UV.x >= 0 && UV.y >= 0 && UV.x <= 1 && UV.y <= 1; } float4 ReplaceColor(float4 Color, float4 RedReplacement, float4 GreenReplacement, float4 BlueReplacement) { if ((Color.r > 0 ? 1 : 0) + (Color.g > 0 ? 1 : 0) + (Color.b > 0 ? 1 : 0) > 1) { return Color; } else if (Color.r > 0) { return float4(RedReplacement.rgb * Color.r, RedReplacement.a * Color.a); } else if (Color.g > 0) { return float4(GreenReplacement.rgb * Color.g, GreenReplacement.a * Color.a); } else { return float4(BlueReplacement.rgb * Color.b, BlueReplacement.a * Color.a); } } float4 HandleFullyTransparent(float4 OriginalColor, float4 HorizontalNeighbor, float4 VerticalNeighbor, float4 CornerNeighbor, float2 OffsetDistance) { // Keep the original color if it is not fully transparent. if (OriginalColor.a != 0) { return OriginalColor; } // Take the horizontal neighbor's color if the vertical neighbor is fully transparent. else if (HorizontalNeighbor.a != 0 && VerticalNeighbor.a == 0) { return float4(HorizontalNeighbor.rgb, 0); } // Take the vertical neighbor's color if the horizontal neighbor is fully transparent. else if (VerticalNeighbor.a != 0 && HorizontalNeighbor.a == 0) { return float4(VerticalNeighbor.rgb, 0); } // Take the corner neighbor's color if the horizontal and vertical neighbor is fully transparent. else if (HorizontalNeighbor.a == 0 && VerticalNeighbor.a == 0) { return float4(CornerNeighbor.rgb, 0); } // Neither the horizontal or vertical neighbor is fully transparent, so we take the closest color. else if (OffsetDistance.x < OffsetDistance.y) { return float4(HorizontalNeighbor.rgb, 0); } else if (OffsetDistance.y < OffsetDistance.x) { return float4(VerticalNeighbor.rgb, 0); } // The horizontal and vertical neighbors are equal distance away, so we mix them. else { return float4((HorizontalNeighbor.rgb + VerticalNeighbor.rgb) * 0.5, 0); } } void PixelHDReplaceColor_float(UnityTexture2D Texture, float2 UV, float4 RedReplacement, float4 GreenReplacement, float4 BlueReplacement, out float4 Color) { // Convert UV coordinates into texel coordinates. float2 texelCoordinates = UV * Texture.texelSize.zw; // Get the bottom left of the nearest texel. float2 texel = round(texelCoordinates); // Get the change in uv per pixel. float2 uv_ddx = ddx(UV); float2 uv_ddy = ddy(UV); // Get the uv coordinates of the center of the 4 surrounding texels. // Multiply by texel size to convert from texel coordinates to UV coordinates. float2 uv00 = (texel - 0.5) * Texture.texelSize.xy; float2 uv10 = (texel + float2(0.5, -0.5)) * Texture.texelSize.xy; float2 uv01 = (texel + float2(-0.5, 0.5)) * Texture.texelSize.xy; float2 uv11 = (texel + 0.5) * Texture.texelSize.xy; // Sample the texture at our 4 points. // If we are replacing colors, mipmaps are not used as they pollute the data. float4 c00; float4 c10; float4 c01; float4 c11; if (all(RedReplacement == float4(1, 0, 0, 1)) && all(GreenReplacement == float4(0, 1, 0, 1)) && all(BlueReplacement == float4(0, 0, 1, 1))) { c00 = UVInBounds(uv00) ? Texture.SampleGrad(Texture.samplerstate, uv00, uv_ddx, uv_ddy) : 0; c10 = UVInBounds(uv10) ? Texture.SampleGrad(Texture.samplerstate, uv10, uv_ddx, uv_ddy) : 0; c01 = UVInBounds(uv01) ? Texture.SampleGrad(Texture.samplerstate, uv01, uv_ddx, uv_ddy) : 0; c11 = UVInBounds(uv11) ? Texture.SampleGrad(Texture.samplerstate, uv11, uv_ddx, uv_ddy) : 0; } else { c00 = UVInBounds(uv00) ? ReplaceColor(Texture.SampleLevel(Texture.samplerstate, uv00, 0), RedReplacement, GreenReplacement, BlueReplacement) : 0; c10 = UVInBounds(uv10) ? ReplaceColor(Texture.SampleLevel(Texture.samplerstate, uv10, 0), RedReplacement, GreenReplacement, BlueReplacement) : 0; c01 = UVInBounds(uv01) ? ReplaceColor(Texture.SampleLevel(Texture.samplerstate, uv01, 0), RedReplacement, GreenReplacement, BlueReplacement) : 0; c11 = UVInBounds(uv11) ? ReplaceColor(Texture.SampleLevel(Texture.samplerstate, uv11, 0), RedReplacement, GreenReplacement, BlueReplacement) : 0; } // Exit early if the result is fully transparent. if (c00.a == 0 && c10.a == 0 && c01.a == 0 && c11.a == 0) { Color = 0; return; } // Exit early if all colors are the same. if (all(c00 == c10) && all(c00 == c01) && all(c00 == c11)) { Color = c00; return; } // Get the offset of the texel coordinates from the bottom left of the texel. float2 offset = texelCoordinates - texel; // Handle fully transparent colors float2 offsetDistance = abs(offset); c00 = HandleFullyTransparent(c00, c10, c01, c11, offsetDistance); c10 = HandleFullyTransparent(c10, c00, c11, c01, offsetDistance); c01 = HandleFullyTransparent(c01, c11, c00, c10, offsetDistance); c11 = HandleFullyTransparent(c11, c01, c10, c00, offsetDistance); // Get the pixel size of a texel. float2 pixelSize = sqrt(uv_ddx * uv_ddx + uv_ddy * uv_ddy); // Get the ratio of texel size to pixel size. float2 texelToPixelRatio = Texture.texelSize.xy / pixelSize; // Multiply the offset by the texel to pixel ratio. // This means if the texel takes up a lot of screen space, then small changes in the offset matter very little. // If the texel takes up very little screen space, the offset matters much more. // We add 0.5 since the offset is between 0 and 0.5, because we rounded to the nearest texel. // We clamp this value so that it can't leave the texel. float2 blend = saturate(offset * texelToPixelRatio + 0.5); // Interpolate between colors. Color = lerp(lerp(c00, c10, blend.x), lerp(c01, c11, blend.x), blend.y); } #endif The color replacement happens before the bilinear filtering, then it handles any transparent texels, then it performs bilinear filtering. The amount of filtering is based on the ratio of texel size to pixel size on screen. You don't even need to pad the sprites any more, since I treat all UV's less than 0 or greater than 1 as transparent. The result was perfectly crisp pixel art with no shimmering. At this point I thought my problem was solved. However, there was one glaring issue: Sprites could no longer be seamless.
You can see there are seams between background tiles, and very small seams between the character's legs, toros, and head, since they are all separate sprites. This may not seem like a big deal, since the background is currently a solid color, but that will eventually be replaced with tiles which may not always share the same color at their borders.
This is ultimately where I got stuck. At first I though the solution would be to not sample alphas, making each edge that borders an alpha a hard edge instead of a soft edge. I accomplished that by doing this at the end of the shader:
float w00 = (1 - blend.x) * (1 - blend.y) * (c00.a != 0 ? 1 : 0); float w10 = (blend.x) * (1 - blend.y) * (c10.a != 0 ? 1 : 0); float w01 = (1 - blend.x) * (blend.y) * (c01.a != 0 ? 1 : 0); float w11 = (blend.x) * (blend.y) * (c11.a != 0 ? 1 : 0); float totalWeight = w00 + w10 + w01 + w11; // Interpolate between colors. if (totalWeight > 0) { Color = (c00 * w00 + c10 * w10 + c01 * w01 + c11 * w11) / totalWeight; } else { Color = 0; } This actually does solve the seams problem, as sprites that share a border connect properly, but the shimmering returns since that part of the sprite is no longer being filtered. At this point I've been struggling for about a month on this, and would appreciate any insight.
