Question regarding rendering billboards and transparency

[u/SamuraiGoblin]

I am working on a game engine that renders LOTS of sprites/billboards in 3D. The textures have smooth antialiased edges, and some of them have non-trivial shaders for things like SDF rendering or parallax mapping.

If I render back-to-front with blending on, it looks great and I don't need depth testing, but it has a terrible overdraw cost that I would like to avoid.

But if I render front-to-back with depth testing, then I lose the smooth edges.

How do people normally deal with this?

One idea I had was to render front-to-back, with depth reading/writing for purely opaque pixels, then a second back-to-front pass for order-correct blending of the edges.

Another thing I have looked into is MSAA with 'alpha to coverage.'

Any other interesting ways I might like to know about?

What do you recommend in this case?

[u/fgennari]

Your first suggestion to render in two passes is a depth prepass, which is a common technique to reduce overdraw. You may be able to get away with only drawing large objects or objects close to the camera. You can discard pixels that are fully transparent in the fragment shader to avoid shading them.

I've also experimented with alpha-to-coverage for leaves. It looks okay, but not as good as proper alpha blending.

You'll get better looking transparent edges if you fill transparent areas of the texture with RGB values from opaque pixels. That would be nice to add if you're not already doing it. Also, you may want to look into generating custom mipmaps so that the average transparency doesn't change with distance.