One significant limitation here is that the polygon needs to have constant colour, unfortunately.
One significant limitation here is that the polygon needs to have constant colour, unfortunately.
Pixar's Renderman used to use a (slightly) related technique. This was along time ago, and I have no idea if they still do things this way - but they would subdivide patches until each primitive was smaller than a pixel.
There's also this work on analytic antialiasing by Michael Mccool: https://www.researchgate.net/publication/2524514_Analytic_An...
The problem you run into with non-constant polygon colors is that you’d have to integrate the product of two different functions here - the polygon color and the filter function. For anything real-world, this is almost certainly going to result in an expression that is not analytically integrable.
Ideally you also want motion blur and probably some other effects, so IMO it just makes sense to use a 2D BVH and high efficiency Monte Carlo importance sampling methods.