Fast 2D Rendering on GPU

It's a good question, as honestly the knowledge for 2D graphics is pretty arcane, as opposed to 3D being so widely taught. I actually started a github repo for a book on 2d graphics but have no idea whether I'll actually finish it.

In the meantime, antigrain.com is one good (if old) source. The original PostScript "red book" was extremely influential in its time (it's where I learned a lot of this stuff) but is quite dated now. Best of luck, and I'm also happy to field requests for more specific areas. For example, for color theory (an important aspect of 2D graphics!), handprint.com is quite a remarkable resource.

I recommend the blog of the OP, so many gems

I absolutely have data visualization in mind for this, as I think it can benefit greatly from the scale. But the pipeline I've built is very agile, it will easily handle a diverse mix of items. It's not like OpenGL etc where there's a certain amount of overhead for a draw call so there are significant gains to be had from instancing and batching.

It is likely that CPU-side encoding can be made more efficient, though, by just filling in quantities to a template, rather than encoding from scratch.

Yes, there's been a lot of influence in both directions. The approaches have a lot in common, but also have significant differences. I plan to write up a description of the Pathfinder compute work soonish.

As Patrick indicated, there's a lot of cross-fertilization of ideas, and one of the best outcomes of this work would be for high performance compute rendering to ship in Pathfinder.

> - Shouldn't 2D rendering be a solved problem given that it's basically a subset of 3D rendering?

The problem is that primitives artists use are different. 3D rendering tends to all consist of polygon meshes, which are relatively easy to render. 2D rendering (basically) consists of Bezier paths, which are harder. The equivalent in 3D, which is adaptive subdivision, is not really a solved problem in real-time either.

Additionally, 2D rendering quality tends to be more important than 3D rendering quality. Whereas you can get away with 4xMSAA or hacks like FXAA in 3D, true 16xAA (without hacks) is the absolute minimum for 2D rendering quality nowadays, and even it isn't considered great for some tasks like font rendering (Pathfinder and piet-gpu both use analytic AA which is effectively 256xAA).

> - Don't libraries like Skia, Qt, Cairo use GPU rendering? I've always assumed so. I mean, this is 2020, GPUs have been around for decades.

There's a difference between renderers with GPU support and renderers that are oriented around using the GPU efficiently. In many cases this results in an order-of-magnitude speedup. On the GPU, state changes are expensive, and many such renderers that have GPU support don't really go out of their way to avoid them. There are also occlusion culling optimizations that most renderers don't do, but piet-gpu and Pathfinder do.

Things like multi-resolution font-rendering are actually more complicated than one might imagine.

The easy way is to just tesselate the font into polygons -- but this tesselation often depends on the zoom level. The same thing can be said about implicit curves etc.

Most libraries do use GPU's for basic draw operations (e.g rendering a gradient), but to build something like Photoshop -- you need much more complexity.

Others have spoken to this, but as a general introduction I highly recommend Jasper's post "Why are 2D vector graphics so much harder than 3D?" In short, no, it's not a solved problem.

Also, I see a lot of variations of this question, but I should state this more clearly. There's been accelerated graphics in one form or another for a long time, but what I'm doing is a completely different type of thing. In my world, on the CPU you just encode the scene into a binary representation that's optimized for GPU but in many ways is like flatbuffers, and then the GPU runs a highly parallel program to render the whole thing. In previous approaches, the CPU is deeply involved in taking the scene apart and putting it back together in a form that's well suited to relatively dumb pixel pipes. Now that GPUs are really fast, that approach runs into limitations.

It also depends what you're trying to do. I'm focusing here on dynamic paths (and thus font rendering), while most of the libraries optimized for UI put text into texture atlases and then use the GPU to composite quads to the final surface, something they can do well.

https://blog.mecheye.net/2019/05/why-is-2d-graphics-is-harde...

> Don't libraries like Skia, Qt, Cairo use GPU rendering? I've always assumed so. I mean, this is 2020, GPUs have been around for decades.

As a Qt user / developer - you can use the GPU for your app, e.g. with Qt Quick or QGraphicsView, but there are sometimes good reasons to stick to CPU & software rendering, e.g. it is somewhat common to want to have intertwined "native" OS widgets (which are all CPU-rendered raster things) and custom GPU-drawn scene - this is a case where things fall a bit apart.

Another thing is that pretty, freetype-like font rendering is super expensive when you have a lot of text to show, and can't really be done (at least I have definitely not seen infinality-level beauty from the state of the art) on the GPU yet... next to that filling some rects (read: 95% of UI) with SSE/AVX/AVX2 as Qt does is stupidly fast.

No, it is not a subset, unless you are talking about drawing textures.

Those libraries use GPU rendering using textures, like distance fields(Qt) or simply calculate fonts in 2D and draw it on textures using the GPU, like Apple or cairo usually do.

Distance fields are blurry when you have small fonts on display.

Things like calculating the exact area order the 2D curve is something you could easily do in CPU, but it is extremely difficult to do on the GPU. You need decoupled data in order to parallelize it.

Not very, and not very. This is research, it depends on compute capabilities that have really only gone mainstream in GPU lately, and the current codebase is about trying out the ideas. Best of luck in your project!

I pondered on the same subject recently as I was implementing the same algorithm (i.e. Mandelbrot set) on the CPU (scalar vs SIMD) and GPU compute using fixed-point and floating-point for comparisons (if interested: https://tayfunkayhan.wordpress.com/2020/06/03/mandelbrot-in-...).

It bothers me how little progress has been done on "shading" languages front compared to overall many-core computation models and capabilities over the years. And, that is despite the fact that shaders are very often where the most time is spent in modern workloads.

Compute with Vulkan is another story. It offers some nice abstractions, but it shows that it's mostly intended for async-compute/work-offloading for rendering, IMO. Too much fruction.

That's a big topic. I've been able to work around the missing primitives (for example, I autogenerate code for Rust-style structs and enums), but have had much bigger struggles around two issues: tools, which are still quite primitive, and understanding performance, which is extremely difficult. These two problems intersect because I can imagine a lot better tools for digging into performance issues. One that I would have paid good money for is an instruction-level simulator that would highlight the source code to tell me where the stalls, bank conflicts, divergence problems, etc. are in the source code. Such a thing is possible (there are academic papers like [1]), but not as far as I know usable in daily development.

The "impedance mismatch" is that you (generally) have to write in a style to extract lots of parallelism. This tends to be very different than the way you'd write scalar CPU code, but not completely alien to me as it has a lot of similarity with the way you'd write SIMD. I've pretty much gotten the hang of it now. I'm thinking of a blog post of redoing path_coarse.comp from its current basically scalar style to a more parallel version, as that would I think illuminate the issues.

[1] http://comparch.gatech.edu/hparch/papers/gera_ispass18.pdf

For the OP's effect, remember you're probably going to want motion blur in each frame, which means you can render at a much lower resolution in the direction of motion (as long as you can do multipass rendering)

Another need for that kind of effect is for real-time events.

I work on software that is used to create visual effects in real-time. It's used in live events such as concerts, clubs, corporate presentations. There you want to be able to render high-quality text in high resolution, even higher than 4K, and still allow a performer to add effects on text such as zooming, scrolling, doing 3D transition effects...

Today the best way to do this is via a font atlas but it does not look nice when zooming. Also if you need non-European characters it may heavy preparing the font atlas.

So I am looking for a library that would allow this kind of rendering manipulation for text rendering. Do you have any suggestions?

I don’t believe this is a problem. It will only need to render it when things change—so maybe the GPU will be sitting at 100% while you’re flipping through pages, but other than that typically very brief time, it should be at 0%.

Well said, thanks. It's also a goal of this work to do compositing inside the "fine rasterizer" compute kernel, to avoid multipass as much as possible. Of course, in the general case for stuff like blur that's hard, but that's one reason why I've been working on special cases like blurred rounded rectangles, which I can render pretty much at lightspeed. (It's not integrated into the current code but would be quite easy)

What does latency have to do with 2D? Are you suggesting UI toolkits are writing GPU commands synchronously? You should fill display list, fire it and forget about GPU until the next update.

At the very beginning x86 PC computers didnt have DMA capable of transferring ram to ram, not to mention DMA was slower than CPU http://www.os2museum.com/wp/the-danger-of-datasheets/ First PC 2D acceleration was very much on graphics chip, in form of either IBM 8514 or TIGA compatibility ($1K cards when introduced). Amiga also doesnt fit the profile, seeing most models shipped with unified shared memory (chip ram).

Try windows3.1 instead of windows xp: 2d acceleration cards started early 90 , see https://en.m.wikipedia.org/wiki/S3_Graphics or trident pages by example.

I kind of meant more in terms of how they work and what they can do but I suppose I could do the work in answering that question myself by figuring out how nanovg works then comparing it to what you've written.

I've been finding Patrick's pathfinder work fascinating so it's nice to have yet another project to follow.

That is very exciting to hear. I'll give it a shot. Thanks.

Is it true for all 2D rendering?

From my intuition, this seems pretty specialized for vector-like rendering, with a lot of small bezier shapes.

> all the ways I know of to do that either involve (a) an expensive CPU process or (b) a cheap CPU process but way too much overdraw on GPU side

That's fixable, take a look: https://github.com/Const-me/Vrmac#vector-graphics-engine

My CPU process is moderately expensive, and I use quite a few tricks to reduce overdraw, e.g. both draw calls (the complete vector image, however complex, takes 2 draw calls to render) use hardware depth buffer and early Z rejection.

I understand.

Problem is that any practical 2D rendering solution shall support as GPU as CPU rendering unfortunately.

It would be interesting to see any GPU equivalent of something like AGG by Max Shemanarev, RIP.

Interesting, thanks - I remember hearing at one of the WWDC that Coregraphics got a 10x speed improvement using metal. I just read the fine print - it seems draw calls only have a 10x speed improvement, I assume because they render through a layer of some sort. The CA* libraries may use metal - animation and layers, which I guess is where the 10x draw call improvement comes in maybe.

Some discussion here https://arstechnica.com/civis/viewtopic.php?t=1285571 , though a lot of guessing.