I’m curious if anyone has a variant of this idea where an invisible DOM lives alongside the canvas for accessibility reasons. It should work but I imagine it would also take a lot of work.
Pax renders vector graphics to canvas layers and composites + clips + occludes native elements (web DOM nodes, iOS SwiftUI Text elements, etc.) in the same space to make a cohesive scene.
This solves accessibility and SEO, as well as dramatically reducing runtime size (~100KB WASM network footprint) because text rendering and atomic native UI elements don't need to be reinvented.
To my knowledge we're the only ones taking this particular approach so far (I'm on the team behind Pax) but it works quite well. Yes it was a lot of work.
[0] see pax.dev or check out a simple example: https://static.pax.dev/space-game/ ; source code for space game in examples on github https://www.github.com/paxengine/pax
I checked it last a couple of weeks ago on my Android phone, and the result was so poor, I didn't check with other platforms.
You need to "opt in" to accessibility as a user, so in the beginning you have one button visible to screen readers, then your screen reader sees that single button, then you need to "double tap to activate" to make the rest of the document screen readable.
Then, somehow they made a funky screen reader voice read stuff out loud, not my system's voice, with all the awkwardness and issues that come with it.
On lists, it only recognized the items that were on screen, if you had to "scroll", you are out of luck.
Now, I'm sure someone will come and tell me how I'm holding it wrong, but somehow, every website and even Flutter on mobile got the accessibility mainly right by default, on Flutter web the accessibility situation is very poor.
Edit: unless you're doing something like flutter where your goal is to escape the DOM/JS paradigm in your code entirely, the fallback DOM nodes don't tend to be implemented too often.
The biggest reason is that canvas is for things with interactive/consumptive patterns that don't work well without vision.
Cute animations, 3d explorers and configurators, that sort of thing.
For users who rely on screen readers, they are better served by a DOM and interactive patterns that are not strictly recreating the canvas.
As an example, canvas based animations might have a fallback equivalent to an alt attribute on an image. A 3d explorer or configurator might be represented in DOM as a multi step form with verbose descriptions- no need to recreate drag and drop, camera controls, or anything else.
So, if you're not automating it like flutter is (and even if you are) what you really need to do is effectively an entirely alternate UX for screen reading users... Trying to get your product team to budget that is difficult despite being a Good Thing to do.