It's just an html canvas powered by a low level language through WASM, plus a bridge interface. Perfect use case for Rust.
Rest of Figma is, I guess, just React.
But just as you mentioned I believe there's a lot of business potential there, and someone will get it done sooner or later.
But they've done all of this work to get this far and have basically abandoned their pen tool before it was able to be used as an actual tool for drawing. It has all of the underlying technical aspects worked out but it is a UI nightmare to use—it's painfully user unfriendly and doesn't have a curvature tool, an easy way to edit/delete nodes, etc. It has such great potential, and is so close, but Figma has chosen to go another direction.
Anyways, the only rendering engine that I am aware of that rivals Figma's is Google's Skia (which in turn is the renderer for Canvas itself in most browsers I believe) but it's 20+ years old & painfully low level.
You could theoretically do whatever you want on Skia and port it to wasm to get it into the browser, but it doesn't itself have any tools that you'd want for drawing, or creating anything that looks anything like CSS/HTML.
Literally the second someone releases something high level enough in this area I am going to build an animation library on top of it. I've had the desire to build a library that could programmatically create and edit vectors for a decade now, but there's still nothing easy enough to work with to incentivize me.
It has text shaping & editing (might have helped OP) and can play back Lottie animations
It is developed by Google in part for Flutter, which uses CanvasKit by default when you compile Flutter apps for web.