How do you even start to build something like this? What frameworks could I use?
Quick sketch:
- Learn WebGL (or OpenGL and transition)
- Learn some linear algebra, don't worry, all you need is 4D matrices, so you don't need to know proofs, just how to use it.
- Create a 3D graphics engine like this: https://www.youtube.com/watch?v=PH6-dLvZEiA (that's mine, I haven't progressed since in this area, so this is the level I'm at, enough to know what fields are needed to learn this but not enough to know all the specific techniques)
- Look up names/techniques people have used to create something specific like OPs post, see [1]
- Learn that stuff as well / related math
- Build it in WebGL
[1] https://www.youtube.com/watch?v=J9CExYF8yrU&ab_channel=V%C3%...
You don't need a framework for anything. You need to know your problem and have an idea on how to solve it.
To do something like this, you need to know a bit of WebGL and a bit of JS. No frameworks really needed at this level. The best guide to WebGL I know of is this:
https://webgl2fundamentals.org/
Complexity starts when the height map isn't enough and you want to make arcs or otherwise arbitrarily shaped structures. This is where you'd need a framework, unless you want to implement all the math and glsl yourself.
And as always, the idea is really the 1% of the work. Anyone can say "it's just a height map on a sphere". 99% of the work is materialising the idea, debugging glsl, fixing ugly bugs on smartphones and esotheric browsers, dealing with partial WebGL API and so on.
How do you even start? I started with `three.js` about two years ago and a ton of documentation. I didn't write a sculpting tool, but I wrote a PoC for a tool that "renders" custom tessellated data, pan/tilt/zoom, by the documentation from their site in about a weeks' time[0].
[0] I do more back-end than front-end, but I did the PoC in React/TypeScript with a kludge for the Canvas and a few ugly functions for handling loading/camera positions.