You could still keep Three.js for bits like vector calculations, but it just doesn’t feel worth it, it’s easy enough to implement yourself—or copy and paste from some such library and modify as needed—and will be much lighter. And you build up the bits and pieces you need over time.
Some sources
> Math & motion: – The Nature of Code https://natureofcode.com/ – Coding Math https://www.youtube.com/user/codingmath
> Shaders / math-based rendering: – The Book of Shaders https://thebookofshaders.com/ – Inigo Quilez https://iquilezles.org/
> Interactive explanations: – Red Blob Games https://www.redblobgames.com/
You can also find some insight into his work process here: https://x.com/BCiechanowski/status/1387827101294686210?s=20
Most people doing this level of work built their own tools over time, learned a bit here and there, and kept refining things throughout their career.