I've built "vanilla" Three.js projects, and worked with react-three-fiber. It is incredibly useful.

Having a new set of "primitives" that map to three.js objects and being able to render them in a React application makes so many things easier. It's also just way less code to write compared to standard three.js.

RE: animation performance, I haven't had any issues with it. I presume either they're doing a bunch of optimizations, or the concerns are greatly exaggerated. You can check out an app I built using it if you want some proof: https://beatmapper.app/