If you care that much, you use intrinsics, asm in the C file, or asm to create an object you call into.

Most of the code doesn't need nearly that level of optimization, so you write it in a higher level language.

Making your own compiler is a lot of work compared to just doing what I've outlined above. So you don't. I's not worth it, and you'd still end up back where you are.

Writing assembly really isn't that scary, especially in small doses for performance-critical code. That was a very common practice in the 80s/90s, though it's faded as compiler optimizations have improved.