>but you need to tail the code for the hardware, which is niche

Not really.

You just need to declare the data types (Common Lisp supports type declarations) and tell the compiler you want the code to be fast, which reads just like that:

(declare (optimize (speed 3)))

You also need to use the same tactics you would use in C to gain speed: Use arrays instead of lists, use threads. All of this is easily doable in Lisp and IMO while it takes more effort than writing typical Lisp code, it is easier than using C and the end result is more elegant and debuggable.

This is not hardware dependent, mind you.