Check out this port of microgpt to C, posted 5 months ago. It got a 2500x speedup over the python version. https://github.com/moebiusV/cugpt
[1] see eg https://github.com/OpenMathLib/OpenBLAS/tree/develop/kernel/...
Even if counting that it is basically calling into C or C++ libraries, there is lots of boilerplate that the CPU has to consume, that just won't happen on the pure C implementation.