Would Numpy be of use? Haven't looked at the source yet, but for a keyboard, 4 ms per matrix would be noticeably laggy, especially if you add debouncing (is that even a thing in keyboards?)
The controller is running CircuitPython which is a fork of MicroPython - a tiny re-implementation Python for microcontrollers.