Not only that, but the matrix scanning could be done via interrupts instead of via polling as is typically[0] done. An interrupt could fire on a level shift of any of the column GPIOs, then the MCU needs to search the rows to find the selected key. After the interrupt the MCU will need to poll until all the keys are released. So there's another source of latency that could be improved.

I think the only source of polling left is in USB, but I think that's inherent to the USB protocol (someone correct me if I'm wrong here). Without the USB polling I think it would be possible to have key-press-to-USB-packet completely interrupt driven which should make the latency in the keyboard itself negligible.

[0]: I say typically but like you said a lot of implementations are closed source, so who knows. All of the discussions I've seen on matrix scanning use the polling method, and the open source implementations use polling as well (e.g. Keyberon[1]).

[1]: https://github.com/TeXitoi/keyberon-grid/blob/master/src/bin...