"Since the whole protocol is bit-banged, it is susceptible to delays caused by other higher priority task"
Ermmm no. Not utilising both cores properly on the ESP32 is a common mistake which causes this (described) behaviour https://randomnerdtutorials.com/esp32-dual-core-arduino-ide/
Ref. me, I wrote a Java Gameboy 2BPP encoder/decoder library and have done multicore ESP32 dev. https://github.com/KodeMunkie/gameboycameralib