Nah, low latency audio is absolutely possible, but not necessarily using pulse audio. Using either Jack for the low latency stuff has been the advice for years, but with the advent of pipewire you can mix and match without too much difficulty.

A quick skim didn't turn up what audio backend this project is using, but I'm using patchbox with modep on my Pi4 as a bass pedal board and it's pretty much flawless low latency. I do need to add a fan to my pi since it's mounted underneath a pedal board and doesn't get quite enough air though.