If you mix in some of the diffused signal directly, it bridges the gap between the initial sound and the feedback echoes: https://signalsmith-audio.co.uk/writing/2021/lets-write-a-re...
---
You can tune the diffuser to have an almost-instant onset. I can't remember what I did last time, but at a guess, having the largest diffuser stage increase by a factor of N (number of channels) instead of 2 might do it.
But also, if you're playing acoustic drums in a big space like a concert hall (instead of a long one like a staircase), the first echoes coming back from the walls are actually a bit delayed (1 foot ~= 1ms, at the speed of sound). So if your nearest wall is 10ft away, the first wall-echoes will come 10-20ms after the initial direct sound.