> It bumps up the debounce for next time

...

> But when things are smooth for a while, the debounce comes back down

it sounds like he's just incrementing a variable in the first case and using timers to decrement it in the second case. that's a bad implementation. the correct implementation here is an envelope follower, or maybe-perhaps a kalman filter. OP's implementation will exhibit motorboating: https://en.wikipedia.org/wiki/Motorboating_(electronics)