The slowdown is actually in the startup of the backdoor, not when it's actually performing authentication. Note how in the original report even sshd -h (called in the right environment to circumvent countermeasures) is slow.
As for the rest of us, we got lucky. In fact, it’s quite hilarious that some grump who’s thanklessly perf testing other people’s code is like “no like, exploit makes my system slower”.
Andres is one of the most prolific PostgreSQL committers and his depth of understanding of systems performance is second to none. I wouldn't have guessed he would one day save the world with it, but there you go.