I think that's a really classical case of O(n^2). You have a linear algorithm (n), and then run it on each instance you insert (n) so n^2.
It bothered me for several years, until I eventually did some profiling and contacted the readline (and bash) maintainer who fixed it.
This is about 5 years ago.
There's also another readline-related problem in several shells that they don't discover if you resize the terminal window. I can't remember if I raised that too. It sometimes amazes me that very few people appear to ever look at or debug our common infrastructure.