It only takes a couple million years for humans to colonize the entire galaxy.

As long as a species like humans is confined to their own rock, there's no threat of them doing much harm. Once they get to a second rock, there's a good chance they'll get to a 3rd, 4th and nth in another star system, within a couple decamillennia. It's exponential grown, limited only by the speed of light.

They'll probably start building Dyson swarms too and taking over stars that way, which is a lot more annoying than them taking over planets.

This is scary fast to observe if you're a long-lived species, or if you live around a gravity well and time dilation makes it seem like humans are buzzing along and taking over stuff at incredible speeds.