A tech support scam attacker would have many first-time connections to many other first-time TeamViewer users who are generally seniors instructed to run the TeamViewer app over the phone. While they may use a pool of computers/TeamViewer IDs, and a pool of IPs, there's limits to the cost-effectiveness of scaling that variation, and a pattern should definitely be visible.
"Assuming proper rate limiting" seems like a large assumption, given that the possible attack vectors are guessing the random alphanumeric passwords and testing password dumps for account pairs from other services that work with TeamViewer.
Defaulting to accepting any connection from anywhere seems like a great example of poor security configuration by default.