Lots of big businesses use recaptcha. Quite often unnecessarily. If I need to login with 2FA touse a service does it really need recaptcha?
Similarly, cloudflare sends you emails telling you how many bots and attacks it has stopped - but you do not know how many false positives there were.
I would guess that simple rate limiting would do the trick for the rest
As far as I can tell, most startups resolve their technical debt by failing, and the majority of the rest resolve their debt by being acquired by a company which replaces the original service entirely in 1-3 years because it's too hard to integrate as-is.
In fact I used to fake user agent all the time because Microsoft 365 is so retarded. With the Firefox/Linux user agent a lot of features don't work. When it pretends to be MS Edge it works fine. Clearly trying to force people to use the 'invented here' browser :(
But as I was getting captcha's I moved to using it only for the MS365 sites and nowhere else. It seems to have reduced the captcha's somewhat, especially the ones that never end (keep looping). But I still get a ton of "Your browser is suspicious, here's an extra check" nonsense from Cloudflare in particular.
At least with captchas, it's somewhat understandable with the arms-race aspect. The third party does the work of engaging in the arms race, so you don't have to, but the tradeoff is what you describe.