For example, Google proposed https://github.com/explainers-by-googlers/Web-Environment-In... and this was shot down by privacy advocates (for very good reasons).
So basically the choice for website operators is either to fight the bots and accept that their service will be unusable for some subset of their users or not fight the bots, which will lead to their service becoming unusable for everyone.
More and more, you see services pushing you very hard towards using their app and the reason is that with the app, they are able to actually verify that you are likely not a bot (or rather, in reality, that at least the app is running on an actual physical device, mobile phone bot farms are unfortunately also a thing).
As for Cloudflare - they offer it as a service, so when the website operator has a choice between using them or allocating several engineers for bot-fighting, why would they not just go with Cloudflare? Doing it yourself can be slightly higher fidelity, as you know your customers better, but it is also a lot of effort which could be better spent elsewhere.