It's hard for employees to know how valuable it is for them to sign a non-compete. It's hard for companies to advertise their desirability based on, "those other guys will make you sign a non-compete. We won't." So it becomes low cost for people to ask for it.
In the current environment, non-competes should be banned: California proves this is a viable approach. But perhaps a better long term solution that seems within technical reach right now would be some kind of AI legal assistant that could demystify and educate people about the contacts they're asked to sign.
Like imagine if you had a smartphone app. You scanned the contract with your camera and the app said, "This is a non-compete agreement. 60% of the companies in this field use them. Best estimate is that signing it costs you $30k in lifetime earnings."
Then a job seeker could just negotiate to their own best interest.