It totally depends on the state you are a resident of. Or the state they sue you in. Some states hold IP arrangements to be toilet paper. Some will enforce even the most egregious ones.
I am not a lawyer. It appears to me that being a contractor - rather than a salaried full-time-employee would be the route to go if you are interested in working more than one job. As a salaried FTE, the legal expectation is that you devote yourself 100% to your employer.
In business law classes, this is referred to as the "principal agent problem".
https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...
Business-type people are going to use this viewpoint in order to perceive the situation. In India, there are some laws that basically say that you are required to put your main employer first and that it is effectively illegal for you to work more than one job - no matter how poorly that job pays you.
This is one post from very long ago about "work for hire" and IP assignment agreements:
https://www.perlmonks.org/?node_id=153046
Another case that I can't fully remember the names of (so no links or references) was of a software developer in Texas or Oklahoma who developed something on his own time. His employer wanted the IP and pursued him in court for almost 2 decades bankrupting him and getting him thrown into jail (for contempt of court) when he refused a court order to turn over the source code and documentation.
How far will they go? Well, how much to they hate you? How much spite are they powered by? Narcissists and sociopaths will go to extreme lengths to destroy their "enemies". For a recent example, see ex-President Trump.