I think you are confusing non-compete and NDA, IP rights, non solicitation agreements and the like. You can go work for competitor, but it doesn't allow you to use any of your previous employer's IP, poach colleagues, clients etc. Doesn't prevent you from using your expertise and come up with novel things.
And some big companies do care that you don't do it, because they don't want to go into litigations because layers like to target fat cats.