If you're on an H-1B and you get fired or laid off, you have 60 days to find a new job or be deported. That creates an underclass of workers who are willing to put up with much worse working conditions and work longer hours. That drives down working conditions and wages for everyone.
A $100k per year fee doesn't fix that, but it does make them so expensive that they are really only viable for $300k+ positions.