But if, say, a mere 99% of miners switch, it’s far from a given that people would follow. Having more mining capacity makes the chain more secure, but it’s not that big of a deal.
If miners suddenly fled en masse, it’s possible for the chain to be left stranded where the small number of miners remaining couldn’t realistically get to the next 2016-block interval to adjust the difficulty down to match the drastically decreased mining capacity. If mining capacity dropped by a factor of 1000 and it happened right after an adjustment, then bitcoin would be producing about one block every week, and it would take about 40 years (if mining capacity stayed constant) to reach the next adjustment.