Might be wrong about what Aperocky is alluding to but there is an entirely theoretical edge case. The time to the next difficulty adjustment is based on the current speed of mining, and the possible change in difficulty is capped. With enough minors leaving it will drop the speed of mining/network speed/ and push out the expected time to the next difficulty adjustment. I can't think of any realistic way this can occur given the miners that stay will (personally) be producing blocks as often, the increase in time being balanced out by being a larger proportionate of the mining rate. They don't care if they get 1% of the blocks, which average about 20 mins per block or 5% of the blocks that average 100mins per block.

Difficulty only adjusts every 2016 blocks. If the system gets out of whack enough it could slow down to a crawl for an extended period of time.

In practice it’s not much of an issue because bitcoin is not use for commerce but it’s a store of value and it some of the trades are not even on chain.