The difficulty will adjust for the reduced compute power
So the scenario is that miners drop off at a pace where you don't reach the next adjustment?
Taking that into account, the "mining death spiral" theory doesn't seem too far fetched? In bad circumstances, if the network drops a substantial amount of hash rate early in the adjustment, and price took a hit, you could see some major disruption from the miners following suit.
...assuming very slow change in hash rate, yes. A rapid increase makes it happen sooner, a rapid loss can stretch it out.