So the scenario is that miners drop off at a pace where you don't reach the next adjustment?

Taking that into account, the "mining death spiral" theory doesn't seem too far fetched? In bad circumstances, if the network drops a substantial amount of hash rate early in the adjustment, and price took a hit, you could see some major disruption from the miners following suit.