If all the miners go away the difficulty requirement will drop on its own to maintain the 10min block rate. No need to fork anything: https://en.bitcoin.it/wiki/Difficulty
As I commented in another thread: the difficulty adjustments happen periodically after a certain number of blocks, not after any given period of time. If the hash rate decreases, the adjustment slows down along with everything else.
A 5%/day sustained drop is enough to halt transaction processing entirely.
Interesting weakness! I hadn't considered that. (Potentially also an interesting attack vector for states wanting to shut it down?) It would indeed force the community to change the difficulty algo to work around it.