Let's call this cycle A and cycle B.
If A is too hard, miners drop out, cycle B gets easier, miners flood back, cycle A gets harder.
This results in the hard cycle getting longer and the easy cycle getting shorter.
This isn't completely critical as there is I believe a small damping effect, so it isn't completely lethal to bitcoin, but a key thing about bitcoin mining is that whether other people are mining or not doesn't actually affect your own profitiability.
Other people dropping out doesn't actually mean you get more bitcoins per hour/watt, it only affects the next difficulty adjustment as a secondary effect.