I suppose one source of risk is that if enough miners exit then it becomes viable for a bad actor to purchase enough computing power to mount an attack.

IIRC 51% attack happened on one of the bitcoin forks.