Unless it impacts your performance during working hours
I broke my leg once playing paintball, my personal time hobbies impacted my performance during work hours for a couple months due to mobility issues + appointment followups. Is it in my employer's purview to tell me I can't play sports?
Actors will sometimes have clauses which prevent them from doing dangerous stuff they like to do (skiing, race car driving) while they're shooting a film. An actor becoming unavailable is very expensive.
For any given worker, losing their contribution costs more than just their share. There's always a certain amount of working around the gap in the roster. But that's priced-in, if you will: people go on vacation, take sick leave, parental leave, and they quit outright.
So I don't consider it ethical for employers to add that sort of clause under normal circumstances, as indeed, they don't tend to. I do consider it ethical in specific circumstances, however, such as the one I just described.
I know games companies that let their employees work on games on the side, and publish them. It doesn't hurt the company, and the employees are happier.