Programming is by definition technical work that requires a significant amount of brain power and focus and if I am an employer (a good one!) I would intuitively expect a certain level of focus from each employee that also entails a certain amount of downtime in order to stay fresh and alert.
This is my attempt at a steel-man of their argument. If your employer(s) is happy with your output and you aren't lying about your availability in order to juggle everything, then there is no harm imo.