Baking for friends is a fun way to pass the time. Baking for money is a quick road to a second burn out; it was literally years before I was able to stomach the thought of baking cookies or bread for my wife after I left the bakery I worked at.
Maybe it's because I grew up in a working class mexican neighborhood where trades and unskilled work are the majority of occupations, but sitting in an office chair for 7hrs a day is paradise compared to baking, mechanics, or any type of manual labor.
This is the real problem with manual labor, low-skilled labor, and jobs like baking.
The real stress isn't the day to day work, it's knowing that you are slowly wearing down your body but not earning enough to "get out of the game".
It's lack of financial security.
If as a baker you could buy a nice home and pay it off in 7-10 years it wouldn't be such a tough gig.
8+ hours a day, 5-7 days a week, nearly every week for years on end is liable to turn any hobby in to a slog eventually. Except for some people and some hobbies it doesn't. Somehow some people make it work, and enjoy it in the long run, without burning out.
Don't ask me how. I haven't found that magic job myself, and always tend to burn out on anything eventually. I need variety. I can't do any one thing for the long run without getting sick of it. Unfortunately, making career changes is difficult and stressful. So I'm not sure what the solution is. Maybe someone who's made it work for themselves can chime in.