I don't your employer can characterize your entire life as 'work hours'. Do you log what time you work for them (you should)?
It also might be relevant to ask, if your side project has anything to do with your job or what you do for them. If it's completely unrelated, I think that would also think that would help.