Yes, you are within your legal rights, as you say, "to do whatever the hell you wish to do." Unless, as the others who have made comments indicate, you have signed away that right in your contract somehow- via IP assignment rights, etc.
If you don't want to 'go see a lawyer', you may want to check out an item posted to HN last week- from This Week in Startups, "5 Biggest Legal Mistakes That Startups Make" - http://www.youtube.com/watch?v=-F_vbfiScoA#t=746
Specifically, what you want starts at the 10:20 mark.
Also, if you feel that your employer is the type to create problems for you, it would be a good idea to be strict about not using company resources at all. Even though other employees may be doing it, don't google things on the company computer that can be traced back to your project.