I'm 23 and I just left my first job of two years to build a startup about three months ago. I'd like to believe that while you're young (if you are) and/or while you don't have to support anyone (e.g., a family), there's no reason why you shouldn't give it a try. If it doesn't pan out, you can always try again or just go back to having a regular job.
And I'll agree with some of the previous comments that all you really need to worry about is the money. Be sure that you can afford to not have any reliable income for a long while. Otherwise, if you think you have a good enough idea to pursue, just keep your morale up and you'll be fine.