Being in a situation where you only have yourself to blame, when you can't say "oh if only I didn't have these coworkers and that boss and my family and everything else holding me back (holding you up, really)," it's a different kind of fear. It's probably healthy, but surprisingly intense.
You're totally exposed and get to see exactly what you're capable of accomplishing. It's like that machine in The Hitchhiker's Guide to the Galaxy that shows you exactly how insignificant you are in the universe.