Had I not done that, I probably would've stayed a physics major in college, done well, gone to physics grad school, and then realized I hated it when I was around 26 and suddenly ended up with no experience doing anything else and no clue what I wanted to do, a far worse outcome.
I think that there is, in general, far too much fear in contemporary American culture. Fear that if you don't do everything perfectly and don't follow the sequence laid out as a "safe" career track, you will end up screwed for life. Yes, if you end up following your own path, you will get burnt by bad decisions at time, and you'll have to scramble to fix them, and some opportunities will close themselves off to you because you didn't pursue them. But you will also gain practice & confidence in dealing with those bad decisions, and learn how to fix them and that there will be other opportunities that open themselves up to you because you screwed up with what you thought you wanted.