Anecdotal, but my cousin did exactly this. I will note, he is probably the most exceptional person I known of, beyond those classic geniuses everyone knows.
Was money a problem in general along the way?
I agree that you should give serious pause. Probably the most serious pause of your life, but still. I wouldn't shy away too quickly. When you know, you know.