My brother is quant physics PhD. Whenever I measure myself up to him, I realize it's good I didn't pursue academia because I'm not smart enough. It's insane how smart other people are.
Just because you don't think you'd succeed in ${world_renowned_and_highly_pressured_university} doesn't mean you can't do amazing things at ${other_university}.
Seriously - if you have an original thought in your head, can write, and can justify your point of view, you can do a PhD.