As far as I can tell, I was drawn to Physics out of sheer masochism. In highschool I excelled in history and English without putting any effort in, but had to work on my math and physics to do well. This trend only intensified in university. I found quantum mechanics to be the most challenging sub-field of physics (except, perhaps, solid state), and guess what I wound up doing in grad school!

Maybe there just aren't enough masochists willing to embrace the pain out there.