When I discovered that wasn't normal for everyone, even people I go to church with, I was pretty surprised.
When I started meditating a few years back, I took to it like a duck in water. I had no problem doing 20-30 minutes right away.
But I don't get the same thing out of it that I do out of prayer. Meditation is like local maintenance - a reboot or a defrag. Prayer is more like a software update, connecting back to the home server for bug fixes and feature upgrades.
It's the feeling of being connected to something greater than yourself that seems to be the key difference.