So. There are sports which don't require competing with other people in any real sense - take a look at rock climbing, hiking, riding bikes, SCUBA diving, tai chi, yoga, dancing etc etc. Some of these can be solitary if you don't like big groups.

On the physical pain side of things - that's just the hump you have to get over, and I agree it sucks. Once your body is trained to do a basic minimum of exercise, the endorphins outweigh the painful aspects, and it feels good man. Like, really good.

To your original question, I suspect 20min a day walking is a bit low, both in terms of your overall exercise per week, and intensity. Intensity counts for a lot in fitness, IMO it's worth pushing it a couple of days a week, go for a run or something. Or join a club that does some kind of fun exercise.