I would attribute this to the following: 1. As a society, and culturally India historically has not given sufficient importance to physical fitness. So while historically there was great importance given to spiritual and philosophical pursuits(they had universities setup more then thousand years back dedicated to philosophy - Nalanda), the physical/bodily aspect was not only overlooked but considered almost an overhead/obstacle to ultimate goal in life. Even Yoga's birth(in India) was related more to spiritual pursuit. Not surprising to see origins of Chess in India(even though Chess is a sport, but it is mental). And India does reasonably good in Chess even now.

2. Even the food habits, historically, in India are geared more towards supporting this bent towards mental/spiritual aspects than physical. Hence prevalence of vegetrianism. Now vegetarianism can be one of the healthiest lifestyle(there has been known triathletes who are pure vegetarians)...but that requires a well balanced diet - which is not the case for majority in India(for most in India, vegetarian diet has lot of grease and less nutrition).

3. There is a reason why Cricket is the most popular sport in India - it does not require great physical fitness yet you can play it for hours. And cricket does grab a lion share of sporting opportunities in India (and most lucrative).

3. Prevalent thinking still is that sports is waste of time, and you are better of spending that time on studies...hence you will see abundance of Indian American spelling bee champs even here in US, yet you will not see many in athletics

4. Combine that with lack of creativity and focus from Government in India in identifying(and persisting) with a sport which can fetch medals (like Turkey etc)