I think the caste system is what’s hampering social and economic progress in India (or at least partially).

It’s a sort of game theory stuff here. A structured class hierarchy makes it inherently more difficult for individuals to challenge the authority in power, even under the democratic government. The system imposes an additional risk of social backlash/punishment/retaliation for anyone who attempts to disrupt the established order, thus people have more reasons to stay back. This kind of risk is largely absent in more egalitarian/classless societies.