Buddhism was prominent in India till Muslim invaders came and conquered India. Buddhist Monastery were sitting ducks for them, Buddhism was eliminated in the first wave of Islamic invaders.
Of-course there were struggles with others before that, such as Brahmins, but those were political struggles, not existential. Many times Buddhists gained the favor of Kings, other times Brahmins.