But I did come across some interviews of great Tantric gurus, and they all say that some people are more in inherent advantage over others in terms of spiritual progress.
This is from Digha Nikaya 14. There is another one, but I don't remember which sutta, where Buddha says that for some people the path is long and hard, while for others it's short and quick.
I interpreted this as being in a better place due to entirely one's own background after their birth- i.e. upbringing, education, work, mindset, intelligence, etc.
Where the tantric gurus say that some people are ahead of other people from their birth - in a very clear and unambiguous way. In a very "the force is strong with this one" vibe.
I don't "subscribe" to tantra, either, but I felt curious and inquired into it.
That's essentially the entirety of the Buddha's teachings on karma. I haven't the time to look for sources right now, but I'm pretty sure the close disciples of the Buddha - who all had distinct and extraordinary abilities, either feats of memory or meditation - were explained to have those abilities due to intense work in past lives. Indeed, the Buddha himself didn't chance upon his enlightenment - he made a vow to do it thousands or millions of lifetimes prior, and worked towards it life by life until he was able to achieve it.