So is everything everyone who's parents didn't act like complete deadbeats just the result of "privilege" and any effect of individual achievement, merit, hard work, etc. is marginal at best?

You can't hand wave and cite some difficult to quantify "privilege" in early childhood to explain all the differences in individual achievement among a group of people who share a common trait.

At what point does privilege stop and taking advantage of opportunity begin?

I'm not denying the role of luck in determining the course of one's life but to treat it as the global predictor of what one will achieve is basically denying the ability of people who do not have good luck in one way or another (e.g. who you were born to) to move up the ladder in spite of that.

Yes, it's better to be lucky than be good but being good can still get the job done.