Usually this stems from an arbitrary distinction between oppression of "the government" and oppression by a private party. When you make the (absurd) assumption that oppression can only come from the government, sure, it makes sense. However... well, it's an absurd assumption. At the end of the day, oppression is oppression (and the distinction between government and private enterprise gets awfully fuzzy in the kinds of worlds these so-called "anarcho-capitalists" envision anyway. If the police and the military are corporations, what's the difference?).
Libertarians (or "anarcho-capitalists") tend to put themselves in the rich person's shoes, so they glaze over the oppression part. Nobody is oppressing /them/ because in their ideal world they are, of course, filthy rich. They have only considered half of the equation...
In short, "It's okay for me to stomp on another's face as long as the government doesn't stomp on mine" is not anarchism in virtually every other sort of serious anarchist's view (and, IMO, is hardly an admirable world view regardless...).