I, too, am irritated by people who advocate raising taxes for (or greater charity in) every bracket above their own. But Harris isn't doing that. He's asking the two former richest people in the world to tell him how to spend the fruits of his own labors.
But beyond that, Harris points out that a significant fraction of your "fruits" are not of your labors but of your favorable circumstances. He's telling you that you ought to share some of the fruits of your good fortune.
Even if you deny the role of chance in your financial success, how you spend your money has strong ethical implications. If other people believe I'm acting unethically in some way, I do consider it their duty to tell me.