Public servants often do not make enough money in public service to make these kinds of ethically gray payments easy to turn down.

I don't know about Yellen's personal finances, but the chairman/woman of the Fed makes $203k/yr. I bet 50%+ of the people reading my comment make more money than that, and unless my comment is graced by a well-known CEO I sincerely doubt any of you have as important a job.

Maybe if we want to hold our public servants (generals, commissioners, congressmen, secretaries of x, etc) to higher ethical standards we should compensate them well enough that a speaking fee like this is "more money" rather than "help pay off my mortgage".