What if you say, "you know, these cards can be worth a lot sometimes, is that going to bother you?".
I mean, part of it is that you should also remain ignorant. If you see an expensive card on the first page, yeah, morally you're in a spot if you act like you didn't. But if you just assume that a collector has something of value, but you don't really go through it other than to make sure that it's actually all the same stuff, I think the terms are fine. Just so long as you're actually spinning the wheel as well.
I'm reminded, though, of High Fidelity. There's a part of the book (maybe a deleted scene on the DVD as well) where Rob goes to buy a collection and he's seeing all these great items, first editions in pristine condition. The seller is a woman who is going through a divorce and doesn't need the money but her husband does, and he asked her to sell them for him, and she's trying to dump them for very low price to be mean.
Rob can't bring himself to buy them, even though he wants some for his collection. So I get where you're coming from. I don't make as much money as I could because I don't fleece people, either. But there's times when people are asking for your expertise and times when they just want to move on, and everyone should be happy in the end.