As someone who shops at Whole Foods all the time, despite the seemingly ridiculously small discounts you get as a prime member, it does add up to cover most of the cost of prime membership.
I do use prime services, so I might otherwise keep my membership — but I would consider cancelling if I were just using it for shipping.