a) RFID is readable from further away than they'd like you to think.
b) You don't know when your RFID card is being read.
c) Points a and b make tracking you really easy... for anyone to do.
d) The only thing that should (ideally) be stored on any RFID chip is a unique number... not any history (recent transactions), personal data (name/phone/picture), or payment system (think public transport) where the actual info about how much money is on the card is stored on the card itself... but that's exactly the type of information which is stored on these cards.
e) Nearly all encryption mechanisms are shoddy, either because they're poorly implemented open standards, or developed in-house by the vendor (security through obscurity). Cards that make use of real encryption would be (are?) expensive to make.
Here in the Netherlands the entire public transit system is being switched to an RFID-based system, and even to a non-security expert (me) it's clear that the system is based on an insecure premise (d), and would be very vulnerable to unknown scanning by someone wishing to track you from a decent distance (a-b-c).
I was interested in the security of this system, and found this video (http://events.ccc.de/congress/2007/Fahrplan/events/2378.en.h...) of some hackers who did an amazing job tearing it to shreds. They're pretty adiment that nobody is doing adequate encryption on RFID cards. If you're interested in this at all it's an amazing hack, involving dissolving the cards layer by layer to see the code.