I agree, however I'm not sure of how far you can get in probability theory without set theory. For instance, I recall my stats prof defining a lot of sets in class, e.g. sigma algebras.
Another point of clarification: I'm all for telling people what sets are. I'm less warm toward discussing things like ZFC, esoteric cardinality issues, and certain paradoxes (and large cardinals, forcing, etc.), which is typically what mathematicians mean when they say "set theory."
Right. What most people need to learn is "algebra of sets," essentially.