Personally, I think an inexpensive 10m rig is the way to go, although if you live in a place with a decent 2m or 70cm repeater network there's an argument to be made for getting a cheap HT first. There's almost an assumption in most clubs that everyone has a 2m HT (e.g. talk-in channels for events and stuff).
A club that I joined years back handed me an old Radio Shack 10m SSB on my second meeting, it was sort of the "village bicycle" radio that everyone got to use until they bought their own. That was a fine way to start; I think it put out 10-12 watts or so (I'm dead sure internally it shared a lot of parts with some CB radio they must have sold at the same time), but that's enough to get a feel for things without too much frustration, even if your antenna isn't optimal.