If your frame of reference is an economy coach ticket, if you say "it's more expensive to fly private", you'll almost never be wrong. (Your economy coach ticket is
heavily subsidized by the first-class and other full-fare tickets.)
If your frame of reference is a first-class or full-fare business class ticket, I can often beat or near the airlines on both cost and door-to-door time. (If the keys to the plane are in your pocket, there's no getting there an hour ahead, dealing with the parking garage, the TSA theater, maybe a delay on takeoff for crew or aircraft availability or ATC/ground holds, perhaps changing planes, waiting at baggage claim, etc. Plus, I can often start and end the flight much closer to the "doors" I want to be at, leave my shoes on and carry 4 oz of gel deodorant in my bag if I want.)
As to "that's worth it", I bought a Cessna 182/Skylane (4 seat, single-engine, 160 mph high-wing) a little over 12 years ago, and have flown it for business travel as well as most of our in-US, east of the Mississippi, personal travel, as well as a few trips west of the Mississippi. It costs me under $2/mile on average, and the direct operating cost (aka marginal cost: the cost of an incremental trip within a year) is just under $1/mile. If I had to, I'd pay 2x what flying actually costs me without thinking about it.
If you want to travel in your aircraft (as opposed to just going flying), I'd suggest you take the traditional route and go all the way to the private pilot's license. Far fewer restrictions, and if you enjoy flying anyway all of the "extra" time is hardly wasted. IMO, there are too many restrictions on LSA/recreational pilot's certificate to make it usable to actual go places with 3 friends, etc. It's nice to be able to fly into BWI for an Orioles game, or fly up the Hudson River at night, cross Central park and Laguardi tower and head out on course over the Whitestone bridge. I think it's great that the FAA created the easier licenses, but I don't think they are practical for travelling.
Used airplanes are reasonably cheap now, and loans and insurance are easy to get. (Most airplanes are made of aluminum and have nearly infinite lifespans, with avionics, engine/props, interior and paint being replaceable items over the life of the plane. Think of a 30-year old airplane as much more like a 30-year old office building than a 30-year old car in terms of "acceptable to rely on". They're inspected annually [or ever 100 flight hours for rentals/commercial usage], and no reason a 30+ year old plane wouldn't be perfectly reasonable.)