Ouch. Yeah, that's a really good point about travel being both a big commitment/energy-drain on your end, and a bad travel experience making it really hard to interview well.
I have, in the past, flown someone who is coming cross-country out for the weekend, covered the hotel costs for the whole time, done something non-structured on Saturday or Sunday, then done the interview on Monday. And a few times people have brought family members. (We didn't pay for the family travel, I don't think, but I don't remember for sure.)