I've thought a lot about what I would do in that position, and came to the same conclusion you did. The right thing to do probably would have been to move far away, live simply, and take up a hobby so he could meet people on an equal footing.
Of course, it's easy to decide how other people can fix their lives :)