I don't think this is intrinsically true - a lot of people just don't personally know anyone who desperately needs help. Is it really overall better to be 100% efficient at getting a rich private school a new telescope instead of being 50% efficient at getting people with no safe drinking water a means of access/production of safe drinking water?
If getting maximum impact is your main concern, I'd go with GiveWell's recommendations: