I think that the united states could absorb the entire population of Haiti without much impact (ie., from 81 to 80), and eventually we'd probably benefit. But how about all the desperately poor people everywhere else in the world? At some point, I think we're going a lot further down than just one point on the scale.
Right now, this reality is reflected in our (remarkably badly run) immigration system, where the United States takes about 1.2 million immigrants legally into the country every year, by far the most in absolutes and high up there in percentage of population, but still faces a surprising amount of criticism for being anti-immigrant because we don't take in even more. Much of this criticism comes from people who live in countries that are remarkably stingy with their immigration systems.