I completely agree. Dealing with remote team members has always been difficult with teams I've been on, and I would guess they have been only half as productive.
This isn't through any fault of their own, but they just can't be "in the loop" as much. Most communication takes way too long to type out in e-mail or IM, so you need to get them on the phone -- but calling and getting transferred, and then they're not at their desk, etc., is just too much of a pain. There's a HUGE difference between waiting until you see that someone is at their desk, and repeatedly remembering to keep trying to call them -- or sending them an e-mail to call you, and then you're out.
They can't see when someone on your team is having a little difficulty and instantly help out, turning someone's hour task into 5 min. They're not involved in the informal meetings behind someone's screen where a feature is changed slightly. They're just missing out on a lot of "informal" information that is crucial for productive development.
And even over the phone, I find it's much more difficult to explain things and make sure they're understood crystal-clear. Sometimes you need to see the understanding in their face. Sometimes you need to draw, sometimes you need visual aids, sometimes you need to pull up a web page, but it's too much work to pull up videoconferencing or something.
In the end, all these problems are "technically" surmountable, sure, but I've never seen it work. Maybe remote work would be fine in a waterfall-type development model, but with modern-day "agile" development, it just doesn't work as well for collaborative teams.