I'm not quite on Pitcairn Island, but I live and work in rural Vermont, and by far the biggest problem with it is precisely the lack of a reliable internet connection. Right now for me, the only viable option is satellite, which is horrible (225 MiB/day download cap, 800 ms ping times). I'm currently investigating a T1 line, which will be expensive, but (slightly) cheaper than renting an office in an area with DSL access. (Moral of the story here is clearly to not fall in love with a girl who owns a house on the end of a dirt road...) I'm sure many (though certainly not all) other remote locations are as bad; so don't make assumptions on that before you pull the trigger on a specific area.

Another factor of living in the middle of nowhere, especially after living in DC and NYC, is the technical/social isolation is pretty intense. I'm about 10 miles from Dartmouth, and there is small tech culture there, but it's not like there are monthly Lisp or Scala or security meetups. Or even a student ACM chapter, for that matter. No matter how you slice it, it's hard to replace quality in-person interactions with online. IRC and mailing lists are great and useful things, but being able to hang out drinking coffee with someone while you sketch out ideas in a notebook is pretty damn important too. (There are plenty of interesting people out here, but most programmers, especially the good ones, will of course gravitate to urban areas, so the population of tech people out here is near zero).