Ask HN: How to upload lots of data? | Hacker News Reader

Ask HN: How to upload lots of data?

6 pointsggruschow6 comments

I've got monthly 10-30gb lumps of data I need to upload to S3 for parallel analysis on EC2. It takes days to upload it on my connection, and my ISP will get mad eventually.

I ordered a second line to at least avoid pissing off my primary ISP, but I'd rather it was a lot faster. I don't think I can't get yipes, cogent, FiOS, or any of that good stuff where I am (60614), and it's not worth it for me to get a T3 (yet). It seems like I should just be able to drop off a USB drive somewhere that's got bandwidth to spare (any time of day.. I don't care), or maybe there's some multi-megabit upload technology I'm not aware of, or maybe somewhere to do a cheap mid-tower server colo with easy physical access somewhere in Chicago?

P.S. The data is already heavily compressed. P.P.S. I want to do it on the up and up.. I'm not going to sneak it onto someone else's connection.

I had a box downtown in the loop (in the Chicago Board of Trade building) here about 4 years ago:

http://www.fdcservers.net/ContactUs

Like all colo, it's not cheap (if you buy for a year, it's $82 a month). But you get a great pipe and 24hr access (it's manned at all times) to drop DVDs in the server (you can colo towers, it's not just limited to rackable).

If you only need to do this once or twice a month, I'll bet you could email them and they'd work something out where you could pay something small to come in with a laptop or even maybe drop of a whole USB drive for them to upload (you'd be willing to entrust your S3 keys like that?).

It wouldn't really take that long with their network speeds. Let's say you could get 10MB/s to S3 from there, 30GB would take, what, like an hour?

Anyhow, they're reasonable, won't hurt to ask.

Some similar talk here

http://solutions.amazonwebservices.com/connect/message.jspa?...

I talked to jeff barr @amazon a while back about these kind of issues (if you have large datasets to shuffle on/off EC2, is there an efficient & cost effective way to do it?) - shoot him an email or comment on the aws blog so they know more people have this need. Amazon public datasets can help with this for some use cases, for example certain bioinformatics or other academic use cases.

can you just send the data on dvd's with regular mail (ups etc)? of course I don't know where the server stays, if it's human accessible..

to the amazon people? Good luck, I don't think it's possible, but it's the only option I can think of.

I smell a business opportunity.

Reply on news.ycombinator.com