Hacker News Reader

I presume this file doesn't change very often and doesn't require any security in front of it? To be cheap, you could sign up for a few crappy (like GoDaddy) hosting accounts with absurdly high bandwidth limits. Then, provide users with links to them sorta like mirrors. It's not the greatest user experience, but it might be the easiest cheap solution.

If you care a lot about user experience, go with S3.

Instead of multiple links to mirrors, I'd just put a bit of scripting in there that randomly chooses the link out of, say, 3 accounts.

S3 does look pretty expensive for this case. In general, I think S3 is best for storing files that aren't retrieved often. For hosting a download, there are better options.

I have an BingoDisk account that supposedly provides unlimited transfer. If you are just uploading and downloading stuff, it sounds like BingoDisk is ideal. (http://www.bingodisk.com/) I use it for hosting big files, like a 600M SQLite database with information about CPAN modules. It gets quite a few downloads, and nobody has ever told me I'm abusing the service.

(FWIW, I used to use BingoDisk for backups, but Linux's WebDAV support does not work well for backups -- way too much caching going on. S3 turned out to be infinitely more reliable. I don't think this has anything to do with BingoDisk, though, but rather how davfs2 is implemented. This may or may not be a problem for your use case.)

If you don't mind offloading the cost to your users, though, S3 is probably the way to go. I think Amazon will even bill your users for you now.

Using Amazon S3 you have the advantage of using a tested and scalable system, easy to implement and use and it's not expensive.

Maybe my math is off or something. But according to the S3 calc: http://calculator.s3.amazonaws.com/calc5.html

If a 150MB is downloaded 100 times it would cost $2.59. If it was downloaded 1000 times it would cost $25.55. Is that cheap?

Compared to the cost of building and maintaining your own system I think it is cheaper. Add the cost of servers, colocation, bandwidth, people.

Also, on S3 you pay exactly for what you use, if you use your own server you are paying for them up front. Another bonus is the using their CDN if you have frequently downloaded files.

Check this comment by Don MacAskill, CEO of SmugMug which stores a huge amount of data on S3: http://news.ycombinator.com/item?id=422574. Check his blog too: http://blogs.smugmug.com/don/tag/s3.

Not cheap but reasonable. They pushed 1.5T of traffic for you at that point.

If you're going on a base10 system then it's only 150GB, not 1.5TB. Technically it's 146.5GB.

Argh, you are right, off by a magnitude.

Well, I'll try to make up for it by pointing the original asker to http://simplecdn.com

From their frontpage:

A software developer with 100MB worth of downloads would pay $24.90 for lifetime storage, and would then receive a transfer rate of 3.9 cents per GB.

So, pay $24.90 once and serve your 1000 downloads for $5.80 then.

Cloudfront!

Like hell S3 isn't expensive.

You contribute nothing to the conversation.

This has already been discussed in some detail at http://news.ycombinator.com/item?id=422225. I suggest anyone interested in talking price/value reads up on that thread before commenting here.

I run a fairly small site. We're upto around 7 million hits a month, still pretty tiny. Our costs right now are fairly low, but recently I made a post inquiring how to cluster efficiently, because we needed more bandwidth available (And we were going to use a few 10mbit servers).

Someone suggested that we use s3. This would've tripled our bandwidth costs.

This is why I believe it's expensive.

And how would your s3 bandwidth costs compare to running your clustered servers? Is the DIY storage triply redundant?

I'm not saying you made the wrong decision, or the right one, but is the comparison apples-to-apples?

Also, that discussion there seemed focus solely on the point of STORAGE.

He seemed to be concerned about bandwidth also, for which S3, frankly, is not cheap.

Amazon S3 is actually relatively expensive. It's far cheaper in the long run to host it on your own server. However, if you don't have experience setting up scalable systems, tuning your configuration, or do not want to spend time doing that type of sysadmin stuff, it can be a good way to save time. It's like buying Coke at a vending machine instead of at a wholesale store. You're paying for the convenience more than anything else. Just be aware of how much you're being marked up.

Yep. Someone needs to build a S3-like solution and open source it, down to the cluster setup and harddrive specs and brand names. I have set up a storage cluster once with mogilefs and perlbal but finding documentation was a nightmare. There is none, you have to dig through the mailing-list archives.

But then again, that is the kind of stuff that people get paid ridiculous amounts of money to build, heck, I know I did. So I wouldn't bet on it. But if I had the time I would probably do it.

How many users would you have?

Because I'm tempted to say you should just try one of these cheapie web hosts and see if that fits your needs. For example, and I have no affiliation with this company other than using them a few times, webhost4life has an account with unlimited* bandwidth and unlimited* storage for $6.95 a month (http://www.webhost4life.com/linux.asp). That's a pretty standard price for linux based hosting.

The disadvantage to these types of deals is that they stuff a bunch of websites all on one server which makes performance slow. But it sounds like you'd only have a 1 or 2 page site so that might not be an issue for you.

* when these providers say unlimited there is often a clause in the fine print that sets a limit. You should check before signing up for one of these services.

Using Amazon services your initial cost is low and it's easy and fast to set it up. Using wisely AWS is a great service, especially bootstrapping a new service.

There are use cases where AWS is not recommended, having too many machines up and running 24x7 or transferring huge amount of data out of the cloud. For a site with heavy uploads it's even possible to store your data on S3 but serve your clients from a machine hosted outside where data transfer is cheaper.

What is the actual site doing? Is it just a file sharing site where you have a 150mb limit, meaning the average size will be a few megs...or is this something special where people upload some special 150mb files?

Good question. It is a site where people can upload 1 special 150MB file (like a zip file) and people can download the entire thing. So the average file size will not be a few megs.

I think S3 includes support for torrents, so it might help to reduce costs.

You need to bound the problem before you can figure out whether S3 makes sense.

* How many new files do you expect to have uploaded every month? * How many files do you expect to be deleted every month? * What's the average file size? * How many times, on average, do you expect each file to be downloaded per-month?

Use that + Amazon's pricing to figure out how much you'll have to spend on S3 every month. Then compare that to your alternatives.

In my mind, the biggest reason to go with S3 is if you think you are going to end up with a lot more files than you could easily store and mirror on a few leased servers. S3 is a pretty good price for scalable, highly-available storage with a nice fat pipe to the Internet, but the cost per GB transferred isn't great. You can take advantage of the lost cost of storage by keeping track of how often files are downloaded and keeping a cache of locally hosted files that you serve directly off a leased server or VPS.