S3for.Me – cheap alternative S3 cloud storage

These headers are part of S3 implementation an we left them to avoid any possible problems with S3 clients. Some clients are very strict on protocol validation and refused to work in some cases. The purpose of these headers are to track all requests and be able to find any problems with any particular request. You must include them when contacting support with problems with some request. This is true both for Amazon and S3For.Me

(Potential) infringement doesn't just occur upon verbatim copying of a trademark but also using terms that are similar or confusable with it: http://en.wikipedia.org/wiki/Confusing_similarity .. For reasons unrelated to the above, it didn't quite go Microsoft's way but consider the Windows vs Lindows case: http://en.wikipedia.org/wiki/Microsoft_v._Lindows

Of course, and amazon S3 is also storage, not Audi S3. So you might expect action from them. Play it safe.

This is clearly a reference to Amazon S3. So much for having a unique name. This sucks.

Hmmm... Well you may want to actually put that on your website. And you probably want to be careful, since you've posted comments on here like "all the rest documentation can be found at Amazon S3 documentation site, it is basically the same with small differences" and you have "Amazon" on the sign, which is above the "fold" on my screen and pretty much the first connection I made on S3. If your entire website doesn't scream "we are just like Amazon S3" then I must be missing something.

I've been happy with unmanaged servers from Incero. Their severs use ECC RAM, their support seems competent, and they offer buy-down prices for things like RAM.

Back to pgrove's point, though — there are trade offs with using a dedicated server. The storage isn't as reliable as S3 and the server itself is likely less reliable than a VPS (since some hardware problems on a VPS can be solved by migrating to new hardware and only involve a small downtime, rather than the large downtime of restoring from backup).

A major advantage to dedicated servers that I don't hear much about is simplicity. Dropping everything on a dedicated server means I can use local services like the file system rather than having to deal with S3 and latency and bandwidth costs.

Edit: I just realized that the idea was to use Rackspace files + one of their servers rather than S3 + EC2. Somehow I missed that bit and was thinking about just storing everything on the dedicated server.

Interesting, and thanks for the response. If I may probe a little further, I have a couple of follow-up questions.

1) Server groups of at least 3 mirrored servers, with a max of 5TB.

This seems like an interesting design choice. What do you mean by "at least"? Does this mean you'll have some data with more replicas? Are these server pools filled up and then powered down until they are needed? How do you choose which server pool to send the data to? And since you have a mirrored set of servers, when do you send a response back to the client?

Is the 5TB number something that is a limit for the storage server (ie 15TB total for a cluster of 3)? That seems rather low. It also doesn't divide evenly into common drive sizes available from HDD vendors today. So what kind of density are you getting in your storage servers? How many drives per CPU, and how many TB per rack? Since you're advertising on low price, I'd think very high density would be pretty important.

2) You say you split data into smaller chunks if it crosses some threshold. Let's suppose you split it into 1MB objects, once the data is bigger than 5MB. And each 1MB chunk is then written to some server pool which has replicated storage (via the mirroring). How do you tie the chunks back to the logical object? Do you have a centralized metadata layer that stores placement information? If so, how do you deal with the scaling issue there? If not, another option would be to store a manifest object that contains the set of chunks. But in either case, you've got a potentially very high number of servers that are required to be available at the time of a read request in order to serve the data.

Just as an example (and using some very conservative examples), suppose I have a 100MB file I want to store and you chunk at 10MB. So that means there are now 10 chunks replicated in your system, for a total of 30 unique drives. Now when I read the data, your system needs to find which 10 servers pools have my chunks and then establish a connection with one of the servers in each server group. This seems like a lot of extra networking overhead for read requests. What benefits does it provide that offset the network connection overhead?

And what happens when one of the chunks is unavailable? Can you rebuild it from the remaining pieces (which would essentially be some sort of erasure encoding)?

Overall, the chunking and mirroring design choices seem to me like they would introduce a lot of extra complexity into the system. I'd love to hear more about how you arrived at these choices, and what you see as their advantages.

In order to not make my long post even longer, I'll not pursue more questions around listings, failures, or hot content.

Right now it is $0.04 per GB which is 3 times cheaper than Amazon and customers with a lot of traffic will get additional discounts.

Yes, of course we have our reasons. One of the most important are: - We didn't like internal storage implementation, we believe that we can do better. Very self-confident, I know, but I still think so after almost a year of development and production use. - Production deployment from the start required a lot of hardware investments, we decided to put these investments into development.

Personally I think that we made a right choice and I don't regret about it. We learned a lot about cloud storage, scalability, performance, possible problems and I believe that this knowledge is very important for every team who works in cloud storage business.