Amazon S3 and Glacier Price Reductions

That's exactly why I started Ploid.

A consulting customer came to me a year ago, with a growth from 200TB/year in data production to over 6PB/year and their budget couldn't sustain that jump (or anywhere close to it)

Having come from the mass-facilities and data center space with MagicJack, I knew the wholesale cost of bandwidth, power and drives were continuously falling.

There are certain clients and use cases that need access to their data all of the time and the very bones they are built on is based on collaboration (Genomics).

For example, this client is now storing 6PB of data with us, 3 copies in separate data centers. We are half the price of S3, and we include all the bandwidth for free, but limited to a 10GigE per PB stored. This has worked out extremely well - we were about 20% (!!!!) the price of Amazon after you factor in bandwidth.

There are lots of challenges we faced, like over zealous neighbors in the environment, storing lots of small objects and high usage of ancillary features like metadata but for customers of any size. By putting the "tax" on bandwidth, a lot of these business cases are solved. I see why Amazon does that.

AWS is truly great, but as you get into very high scale (specifically in storage - 2PB+), it becomes extremely cost prohibitive.

> The bandwidth costs are so far out of line with what the network transfer actually costs, it just feels like price fixing between the major cloud players that nobody is drastically reducing those prices, only storage prices.

Yes, it makes me wonder. For many applications, the ridiculous bandwidth costs will be substantially higher than compute costs, so it makes no sense that nobody in this supposedly competitive market wants to compete in this area.

> There are several providers that offer a unmetered 1gbps line PLUS a dedicated server for ~600-750/mo. Providers like OVH offer the bandwidth for as little as 100/month. ( https://www.ovh.com/us/dedicated-servers/bandwidth-upgrade.x.... ) I am just not sure how amazon can justify a 160x price increase over OVH or a 30x increase over dedicated server + transfer.

OVH and other large volume providers are probably relying on the fact that many customers won't use 100% of their purchased capacity, but even taking that into consideration, AWS/GCE/Azure bandwidth pricing is insane.

It is indeed curiously unspoken among the major providers. (Softlayer too raised all their bw rates when IBM bought them.)

Part of building a moat? On one side we have a big stack of custom APIs. On the other -- once you have a few hundred terabytes of data in our systems -- it will be very costly to emigrate.

Exactly!

I knew as soon as I saw the headline that the would be reducing storage costs (who cares????) and not reducing the bandwidth costs which have been stuck for years, which are price-fixed with the other providers, and which account for the vast majority of our S3 bill.

Are bandwidth in costs free? Are you referring to bandwidth out costs? I don't work for them but I imagine they model bandwidth out as disk reads for accessing and reading your data. Think about how much it would cost to read at the rate of a 1 gigabit line from disk continuously for a month. Put a CDN in front of S3 if you have high bandwidth costs and low diversity of files.

CloudFront costs are super expensive outside of the US and Europe. We are a small company and just in south America we are paying 1000USD per month for 4 TB of traffic (0.25$ per GB). Based on traffic alone we are loosing money with some customers.

For many hosts, bandwidth is what they are really selling. You may rent a server from them and they may make some money from that over the capital costs, but the amount they are charging for bandwidth in comparison to what they pay for it is where the real money is made.

Also from OVH, have a look at OVH Object Storage, which is pretty much S3 but for 1/10 of the price [1].

There is also a GitHub comparison for all object storage providers (this price reduction isn't included yet as I see) [2].

[1] https://www.ovh.com/us/cloud/storage/object-storage.xml [2] http://gaul.org/object-store-comparison/

I use s3 as a webserver for static pages. Total bandwidth costs: 17 cents a month. 14 or so cents now. And I have some tens of thousands of visits per month. The only cognitive thought I've had to put in since setup was questioning one aspect of a bill (it amounted to pennies, but I couldn't figure it out, and maybe they were overbilling bigger entities). And they gave me a month free (including DNS costs).

I wish the Route53 would go down... Hard to beat, even if I were running job-replacing amounts of income.

I think it is possible to draw some conclusions from the volume discounts AWS is giving on data transfer. From 10TB/month to 150TB/month the price drops $0.09 => $0.05. I don't see how this kind of change in individual customer's volumes would make any difference on how much Amazon pays for the traffic. So I assume it is quite profitable for them to be selling at $0.05 (and lower) and the higher prices are just because smaller customers accept them.

The bandwidth seems to be used for trapping users in. You don't get charged when you're uploading data to AWS but downloading it back is quite expensive.

I wholeheartedly agree. The bandwidth cost reduction is long overdue.

> The previous model, if you made requests too rapidly, you might be hit with thousands of dollars of bills for relatively small data retrievals -- very easy to make a very expensive bug.

Glacier has supported for something like 18 months now, the ability to put a policy on your vault that capped your maximum retrieval cost. Whenever your request would cause you to exceed that limit, it would get throttle response that the SDK handles happily. I've used it when I needed to retrieve a whole bunch of data and wanted to do it faster than the free tier supported. I set it at $5 and just left the retrieval running.

A t2.large costs 10 cents/hour, a t2.medium RDS instance costs 7 cents/hour. If you put in 50 hours/month on this side project, that's $8.50 for compute plus maybe $3 for ~ 30GB of storage.

$11.50/month doesn't sound too hard to budget for.

I run dozens of small side projects off a $10/mo DigitalOcean server. I do not mind paying that absurdly low amount.

Agreed.

Glacier was, from my point of view, literally unusable. Now, it's usable. I still may not use it but at least I can.

RDS does have a free tier: https://aws.amazon.com/rds/free/

Dynamo might fill this need for you; you can provision a 1/1 table for less than a dollar per month and just put everything in it. Dynamo will bank some provisioned throughput for you to allow bursting, and the client apis all implement exponential backoff in the event of a throttle, so a 1/1 table isn't nearly as scary as you might initially think.

Of course, then you're using Dynamo.

Same about prototypes! I wonder how many other people have created really great products for their company doing the same thing. It would be in any company's best interest to provide free tiers for experimentation.

Can you share your needs with me so I can pass them along to the S3 team?

This is really something that needs to be setup/supported at the CloudFront layer (as CloudFlare and other CDNs already do) instead of S3. The danger is that somebody sets an HSTS or other security-related header on their bucket and breaks access for all other customers inadvertently for their customers that fetch from the S3 domain.

> While I'm not going to complain about a price reduction, I'd honestly be more excited if S3 implemented support for additional headers and redirect rules. Right now, anyone hosting a single page app (e.g. Angular/React) behind S3 and Cloudfront is going to get an F on securityheaders.io.

You can setup "Origin Custom Headers" in CloudFront ;)

Take a look at Netlify (disclaimer, I'm a co-founder), you'll get redirects, built-in prerendering, automated deployments, custom headers, etc. All with full cache-ability on our CDN.

https://www.netlify.com

Hmm, as someone who is about to move from using a webserver to hosting our single page app on S3 you may have just convinced me not to.

What exactly are the benefits over simply setting up nginx if not simplicity? Yeah, it's great to just serve the asset from s3 but the complexity of what you just described negates it almost entirely.

If i'm not wrong, cloudfront actually follows 302 redirects internally instead of passing the redirect on to the client.

but since ?_escaped_fragment_= is a suffix, not a prefix, i don't think redirection rules help.

I use Arq (https://www.arqbackup.com/) and it works very well. I've only tested retrieving small amounts of data from it so I can't comment much on a large bill. I only wish it worked on Linux. I've been thinking about seeing if it would work with Wine.

I have about 150GB in /home backed up daily using Duplicity, which encrypts & compresses everything and saves to a second internal backup drive. Data is kept for 6 months minimum. After several years, the total backup size is 190GB which syncs daily to an S3 bucket and my monthly bills are about $11USD. If I ever had to restore all that from S3 it would cost extra but would not be prohibitively expensive.

Install the AWS CLI (https://aws.amazon.com/cli/) and choose whatever method you like for making an encrypted local backup. Then sync that backup partition to S3 every day.

Here's an example command you can adapt for cron to call via a shell script:

    /home/tom/.local/bin/aws s3 sync /mnt/backups/daily s3://your-s3-bucket-name --storage-class STANDARD_IA --acl private --sse

The -sse means server side encryption which is redundant since I've encrypted the data prior to uploading, but why not?

I use Tarsnap (https://www.tarsnap.com/), which uses S3. It's true, though, that I only backup a small enough subset of my files, not anywhere close to a complete image. OTOH, it's very easy to use (CLI only), and as secure as it gets.

Using S4cmd on my Debian box I backup copies of my entire Lightroom folder structure contents. All my personal photos and videos get uploaded to a bucket on S3, then I convert the entire bucket to Glacier. Now with the new pricing of $0.004/GB and easy retrieval, it's a very nice setup.

To backup nearly a terabyte of photos costs about $4/month in storage costs. Uploading costs a bit extra due to the pricing for requests.

https://github.com/bloomreach/s4cmd

I was a masochist and hand-rolled incremental tar snapshots to Glacier using a cronjob. It works great for keeping price down, though I learnt the "lots of retrievals" lesson the hard way once I actually tried to retrieve my data. I now compromise by doing a full snapshot monthly and incremental daily, so I'm upper bound to 30 retrievals.

I'd highly recommend not repeating my mistake - use a real backup service for your actual data. Though rolling your own can be fun and interesting, it's probably a bad idea to bet your data on it.

I use my Synology NAS pointed at Google Nearline (through the included S3 support!). GCS also supports customer supplied encryption keys, but I store the bytes as encrypted on the NAS box itself.

As others have said, if you're just trying to back up your Mac, take a look at Arq.

Disclosure: I work on Google Cloud (and get $30/month of credit, so my first 3 TB are free).

Check out rclone [1]. I haven't used it yet but it's free and supports lots of backends.

[1] http://rclone.org/

I use duplicity to backup my VPS to S3, and Fast Glacier to archive stuff to Glacier from my Windows machine (things like photographs, important documents etc.)

Note: Glacier is not a backup service, it's an archive service. It's for long term backups, vs the relatively short term of standard backups (Full, delta, delta.. cycles). With Glacier if you delete an archive before 90 days, you'll still be charged for the full 90 days of storage.

Arq on OSX is pretty great.

Check out Duplicity or Duplicati. Also Arq for OSX I believe is very nice and has a great GUI but isn't free.

I work on quality control medical data (MRI images) and have huge data sets from machines going back over a decade. Most of the useful stuff is extracted metrics (stored in a db), but every now and then we need to pull up a data set and run updated analysis algorithms. We'll usually keep the latest couple of years in S3, and the rest in Glacier.

The data trove is fairly unique, and valuable in being the only of its kind, but we don't need anywhere near instant access to most of it.

It may be for backing up infrequently accessed data (compliance logs, etc) for example.

Hypothetical: you create a logging service for users to send all their log data to you. You promise 365 days of archives, but 30 days of data accessible at any time. You create a lifecycle rule on your S3 bucket to automatically archive data to Glacier 30 days after creation. On the 31st day, your user decides they want to look at an old log. They click the big Download button. You display a message saying they'll get an email from you when that data is ready to download.

Why not? If you have ever worked with any asynchronous API, you have already been introduced to the "come back later" model. Does it really matter how much later it is?

Glacier is not for data you want readily available. It's for when you care more about storage than access.

Yes, they are high for all 3 major cloud providers. I guess it's to keep you from using the cheapest part of each provider. That way you have to use their whole ecosystem and they can offer some products below costs to lure people in.

In theory, tape [1], optical [2], or spun-down disk [3] are cheaper but slower than spinning disk. Erasure coding [4] is also cheaper but slower than replication. One could even imagine putting cold data on the outer tracks of hard disks and warm on the inner tracks. In practice I suspect Glacier is market segmentation.

[1] http://highscalability.com/blog/2014/2/3/how-google-backs-up...

[2] http://www.everspan.com/home

[3] https://www.microsoft.com/en-us/research/publication/pelican...

[4] https://code.facebook.com/posts/1433093613662262/-under-the-...

Pure speculation here, so I'm probably completely wrong, but I imagine it's so that they can essentially reduce the amount of seeking. Glacier uses magnetic tape storage, I think, and I believe each tape has to physically be inserted into a machine to be read, and then be removed afterward. So there would be some downtime as tapes are swapped in/out. Therefore, it would make sense to aggregate reads ahead of time and maybe even physically reorder whole tape accesses to reduce the time it takes to load them.

But this wouldn't explain why the read rate factors into cost. Maybe they scatter data across tapes as well, and higher bandwidth requires loading more tapes concurrently?

Again, total conjecture. Please let me know which parts are wrong and which are right.

Storing the data in tape systems or some other system which is automated but not available in a matter of milliseconds.

Tape is cheaper than hard drives are cheaper than flash drives are cheaper than ram.

If you don't want to mess with all of that, you could use the standard "aws s3" command to upload your files to your s3 bucket like normal, and just apply an archive policy to your bucket or archive/ prefix or whatnot and it will automatically transfer your files to glacier for you.

Perhaps you can script in python using boto3.

I love the folks at backblaze but the single datacenter thing really worries me (and again, disclosure, I work on Google Cloud). If you're just using it as another backup, maybe that's less of a concern: your house would have to burn down at the same time that they have a catastrophic failure. But it is part of the reason you see S3 and GCS costing more (plus the massive amount of I/O you can do to either S3 or GCS; I'd be curious what happens when there's a run on the Backblaze DC).

Again, huge disclosure: I work on Google Cloud.

I hit that scenario, when we create tons of small files (e.g. < 10KB ones). In that use case it is often cheaper and easier just to use database such as DynamoDB.

See my other comment, it got link to article about S3 costs optimizations which got more detailed recommendation.

What app/site are you using to upload to S3? I use a combination of CloudBerry Backup and Arq Backup on my Macs/PCs here and the requests aren't that high (on average about 30Gb of data per machine in around 300K files).

I am guessing it comes down to the algorithm used to compare and upload/download files. I believe the two solutions above use a separate 'index' file on S3 to track file compares.

The new glacier pricing is $0.004/GB/month, those prices above look to be the new S3 pricing. Also you get 99.999999999% durability.

S3 data is backed up in three geographically separated data-centers, so you need to budget for 3 copies.

It's a running service (rather than cold storage) which means you need servers, power network ports, etc. Again, times 3.

Finally, include the software development to build the server stack and the staff for 24x7 ops and security.

It's possible to beat their pricing but most people who actually do manage it by cutting in an area which they don't need.

That's comparing apples and oranges - your hard drive doesn't live in a secure, fireproof data center in the cloud you can access from anywhere.

I do miss my little 3 TB seagate that I rsync'ed from home to my office. That said, what if someone broke into our little office (encrypted backup of course)?

The reality though is that no business would be comfortable with the plan being "our data is replicated offsite at Steve's house". And, other than maybe a pair of NAS boxes (one at each end) the cheapness of the solution assumes you have a great network connection between the two, a machine to plug it into, and only need a single hard drive. That is, how would you do offsite, active backup of say 50 TB?

Disclosure: I work on Google Cloud (and do offsite backup to GCS Nearline, that I'll move to Coldline when I get a minute to play with our new per-object lifecycle rules).

Where are you going to keep it? On site? Not so good for disaster recovery.

It's kind of funny when Feral hosting (I know I've been mentioning them a lot but trust me I'm in no way affiliated with them, I'm just satisfied customer, though they have trouble now with OVH) offers 2TB + 10Gbps unlimited bandwidth connection (it is unlimited, I abused it as much as I could with no warnings or anything) for 120 british pounds per year. That's good enough for me for mass storage of non-important data like Movies, Music, TV series, etc., that I convert then serve to users.

There is value to having off-premises replicated storage on something more durable than home-user targeted drives.

Google cloud nearline costs $0.12 per gigabyte-year with prices that will continue to fall. For a typical 500g hard drive that saw perhaps 700g of unique data, that's $84/year to have an outside-the-house replicated backup using something like Arq.

They are not the same thing at all. Glacier is the backup of the backup. It's where you go if your house burns down and the offsite backup at a relatives house is destroyed as well.

If you want to compare them, you have to buy space on a different continent, and store your backup there.

> A 3TB drive costs about 110eur, so if you'd have to buy a new one every year (you don't) that'd cost 110/3/1000/12=0.31 cents per gigabyte per month. Glacier? 7 times more expensive at 2.3ct.

Your pricing assumes that the drive is never powered.

I had such a setup when the Joplin 2011 tornado hit: http://www.ancell-ent.com/1715_Rex_Ave_127B_Joplin/images/ and I got off lightly. But the separate room my BackupPC hard drives were in was breached (see 302-2nd-bathroom-with-hole-of-unknown-origin) and those drives become fit only for Ontrack's $$$ recovery service, maybe, and one of my computers with e.g. my email was seriously damaged. The data on it was easily recovered from rsync.net's off site Denver location, who's service I love and will continue to use for my most important and "hot" data.

LTO (-4) tape had gotten capacious and cheap enough that I went back to tape (I'd outgrown DAT); if I didn't have a big sunk cost in a well working tape system and pool of tapes, which are very easy to put in e.g. a safe deposit box (they're a bit fragile, but nothing like a hard drive), I'd already be using one of S3, Glacier, or Backblaze, maybe even GCS since suddenly and irretrievably losing access to my backup data because a bot decided I was evil would not likely coincide with a total data loss at home (Google simply cannot be trusted if you're small fry like myself, as HN has been discussing as of late).

As Glacier has gotten sane enough to use without twisting your mind into a pretzel, with the new price reduction for slow retrieval I can seriously think about adding it to the mix and switching to it when my LTO-4 tape drive dies someday (e.g. ~3TiB for ~$12/month per my quick calculation just now), instead of buying another tape drive.

The biggest difference is that Glacier is still a "suspend/resume" type of accesss. However, if you just want to compare pricin, it'll depend on your access pattern and object sizes.

Retrieval in all Google Cloud Storage is instant and for Coldline is $.05/GB (and Nearline $.01/GB). If you value that instant access, it seems the closest you'd get with the updates to Glacier is via the Expedited retrieval ($.03/GB and $.01/"request" which is per "Archive" in Glacier). Then you have to decide how much throughput you want to guarantee at $100/month for each 150MB/s. (It's naturally unclear since it was just announced what kind of best-effort throughput we're talking about without the provisioned capacity).

If you're never going to touch the bytes, and each Archive is big enough to make the 40 KB of metadata negligible then the new $.004/GB/month is a nice win over Coldline's $.007. Somewhere in between and one of the bulk/batch retrieval methods might be a better fit for you.

But again, it's still a bit of a challenge to go in and out of Glacier while Coldline (and Nearline and Standard) storage in GCS is a single, uniform API. That's worth a lot to me, and our customers. But if Glacier were a good fit for a problem you have, and you're talking about enough money to make the pain worth it, you should seriously consider it.

Disclosure: I work on Google Cloud, so naturally I'd want you to use GCS ;).

Google has a "Glacier vs Nearline" calculator, which doesn't appear to have been updated yet:

https://cloud.google.com/pricing/tco/storage-nearline

(work at Google)

Hey, did you mean Reduced Redundancy (not Infrequent Access)?

I just noticed that the new pricing for Standard (2.3¢) is now less than the pricing for Reduced Redundancy (2.4¢)! So there appears to be no reason to use Reduced Redundancy anymore.

I wrote the post after spending all of about 15 minutes learning about this new feature. We did make things simpler.

> AWS if you read this: Please make your offers and your pricing simpler, NEVER more complicated.

I am reading this. We made them simpler. Decide how quickly you need your data, express your need in the request, and that's that. If you read the post you will see that these options encompass three distinct customer use cases.

I believe this HUGELY simplifies glacier pricing.

I don't think you understand just how insanely, incredibly, bizarrely complicated the old glacier pricing was.

I appreciate your enthusiasm for our products, but this is a huge step forward for S3 customers. I think it's fair to say you find Glacier too complicated (it's part of the reason Coldline doesn't look like that), but to say this new setup is worse just isn't true.

I can assure you that if AWS had announced a simple flat-rate structure that was more expensive for everyone there would be plenty of unhappy existing customers. It's a tough call to balance simplicity and efficiency, by making this complex for you, they allow you to opt-in to the "screw it, give me my bytes back as fast as you can" model. You could just pretend that's the only option ;).

Again, I'm not criticizing your unhappiness with the complexity of Glacier. But I think it's only fair to recognize that the folks at AWS have just released a major improvement that provides real value for some customers.

Disclosure: I work at Google Cloud.

This is a good change. The stupid former "rate-based" system meant that if you stored a large file (say, 10GB), you'd have had to pay quite a bit of money if you tried to retrieve just that file. After all, you can't control the rate at which you retrieve one file.

Simple is easy to shop around. Their goal is not to make pricing simple, don't hold your breath.