Which is why you should be relying heavily on a caching layer outside of AWS. You're not serving objects directly from S3, are you? A big no-no if you care about latency at all (S3 from time to time will take upwards of ~500ms to serve HTTP GET requests).

EDIT: Example: We use Akamai as our caching layer, as their bandwidth costs are much less than Cloudfront (their analytics are better as well, and they don't charge for invalidations like Cloudfront does). We use S3 as the origin as its a lower cost per GB than Akamai's NetStorage, and the S3 API is superior to work with compared to the interfaces exposed by NetStorage (scp/ftp/rsync).

In my opinion they're doing that on purpose to discourage people from using S3 as both a storage and a transfer method.

Storage and Delivery should be considered separate, as there are benefits to it being a separate service. If you throw a CDN between your customers and S3 you'll not only get a significantly lower cost, you'll also give your customers much better performance.

Lets put some math on the network costs. An oft quoted cost is $1/mb at p95 for decent transit. 1 mbs * 2,592,000 seconds = 2,592,000 mb per month, / (8 * 1024 gigabytes) = 316GB transferred per month. Dividing our $1 / 316GB = $0.0031 per GB transferred _at a constant 1mbs_.

However most networks have sine wave pattern diurnal utilization. A common peak to average ration would be 3:1. Our $0.0031 is effectively the rate for an average of 1mbs. To get that average rate we'll be paying about 3x, or $0.0093 per GB transferred. So we're paying $0.01 per GB for the transit alone. This is also in a major metro area in North America. Transit costs can easily be 3-10x in EU or 10-20x in AP.

On top of that there's capex and opex for that 10gb port. Assuming you've got some decent scale you're going to get 96 10gb ports in a $300,000 to $500,000 router, or $3-5,000 per 10gb port. You need ports both north and south, so you get 480gbs throughput per chassis. But you can't run above 80% util, so call it 384gbs of peak throughput. Opex for rack position, data techs, power, etc will cost $2,000 to $5,000 per month. Amortized over 36 months let's call it $10,000 to $17,000 per month all up.

That 384gbs peak throughput is actually more like 128gbs averaged over the month. At 128gb/s average we transfer about 41,472,000 GB per month, or $0.00024 to $0.00041 per GB. Now double that cost 'cause routers actually come in pairs for fault tolerance. So we're paying $0.0005 to $0.0008 per GB of transit connectivity.

And how many hops are there back to your application? I'd hazard around 6 by the time you go transit -> edge -> border -> core -> dc -> agg -> tor. Paying for that internal network infrastructure is going to add $0.0030 to $0.0050 to our bill. All up that gets us to $0.0128 to $0.0151 per GB transferred from our host to a third party network.

Expanding the error bars lets round to $0.01 to $0.02 of network costs per GB that we deliver. To get these prices you're going to have a capital budget of tens of millions per year. OpEx would be at least $500K, or more likely a million plus, per month. If you can actually beat my pricing I'm hiring and or would like to subscribe to your news letter.

Apologies if I missed something in the math. I had to do some mental acrobatics to go from dollars per gb/s to dollars per GB/month.

Not to disagree, but what other options? Google and Rackspace charge the same. DreamObjects charges $0.05/GB in transfer, but still charges $0.05 for storage. Azure is still the same $0.12/GB.

I don't disagree that $0.12/GB could be considered expensive, but S3's competitors seem to be charging the same.

On the other hand, S3 <-> EC2 transfer is free of bandwidth costs (there are still per-request costs though).

I hesitate to even post, as offsite backup is very different than the use case for S3, but we're the opposite...

That is, our storage is expensive (10c per GB, per month, with the HN-new-customer discount) but our usage/transfwer costs are zero.

related:

  ssh user@rsync.net s3cmd get s3://rsynctest/mscdex.exe

interoperability!

This was my take-away from the Google Cloud pricing changes yesterday. By reducing CPU costs, my bandwidth just becomes even more the dominant cost, so further reductions help me less and less.

AWS users will end up with the same problem in time. I wonder why the bandwidth costs are so subborn, does the cost price per traffic unit really never come down, or is someone (some people) in the infrastructure layer(s) screwing us?

What "other options"? You mean sketchy "free 9000 exabytes/month of bandwidth" low end hosting places like hetzner? Those are not alternatives to AWS or azure or google cloud. And those bandwidth costs do not reflect reality, they assume most people use almost none of that bandwidth. The bandwidth still costs the same, just the big users are getting the small users to pay for their bandwidth costs.