Aurora I/O optimized config saved 90% DB cost
graphite.dev
graphite.dev
No surprises here. Come what may, Amazon has always strived to lower costs (the low costs, more customers, more volume fly-wheel?). This is but one example.
AWS adopted cost-follow pricing for S3 (very different from value-based pricing) after apparently a lengthy debate: As they get more efficient, they want to pass down those savings to customers (as price reductions):
S3 would be a tiered monthly subscription service based on average storage use, with a free tier. Customers would choose a monthly subscription rate based on how much data they typically needed to store. Simple ... The engineering team was ready to move on to the next question.
Except that day we never got to the next question. We kept discussing this question. We really did not know how developers would use S3 when it launched. Would they store mostly large objects with low retrieval rates? Small objects with high retrieval rates? How often would updates happen versus reads? ... All those factors were unknown yet could meaningfully impact our costs ... was there a way to structure our pricing [to] ensure that it would be affordable to our customers and to Amazon?
... the discussion moved away from a tiered subscription pricing strategy and toward a cost-following strategy. "Cost following" means that your pricing model is driven primarily by your costs, which are then passed on to your customer. This is what construction companies use, because building your customer's gazebo out of redwood will cost you a lot more than building it out of pine.
If we were to use a cost-following strategy, we'd be sacrificing the simplicity of subscription pricing, but both our customers and Amazon would benefit. With cost following, whatever the developer did with S3, they would use it in a way that would meet their requirements, and they would strive to minimise their cost and, therefore, our cost too. There would be no gaming of the system, and we wouldn't have to estimate how the mythical average customer would use S3 to set our prices.
From: https://archive.is/lT5zTI wonder what explains AWS' high egress costs, though.
Vendor lock-in. It prevents people from otherwise picking the best provider for the task at hand - for example using some managed AWS services, but keeping the bulk of your compute on-prem or at a (much cheaper) bare-metal host.
It makes sense, but I wish there was an option to opt-out, as to allow high-bandwidth applications that are fully on AWS (at the moment AWS is a non-starter for many of those even if you have no intention of using AWS competitors like in the scenario above).
Maybe they should just price end-user egress vs competitor egress differently (datacenter and business provider IPs are priced like now, but consumer-grade provider IPs are much cheaper/free)? That would discourage provider-hopping, while making AWS a viable provider even for high-bandwidth applications such as serving or proxying media.
If subsidizing the networking products would require a 10x or higher markup on egress, then that is not the correct way to organize prices.
Egress costs are where AWS makes money. You can get around that by negotiating lower cloudfront costs then basically exfiltrating your data via cloudfront. We were getting < .01/gb pricing (.0071?) for only like a $2k/mo commitment.
Maybe it's just a goal to reduce costs, but it seems likely that this is a response to Google's introduction of AlloyDB, a Postgres-compatible database competing with Aurora that is advertised as having "no...opaque I/O charges". I doubt Amazon was feeling generous.
Agreed.
Azure on the other hand is notoriously expensive and do everything possible to raise prices.
And then tie in how some AWS services only a binary choice between logging ridiculous volumes of data or nothing at all (looking at you, AppSync) to make the log ingestion fees high too.
I have a hard time reading "amazon" and "lower costs" in the same sentence.
The cloud is consistently an order of magnitude more expensive than self hosting, even when including people cost, unless you need every single bit of load balancing/high availability/normalization they provide.
The vast majority of projects don't.
We have a similar story with DynamoDB too.
Sounds like good material for a technical blog post.
#AWSHistory
Even your history thing makes no sense, Aurora Postgres was launched 9 months after Mysql version in July 2015.
What? That's not what that comment says at all. They're saying that aurora mysql was a plausible interpretation of what OP moved from, before OP clarified.
https://aws.amazon.com/blogs/aws/highly-scalable-mysql-compa...
Aurora Postgres hit general availability October 2017. (Sorry, betas and early release offerings don't count.)
https://aws.amazon.com/blogs/aws/now-available-amazon-aurora...
Quick math… carry the two… compute the partial differential equation…
Looks like 3 YEARS between the release of Aurora MySQL and Aurora Postgres.
…which was November 2015.
https://aws.amazon.com/blogs/aws/now-available-amazon-aurora...
The majority of the time, it was identified as "aurora" in CloudFormation templates. 5.6 hit EOL in February of THIS YEAR, aka 2023.
https://aws.amazon.com/blogs/database/upgrade-amazon-aurora-...
So… that would make it the majority of time, wouldn't it? Why does this upset you so?
FWIW, I'm actively exploring native partitions on Aurora with Postgres and I'm seeing very little benefit. Two identical tables, each with 500M+ rows, and I'm not seeing any meaningful performance/IO changes. A composite index with the partition key as the first entry has been as effective for reducing IO and query time as partition pruning. I'm sure there are workloads where partitioning makes more sense, but I've been surprised by how little difference there was in this case.
Agree. It feels like an actuary ran some numbers and found that high I/O customers spend more on other AWS services and are more profitable than low I/0 tenants, so they changed the formula to reduce churning those high I/0 tenants.
Mine have always been helpful and have kept us up to date on releases like this, sometimes we even get them before they are GA.
"AWS announces Amazon Aurora I/O-Optimized" - https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-au...
"New – Amazon Aurora I/O-Optimized Cluster Configuration with Up to 40% Cost Savings for I/O-Intensive Applications" - https://aws.amazon.com/blogs/aws/new-amazon-aurora-i-o-optim...
"Getting Started with Amazon Aurora I/O-Optimized" - https://youtu.be/OlFeaVd6Ll4
"AWS On Air ft. Introducing Amazon Aurora I/O Optimized" - https://youtu.be/FBZmxLm0Bhw
"AWS Aurora IO Optimized Storage — Is it worth it?" - https://towardsaws.com/aws-aurora-io-optimized-storage-is-it...
Outside of that though, you can go to your AWS billing, filter on RDS, and slice by usage type. They make you dig, which is why I pref Datadog
Disclaimer: I work for Vantage.
PS. I maintain https://ec2instances.info/rds and recently added support for Aurora I/O Optimized pricing.
(The article is about the PostgreSQL variant.)
Also, it sounds like they could save money by not using cloud.
This always comes down to opportunity cost and staffing. If their team doesn’t include Postgres or MySQL experts, or security experts, etc. outsourcing that to AWS allows them to spend their staff budget on things they cannot outsource.
One thing to remember is that your organization can only do so many things a year. That number varies based on complexity, internal overhead, customer and regulatory overhead, etc. but it’s finite and lower than you think. If you’re spending time on MySQL patching or trying to figure out how to patch your servers, that’s taking decisions away from your product. That can still be the right call but one thing I’ve noticed over the years is that many people are prone to dramatically underestimating that cost.