Amazon Aurora Limitless Database | Hacker News Reader

Amazon Aurora Limitless Database

aws.amazon.com

69 pointssamaysharma68 comments

Sharing here a wonderful MIT lecture on the architecture of Aurora I recently watched https://www.youtube.com/watch?v=jJSh54J1s5o

I can't wait until people find its limits!

Reminds of unbreakable linux.

I can't wait until people find its limits... in production!

Production is the best testing environment after all.

I don't always test my code. But when I do, I test it in production. -The Most Interesting Man in the World

For 90%+ of our customers (small-to-mid sized US financial institutions), production is the only environment available to work with.

For the other 10%, we take them aside and politely explain that they almost certainly have an unusable staging environment per the scope of our B2B project.

Testing in production is a wonderful path if you are comfortable talking to business people and making lots of compromises.

Everyone has a testing environment. Some people even have separate production environments.

Did you know that Postgres has a max table size of 32TB? Its really really fun to find that out the Wednesday evening before Thanksgiving.

Make sure to prune old data from your tables. This one got to this limit because it eventually got too large that queries to delete old data would time out... so it just kept growing.

https://www.postgresql.org/docs/current/limits.html

I just put a maximum Postgres table size I’m willing to manage in my employment contract. By the time we’re measuring in Terabytes I’m out of there.

For an interesting presentation on Postgres at different orders of magnitude, all the way up to petabytes, see https://thebuild.com/presentations/2019-fosdem-broken.pdf

Wow. That is still pretty large for one table.

it is partitioned underneath with limit of 32TB for single partition. (hello ctid). there could be many of them.

I'm going to guess that Amazon.com will be one of the first to find all of the limits.

This sounds very exciting, but I would have greatly appreciated some actual technical details. From this statement

> Sharded tables – These tables are distributed across multiple shards. Data is split among the shards based on the values of designated columns in the table, called shard keys.

It sounds like this is very much managed CitusDB on top of Aurora, but without any details about the implementation its impossible to know if its just repackaged Citus, or some new novel technology built specifically for Aurora.

Likely it'll be talked about in-depth this week at Reinvent. This intro article is par for AWS "product announcements".

Aurora Limitless Database is based on our own investments in database-optimized virtualization (Caspian), in scale-out log-first database storage (Grover)[2], and in a custom approach to cross-shard transactions that makes use of the high-quality hardware clocks available in EC2[1]. We'll be talking more about the internals over the coming months.

[1] https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-ti... [2] https://assets.amazon.science/dc/2b/4ef2b89649f9a393d37d3e04...

> makes use of the high-quality hardware clocks available in EC2

So are you using vector clocks in the backend to handle transaction ordering and maintain consensus?

vector (or matrix) clocks are awfully slow in real life. have shipped product with this, will not recommend for future.

I appreciate the context, but this just adds so many more questions. How are you integrating hardware clocks into the Postgres transaction model? Is this just postgres wire compatible like Cockroach or Yugabyte, or have you modified Postgres MVCC implementation to not use the standard TID. Does this support all of postgres at the Transaction coordinator ?

Super exciting announcement, and I am really looking forwards to learning more!

They’re probably using it similar to how Spanner uses TrueTime: https://cloud.google.com/spanner/docs/true-time-external-con...

but Spanner isn't Postgres compatible, they have their own transaction processing layer that is built for TrueTime. If this is using the Postgres frontend how are they incorporating TrueTime into the normal Postgres MVCC model that uses the monotonic xid.

edit: More details make it seem like this _isn't_ Postgres on the frontend, its just mostly postgres compatible, which would explain how they got away from the xid based MVCC. So it seems like this is an entirely new distributed database, rather then a modification to existing postgres.

I think they are internally using postgres only. aurora uses a custom storage process on host which internally supports the custom storage engine (s3)

given the mentions of "log as a database", I believe that depending on the transaction the storage responds differently. like how mysql mvcc uses undolog to essentially rollback database internally so that transaction sees data consistency. they could be doing something similar i.e regardless of whatever postgres uses, if they can get a reference of transaction and its start time then they can use the custom storage engine to rollback the log and respond in that way

remember it is aws, "leverage" is built into their DNA, I don't believe that they would go on the route of building a totally new SQL database like google did

https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-ti...

Here's the deep dive talk from re:Invent 2023

https://youtu.be/a9FfjuVJ9d8

I'm glad someone from AWS is here! Some general info on the sharding approach would be interesting, e.g. whether secondary indices on sharded tables are scatter-gather, or separately sharded on the index columns, or you configure it per index, or what.

From what I hear from folks using things like Vitess, if you're used to a monolithic SQL database there's often some things to learn to mentally model your query cost well after you move to a sharded world, and understanding more up front can save heartburn later. Writing up those details well is a good thing that AWS could do.

This does look exactly like CitusDB even in the naming of some things (reference / distributed tables) and how they describe it being added to an existing cluster as extra nodes with an extra manager node that will redirect requests!

Things I want in a serverless database:

1. Scales to zero, no cost when not using

2. Allows for SQL over API like aurora v1. This point is important, it allows for faster access from other serverless technologies, i.e. lambda

I want to better understand the reasoning behind the "scale to zero" point that many people mention.

only case I could think of is that businesses that want "scale to zero" have very low total expenditure on the database. is this the case for you as well?

because serverless without going to zero still solves a major problem for a lot of companies with some decent scale, where there is some decent traffic and its too hard to implement database autoscaling and most businesses have unequal traffic (daytime and nighttime etc,...)

Yeah, can’t really see any overlap in the camps of folk who want “scale to zero” and those who want the “automatically horizontally sharded ginormous transaction scale” that this post is about.

Why not have both? That's exactly what DynamoDB offers, and people want the SQL equivalent promise of "it'll keep up no matter what scale you're at".

The point is: You don't want to need to switch technology at a later stage, you'd like to build with the right technology from the get-go.

I almost always go for DynamoDB first, since for small projects it's essentially free, and for huge projects I do not have to worry about the ops overhead that follows normal SQL databases.

A lot of people also have traffic patterns that go:

- Very high volume during the day - Almost no or little volume during the night when your users are sleeping

Aurora Serverless v2 gets us closer to this for SQL, but does not scale to 0 so it's not too nice for the initial phases of projects.

This is some seriously impressive engineering. Spanner is the only DB that can achieve this.

What are you referring to? I didn’t see anything described that’s exclusive to Spanner. CockroachDB is a Postgres-compatible-ish distributed DB that does multi shard transactions. YugabyteDB is another that comes to mind. CitusDB is a Postgres extension that manages multi shard tables. Or are you referring to automatic scale-out?

The scale of Spanner is unparalleled. They had to use atomic clocks to manage time in a distributed set up.

Aurora uses Amazon's equivalent of TrueTime using a more pragmatic high-quality GPS time solution

Not sure i would say Spanner is the only database that can do this... first of all, this new thing from AWS can't do what spanner can do and second, there are few that can deliver distributed transactions in addition to Spanner, such as CockroachDB (ex-emplyer), Yagbyte and TiDB.

> Join the preview of Amazon Aurora Limitless Database

would be the non-editorialized submission, and would have advised potential readers that one cannot currently play with it without a ton of hoopjumpery

one will also want to watch out for this, buried 7 paragraphs in:

> The preview runs in a new Aurora PostgreSQL cluster with version 15 in the AWS US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland) Regions.

So basically 70% of AWS users can use it if they get accepted to the preview? Those regions seem chosen because they’re the biggest ones.

Keep in mind that the California region has been devalued for a while. AWS would turn it off if they could.

https://www.lastweekinaws.com/blog/us-west-1-the-flagship-aw...

> But AWS isn’t gouging customers, otherwise you’d expect to see some things that I don’t find evident here; for example, data egress charges are the same price in us-west-1 as in other U.S. regions.

As if data egress charges aren't gouging customers everywhere!

Thank you for the word "hoopjumpery".

Ah yes, Reinvent season, when all of the AWS announcements make the front page of HN but most won't be generally available for 6-12 months.

It’s just the opposite of what happens with Google. They announce a product that is available today. But won’t be available in 6-12 months.

These architectures are super interesting to me.

We are using the ~equivalent SQL Server architecture in Azure right now (Hyperscale tier) and really enjoy the experience. Not having to worry about specific machines and being able to treat it the same way as an express instance (from an app perspective) are extremely compelling selling points.

The only downside I saw in the Microsoft offering was the fact that there was a throttle for transaction log writes (100 megabytes/s) which is required to maintain resilience across the cluster. You don't want your replicas to wind up 3 weeks behind, etc. My hope is this limit will only increase with time, but even so it is quite workable (for us) today.

I may spin up an instance of Aurora (when available) to take a look at how it behaves around our worst-case serialized txn paths. Our use of SQL Server is very "light" in terms of its weird claws/hooks, so we could pretty easily shift to another RDBMS vendor in the future, assuming we can sell our clients on the 3rd party mix.

I used aurora for a while and liked it quite a bit (especially compared to Alloydb which was absolutely horrible).

I'm curious to hear more about your troubles with alloydb. It's something I just started to test out at work

https://news.ycombinator.com/item?id=34304376

Maybe they improved since. Let me know how you find it:)

That is quite the ambitious name, obviously it can't really be "limitless" even if that limit is just whatever spare resources AWS has to run your database.

But they must be confident that that whatever the real limit is, is just not realistic for any use case.

I believe it means “the bill will have you on the phone with your TAM discussing alternatives before you exceed what we can host”.

AWS team needs to automate migrations from RDS to Aurora

Literally just today I was looking into a new project that requires postgres, and read on the aurora doc that this exists. Didn't look further into it considering the project is new and no migration is involved, but if you need this, worth searching.

I did it it was the craziest process but it worked eventually even though that was quite a while ago.

Isn't it more or less automated? Create an Aurora replica cluster from your normal RDS database, switch your application over and then delete the old RDS instance.

what if your postgres instance has replicas? They support blue/green for RDS postgres, but not for RDS/Aurora. It's slated for later next year is what I heard.

This is a great blog post, I feel that this level of database knowledge doesn't get enough coverage.

Maybe it's because most projects will never need it, but when you eventually do it feels like you're just left to yourself

What type of consistency guarantees are involved?

Are we giving away something to get multiple writers to same table? Or does the routing to the correct shard somehow solve for that?

It mentions it can scale write transactions to millions per second, but I see no mention of how it can scale read capacity?

"With limitless databases come limitless costs" (c)

It's web scale!

Is this available for Aurora Serverless V2 as well?

limitless is hyperbole

> a new capability supporting automated horizontal scaling to process millions of write transactions per second and manage petabytes of data in a single Aurora database.

Limitless horizontal scaling is really cool and all, but does anyone not running a Fortune-500 Tech company actually need this kind of firepower?

The people running a fortune 500 company are the majority of AWS income? I’d say it’s very relevant to them.

Most AWS products outside of the core group (EC2, S3, RDS, Lambda, etc) are for the enterprise or venture-backed startups.

At Notion we run ~100 Postgres instances and we’re pretty far off from being Fortune 500, although we have many Fortune 500 clients. We could have used this Aurora thing instead of doing sharding in the application ourselves: https://www.notion.so/blog/sharding-postgres-at-notion

Separation of compute and storage, and storage having tables being sharded is now in many modern databases (cockroachdb, clickhouse and singlestore come to mind).

This being part of native aws fully postgres compatible is exciting though.

I wish AWS brings columnstore or custom storage engine features.

But I can see why AWS wouldn’t want to bring columnstore tables - it would directly compete with redshift.

AWS does a lot of enterprise business but I’d also think about it as setting the upper bounds: you probably don’t need millions of transactions per second but if you’re a startup needing, say, thousands it’s nice knowing that you don’t need to fundamentally rearchitect for a while even if you become popular. The pain of hitting the limits of a single database write node can be high because you need to make substantial changes to what is by definition a busy system, but you have an incentive not to make the big changes until you need them since they make everything else more work. This gives you the option of effectively never needing to do that, which is going to be popular with anyone anticipating lots of growth if it delivers as advertised.

Reply on news.ycombinator.com