Amazon Aurora DSQL
aws.amazon.com
aws.amazon.com
No temporary tables, no foreign keys, no views, no more than 10k rows in a transaction.
Except for some basic wire compatibility with the postgres protocol, I'd hardly call this a "database", and more a key-value store.
https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
Hopefully that keeps the pricing reasonable. :)
But seriously, for a smaller CRUD app, this could be sufficient, even "magical," if the price is right. For my part though, the lack of multiple databases per cluster puts multi-tenant systems completely off the table. Now that you mention it, I almost wonder if this is a giant hack on top of Valkey/Redis...
I don't think that will matter. This feels much more like DynamoDB where you're charged for GB used/stored and no infra cost, so no reason to nest databases
That makes it very easy to manage access to sensitive data.
Having it be Postgres compatible is the big selling point.
Who else would use it? Not sure, gaming perhaps?
I do not want to denormalize my data model, i do not have a high performance usecase, I simply want infrastructure that scales with usage (down to 0) and a flexible normalized model I can build more on top of easily.
DDB is great and all until you're asked to add 3 filters and orderby feature and suddenly you're adding elastic search to your project
Not that it matters but it's D1 but I noticed people sometimes call it D2, maybe by analogy with R2.
This is the first release of something new and groundbreaking.
Identifies as PG 16.5
No views/triggers/sequences
No foreign key constraints
No extensions
No NOTIFY ("ERROR: Function pg_notify not supported")
No nested transactions
No json(b)
There is no 'standard' or 'spec' for what makes something Postgres wire compatible.
This feels like a strong overreach on the marketing front to leverage the love people have for Postgres to help boost what they've built. That is not to say there isn't hard and quality engineering in here, but slapping Postgres compatible on it feels lazy at best.
I actually think that they'd argue they intend to close the feature gap for full Postgres semantics over time. Indeed their marketing was a bit wishful, but on Bluesky, Marc Brooker (one of the developers on the project) said they reused the parser, planner, and optimizer from Postgres: https://bsky.app/profile/marcbrooker.bsky.social/post/3lcghj...
That means they actually have a very good shot at approaching reasonably full Postgres compatibility (at a SQL semantics level, not just at the wire protocol level) over time.
They'd save space by listing the supported features. It's not useless, but no existing application would run on it.
> Unsupported objects
> * Databases - Aurora DSQL supports only one database per cluster at this time.
> * Views
> * Temporary Tables
> * Triggers
> * Types
> * Tablespaces
> * UDFs / Functions other than functions using language = SQL
> Unsupported constraints
> Foreign keys
> Exclusion constraints
> Limitations
> ...
> A transaction cannot modify more than 10,000 rows
If you want a distributed PostgreSQL-lookalike, look at CockroachDB.
So it lacks Serializable isolation also, but I guess that's the secret to how it "offers the fastest distributed SQL reads and writes".
The quotas page is always the most interesting for any aws service.
10Mb transaction size and 5min transaction duration limits are interesting limitations that kill any "everything lives in the db" notions that one might have.
A fully transactional, scalable, distributed data store + queue + db + data stream with triggers is my endgame
Also, the 128mb limit on transaction temporary storage seems _very_ limiting for big ol' analytic queries
Wat?
As far as performance hits go, I guarantee I could find a half dozen other, larger problems that would have a bigger impact than disabling FK constraints.
It is of course possible to maintain referential integrity without them, but it requires much more diligence from devs, and I have yet to see it implemented well anywhere.
Until a single DB instance is serving well in excess of 100K QPS, I wouldn’t be concerned with the impact of FK constraints. Let RDBMS do what it’s good at – keeping your data safe.
- many data stores. even if you're not doing microservices persay, you're almost certainly going to end up with different domains/buis units/systems having their own data stores. FK constraints obviously dont work across system boundaries. They could be vendor systems too, not even just your own services.
- soft deletes. you rarely hard delete root objects, so you dont need the FK constraint to protect those dangling refs
- eventual consistentency/idempotency - you end up creating data in different systems as part of a workflow (eg signup), and you have to tolerate partial creation for resiliency.
- some sort of ORM or sql framework that makes it very difficult to create a leaf row with an invalid FK parent pointer.
So in a fundamentally distributed large scale tech ecosystem, FK constraints end up protecting against.... not much. Ive worked at three large scale (10m+ users) companies which all ended up deleting all their FK constraints. And... nothing happened, nothing ever went wrong due to their absence. Its theoretical protection, not practically needed IMHO.
Not to their full extent, but they can still be used. At the simplest level, it is of course entirely possible to give different services their own schema in a given database, and FK constraints are supported across schemata in both MySQL and Postgres. Vertical scaling can take you enormously far with properly architected schemata and queries.
A more flexible, but still easy to reason about way to accomplish this is to have local versions of certain tables in each DB. This can be manually implemented (though this is not an easy problem to solve, for a variety of reasons), or by using something like Citus [0], which accomplishes this using 2PC. This is of course slower, but if your data model is carefully designed, it can be managed.
> soft deletes.
Sure, but now you have a new problem - needing to add a `is_deleted` or `deleted_at` column to a bunch of tables, and indexing that column on every table. In Postgres you might get away with this by using `DEFAULT [FALSE, NULL]` (respectively), and then creating a partial index with `...WHERE <column> IS NOT [FALSE, NULL]`; that way the index size stays reasonable, and the cardinality isn't as horrible (well, it is for bools, but since you're only using it as a filter, it can work OK). Also, of course, you have to include this predicate in most queries.
> Its theoretical protection, not practically needed IMHO.
Different subjective experiences, of course, but IME it very much saves you time and headaches.
I also now don’t get why they are emphasizing PostgreSQL here when they should be emphasizing CQL compatibility. Being compatible with Postgres wire protocol but not being relational is strange.
https://brooker.co.za/blog/2024/12/03/aurora-dsql
Specifically, transaction latency is constant relative to transaction statement count, even across region, so I guess that's something
I don't trust them enough to use non-portable technology like this until they give me confidence they are committed to lowering prices.
If the price hasn't been going up in line with inflation then it's been going down in real terms. Not as nice as the number going down, too, but it's not nothing. Over time AWS prices have significantly declined in real terms simply by not having been raised in nominal value.
https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
Especially when you consider the recent feature additions to S3.
Serverless Aurora is stupid-expensive so I can't imagine this is going to be cheap but I keep hunting for that perfect DB that can scale to 0 (or very low) and be priced well.
Right now neon.tech is what I'm using and I'm very happy but this looks like it would be interesting IF the pricing is good.
we have multi-region disaster recovery and replicas coming in 2025. Switch over time will be greater than this active-active system but the overall latency for Neon should be much less on a consistent basis.
AWS' recent "serverless" products, Aurora Serverless v2 and MSK Serverless for example, had been very misleading with "serverless" title.
I hope it is not a lie this time.
However, the latency scaling from zero is still painful, "up to 15 seconds".
I'm itching to read more details into what this actually is under the marketing blab.
Seeing the No to configurable for some of the settings is the most telling hard limits that we can see up front.
Some Very Noteable ones are: Maximum size of all data modified within a write transaction: 10 MiB Maximum transaction time: 5 minutes
https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
It may be even better in one important way: (if I understand correctly) for Spanner, you have to provision servers -- and keep them running. So GCP version of Spanner really makes no sense for anything that isn't huge.
This sounds more like "Spanner as a Service" -- which is something I've always wanted (for my hobby projects, no less :)
Unless I misundertood, anyway.
UPDATE kv SET value = 'alpha' WHERE key = 'beta.charlie';
UPDATE s3 SET value = $b64$good luck$b64$ WHERE key = '/some/s3/path';
LISTEN whatever;The type of situation I'm thinking about is for example storing a blob in S3, storing metadata and a reference to the blob's path in a database row, sending a message into a queue to trigger some async processing, and updating a cache. It would be nice to be able to do this through a single API or service, and it would be really nice to do all this within some type of transaction abstraction that would allow all operations to pass or fail collectively, really really nice if the whole thing could be pay-as-you and scale horizontally-ish on shared infrastructure without managing nodes or slots or whatever.
I'm not a Postgres user so I don't know how far you can get currently and I should probably look into it in detail. Coordinating blob/ject storage, database, and pubsub operations is a pain point for me presently. I think that overall system design is going to prevent a database-type system from being a good idea for blob storage but I would still like to see someone try to put three systems in a trenchcoat and try to make it work behind one interface.
https://github.com/turbot/steampipe#steampipe-plugins and https://steampipe.io/docs/steampipe_postgres/overview may be relevant, although watch out for Steampipe's license
https://github.com/topics/foreign-data-wrapper and https://github.com/topics/fdw are some other examples
1: https://www.postgresql.org/docs/17/fdwhandler.html (although strictly speaking that page is for _authoring_ FDW, not a tl;dr of the concept)
> With its innovative active-active distributed architecture, Aurora DSQL is designed for 99.99% availability in single-Region configuration and 99.999% in multi-Region configuration, with an innovative active-active, distributed architecture, making it ideal for building highly-available applications
We get that its architecture is innovative and active-active
Maximum storage GB per cluster: 100GB Maximum size of all data modified within a write transaction 10 MiB Max: 10K rows per transaction
I imagine that number is tunable up to multi digit tb if you your monthly bill is worth Jassy's attention
https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHA...
The more interesting (i.e. constraining) limits are transaction size (both row count and data size) and transaction time, which aren't configurable.
>Aurora DSQL is PostgreSQL compatible, which means that it provides identical behavior for most supported features, identical query results for all SQL features, and supports many popular PostgreSQL drivers and tools with minor configuration changes. Supported SQL expressions return identical data in query results, including sort order, scale and precision for numeric operations, and equivalence for string operations. With a few documented exceptions, such as synchronous replication, no-lock concurrency control, and asynchronous DDL execution, Aurora DSQL behaves comparably to PostgreSQL.
Aurora DSQL supports core relational features like ACID transactions, secondary indexes, joins, insert, and updates. See Supported SQL expressions for an overview of supported SQL features.
Aurora DSQL doesn't support all PostgreSQL features. For more information, see Unsupported PostgreSQL features.
https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
No pricing that I can find
PG::FeatureNotSupported: ERROR: ddl and dml are not supported in the same transaction
https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
This appears wire-compatible with postgres but does things differently like locking.
Depends a lot on how you use postgres as to whether its drop-in compatible or not.
It feels like the compatibility claim is a bit abused (not only in this case) mostly because there’s no clear and strict reference (maybe with mandatory and optional requirements) for what “compatible” means
We are also building a hosted Postgres service based on Firecracker and with the ability to create unlimited databases. Will have to dig in and see how we compare to this. https://www.prisma.io/blog/announcing-prisma-postgres-early-...
It provides strong consistency for cross-region transactions
I will ignore their claimed almost unlimited scalability (which I somewhat believe as it's in their best interest to make it expensive fast), and "highest" (as compared to what?) availability. But my pet peeve is zero infrastructure management. This is blatantly false and it always was - when they first announced their cloud (at that time called "web") services, when they marketed their solutions as "serverless", and now. You need to have well-trained staff to maintain your AWS infrastructure otherwise you will come across many problems.
Managing a SQL Database at scale is more like four thousand calories daily, if I continue stretching this analogy. :)
[0] They disabled the trap a few weeks ago, finally: https://aws.amazon.com/about-aws/whats-new/2024/10/aws-lambd...
I would believe it can scale your cost easily, but I'd be surprised if it can scale well in terms of capability. How many million QPS is "virtually unlimited scale"? The 10k row transaction limit and no FKs (or even views???) also just make this thing sound completely non-serious.
- Yes, it's SQL on distributed transactional kv
- SQL layer is stateless, runs in multiple containers
- SQL layer would push down compute logic down to the storage(like, filter/predicate)
....Yes, that reminds me of TiDB
I think this is pretty big contrast to something like S3 where the product feels far more clearly and coherently managed, and where they have released lots of big improvements/changes over the years without disrupting or muddying the core product in major ways.
Not to worry -- you can use Rufus...uh "Q"....uhh...Nova...to summarize the docs into a decision tree on which confusing choice to use!
Even asking it something on the same page that it's advertised 'now supports asking about' completely failed for me - think it was ENIs or something; just said something completely generic, not at all about the instances I'd asked about which it advertised being able to do (and I happened to have a concurrent need for).
As a very naive person myself, isn’t “being very confusing” one of the main points of AWS? They know they are big and nobody will get fired for using AWS. So, the more confusing their products are, the more benefits for them (e.g., because you don’t know how much you will pay, because you are misusing their products, because perhaps you don’t actually need their products, etc)
So, any manager at AWS probably thinks this is all working very fine.
If you don't understand, for example, the difference between MySQL and PostgresQL then RDS/PostgresQL and RDS/mysql are the same. How does aurora mysql and aurora PostgresQL differ from standard RDS versions? Well, you have to actually read and understand...or watch some yoyo's YouTube video explainer.
AWS is complicated because it offers a lot. Do you want a satellite downlink? AWS has that. How about a private 5G cell network? Yep. How about a distributed data store with basically fixed read latency? Well, there are a bunch of different options depending on what you want.
Because you're obviously not used to being in the enterprise space, you think that confusion is what they want. They don't. In the space where people spend the big bucks being totally clear is what every good vendor strives for. You're in a multi-year relationship with these companies, and you want them to succeed. AWS isn't Oracle.
Obviously there are tons of shops that will sell people the wrong thing. But then another shop comes along and takes their business (which is one project I'm involved with now - they have like some ridiculous RDS bill that shouldn't be more than $500/mo).
Otherwise if you're just staring from scratch, I'd pick provisioned aurora if your load is steady and serverless v2 if not. Then IO optimized if the db doesn't fit into memory and non-IO-optimized until it does.
Just random example I was poking around AWS Backup recently, and there you have option to enable Backup for different services. Among the other options, you have separate toggles for "Aurora" and "RDS".
Overall the whole RDS API has lots of weirdness because there are all these different things (and combinations of things). Another random example, configuring Data HTTP API, you got this thing:
https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/AP...
> This operation applies only to Aurora Serverless v2 and provisioned DB clusters. To enable the HTTP endpoint for Aurora Serverless v1 DB clusters, use the EnableHttpEndpoint parameter of the ModifyDBCluster operation.
At least that specific case was nicely documented now. Not all the pitfalls are thst clear.
This is deprecated.
* Aurora is Amazon's own relational database. You can't run it yourself, only with Amazon. It can pretend to be either Postgres or MySQL. And it'll be cheaper and faster and have higher availability. But it won't be the exact same bits and bytes as your own Postgres, so there's some risks.
So far the stuff we said runs on clusters. You pick how big and how powerful and how many and in which AZ and configure how to scale it.
This has fairly pragmatic limits before at a certain point sharding and continuous horizontal scaling just gets too hard.
* Until Aurora Limitless that is which is, well probably not limitless (I don't know) but effectively.
* But you're still configuring cluster sizes and scaling policies. If you don't want to do that, Aurora has a Serverless option. It's the same Aurora but now you don't have to worry about scaling it yourself. The first version was Aurora Serverless but people said it wasn't very good.
* So they put out a 2nd one which is great and scales to zero.
* Now if you have globally distributed customers or care a lot about resiliency, you probably want your database in multiple regions. And you're going to be setting up eventual consistent updates for that. To make that easier there is Aurora Global Database, which is the same Aurora, but now with cross region replication.
* But it's not strongly consistent across regions. Aurora DSQL is. It's an even more bespoke version of Postgres. Actually it's not a relational database one at all, it just pretends to be one. But it uses atomic clocks and shit to cheat the cap theorem and give you global distribution and resiliency with strong consistency.
So in conclusion there's basically only 3 things here:
* RDS which is an unopinionated way of running a relational database in the Cloud
* Aurora, Amazon's highly opinionated relational database. That has Global, Serverless, and Limitless configuration options.
* And Aurora DSQL which is not Aurora a relational database at all but plays one on TV. But it gets to have the best of all worlds - SQL and NoSQL.
They probably should have called it something new but Aurora has good brand recognition. People know and trust Aurora.
In the long term all will probably continue to exist depending on where you are in your cloud journey. But I also expect that in the fullness of time if you're building a new Cloud native application and you don't have to worry about legacy migrations, you'll probably choose either DynamoDB or Aurora DSQL in 99.9% of cases.
It is public information?
See at this timestamp: https://youtu.be/LY7m5LQliAo?t=3507
That doesn't ring true to me. From my understanding rds aurora mostly replaces the storage engines of these DBs and the frontent mostly follows upstream releases and is compatible with them.
In reality, Aurora MySQL is clearly heavily based on the MySQL codebase, but with major changes specifically to InnoDB's persistence and MVCC layers. Aurora MySQL implements a physical replication system using shared storage, undo, and redo logging. The rest is otherwise compatible with MySQL because it is using a lot of MySQL code.
I haven't read up on Aurora Postgres, but I'd bet it's something similar being applied to the Postgres codebase.
It's entirely possible the custom storage components may share some common code between Aurora MySQL and Aurora Postgres, but a custom storage layer alone isn't a "relational database" which "pretends" to be other DBs. (In contrast, it appears that DSQL actually might be that – a custom system with a reimplementation of Postgres wire protocol.)
Not trying to knock Aurora btw, it's really interesting technology and is quite an achievement by Amazon. But the constant confusion about what Aurora even is creates some support burdens in the DB ecosystem.
If cloud-native means spending your life in AWS, that is.
But I use "cloud native" in a way that actually makes sense, not the way that GCP dubbed Kubernetes and all of it's ecosystem of friends "cloud-native".
"Native" development on a desktop OS means using primitives specific to that OS. As opposed to using something like Java Applets or Electron which will give you cross-platform compatibility, but you're not developing "natively" for the platform. That has the obvious pros and cons to each way.
Kubernetes is very much the Electron way: Rely on a generic high-level abstraction that will then create awkward bindings of running on abstract generic servers - whether they're in your data center or in the cloud.
Calling that "cloud-native" is some Orwellian post-truthness. I hate it.
If you want to develop in the cloud and make the most of it, use Azure, use GCP, use AWS, I don't care (not true: AWS is the best; use AWS). Just use the cloud provider's NATIVE primitives.
Otherwise, you might as well go back to running your own data centers, what are you even doing.
(Note: i'm just some guy. You don't have to listen to me. It's just what I think)
It appears to be a recent generational trend to openly flaunt one's ignorance...Admit you've never read the documentation but prefer to "learn on the job," criticizing Certifications, and having no patience for attending formal training.
Stay Tuned
We're sorry. The documentation you are looking for is not yet available. Please check back soon.
-The Amazon Web Services Documentation TeamEDIT: Your link is up now too.