AWS cancels serverless Postgres service that scales to zero | Hacker News Reader

AWS cancels serverless Postgres service that scales to zero

datanami.com

118 pointsgoeldhru93 comments

I guess a good opportunity for someone to build a good alternative. Although it's hard to get it right because if you scale to 0 people will complain about latency and if you don't it's not really serverless.

What alternative does someone need to make to compete with AWS?

I think the problem is that we're trying to take paradigms that are designed to be serverful (is that a word?) and shoe-horn them into working in a serverless way.

(Neon CEO).

It's possible and here is the recipe: https://twitter.com/nikitabase/status/1725394843285766463

Always gotta watch what you say on Hacker News or the CEO of the company you're talking about might just arrive outta nowhere with the elbow drop of justice.

It helps but does not fully solve it if I need PG to have 512 GB of memory buffers under load to have reasonable latency starting from 0 and going through a number of VM migrations to finally get to properly sized VM with 512 GB of hot data in memory gonna cause a ton of latency.

The buffers will be cached in remote storage - so it will just work likely.

In the future we are considering separating the buffer pool from compute and turning off compute doesn't drop the buffer cache. It's a research project but PolarDB did that for AliCloud.

"Remote buffers" sounds like it'd introduce a base level of latency anyway.

Though computers are pretty fast these days (and RDMA exists), so if the pieces are in the same data centre and using good interconnects it's probably not going to make much of a practical difference.

... which does not refute the parent post's statement ;)

There is a good alternative, https://neon.tech check them out. Really good bunch. Product is great and it's open-sourced https://github.com/neondatabase/neon

I'll note that the storage engine is open-source as well as any Postgres modifications, but our control plane is closed-source. I work at Neon.

Neon is only scale-to-zero from the point of view of the tenants; whoever runs the software, must keep nodes running, including triple-replicated WAL storage.

I mean if you scale to 0, you should expect cold start latency.

Well yes and no if you had enough resources imaging something like Spanner but highly optimised for multi-tenancy. There will be higher latency because inactive tenant's data got purged from caches but it would be nowhere near the latency of spining up a postgres instance. You could even have a smart stats layer that tracks usage and keeps some minimal amount of inactive tenants data in memory to minimize latency.

Corporate customers that runs AWS, GCP or Azure at not looking for alternative outside of their cloud providers. Most solution architecture are designed with the assumption that all your ressources will be inside the same vnet or subnet, in a given cloud provider. And I am not talking about any egress fees yet.

I challenge this. I work at Aiven and we provide alternate cloud-independent solutions, sometimes in the customer’s own account, and enterprise customers definitely see the value in using a fixed cost, no-lock-in service.

Well just one counterexample Snowflake

SQL Server on Azure has an auto pause feature. All of our QA and dev instances use this. Seems to work reasonably well for us.

Do they auto-wake when you query them? How long does auto-wake take?

Yes - takes about 10-30 seconds in my experience.

We wanted to switch to Aurora in our company about a year ago. We tried serverless first but decided against it. We ended up using Aurora provisioned. Main reason was cost. Aurora Serverless was 20%-40% more expensive than Aurora provisioned.

One area where Aurora is superior to non-Aurora in AWS is replication lag. When running non-Aurora MySQL servers, we had replica lag sometimes reach an hour making read replicas useless. With Aurora provisioned, replica lag is always under 30ms.

If your app is database heavy and you end-up using database pretty much 24/7, don't fall for serverless hype and go with Aurora Provisioned.

Honest question, why not just use dynamodb? If I'm already mostly server less and AWS, dynamodb seems like a natural choice for a persistent store in a world where cost and zero-scaling is important.

Because you need SQL. Dynamo can be quite powerful for querying if you use all of its features but it won't be as flexible as just using SQL

Almost any SQL workload you can move to DynamoDB

https://aws.amazon.com/blogs/database/how-to-determine-if-am...

> Some unsuitable workloads for DynamoDB include:

> Services that require ad hoc query access. Though it’s possible to use external relational frameworks to implement entity relationships across DynamoDB tables, these are generally cumbersome.

> Online analytical processing (OLAP)/data warehouse implementations. These types of applications generally require distribution and the joining of fact and dimension tables that inherently provide a normalized (relational) view of your data.

> Binary large object (BLOB) storage. DynamoDB can store binary items up to 400 KB, but DynamoDB is not generally suited to storing documents or images. A better architectural pattern for this implementation is to store pointers to Amazon S3 objects in a DynamoDB table.

I said moving the workload to DynamoDB, not doing the same query pattern with DynamoDB ... :-)

For each of these workloads you switch the query pattern to accommodate for DynamoDB and provide a solution for your workload and/or app. That is actually the secret of NoSQL. You do the work upfront.

So your reading of the recommendations is correct on the surface but incorrect on the fundamental usage aspect of it. Lets have a look at each:

> Services that require ad hoc query access. Though it’s possible to use external relational frameworks to implement entity relationships across DynamoDB tables, these are generally cumbersome.

A: Dont do this. Don't do ad hoc query access. Define a series of access patterns and lay out your data to support them. You dont get ad hoc query access for Netflix, Amazon or your Airline Travel website...

These are not most SQL Patterns. This is OLAP and was always or should be done with MPP systems, not with your relational database like your SQLServer or your Oracle. So this is going on an edge...

Your Relational Database will not store BLOBs more efficiently than S3 anyway....

DynamoDB lacks consistency (in the ACID sense, it has been it in the CAP sense which is more related to Atomicity in ACID), so you can move an SQL workload to DynamoDB by moving the consistency parts out of the DB into the code of any application that hits the DB.

You hardly get the ACID on a relational database :-)

"When is "ACID" ACID? Rarely." - http://www.bailis.org/blog/when-is-acid-acid-rarely/

That's an article about isolation, not consistency; and even there, both the RDBMS’s relevant to thus discussion (those on Aurora Serverless that it is suggested in this subthread be replaced with DynamoDB) supported serializable isolation even in their current versions at the time of that article a decade ago.

So your post is both irrelevant as a response to mine about consistency and more generally irrelevant to the entire discussion here.

The article is about ACID, both consistency and isolation not just about consistency and you can't isolate them (pun intended). The underlying of the discussion was not on consistency, instead on the possibility of using DynamoDB where you would use a typical relational engine.

From the article: "The textbook definition of ACID Isolation is serializability (e.g., Architecture of a Database System, Section 6.2), which states that the outcome of executing a set of transactions should be equivalent to some serial execution of those transactions. This means that each transaction gets to operate on the database as if it were running by itself, which ensures database correctness, or consistency."

" This means that each transaction gets to operate on the database as if it were running by itself, which ensures database correctness, or consistency. A database with serializability (“I” in ACID), provides arbitrary read/write transactions and guarantees consistency (“C” in ACID), or correctness, of the database. Without serializability, ACID, particularly consistency, is generally not guaranteed"

I am of course ignoring the Consistency you are certainly aware that exist in DynamoDB with Strong Consistency and DynamoDB Transactions.

Moving consistency management outside of DynamoDB as you propose doesn't circumvent the CAP theorem's limitations.

DynamoDB can't even represent SELECT * FROM items LIMIT 25 OFFSET 100. It's just not designed for that. It's not meant to be a relational DB replacement.

How would you do it? Assume we want proper pagination, and not rewrite the app for cursor based "Load more" style pagination. Why? Because the React Admin provider API insists. https://github.com/marmelab/react-admin/issues/1510

Your example itself shows why you would not want to do this. The database still has to read through all the rows up to the OFFSET point...

You are not supposed to do the same query patterns. My argument is that you can substitute your relational database, by changing the app and the layout of the data to match the proper patterns for DynamoDB.

"Migrating to DynamoDB from a relational database" - https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

""How to model one-to-many relationships in DynamoDB" - https://www.alexdebrie.com/posts/dynamodb-one-to-many/

If you place arbitrary restrictions upon it then of course it won't fit your model. Dynamo can handle pagination easily by passing in the last seen ID in the request. We used IDs that were lexigraphically sortable by time which has some nice properties such as never losing your position due to extra items being inserted in the middle of your query. Offset/limit aren't fun to work with.

You can go further by using the streams feature to dump your data into an analytical database for your querying needs.

Yeah agreed. It's about tradeoffs. If you want scale to zero with low latency DynamoDB already exists and provides that. The tradeoff is you give up all the nice features of a RDMS.

DynamoDB is _fantastic_ for certain scenarios. But, you also need to know your access patterns upfront and more or less end up hardcoding your indexes into the "table design". It's not a 1:1 replacement for a typical relational DB's use cases.

That's like telling people to use S3, those are completely different storage systems.

Dynamo is NoSQL, with all the architectural tradeoff that brings: Say goodbye to joins! So depending on how your data looks like, dynamo can an pretty bad fit. Do not run your reporting engine in Dynamo.

It gets worse than that though, as Dynamo's usage on large tables depends on its hottest shard, and other than setting primary keys, you have no control over how it shards. The story of people moving to dynamo and touting its advantages, just to move out a year later because the shape of their data makes dynamo prohibitively expensive as the data grows is pretty common. I once worked at a place where they stored historical data in dynamo, leading to a substandard hash key. Most of the time the db was idle, but when it wasn't, the sharding scheme made them pay a good 500x capacity than it was actually in use, because while a shard was red hot, others were completely idle. The monthly price for this relatively unimportant feature ended up being higher than the office's rent, in San Francisco.

If your table's key is UUIDs, and the chances of querying one record or another is almost perfectly flat, then sure, dynamo away! But if you walk away from it very best use case, and you suddenly start having any amount of data... dynamo can become really, really expensive. I'd not say you should never use it, but it's a really scary first place to go, precisely if you care about costs.

The title includes Postgres, but if I’m understanding correctly it’s actually both flavors of Aurora - Postgres and MySQL

Ignore requirements of paying customers because product guy or developer guy knows best. - The Amazon Way(tm)

They are end of lifing v1, which could go to zero. V2 can't.

Are there any truly serverless SQL databases out there?

I've never used it and it's still in beta, but I think Cloudflare D1 would qualify: https://developers.cloudflare.com/d1/

"Maximum database size 2 GB (Workers Paid) beta" (but you can have up to 50.000 databases)

https://developers.cloudflare.com/d1/platform/limits/

Oh strange, what a tiny limit. Is it kept in memory on a 32 bit machine? lol. Half our tables are bigger than that

Actually, yeah, I fully expect them to be running it inside WASM, so 32 bit...

I use TiDB Serverless for some smaller projects I am working on.

https://www.pingcap.com/tidb-serverless/

> Are there any truly serverless SQL databases out there?

sqlite ;)

Neon is serverless Postgres and does scale to 0. I work at Neon.

https://neon.tech

I am the CEO of Nile. We are serverless Postgres built explicitly for modern SaaS workloads. We do provide true serverless with no cold start time. We are able to do this because we made Postgres tenant aware. We hope to share more details on how we do it soon. https://www.thenile.dev/

Serverless V2 is also shit. The minimal one equivalent ACU to a regular instance has lots of latency spikes and much slower than a t3 instance.

We also tried serverless Redis. We have a tiny machine at %10 CPU utilization, but sustained queries. It costs 20x more. We were very afraid of the result we just ran it for an hour to get an idea.

Serverless, pay per use sounds only good if you have very few or unusual traffic.

We just run a monolithic but stateless Spring Boot Application and a dedicated RDS instance with reader in stand by. Only serverless dependency is SQS because it works nice enough. This simple infra has allowed us to iterate and pivot our startup very fast as needed. We still need more customers though.

Yeah, the only systems likely to see real benefits from serverless are those with infrequent and intermittent traffic.

> Serverless, pay per use sounds only good if you have very few or unusual traffic.

And most of those cases can be handled by a $5 a month instance. Thus, usually, the most serverless can save you is $5 a month.

I suppose the real appeal of serveless is you don't have to worry about maintaining the server, doing updates, etc.

(CEO of neon.tech here)

Scaling to zero is tricky. In order to make it work you need to

1. Separate storage and compute which aurora has done

2. Have a proxy between the user and an actual VM in which you host Postgres

3. Spin up this VM after authenticating a connection, stand up Postgres inside the VM and run the query

You need to do all that very quickly- ideally in a couple hundred ms. This can only be accomplished with having a warm pool of VM or having extremely fast microVMs like firecracker.

I believe AWS aurora just uses ec2 and that’s why scale to 0 is turned off

So how does your company do it?

reposting Neon ceo twitter post: https://twitter.com/nikitabase/status/1725394843285766463

Scaling from zero did use quite some seconds so there would be some leeway for an alternative implementation.

The term "serverless" is so confusing if to me.

I had settled on "scale to zero" as my chosen definition for it, because it was the only definition that truly made sense to me and that I very strongly valued.

Apparently AWS don't think it means that.

I guess it just means "you'll never have to SSH in and upgrade anything"?

The other thing that surprises me here is that one of AWS's biggest selling points is how rarely they break existing apps by turning off services developers depend on.

Serverless is a paradigm where you pay for capacity, not for a specific number of instances. It's extending "cattle, not pets" beyond VMs to databases.

In theory it provides better incentives for both parties:

Customers don't have to pick instance size, etc. to try and accomodate their workload. They can buy throughput, etc. in fractions of an actual instance. They can look at their quota usage to determine whether to "scale up" or "scale down".

Service providers are incentivized to optimize their software. If you sell instances + a service on top, there's a perverse incentive not to make your code too fast or people will buy fewer instances. In serverless, providers are incentivized to reduce their costs and optimize the entire stack.

Scale to zero isn't necessarily a property of serverless - a service provider can agree to keep your data on ice for free, but that's a business decision.

I mean, the inverse incentive to hobble performance is still there for all the serverless stuff that bills by time spent.

Also, it's a way for AWS to reuse very old hardware where customers don't think about the specifications of the underlying hardware.

I still consider "serverless" to mean something that does in fact scale to zero. You also don't necessarily need to SSH in and upgrade stuff with a managed service either but RDS isn't exactly serverless even though you don't really need to service the database server yourself.

It typically means the unit of management (and billing) are not physical constructs (or abstractions over them). So no servers/instances/nodes etc.

You manage purely logical containers like databases and tables, the serverless service takes care of the logical to physical mapping. This lets them get better density, manage patching scaling, and a ton of other things

> I guess it just means "you'll never have to SSH in and upgrade anything"?

To me that is just "managed". I agree, serverless means scale to 0 (compute and costs). I'm so glad I left AWS Aurora Serverless after how much I disliked v1 and never considered v2. If I was still stuck on them my costs would double or more for my dev/qa environments (which I had scale to 0).

You may be interested in this article in Communications of the ACM: Serverless Computing: What It Is, and What It Is Not?

https://dl.acm.org/doi/pdf/10.1145/3587249 (pdf)

> "you'll never have to SSH in and upgrade anything"

We have "managed" for that.

But if you manage to get a meaning for "serverless", I would love to know it too.

"Serverless": We don't bill you if you forget to turn it off.

That's a synonym for the GP's "it scales into 0", that is evidently a bad meaning to attach to the word.

Somebody answered the GP with "it's SaaS, not IaaS". The more I think about it, the more I think this actually represent what people mean when they say it.

I learned the expensive wya that Aurora Serverless v2 does not, in fact, scale down to 0 when I created three instances for a few days when I was testing out configuration changes. It ended up costing us $300.

FWIW the base cost is around $44/month on paper (plus storage). You're always paying for 0.5 ACU.

$100 a month for a single Postgres instance is insane

Aurora Postgres is an interesting product where AWS made significant changes to Postgres storage architecture, however it has a fairly high fixed cost.

It seems in many cases RDS Postgres is cheaper and even when using Aurora, using fixed-size instance classes instead of "serverless" autoscaling classes (with ACUs) is a lot cheaper. Fixed Aurora on-demand instances are roughly 17% the cost of ACUs. Fixed instances are usually >= 30% cheaper on top of that if you reserve.

Aurora heavily favors performance and availability over cost.

AWS has a decent implementation transactional distributed block storage that they leverage for a few technically decent offerings. Aurora being one of them.

I used to work for a business that was predominantly used by women.

Like clock work, at 3pm PS most weekdays there was a spike in traffic as our customers became active on our system.

I would love to see the real numbers behind the costs. Is time of day dependency the real price driver here? Are some customers getting more value than others out of this system?

> [...] predominantly used by women [...] Like clock work, at 3pm PS most weekdays there was a spike in traffic

what knowledge gap do i have about the collective womanhood that I don't understand why a service which is predominantly used by women gets traffic spikes at a specific time of day?

Is 3PM PST when most women sync their in-built RTC?

3PM is around when school ends for most k-12 schools; if its something someone might check quickly in a free moment at/after work and/or in the car waiting to pickup kids, I can see a big spike from teachers (disproportionately women) and primary child-raisers (disproportionately women) around 3pm.

To answer, daily peak, 3pm PT to about 11pm PT...

That starts 6pm ET, so post work and kids, it's "free time". By 6pm PT(9pm ET) we would be hitting a peak, for the next hour or two and then winding down till 11. It's the prime time for "adult working women" as a demographic.

Ah I see. So it's just generally the "adult working parent" prime time, but you had a focus on women because they were the service's target demographic.

So if I understand correctly, we're (economically) back at the classic model of continuously running a db in a process, right?

Not yet. Dynamodb is killing it

I think the big takeaway is that scale-to-zero customers aren't where they're making their money.

Or that certain "minimum delightful product" features can be abandoned when they are inconvenient to provide.

I started using the smallest VM I can that's free. On google cloud it's 1 GB RAM, 30 GB hard drive, small CPU. I run apt install -y postgresql after I create it and run my golang app and the database ON THE SAME VM! It's amazing how much you can do with that all for free.

Best solution for small scales, doesn’t work anymore when you need reliability (unless you want to be always on call to restore from backup and are willing to accept data loss)

Obviously cloud computing has done wonders for building gigascale infrastructure, but it's so incredibly useful for building microscale too. You can't run half a CPU at home.

Of course you can run half a CPU at home. With Ubuntu LXD, you can easily run actual, hardware 50% cpu time, or .5 priority container. Why wouldn't you be able to have complete control over what "microscale" tasks you run on your own hardware, and own OS, that you control?

On the contrary, vCPU a completely cloud-centric product oriented concept, intended to abstract away what CPU you're actually running on, and how fast. These abstracts make orchestration and availability easier, but I think the key is a stable, public internet access. If you get a static IP, I don't see what wonders this does for "microscale". In fact, you probably have 90% unused "microcapacity" on all the personal hardware that's often idling, that is a lot less micro than any cloud provider will give you for free.

I have a feeling most people chasing "serverless" postgres or mysql with the ability to scale to zero are perfect candidates for sqlite.

Reminder that AWS Aurora claims to have broken the CAP theorem. The literature doesn’t clarify whether they are consistent or available, and when you ask AWS directly, they tell you the CAP theorem isn’t relevant due to the way Aurora is built.

They claim 99.99% availability or whatever the figure is, that it’s self healing, allows up to 2 replicas to be down for write availability and 3 replicas to be down for read availability. They promise “millisecond consistency”, whatever that even means.

Some of the blogs mention that they use distributed state to extract slivers of consistency across the nodes, which is just scary stuff to hear.

I’m seeing more and more usage of Aurora due to the magical component it offers, but I can’t wrap my head around what it’s actually supposed to be, and what the failure model is, like I can with RDS PG and MySQL.

Can’t you bypass CAP if you just have a ton of compute in the background handling all of these tasks in real time? Aurora isn’t just some binary running on ec2 it’s probably 10 different things going on

CAP is a property of any distributed system. You have to tolerate P (considering it’s distributed), so you can only be CP or AP.

CP means you can’t be available, meaning your system will stop replying to queries.

AP means you’re always available, but you could receive stale data when querying.

It’s irrelevant whether it’s a single binary running on EC2 or a billion services working in unison. It’s a property of the system.

AWS should show us the Jepsen or GTFO

Isn't Aurora similar to Googles Spanner in this regard, giving pretty much same characteristics?

https://cloud.google.com/blog/products/databases/inside-clou...

https://news.ycombinator.com/item?id=15923507

I've not seem them claim to have broken CAP. AWS is quite an academic organization I think it'd be quite unlikely for them to say something like that.

They do actually provide quite a lot of detail about Aurora storage this works. This 2019 reinvent talk gets pretty deep into the weeds.

https://www.youtube.com/watch?v=uaQEGLKtw54

https://d1.awsstatic.com/events/reinvent/2019/REPEAT_Amazon_...

"Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases" - https://assets.amazon.science/dc/2b/4ef2b89649f9a393d37d3e04...

AWS Serverless MySQL/Postgres offerings are straight trash. I used v1 to build a new app but had nothing but problems. Extremely slow starts (from zero), horrible scaling (it would always get stuck), (relatively) huge bills for the smallest capacity, limitations all over the place. After the first year on that I looked into v2 but my costs would have doubled and I didn't believe their promises of faster scaling. I moved to PlanetScale [0] and was very happy ($30/mo covered prod and dev/qa vs well over that for v1 even with having scale to zero on the AWS dev/qa instances). Also you can quickly be forced into paying for RDS Proxy if you are using lambdas/similar which is not cheap (for me). PS doesn't scale to 0 but at the time $30/mo was a decent savings over AWS Aurora Serverless.

This year I started to run into some issue with PS mainly around their plans changing (went from pay for reads/writes/storage to pay for compute/storage). Yes, yes, I know they still offer the $30/mo plan but it's billed as "Read/write-based billing for lower-traffic applications" and they dropped all mentions of auto-scaling. That coupled with them sleeping your non-prod DB branches (no auto-wakeup, you had to use the API or console) even after saying that was a feature of the original $30 plan rubbed me the wrong way. Eventually the costs (for what I was getting) were way too out of whack. My app is single-tenant (love it or hate it, it's what it is) so for each customer I was paying $30/mo even though this is event-based software (like in-person, physical events that happen once a year) so for most the year the DB sat there and did nothing.

Given all that I looked into Neon [1] (which I had heard of here on HN, but PS support suggested them, kudos to them for recommending a competitor, I always liked their support/staff) and while going from MySQL to Postgres wasn't painless it was way easier than I had anticipated. It was one of the few times Prisma "just worked", I don't think I'd use it again though, that DB engine is so heavy especially in a lambda. I just switched over fully last week to Neon and things seem to have gone smoothly. I can now run multiple databases on the same shared compute and it scales to 0. In fact it's scale up time is absurdly fast, the DB will "wake up" on it's own when you connect to it and unlike AWS Aurora Serverless v1 it comes up in seconds instead of 30-60+ so you don't even have to account for it. With AWS I had to have something poll the backend waiting to see if the DB was awake yet, to fire off my requests, if it was asleep. With Neon I don't even consider it, the first requests just take an extra second or two if that.

I don't have any ill will towards PlanetScale and I quite enjoyed their product for almost the whole time I used it. Also their support is very responsive and I loved the branching/merging features (I'll miss those but zero-downtime migrations aren't required for my use-case, just nice to have). In fact if I had written my app to be multi-tenant then I'd probably still be on them since I could just scale up to one of their higher plans. It does seem like Neon is significantly (for me/my workload) cheaper for more compute, I had queries taking _forever_ on PS that come back in a second or less on Neon all while paying less.

All that said, I _highly_ recommend checking out Neon if you need "serverless" hosting for Postgres that scales to 0.

[0] https://planetscale.com/

[1] https://neon.tech/

Reply on news.ycombinator.com