Hacker News Reader

Comment by nikita | Hacker News Reader

Neon CEO here. Happy to answer any questions about Neon technology

This is something I've been hoping to see for a long time. I've long wanted a serverless postgres solution that would enable me to inexpensively put up a db and throw hasura on top of it.

I have a question about performance though. Let's say I have a table with millions of rows. Would it automatically scale to resources to get me sub second queries on it?

Ultimately that's what I desire in such a solution. Something that will let me throw whatever at it and will handle the scaling to keep whatever queries I throw at it fast.

I see there's branching support. Are there plans for version control features? I'd like to be able to treat the database as a collection of versioned immutable artifacts. Diff database branches, merge/patch, revert to a previous known good version. This would allow me to eliminate "migrations" and simply stage a set of changes, test them, and then merge or revert them, knowing I'm getting exactly what worked before without writing additional code to ensure it or walk it back.

I think we need more details to answer this question. Can you describe your workflow with more details? Maybe even with some code examples ?

Use Case 1. Migrations.

Everybody today knows that if you're gonna change your database [schema], you need a system to migrate the DDL changes intelligently so you don't break something, and then run that in lower environments to test, then promote it up to higher envs and run it, then deploy your apps that use the changes. But of course it may be impossible to revert those changes after that point, requiring an entire database snapshot restore. So not only are there serious operational concerns, making any operations around this time-consuming and frought with peril, but you need to set up a migration solution (language-specific or framework-specific or agnostic) and make sure you architect your application to only make changes in a specific way. (all of this, by the way, is only necessary because the database is one big mutable state machine)

...whereas if it worked more like version control, you could make any change, commit it, and get a commit ID. If that causes problems, if you could just `cloudpg revert $change_id`, then there would be no need to carefully architect the app, changes wouldn't be fraught with peril, and we could be more agile with database-driven design. The database would obviously need to be intelligent enough to figure out how to revert any change, which is why this has to be a database-specific feature and not just a git revert.

Use Case 2. Merging Changes

This sort of follows on the above (making changes more agile). If you have branches, and 4 different devs are working on 4 different database changes, how do you merge and deploy those all safely, and handle reversions safely? Well if all database changes had versions, and we could diff the changes between versions, then we could treat the database like a Git repo and merge/rebase all the changes to the database together at the same time as the code. Again, no need to go back and refactor migration scripts or the app design, because the database is essentially just version-controlled code.

The same things would apply to upgrading/downgrading database versions, bringing up or restoring new servers, possibly even making replication easier, maybe other things we haven't thought of yet.

For migrations, how would you want neon (or anything else) to handle all the writes that happened since the commit you want to return to?

It would need to revert only the changes in that commit while preserving all new changes afterwards. I don't have the foggiest how to make that happen, that's up to the ninjas at neon!

There would probably need to be a fallback mechanism if, for example, a new column was created and new data was entered into it, and then the revert removes the column. Probably it could keep pointers to such things ("there is a database D with a table T with a column C and rows [a,b,c,d]"), so that if the change is re-reverted later, an extra merge instruction could pop the reference back into place like nothing happened. Somebody with an actual CS background must have better ideas than me :)

This is a tricky problem to solve. Reverting to the commit - that's easy and we have all the infrastructure already to allow you to do that. The tricky part is

* Make a schema change

* push into prod

* accumulate some new data

* revert just the schema change

In our discussions we call this separating schema and data and allow you to have different schemas on the same data.

It's tricky to do in Neon due to the fact that storage knows nothing about schemas. It stores page with no idea what's on them.

But we have some ideas how to do this with logical replication where we will run a transform on top of logical replication stream to keep two branches in sync. Not this year though.

This would be the killer 2050 alien feature. Git is a really good mental model on itself. Git + Neon = you invent the future!

It would be a killer feature, but the thing is that Git has a conflict resolution feature that puts the working tree in an invalid state until the user has manually fixed certain issues, so that you cannot commit until you manually fix the files and mark those as such.

Doing so for a database seems less desirable from an availability perspective, especially with high-throughput databases.

I've been wondering if it makes sense for databases to start offering CRDTs natively, so that you can perform merges that are algorithmically conflict-free.

CRDTs are pretty hard to implement at the Neon storage level - pages don’t know what’s written on them. I don’t know of a general purpose database that supports CRDTs. It would be cool though!

In a sense, a CRTD is a database. But the 'conflict-free' is only because it defines semantics for conflicting operations.

But, because SQL has conflict resolution by cancellation of one of the two conflicting modifications, I don't think that it is reasonable (or even possible) to merge 2 divergent databases in a single way that always conforms to the needs of the developer and/or application.

But the manual intervention of conflict resolution in a Git repo is done in a working tree. The named branch's HEAD remains the same until the conflict has been resolved and the commit is merged/pushed. So nobody sees the conflict at all, they only ever see either the old code, or the merged resolved code.

I see how having multiple Postgres compute nodes scales up reads, which is great! Does separating the "write" functionality into the Pagekeepers service allow writes to scale as well?

Seems like there's no upper limit for scaling up reads, just wondering how this architecture affects write throughput. Would love to hear more!

Yes you can’t infinitely scale writes. You can make your storage bandwidth infinit but you are still limited to the how much Postgres WAL one instance can pump into the storage.

I think we can do a lot of good things here over time and have plenty ideas. But for now it’s a single writer system. Good news is that there is so much open source tech around Postgres that it might not be a gargantuan task in the future

What’s the ‘page’ in Pageserver? Is it a B-tree page? Or disk page?

I have a hunch that Pageservers contain the disk pages, so storing the B-tree (or may be LSM) pages and compute traverses those pages to find the relevant page/pages. I am curious about how does it fit in together and fetches from disk/Pageservers work

It corresponds to Postgres 8k page. We store them in an LSM tree so we can move layers into S3

On the surface this looks pretty similar to AWS Aurora Postgres Serverless. Could you expand on the differences?

Not to steal the CEO's thunder, but this (tech|company) got quite a bit of commentary last time it was submitted: https://news.ycombinator.com/item?id=31536827

This was a good discussion indeed. The blog captures most of these points. We were pre launch then

Only serverless, only Postgres - that’s focus.

Multi-cloud and open source.

AWS Aurora Serverless can't scale down to 0, at least not on v2, so minimum cost to play is $40/month

I'm sure Neon's design can handle the hobbyist wordpress DB or personal project DB for <=$10/month since it can scale down to 0, and it likely would not be terrible for a cold-start on a small personal website to be 1-3 seconds for the first page load -- sure, bad for SEO, but I think a LOT of people want a DB that is managed and simple for <$20/month -- and again, I'm very hopeful Neon will have a nice pricing number here when they figure out their model, since a very tiny shared tenant should be very inexpensive for their model.

How does shared memory get handled across nodes? I’d imagine there’s quite a bit of in memory state for things like sequences. For that specific example you could preallocate chunks of them as there’s no guarantee of them being contiguous, but I’m guessing there’s more complicated examples.

Our compute node is basically a usual postgres where we intercept WAL write and page read streams. In other words we don't do any compute sharding since we want to preserve vanilla postgres compatibility and any sharded solution will be riddled with a lot of issues like you've mentioned.

can you describe how compute scaling works? I do a lot of work in PostGIS, which can have big CPU needs depending on the function being invoked, so my workload can look super bursty from a CPU perspective.

We are still experimenting with compute scaling, and the tech preview includes only a small fixed compute container. We have a custom proxy in front of a compute, and we can quickly change the underlying compute container with a bigger/smaller one. That works fine if you don't use per-session semantics and transactions are short-lived. But there are a lot of tradeoffs on what to do if there is a long-lived transaction or session-level object. So if we need to scale up the container in the presence of long transactions, we can:

1) roll back long transactions and enforce upscale

2) wait for a better moment to upscale (potentially forever)

3) try to do a live migration of running Postgres to another node (like VM live migrations, or CRIU-like process migration) and preserve long-running transaction

So far, we plan to start with some combination of 1+2 -- should be fine for web/OLTP kind of load. But ultimately, we want to arrive at 3), but that approach has way more technical risks.

Does your Postgres support all the features? What about things like writing SQL functions in Java or other languages?

All the features. It's the same Postgres that you use elsewhere

Huh, interesting. So your postgres can basically run arbitrary code on behalf of the user? I'm talking about stuff like https://tada.github.io/pljava/ where you can use the DB to invoke whatever code you like, outside of any sandbox. Like, could I upload a PL/Java function that probes your internal network? How are you making that secure?

That is true. And that is why we do not have the UI for loading extension binaries and do not give root access to the compute node. Yet. Of course, some containerization is in place, but it is not as tight as we would like for arbitrary code execution.

Still, there are no technical limitations. Our test suite already uses Neon-specific SQL functions from a C extension (https://github.com/neondatabase/postgres/tree/7faa67c3ca53fc...). At the very least, providing a lot of popular extensions out-of-the-box is on our roadmap once we figure out the security, no special repacking needed. As compute nodes should already be pretty isolated from each other, I don't think allowing arbitrary code will require a redesign.

How do you differentiate against Google's AlloyDB, Cockroach Serverless, and fly.io?

AlloyDB is not serverless, cockroach is not Postgres, and fly doesn’t separate storage and compute.

> cockroach is not Postgres

CockroachDB supports the PostgreSQL wire protocol and the majority of PostgreSQL syntax. Not 100% compatible but most stuff works the same.

https://www.cockroachlabs.com/docs/stable/postgresql-compati...

Reply on news.ycombinator.com