kelvich

This thread seems to reflect how the HN audience has shifted — less commenters know what `ssh example.com` does and more commenters concerned about privacy policy.

> I'm not going to SSH to a random server.

Opening a random website likely exposes you to more risk.

To provide more context, our original plan was to launch the GA on March 22nd. However, we decided to move the date to April 15th because a few projects required additional time to complete. Last week we saw Supabase announcement but we didn't know what is that about and decided to not to move our date again.

Thank you, point taken.

The blog post does mention memory limitations, but we could elaborate more on the fact that the index is entirely in-memory and clarify the user-visible consequences of this fact. We'll edit the post for better clarity.

Yes, exploring http/3 is on the roadmap. One of the learnings for us was the fact that v8 is quite good with caching open TCP connection; For consecutive queries the same TCP+TLS session is being reused. Even when queries are sent from unrelated V8 isolates.

There is a second network interface attached to each pod and that interfaces are connected to vxlan-based overlay network. So during the migration VM can keep the same ip address (APR will take care of traffic finding it's destination). That is simplest approach but has it's own downsides when overlay network grows big and painful to set up with some cloud SNI.

Few other options are:

* use some SDN instead of vxlan

* use QUICK instead of TCP in the internal network -- with QUICK we can change endpoint IP addresses on live connection

Main idea was one branch per pull request to better test migrations on a fresh data. But one branch per dev or branch per dev per pr will also work.

> The article ends with the statement that you are pretty much done here for now. Would optimizing your TLS termination not maybe offer some more ways to speed this up? Or is that also already fully optimized?

No, we don't do early termination yet, but it makes sense to try it out too. Here we mostly concentrated on how far we can get in terms of reducing number of round-trips.

> I did not realize before that your approach with Websockets actually meant that there was no application/client side pooling of connections. What made you choose this approach over an HTTP API (as for example PlanetScale did) anyway?

To keep compatibility with current code using postgres.js.

Feel free to try it out on https://neon.tech/

Gotcha. I drew my conclusion based on the mentioned package.json. Now wonder why did you decide to go with rust for query engine? Do you compile it into wasm?

Another angle here is compatibility. With our current driver one can use ordinary node-postgres package, as we can substitute TCP-related calls with WebSocket calls during the build time. With that it possible to use all the packages that do require node-postgres like Prisma, Zapatos, etc.

Right, that is one of the use cases. Auth is tricky part in that setup -- in the browser user may change the query in arbitrary way. But there are some workarounds like obtaining JWT from 3rd party service and checking it via Row-Level Security policy in Postgres.

It is possible to create branch retroactively

And some other projects that DARPA worked on: https://spectrum.ieee.org/tech-history/space-age/the-bunny-t...

Licklider is one the main persons behind the ARPANET. The Imagineers of War by Sharon Weinberger has a lot of context on his time in DARPA. Few things that I found particularly notable:

* Licklider was conscious about not having too much funding compared to other DARPA projects so that in case of a budget cut, there would a more obvious candidates for the slash.

* Initially, universities pushed back on ARPANET because they were afraid that remote access might lead to more candidates for their mainframe time

Postgres tracks maximal LSN among the evicted pages and passes it to the pageserver in the page request. If the pageserver hasn't received that LSN, it will wait for it to arrive.

Currently, we deploy three safekeepers, one in each AZ. We need to collect more stats on failure rates, and maybe we will go to 6 (3 AZs with two nodes in each) as Aurora does.

Right. Our design guideline is to get as much serverless behavior as possible while keeping full Postgres compatibility (in terms of features and expected performance). Single node Postgres can give you hundreds of thousands of small RW queries per second, so competing connections should be a few compare-and-swap instructions away from the shared state to provide this performance. So for the primary, it means it should be just a Postgres in the container or VM, and we have to deal with consequences (cache pre-warm, handle cross-node migrations, etc).

However, read-only nodes require less coordination, and we have way more freedom there, so read-only Postgres as a function seems to be a more feasible concept.

Good catch! Currently, we don't, and we rely on k8s to stop the old node. Technically speaking, if k8s and our control plane are always good at stopping the old primary, we don't need consensus at all. So that is more of a question of what set of problems we can see if there is a bug in our orchestration code. Split-brain seemed to be unacceptable. But with stale reads, we decided that we can only rely on k8s without double-checking that on our side.

The author is here, happy to answer questions if any.

It is about 3 seconds to start now, with about 1.5 seconds out of 3 being k8s time quant. So we hope to drive it below 1 second with some tweaks.

We are still experimenting with compute scaling, and the tech preview includes only a small fixed compute container. We have a custom proxy in front of a compute, and we can quickly change the underlying compute container with a bigger/smaller one. That works fine if you don't use per-session semantics and transactions are short-lived. But there are a lot of tradeoffs on what to do if there is a long-lived transaction or session-level object. So if we need to scale up the container in the presence of long transactions, we can:

1) roll back long transactions and enforce upscale

2) wait for a better moment to upscale (potentially forever)

3) try to do a live migration of running Postgres to another node (like VM live migrations, or CRIU-like process migration) and preserve long-running transaction

So far, we plan to start with some combination of 1+2 -- should be fine for web/OLTP kind of load. But ultimately, we want to arrive at 3), but that approach has way more technical risks.

Our compute node is basically a usual postgres where we intercept WAL write and page read streams. In other words we don't do any compute sharding since we want to preserve vanilla postgres compatibility and any sharded solution will be riddled with a lot of issues like you've mentioned.

Now the most common setup is to copy the production database to the staging once in a while and test migration against staging. With branching, you can test each PR against its own production database branch -- just put branch creation in your CI config. Hence, it has fewer moving parts, is a bit easier to set up, and reduces the lag between prod and staging.

You can, no strings attached

We are planning to open public tech preview next month. We'll go to general availability when we see a decent SLO for several months. Tech preview should be okay for hobby projects and staging workloads, though.

We wrote a custom storage layer for postgres, so in our setup, postgres node (k8s pod actually) doesn't store any data, and it is easy to start/stop/reschedule it. So while we actually are DBaaS, the closest analogy here is Aurora or Alloy, not RDS-like setups.

A bit more details are on https://github.com/neondatabase/neon

Sure, we will

Fixed, thank you!