Architecture Decisions in Neon

Splitting the architecture across compute/storage and then pageserver/walserver seems like a great design decision because the pieces fit snugly onto the underlying layers of cloud services. The design is general enough that it could work for other block-device-based databases (~all of them) so you could put a SQLite or MySQL compute layer on top of the storage layer instead of postgres. Discussion: https://github.com/neondatabase/neon/discussions/1958

The fact that they're targeting vanilla postgres is also intriuging because that means it will be a lot easier to integrate with other interesting PgSQL-based technologies; EdgeDB in particular seems like it would be a pretty great fit.

> In Neon, incoming WAL is processed as it arrives, and indexed and buffered in memory. When the buffer fills up, it is written to a new file

This sounds cool, but what happens if the server crashes? How do you prevent losing the buffered WAL? Do you have WAL of WAL or something?

Cool to see this (I’d somehow missed the early access link until last night).

Your storage strategy makes sense - building from first principles. To be honest, I figured you’d not done that though, and I imagine this makes portability a bit harder.

I’m eager to play with the service, but I wonder how local access patterns (e.g. dev workflows) differ - upon the right abstraction it shouldn’t really matter, right? I.e. a conn is a conn and a cursor is a cursor - the storage is opaque to the client. But does this somehow fall over with transactions and connection limits?

Heikki was behind major decision and has done a lot of prototyping. Happy to add more color