"We separated storage and compute"
No. just no.
Compute where the data lives, else you incur traffic cost and latency.
A lesson learned in life, known from the dawn of time..
No. just no.
Compute where the data lives, else you incur traffic cost and latency.
A lesson learned in life, known from the dawn of time..
The major part of what Neon does is remove the file system abstraction that is between that Storage and Compute, so that we can better utilize the available resources because we can better select what information is or isn't being lost.
A good example of what removing the file system abstraction enables for us is effectively free PITR, (lagging) replicas, and data branching. This is because PostgreSQL's file-system-based storage engine expects to be the only one working on the data directory, which means that any FS attached to a replica cannot be shared. If you remove that file-system based storage engine and plug in a different storage engine, those expectations are removed too, and after some effort integrating into the smgr-APIs, we're now able to provide a storage layer that only needs to contain one copy of the data for N physical replicas, instead of N copies.
The latency of NFS & EBS or EFS is actually the reason many businesses *do not* use them for their databases.
I've seen deployments that had to go bare-metal because the tiny latency of EBS caused their compute times to rise exponentialy (doing AI training).
Talking about the "correctness" of a choice between tradeoffs is weird though. Running your database on hard drives nowadays is not a great choice - yet people still did that because the cost of hard drives was way cheaper than that of memory.
Running your database in a way that doesn't guarantee that a 'committed' response actually retains the changes that it was responding on - yet people still ran their database in such configuration to scrape the barrel on performance.
All Neon does is put up another option: If you don't mind the implications of networked storage, then here's one database system that has zero-cost cloning and does not lose data on single-node failure.
Separating storage and compute at the database nodes is just one architecture method that allows for simpler and more efficient scaling in modern cloud-based deployments where you have large pools object storage that's fast and close. I think you'd be surprised by what you can achieve with some metadata and caching.
Amazon came out with Redshift, a cloud OLAP database, but it tied compute with data so teams couldn't scale compute and data separately and thus had to pay disproportional costs to their required workloads.
Sure, there's good reasons to keep compute and data together. But there is obviously a massive market for technologies that keep them separate...
Apples and oranges.
> "no. Just no."
Conveys the feeling that there is no scenario where doing this is feasibly and you yourself just said it's acceptable for BI but not AI training.
So the "no. Just no." actually means, "my use case does not allow for this, so I believe no one else should use this" which is a fallacy on its own.
In conclusion, this is fairly usable, but like everything else, it's not for all use cases.
else you incur traffic cost and latency
Thus, they are separated so they can charge you (more) for them separately.