Hacker News Reader

Comment by Kinrany | Hacker News Reader

This shows that we lack good abstractions over storage.

Cloud providers are (counter to intuition about their lock-in incentives) improving the space of this. Lots of tools now allow configuring storage by just pointing to various cloud stores, most often a S3 compatible api, not exclusively though.

K8s PersistentVolume is another decent shot at storage abstraction, only a bit raw.

Finally, more and more tools expect you to have a Postgres they can plug into as backend.

All above assumes you want to treat the library data as a big unknown blob. Once data start being corrupted and need bespoke repair, things are less fun. Access and retention is another fun rabbit hole.

Data is complicated.

It’s more like data storage needs are not one size fits all so it’s better left to the user who best knows their storage needs.

This is interesting, makes me wonder if a "dockerised" database is something people could use. I mean a database frontend with its own language/protocols/whatever that allows you to define the data structure but leaves the specific storage engine or format as a backend detail that can change from platform to platform.

> I mean a database frontend with its own language/protocols/whatever that allows you to define the data structure but leaves the specific storage engine or format as a backend detail that can change from platform to platform.

That's more or less a description of SQL.

Postgres wire format is indirectly getting there. Plenty of tools use that with wildly different storage engines on the other end.

A clean room implementation would likely yield different results but there appears to be some appetite for a solution.

Nah. It's not that. We lack a concept that can organize storage. Let me illustrate this.

So, until some years ago there was complete nonsense and anarchy in Linux networking management. That is until we got the "ip" program. There's still nonsense and anarchy, because the "ip" program doesn't cover everything, but it's on the right track to organize everything Linux knows about networking under one roof. So, vendors today, like, say, Melanox (i.e. NVidia) choose to interface with "ip" and work with that stack rather than invent their own interfaces.

When it's extendable in predictable and convenient ways, user will extend and enrich functionality.

Now, compare this to Linux storage... I want to scream and kill somebody every time I have to deal with any aspect of it because of how poorly mismanaged it is. There's no uniformity, plenty of standards where at most one is necessary, duplication upon duplication, layers... well, forget layers. Like, say, you wanted a RAID0, well, you have MD RAIDs, you have LVM RAIDs, you have ZFS RAIDs, you have multipassing with DM (is that a RAID, well sorta' depends on what you expected...) also, well, Ceph RDB are also kind of like RAIDs, DRBD can also sort of be like a RAID...

Do you maybe also want snapshots? How about encryption? -- Every solution will end up so particularly tailored to the needs of your organization that even an experienced admin in this very area your org is specializing will have no clue what's going on with your storage.

Needs can be studied, understood, catalogued, rolled into some sort of a hierarchy or some other structure amenable to management. We haven't solved this problem. But we have an even bigger one: no coordination and no desire to coordinate even within Linux core components, forget third-party vendors.

You cannot abstract away a 3 order of magnitude difference in bandwidth and latency.

Where did you get a 3 order of magnitude difference? Are you still using hard drives for your storage medium?

Adding two numbers together takes on the order of a nanosecond. Doing the same thing using a rest / http service (like an idiot) in the same datacenter takes on the order of a millisecond. Six orders of magnitude actually.

I'm pretty sure the parent comment was about storage media, not about network hops and service boundaries. Also, extremely basic REST/HTTP services definitely do not take 1 millisecond even if you are bad at software - the overhead of that stack is in the tens of microseconds if you are doing nothing.

For the comparison being referenced here, if you want to compare RAM, the storage medium that backs compute, to modern persistent storage, here it is:

* 40 GB/s per DIMM vs 5-10 GB/s per NVMe SSD. At most one order of magnitude off, but you can pack enough disks into a computer that the throughput ratio is almost 1:1. AWS EBS is about 1 order of magnitude different here, and that is with network-attached storage.

* 100-200 ns latency (RAM) vs 10-50 us (fast SSD) - about 2 orders of magnitude, but also possible to hide with batching.

The space is complex enough that I wonder if it's possible to make abstractions that aren't horribly leaky.

I blame the SQL "standard". It's a massive, unnecessary abstraction layer that only complicates attempts to build bridges between code and relational databases (which I believe is the most general-purpose paradigm).

Personally, I am working on a modern Python ORM for PostgreSQL and PostgreSQL alone.

Isn't this what SQL is supposed to be? You bring the DBI for your database and plug it into the app. Shame that it doesn't work out so well in practice.

But that and abstraction exist, because SQL exists.

If the library is designed to send sql to another storage library...

Reply on news.ycombinator.com