NoSQL Data Modeling Techniques (2012) | Hacker News Reader

NoSQL Data Modeling Techniques (2012)

highlyscalable.wordpress.com

127 pointsagmm13 comments

There has been a fair bit of work since this time on automated data modeling for NoSQL databases given a conceptual model and information on the workload. Some of this work is my own[0], but there has been a lot of other related activity. One thing that I've been happy to see explored more is cost modeling[1] in NoSQL databases as this makes any sort of automated reasoning much easier. I doubt we'll ever get to the point where data modeling by an expert is never needed, but I think there's a lot we can do to make things easier and more performant for novices.

[0] http://ieeexplore.ieee.org/abstract/document/7967690/

[1] https://www.researchgate.net/publication/350366905_A_cost_mo...

To my mind, the point of databases is to provide as accurate a representation of some aspect of external reality as possible. The advantage of graph databases is that they model the world as things that have properties and relationships with other things.

This is closer to the way that humans perceive the world — mapping between whatever aspect of external reality you are interested in and the data model is an order of magnitude easier than with relational databases. Everything is pre-joined — you don’t have to disassemble objects into normalised tables and reassemble them with joins.

In this respect, even the simplest graph database such as Neo4j — which models the world as a bunch of JSON documents, some of which may contain pointers to other JSON documents, is much better than even the fanciest RDBMS. Granted there is no taxonomy or schema, and support for temporality is basic, but it’s easy to produce a much more naturalistic model of the world than will ever be possible if you have to break the things up into relations.

Agreed that graph databases are super useful for a variety of purposes. However, it's usually worth consideration as to whether a whole separate data store is needed for that. Postgres, for instance, has uses that would allow for this work within a RDBMS: https://postgresconf.org/conferences/2020/program/proposals/...

Agreed. Actually even the openlink virtuoso database project is kind of using the RDBMS approach in its underlying implementation. https://en.wikipedia.org/wiki/Virtuoso_Universal_Server

However part of the power of the graph database that utilising a linked dataset such as RDFs is that you will be able to use SPARQL to query the graphs, which is more suitable for sets / graphs.

> This is closer to the way that humans perceive the world — mapping between whatever aspect of external reality you are interested in and the data model is an order of magnitude easier than with relational databases.

One approach to modeling data based on mappings (mathematical functions) is the concept-oriented model [1] implemented in [2]. Its main feature is that it gets rid of joins, groupby and map-reduce by manipulating data using operations with functions (mappings).

> Everything is pre-joined — you don’t have to disassemble objects into normalised tables and reassemble them with joins.

One old related general idea is to assume the existence of universal relation. Such an approach is referred to as the universal relation model (URM) [3, 4].

[1] A. Savinov, Concept-oriented model: Modeling and processing data using functions, Eprint: arXiv:1911.07225 [cs.DB], 2019 https://www.researchgate.net/publication/337336089_Concept-o...

[2] https://github.com/asavinov/prosto Prosto Data Processing Toolkit: No join-groupby, No map-reduce

[3] https://en.wikipedia.org/wiki/Universal_relation_assumption

[4] R. Fagin, A.O. Mendelzon and J.D. Ullman, A Simplified Universal Relation Assumption and Its Properties. ACM Trans. Database Syst., 7(3), 343-360 (1982).

The most important problem that DBMSes are solving is how not to lose data in case of a power outage.

If your computer using non-volatile memory (NVRAM) then database is just a data structure.

The secondary problem is how to organize and manage the data, but that is highly dependent on the workloads in question.

Even with NVRAM, you still want to use an append-only log instead of mutable data structures in order to support streaming replication, incremental backup, concurrent writes, and triggers.

> Even with NVRAM, you still want to use an append-only log instead of mutable data structures

Nothing preventing you from implementing AOL in NVRAM

> to support streaming replication, incremental backup, concurrent writes, and triggers

All distributed systems stuff equally applies to both in-memory and persistent data structures

Wouldn't NVRAM only give you durability (D in ACID)?

I think it might be too reductionist to say a database on NVRAM is just a data structure. Even with NVRAM you still need to maintain atomicity, consistency and isolation.

Agree, but not every DBMS supports transactions, and even when they're supported the application code may avoid using them for various reasons.