Hacker News Reader

Comment by stephenr | Hacker News Reader

> NoSQL came into existence because the databases at the time couldn’t handle the scale required.

Arguably the first "NoSQL" database was a DIT, generally accessed via the LDAP protocol. OpenLDAP is one of the better known open source instances, but Novell eDirectory was there in the early part of this century, as was MS AD.

In the case of Directory Services, it's a completely different approach to both SQL and the common "Document Store" approach of modern "NoSQL" options. Unless you go cowboy-mode, you're still adhering to schemas (albeit with the option to combine schemas), you still use indexed attributes to conduct searches, but you have a literal tree of objects with a more flexible layout (e.g. multi-value attributes without needing to use a JSON column) than with a regular SQL RDBMS.

The rise in popularity of 'modern' NoSQL options (most commonly document stores) is IMO driven by a combination of lazy developers and too much Kool-aid consumption.

View on HN

There were NoSQL databases well before LDAP was a thing. We called them "Object databases". The idea was that doing object relational mapping was a PITA and wouldn't it be better to store your domain objects directly in the database? (As an aside, in case you are curious, the answer is: No. Generally it's not a good idea) Of course if your data is not relational, you can't do SQL. :-)

> Of course if your data is not relational, you can't do SQL.

The relational model is, as I understand it, fully general, so this literally cannot be the case. There is a problem of “if you do not know the shape of the data in advance” and “if the RDBMSs available to you do not efficiently handle data of the particular shape and access pattern you are using”, but “relational” isn't an inherent property of data, it's a property of a data model, and any data that can be modelled can be modelled relationally. That doesn't mean it is efficiently handled by every, or any, existing RDBMS, though.

There are plenty of models that don't work well modelled relationally. Few examples here:

(1) Wide table with 100K analytical features keyed against a customer. Now if you need to combine a few dozen of these features for predictive purposes then how do you do that in SQL ? You can't. Since every SQL database has a tiny column limit. In Cassandra that is a trivial O(1) fetch.

(2) Time series data can be many orders of magnitude faster in JSON document stores where you can have embedded/nested records.

(3) Star schema with a fact table and thousands of dimensions. In a SQL database that is thousand joins. In a JSON document store those dimensions can be embedded as well as made available as relations and again that is an O(1) fetch.

> There are plenty of models that don't work well modelled relationally.

There are plenty of data shapes and access patterns for which existing RDBMS implementations are inefficient, as I acknowledged. That's not a feature of whether the data is or is not relational—“relational” is a feature of a data model, and any data can be modelled relationally—but rather of the available RDBMS implementations.

> Wide table with 100K analytical features keyed against a customer. Now if you need to combine a few dozen of these features for predictive purposes then how do you do that in SQL ?

In terms of both the relational model and the SQL language, this is trivial. In most concrete SQL implementations, it is both inefficient and requires a concrete schema that diverges from the abstract relational model of the data, because you need multiple tables with a common key rather than a single table, even if abstractly it is a single relation in a normalized model. There's nothing fundamental about either SQL or the relational model that requires this, its just not the usage pattern common RDBMS storage implementations are optimized for.

> Time series data can be many orders of magnitude faster in JSON document stores where you can have embedded/nested records.

Yes, that's an efficiency of access patterns issue, not an “is the data relational” issue.

Of course, outside the strict relational model of data, SQL based object-relational databases like Oracle and Postgres have supported embedded and nested records (via the combination of composite and collection data types) since before “JSON object stores” existed.

I think you misunderstood the parents use of the term "model". All the things you describe here can be described in the relational model. The parent was arguing that the model is general enough to cover almost all collections of data. You appear to be arguing that for specific use cases specific implementations perform badly. These arguments pass each other unseen like ships in the night.

Actually wide tables for example can't be done at all.

And technically we can use Microsoft Excel as a photo editor. But that's just ridiculous. Just like it's ridiculous to say you can use relational models for any data structure even if the queries will never return.

You are still missing the point. The relational model is not about particular implementations. There is no limit on number of columns etc. It is not "ridiculous to say you can use relational models for any data structure", it is the whole point. That particular implementations may not suite your requirements is not a problem of the model, it is a problem of the particular implementations.

There may be other limitations, but MSSQL supports[1]: Columns per nonwide table 1,024 Columns per wide table 30,000

I think a lot of this RDBMs can't do X, came from using a MySQL. It has improved some, but was garbage until recently.

I'm not sure how wide you're taking about, but PG support up to 1600 depending on the datatypes (there is a page size maximum). Oracle supports 1000 IIRC.

[1] https://docs.microsoft.com/en-us/sql/sql-server/maximum-capa...

As I recall, MonetDB claims "practically unlimited" number of columns per table. And in this particular use case you're probably looking for an RDBMS specialized in analytics, not a general-purpose one like Postgres. But relational model will handle it certainly better than any other.

You can do all of that with a relational database. Whether or not your specific relational database makes it easy, or efficient is a different question from "can it be modeled relationally".

Memories of POET and Gemstone spring to mind...

I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by developers that don't actually need it, but want to be like google and don't want to take the time to properly learn and optimize sql.

There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually consistent" or nosql. The simpler datamodels were probably easier to create and solved the problem, now I think there'll be a rise of scalable and consistent databases, best of both worlds.

It wasn't just devs that didn't know how to properly optimize SQL. Programming w a NoSQL db was a better experience because there was no object impedance mismatch.

ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer.

I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I moved to Elasticsearch and CouchDB awhile ago. I'll still use those where appropriate but now in MariaDB for most. The competition was good.

I've seen some of those applications. It might be easier & faster to start with a schema-less json database, but I saw some serious maintenance issues with applications like that:

* Using the stored data outside the use cases imagined by the original developer is harder than it should be. (Oh you want a customer dashboard with this and this data correlated, ..)

* "Migrations" can be rather error-prone if developers don't pay attention. (It was stored as a string first, and now it's an integer, now how do we query that). Just because there's no database schema, doesn't mean there's no schema.

* It's hard to see what's actually in a json table/collection. You'd need to browse the code, see how it's used, and then browse through the available data.

Some apps aren't subject to new use-cases and some aren't meant to live that long or have another developer either. And SQL isn't great for the use-cases that solutions like Elasticsearch solves.

Migrations are less of an issue if you use an object document mapper which is just a leaner ORM. You can enforce your "schema" in your code while being schema-less. But yes if the type changes it could alter your queryability or you might have to write a script to update all records to reflect the new type. That can be a pain of course.

Viewing JSON collections in NoSQL is nice compared to SQL. Most your data is already composed as a sensible document and if you have subcollections the clients usually allow you to fold them or expand and show all. That's a win over SQL IMO.

Regarding JSON columns in SQL - I tend to store config/settings there and it's better than what it used to be for that stuff (serialized data or base64 encoded). In Navicat JSON columns are viewable in single record mode but pretty print format would be ideal.

Not sure what you are talking about here.

There have been scalable and strongly consistent databases since the invention of the concept of NoSQL i.e. HBase and Cassandra (CL=ALL).

And the idea that "learning and optimize SQL" would instantly change people's rationale for using these databases shows you don't understand them much at all. There are many factors that come into play. For example you can't use SQL databases for large scale feature engineering.

To me “large scale feature engineering” is precisely the kind of problem SQL excel at.

Is it something that makesme grab for SQL it is the need to do adhoc querying exploring data.

Care to elaborate?

He's probably refering to a combination of features that "nosql" databases have.

* Schema-on-read: Makes it easier to ingest large amounts of data, and then do ad-hoc exploration. The schema is only determined when reading the data, which is a bit easier for one-off data exploration, you determine how to interpret the data when actually using it. Not appropriate for production systems though. For example, a customer gives you a few TBs of data, you dump it on hadoop, and query it with spark. It would slow you down if you first have to convert it to a relational schema. Again, only good for one-off stuff.

* Most SQL databases have column limits, so if you have a very large amount of features, I'd imagine you'd run into these limits.

* Scalability. Feature engineering is very parallelizable, most normal SQL databases (excluding stuff like cassandra) aren't trivial to scale.

Cassandra is AP (available and partition tolerant), with tunable consistency. HBase is CP (consistent and partition tolerant, but not always available in case of network partitions).

Spanner is consistent, available and partition tolerant due to the extremely precise clock synchronization that google achieved on their hardware and network, an innovation I hope will spread. So yes, that's new.

I don't think I claimed sql was the only solution that ever made sense. I'm sure for some use cases, schema-on-read or no schema is appropriate. But most of those usecases are rather niche. NoSQL became popular for (web)application software, which in my opinion isn't an appropriate use case for most of these applications.

Edit: I misunderstood. Spanner is CP, but google claims here: https://storage.googleapis.com/pub-tools-public-publication-... that it's also practically CA, which is technically incorrect, and something I misunderstood before.

> Spanner is consistent, available and partition tolerant due to the extremely precise clock synchronization...

This is nonsensical; even in a universe without relativity and with instantaneous communication--an unrealistic world where being able to talk about synchronized clocks can even make sense (as otherwise you fundamentally must deal with an inability to decide the order of two events due to latency)--you still cause indeterminate simultaneity by introducing a partition: your perfectly accurate and infinitely precise timestamps on one side of the partition would no longer be comparable with timestamps on the other side of the partition for purposes of establishing causality between events, as the recent changes being made on one side of the partition "should have" taken place between any older changes on the other side and any transactions being performed right now, and yet they couldn't because they were unknown. Spammer either gives up availability (waiting to verify what happened) or consistency (accepting that it will have to fix the result later) when a partition occurs: they can't do both, fundamentally; the only extent to which there is an engineering challenge here is "maybe you can minimize the number of partitions you have", which I totally believe Google to have done; but, given a partition (which cannot be fully prevented), the tradeoff is clear and unavoidable.

Spanner is CP I think the A part comes from reliability of google network infrastructure which is not part of DB system's design per se.

I think I misunderstood that yes.

https://storage.googleapis.com/pub-tools-public-publication-....

TLDR: It's technically CP, but google claims partitions are so rare people can assume it's CA as well (99.999% available).

I assumed TrueTime bypassed the CAP theorem, but apparently that's marketing bs. It's to ensure something called external consistency, which is important if you want to take consistent snapshots over a distributed system.

Semi-related soapbox: Can we all please stop saying "CAP theorem" and instead say "CAP rule"? The formulation of this supposed theorem is so vague that it can neither be proven nor disproven.

It was proven in 2002, has a formal definition, so I'm going with calling it a theorem. Maybe this'll help: https://mwhittaker.github.io/blog/an_illustrated_proof_of_th... ?

>don't want to take the time to properly learn and optimize sql.

This sounds like an old-wives tail at this point.

> This sounds like an old-wives tail at this point.

Except it's not. I've seen it in person, over and over. I've gone into systems where people were complaining about the 'database is slow', but there were no indexes. I've seen systems pull back 100s of thousands of records and then sort on app server and take the first 50. I've seen what would have been a simple join with exists clause turned into many round trips to the db with loops and app code complexities where a query would have 100x easier to reason about.

It's not even about optimization yet, but taking the time to learn even the surface capabilities of the tool.

I've met developers like that.

Although honestly, sql is more than 40 years old and can be really nasty. Stored procedures, each database has a different dialect, nullability comparisons, CTEs. It's very different from any other programming language. It's not that hard to imagine people finding it hard to learn, especially front-end web developers who'd rather not touch the backend. If people don't know how indexes work, then it'll perform really badly as well, which would make databases like mongodb all the more attractive.

Not all developers are formally schooled, a lot of people don't know what a relational database model is.

I've seen people:

* Iterate over entire tables to get to a small number of records

* Not using joins, having the ORM "magically" handle everything (lazily getting non-joined values in separate queries, resulting in thousands of queries)

* Denormalize everything, and then wonder why things are inconsistent and hard to query

It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. It's a lot harder to write "nasty" sql than "nasty" js, and if you're a front-end dev struggling with sql, then you probably should not be doing back end work. Yes not all databases are equal, thats why there are ORMs and ANSI standards.

From the list of bad practices you've "seen" people do in SQL, I would guess their comfort zone code is probably a hot mess as well.

> It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL.

I used to think this because I learned SQL right along with all my other coding. I've realized though it is a mindset shift to go from imperative to declarative, and to think mostly in set operations. That shift can be hard for otherwise good developers.

> It's a lot harder to write "nasty" sql than "nasty" js

That may be true (though, when it comes to dealing with more complex reporting functions and per-database-implementation differences/quirks outside of the realm of ANSI SQL it may not be).

However, it misses the point. Most engineers that struggle with SQL struggle with it because it's hidden from them partially (they're composing queries from snippets/query builders that come from other code, and never get to see the schema directly since it's hidden behind migrators and management interfaces) or completely (ORMs). Because the actual queries being run on actual schemas are less obvious, people do the wrong thing a lot.

That's not to say that abstractions on top of SQL are always bad--perhaps they're over/mis-used, but having worked on massive codebases where every dev's interaction with the database was "write a query in text or with a select().where().from()-type thin wrapper" and codebases that were 100% Django ORM, I can say with confidence that neither approach scales well absent big investments in correctness and RDBMS education.

Just read through these threads. There are people literally saying these things.

So the developers at Google (BigTable), Facebook (Cassandra), Yahoo (HBase), Apple (FoundationDB), Redis etc are "lazy" and have drunk "too much Kool-aid".

Because these technologies every day power the most demanding applications on the planet.

I think the accusation was against the developers who caused the rise in popularity (i.e. the users), not the developers who created the technologies (i.e. the authors).

In that light, it might be best to exclude BigTable, since it's only available publically as a service.

Facebook, Yahoo, Google etc built these databases to use not as coding exercises. So are they lazy and drinking the kool aid for not relying on SQL databases instead of building their own ?

Or maybe SQL isn't perfect for every use case and other developers should stop arrogantly belittling those that choose to architect their own systems a particular way.

I'd say that 99.9% of organisations using any database technology, don't have the scaling problems of Google, Yahoo, Facebook or Apple.

They objectively can't be "drinking the koolaid" or cargoculting something, because they've specifically created their own solutions, to solve their own problems.

I specifically didn't say SQL is perfect - I said that they're very popular for the wrong reasons, and often are used in situations that a relational database would be a better solution.

So it's okay for developers to build their own database for their specific use case. But it's not okay for developers to use a database for their specific use case.

Makes sense.

This is not what he said. He basically said that in many cases, some developpers pushed something like mongo, ES or cassandra in their project or even company when a small sqlite or an effective postgressql were much more appropriate.

Specific databases solve specific problems. In my current company, for log storage and analysis, InfluxDB was our solution after testing both ES and postgress, but we did compare all the products, and we still do as we expand. In my previous company, the main database in the product was Zope. I agree that mongoDB may be better in some use case, but i strongly doubt that most company are running it because it's the most appropriate product.

Again, the accusation isn't against the builders, but against the popularizers (subsquent users).

To stretch the kool aid metaphor, they're the ones mixing the kool aid, not drinking it.

I believe Yahoo is Sherpa not HBase.

>lazy developers and too much Kool-aid consumption.

When your argument is based on personal attacks of the people using that thing, it sounds more like you have an emotional investment in the alternative rather than a rational argument.

It is a very rational argument people want to copy what Big boys are doing often fairly rationally as they want that stuff on the resume. I've seen countless projects using "BIG data" tools for dealing with 100GB or less of data that has very modest grows rate.

Reply on news.ycombinator.com