Comment by orf | Hacker News Reader

Comment by orf | Hacker News Reader

I submitted this article because of a single paragraph:

> When it came to the different types of NoSQL databases, it was pretty clear that a document store best suited our needs. We needed to be able to query based on the data so key/value was out, there was no need to have the ability to make all of the complex connections between data of a graph db, and nobody actually knows what a column db is (jk but our data is not the right use case for it). So document it was. We would have to change our data layout a bit to really work well with most document databases, our current data structure was similar to what you would expect for a relational db with tables etc since dynamo also uses a table structure. I wanted something that was ACID, none of that eventual consistency nonsense. That took a lot of them out of the race right there.

I don't get it. He needed arbitrary querying, relations, ACID, tables. And yet... his first thought was to go to a document-based no-sql database and is only later hit with the sudden realization that his relational data is bet suited in a relational database.

What has gone so wrong here that an otherwise smart and educated engineer makes these kinds of fundamental mistakes? Is it the success of no-sql marketing departments, and the apparent un-sexiness of traditional relational databases? Is it education? The job market? A fear of SQL (that you never really have to write anyway...)? Is it the quick-fix non-relational databases give you?

As far as I can see there are very very very few nails that a PostgreSQL hammer cannot hit. I guess that's part of the point of the article, but why does it need an article to explain this?

How can we stop these kinds of articles from needing to be written?

> The change drastically increased the speed of our API, some requests are up to 6x faster than they were with Dynamo. The account details request, our previously slowest call, took around 12 seconds with Dynamo and was reduced to 2 seconds with Postgres!

sigh

Edit: Not sure why this has been moved to the bottom of the thread. I wasn't trying to single this particular developer out, I was merely commenting on this troubling trend. Apologies if this came across in the wrong way.

"I’m not a database guy, I’m a node guy."

This school of thought that a SWE can know the entire stack, infra to app, is coming home to roost. You are the technical debt.

Learn your fundementals, and if a relational database isn't part of your foundational knowledge, learn it now or get out.

I get what you mean, but to me that's the same as saying "I'm not a HTTP guy, I'm a web guy".

I mean, sure, he may be a "node guy", but he uses node to make web applications. A large part of that is persistence, i.e databases. For a lot of applications it's one of the most important part of your stack, if not the most important part. It's your treasure trove, it's your crown jewels. Your app is just a thing that shuffles state around in the database.

He knew a fair bit about NoSQL databases, but why did he not even consider a relational one? I'm not trying to single this specific guy out, merely asking what people think is causing the trend of engineers over-using non-rel databases for relational data.

Edit: didn't see your edit. you're spot on.

I spend a lot of time, as a consultant, talking to people who've already made technical decisions that are biting them in the ass and it seems to mostly boil down to good press and an impressive of the five-minute demo. The mental effort required to learn new stuff is scary and always looks worse than it really is; easy demos hook novice and many-years-a-novice developers to the point where, well, of course it seems obvious not to use RDBMSes, they're hard and you might have to spend time on learning them and you might even look silly or dumb (gasp!) when you're doing it.

From the outside, Postgres looks hard. It isn't hard, but it looks it, because it was not designed around a five-minute demo. I use it for everything until I have a very compelling reason not to. But I have an established library of scripts and systems to make it easy, because I am both an infrastructure guy and a software developer so the idea of "I'm going to write a Vagrantfile in two minutes flat and drop in this provisioner" is something that's top-of-mind for me. (Or I just steal the Vagrantfile from another project, whatever.) But I've made that investment and it's paying off--if one hasn't made that investment, it's scary And then doing it in production? Whether it's a hosted service like RDS or doing it yourself, you have to answer a lot of capital-Q Questions along the way and that's effort that seems unrelated to actually getting your software done.

It's not just "why didn't this person think about a RDBMS?". It's that all of these easy-answer-but-not-really systems (fill in your own) are capturing the minds of developers. The quest for abstractions and ways to make ourselves better able to do our jobs is a good one. The uncritical adoption of tools that might not actually do that is not, but that's a cultural thing not limited to datastores.

Same here. We have several clients who got moved to Postgres from a randomly selected unfit technology. Funny thing is with things like Solr you can build a really fast search index as well on the top of doing ACID transactions in the db. This covers almost every aspect of data engineering. There is obviously the scaling limit with Postgres but most of our clients are in the single db instance range by the size of their data.

Yup yup. Our consistent advice to clients:

- By the time you reach the limits of a Postgres setup, you will either be making so much money that porting it is a couch-cushions deal or your company is a smoking crater in a hillside.

- SPAs are fine. SPAs plus server rendering is often a pain in the rear. Walk, then run.

- If you can't express concretely why you want a container fleet, you don't want a container fleet. Your cloud provider's autoscaling will suffice. (This advice is different for non-cloud clients, but the overhead of leaving space in a container fleet for new deploys or for scaling is deadweight and turning around a container in thirty seconds is probably not as important as the blogs you read tell you it is.)

> If you can't express concretely why you want a container fleet, you don't want a container fleet.

I think this applies to most relatively new (hype cycle) technologies.

It does, but relatively few of those technologies have the desperate money being shoveled into their marketing in the way that containers in particular (and non-RDBMS datastores, to a lesser extent) have. So those tend to get called out explicitly because no, unless you are profoundly out of the ordinary, you don't need a hot new Kubernetes cluster on day one.

Exactly. I have seen this container hype exploding in recent years. A quick example: I worked for a stratup who spends ~60% of their AWS budget of operating Kubernetes and roughly ~40% on their own services. I asked their chief IT guy why is that and he could not justify it with anything other than Kubernetes is cool.

This radiates truth. I would like to read more of the things you write.

I certainly drop enough stuff around here. But maybe I should get back into blogging. ;)

Although my business partner and I have talked about a "fundamentals of cloud system design" course on Udemy or something...

NodeJS and MongoDB is frequently used together and I think some fraction of the NodeJS developers thinks that NoSQL is just the better, newer, faster solution to data storage. I regurarly run into customers who hire us to make their data model work and some non-trivial cases end up being moving them to PostgreSQL. Data engineering actually is a more complex topic than some people think and it really bites you when you are actually using the solution for a while.

He likely has no formal training in database and therefore are not exposed to things alike.

Side topic: AWS did agressive move to throw it's resources to University, and pays off hansomely. Those new blood coming to our profession now has the concept of infrastructure ~= AWS (mileage varies between people, but that's the idea).

It's way too easy for anyone to pick up Node and CRUD a few pieces of data into a NOSQL store and voila, I am an architect :)

Selecting the right framework, platform, model for current needs with reasonable forward looking support takes experience and strong formal fundamentals.

Unfortunately job ads imply than many places prefer the former.

Often people don't want to do database admin stuff, dynamodb has none of that.

On the bright side, relational is coming back with RDS, cloud-sql, spanner and cockroachdb. With any luck NoSQL will be forgotten :)

Keep in mind that the Postgres hammer is a non-standard hammer. Sure it is pretty SQL compliant (except there are no dirty reads thanks to MVCC), but if you use JSON or the arrays or the many other wonderful features of Postgres, you are now tied to Postgres. You are also slowly inching your way into Document Stores.

The biggest pro to Dynamo of Postgres in an AWS world is just price. It's very attractive to try to shove a square peg in a round hole to save money. Of course, you end up costing yourself far more in the long run if it's such a bad fit.

The same can be said about any database though. No one is like another.

SQL as a standard is roughly followed, and PostgreSQL is very compliant. That's more than can be said about any non-relational database - each has it's own query language.

I use JSON fields and arrays quite a bit, but that doesn't mean I need a document store. It means I have small portions of data that is non-relational, or is better suited to use arrays/json than strict tables, but I still want everything else consistent, enforced, and also have all the other bells and whistles Postgres gives you (FTS, partial indexes, rich types everywhere, ACID, the list goes on and on).

Being tied to Postgres is often better than being tied to a document store. At least you can dip your toes into non-rel as little or much as you like (where it makes sense), rather than diving in at the deep end and realizing you can't swim.

What about ArangoDB? You can perform joins, even across the cluster. It's a document store. You don't have transactions across the cluster (yet, I'm pretty sure I'm right). You get sharding and distribution automatically. Again, at this point you're out of the SQL world, why not go fully out?

I can't think of a datastore whose purely SQL-compliant feature set is worth using; to that end, using any RDBMS with a reasonable feature set is going to be a product lock-in of some kind. And I've been around and babysat a lot of people's Mongo stuff; I'll die on the hill that being tied to Postgres is a hell of a lot better for the 10th, the 50th, and the 98th percentile of developers (and companies) than a MongoDB or the like.

When I design systems I try to stay SQL compliant. This has benefited me in the past: From SQL Server to Oracle. Internally one can use optimizations like indexes unique to a DB, but in the end the client view is entirely only SQL.

Reply on news.ycombinator.com