Web APIs and the N+1 Problem | Hacker News Reader

Web APIs and the N+1 Problem

infoq.com

14 pointsingve12 comments

Stopped reading (and skimmed the rest) at:

> By moving away from RDBMS we have successfully freed ourselves from the restricting schema-bound limitations of RDBMS and been able to churn code very quickly. This has allowed for higher agility of teams.

The whole article is a fantastic example of why moving away from RDBMS has reduced the agility of teams. They have to bake relationships into their schema (sometimes even duplicating data into multiple different hierarchies), instead of having the flexibility to join tables together in a multitude of patterns on an ad-hoc basis. It makes sense as a performance optimisation if you have a lot of data. But otherwise it's just slowing you down.

Any change to the shape of the data in that database which could be in a backwards compatible way in a "schemaless" database can also be made in a backwards compatible way in a database with a schema (and making the change is literally a 5 minute task - less if you're just messing around in a dev environment). And with a schema you have the added advantage that you'll be notified if you accidentally make non-backwards compatible changes.

See, schemas do reduce "agility" when you write the code that writes to the DB. Without them, you indeed are "able to churn code very quickly". But then the moment comes when you also need write the code that would read from the DB what was written into it... oh.

Indeed, usually when you have an unstructured data problem.. you just haven't thought about the problem hard enough.

More general: When you optimize for one schema of your data, you often de-optimize for other views of the same data. You can try to use views, denormalize, use materialized views, replicate, use document databases, unstructured data, etc., but you always remain in the realm how well the tradeoffs are fit for current and future requirements. You can go with very general approaches, like graph databases, which allow you to retain structure, though at the cost of complexity and less safety. It's still a tradeoff. Schema for graph databases may make sense, but then you need to setup good design and constraints dilligently after all.

What is easy to write now, could be difficult to change or read later. What is easy for one developer, could be confusing to others.

The grass is always greener on the other side, so the lure is to go in circles over time.

This is the bread and butter anyways, as how we use computing always changes. Experts deal in complexity, so others don't have to. But even if we want to, it's hard to escape. Just thinking hard and long about these problems takes time nobody is interested in paying for.

People forget, or don't understand, or don't care about the maintenance. Sure, it's easy to use unstructured data _now_, or in a week. But when the original maintainers have moved on, or not documented how it works, and a year from now when people are making changes, it becomes a slow slog to try to wade through the reality of the data to try to understand how it works and not break anything. Repeat this for every new engineer, for every change that needs to be made, forever. It's a gigantic time sink.

You can write code almost as quickly (like <5 mins slower) with a schema if you're using any kind of migration tool (which probably maybe requires 1 hour of initial setup). When I'm writing code that requires schema changes, I keep an open migration locally, then I can just unapply and reapply it every time I want to make a change.

With MySQL you get quite a few problems when your migration script fails as you can get left in a broken state that you have to sort out manually. But with Postgres schema changes being transactional it's quick and painless.

This article needs to be marked 2014.

No developers in 2021 should be struggling with N+1 or thinking that not enforcing schemas in your DB makes them stop existing. You've just moved schema enforcement to your application, or maybe you've chosen not to do it at all which is even worse for just about anything more than a hackathon project.

I was about to make a comment that this felt like something from 10 years ago, but it turns out it actually was!

This is a strong argument for not using a generic JSON api for your core web application, but rather using server side rendering where you have full access to the data store using whatever the native query language is.

Getting web app UX right typically requires page specific query tuning and introducing those as JSON API end points leads to a gronky, churny API. GraphQL helps in this regard, but that has some pretty serious security implications[1].

I wrote an article on this topic back a while back on the intercooler.js blog:

https://intercoolerjs.org/2016/02/17/api-churn-vs-security.h...

You can and should still create a generic JSON API for server-to-server automations, but it has a different set of design considerations when compared with your web app needs (rate limiting, generic but secure functionality, etc.)

[1] - https://twitter.com/AdamChainz/status/1392162996844212232

I don't disagree in principle, but often you need the API anyway, and it makes little sense to duplicate the effort just for the sake of purity.

If you are in that situation, GraphQL is actually worse, because you end up with less control over the SQL you end up running.

I can see that idea working very well for content-driven, interactive but mostly static websites, something like HN.

Your JSON API, once you pull your web app usage and all the special end points for it out, should be pretty regular and not a ton of code. So the actual amount of duplication cam be low, and your API engineer will be bored and happy. :)

Well the article is from 2014, this should be mentioned in the title. I noticed it because the author was not even talking about the dataloader pattern used often in the context of GraphQL when asking for new patterns.

In Node.JS the dataloader processes N lookups simultaneously in the next tick of the eventloop. So call(1) call(2)...call(n) would be executed as call([1,2,3,..n]) allowing you to efficiently implement it.

Reply on news.ycombinator.com