Hacker News Reader

Comment by holtalanm | Hacker News Reader

One thing that is bugging me is that, while the article mentions DataLoader, it kind of just breezes over it without explaining the reasons why DataLoader isn't a good long-term solution.

> Besides the popular Dataloader approach, another very common way of tackling this problem is by taking a GraphQL query, and coming up with the exact SQL needed to resolve it

But why? Why not just use DataLoader? Is there something intrinsically wrong with it that makes it a bad solution? I want to know! My understanding was the DataLoader was extremely performant and worked for 90% of all situations where a join is needed to get values for a GraphQL query.

> Data fetching remains something to improve

Certainly this might be the case, but the article didn't address the most popular form of data fetching and explain why a graphql-to-sql compiler would be a better solution.

View on HN

I think you might’ve misinterpreted the article. Whilst it does gloss over the Dataloader approach, it doesn’t dismiss it. The conclusion ends up essentially advocating for it.

The Dataloader approach gives you a potentially (arguably likely) suboptimal baseline performance for simple projects, or ones where tight coupling with the database is acceptable. But it’s one that tends to scale better and more predictably with complexity (both codebase size and architecture). A hybrid model where you use Dataloader but include some database query optimisations driven by your selection sets is likely a winning approach, but will probably need to be fairly bespoke to your architecture.

We’ve been using data loader a lot and it’s working ok. We were very careful to dump all sql queries our app was making per request to ensure we were not making more queries than necessary.

One thing I will say as we do make a lot of “IN” queries as a result. You fetch a bunch of rows with a data loader and then you want to fetch a bunch of rows from some other table based on those first results so you end up with an IN query where you might have had crafted a JOIN if you were writing it by hand.

But does the query plan perform a join? The text of your query matters less than the plan it generates.

No, it can't because it's actually making two separate queries. For example, let's say we're fetching classified ads with the users that posted them. First your GraphQL resolvers would fetch the ads (and you won't know that the users are being asked for). Then for each ad, you'd tell your data loader to fetch the user associated with that add. That will ultimately result in a single SQL query with an IN like `SELECT * FROM users WHERE id IN (?)` where you'd pass in all the user IDs you gathered from the results of the ads query.

However, another commenter pointed out above that some implementations allow peeking into the GraphQL query. That way if you know you're going to need the user anyway, you can already perform the join proactively when you fetch the ads.

In a few cases we've done this because we always want the joined results so we just do the join regardless of the query we get.

Erm, are you telling me there is no way to pipeline the ads query to be inline within the IN clause? Because that'd generate a join, generally.

I still don't understand why this abstraction is useful, if not harmful.

I think there is a way to do it, that was the peeking I referred to. But it's not how it works out of the box.

I think the abstraction is somewhat useful if your GraphQL queries vary tremendously (one device asks for the user along with the ads, another device doesn't). Expand this out to a bunch of different data types and you'd find yourself writing a lot of variations of the queries. GraphQL with Data loader allows you to write these things exactly once albeit with potentially suboptimal queries.

All of that being said, we're finding that our queries vary very little and are somewhat questioning the value of data loaders and of GraphQL altogether in our case.

> One thing I will say as we do make a lot of “IN” queries as a result. You fetch a bunch of rows with a data loader and then you want to fetch a bunch of rows from some other table based on those first results so you end up with an IN query where you might have had crafted a JOIN if you were writing it by hand.

I don't know if it's common across other GraphQL implementations, but the graphql-java library lets you peek [0] into the selections so you can opt to limit data you select, execute a join instead of using an IN query, etc.

I've used this with good results when making a prototype graphql interface to our billing software, anything that had a 1:1 relationship with a parent record would get JOINed by the resolver, while 1:N or N:M relationships would execute separate queries.

[0]:https://www.graphql-java.com/documentation/v14/fieldselectio...

  it kind of just breezes over it without explaining the reasons why DataLoader isn't a good long-term solution

nit: I don't think the author argues that dataloader isn't a good long term solution.

If I had to guess, because it's something that you just turn on in your project and it just works -or doesn't-.

I've used it in a project together with graphql-sequelize, which gives you "automatic" resolvers for your queries. Turn dataloader on, perform a query that uses any of those automatic resolvers and all underlying SQL queries to the same table are magically merged together. Now, replace those automatic resolvers with custom ones and you lose this magic. Now you have to write your own ORM query that groups anything that would end up being multiple queries into one, basically the case the author is pointing out.

Great question! I want to know the answer too (hence the comment not just the +1).

Reply on news.ycombinator.com