No More N+1. Period

The problem that I’ve seen a bunch of times with the similar lazy-autoload approach in .NET Entity Framework is that someone accidentally/naively serializes one of the EF objects rather than projecting it to a POCO first, and then Prod blows up when the serializer dutifully follows all the relationships in a complex nested object and ends up pulling a massive amount of data from the DB and then tries to log/send 1GB objects at N requests per second. Oops

This looks pretty great and I'd definitely consider adding it to new projects, but it's not a magic bullet. You still need to think about what queries this will trigger and it's a lot less obvious than includes().

If you have a large application with an ORM like Active Record, you'll often end up in situations where queries inside methods trigger the N+1 problem. Sometimes this loading is highly conditional. You end up trading the N+1 problem for a "loading an entire unnecessary table" problem.

I suspect this is the reason this feature hasn't been added to Active Record. It would be used a lot when it's only narrowly useful and has worse side effects. (It's also yet one more layer of complexity on top of an already complex ORM.)

Glad I immediately saw the gif-memes and could close the article within seconds.

Note: If you're using SQLite as your production database, N+1 queries are generally not a problem [1]. This is because queries are just a function call and don't involve any messaging round trip to a server (i.e., they're very low latency since there is no server). This makes most N+1 queries so fast, there isn't a difference worth noticing. The only caveat is if you're pulling large amounts of data (e.g., binary blobs) off the disk or something really big/complicated in your loop; otherwise it typically shouldn't be of any concern.

[1] https://sqlite.org/np1queryprob.html

related: auto prefetch fk values in django - https://github.com/tolomea/django-auto-prefetch

Thank you very much for your comments! I appreciate that!

So far, I see that N+1 is a "measurable" problem that doesn't have to be removed with an additional layer of hidden complexity.

my opinion, worth what you paid for it:

you shouldn't worry about n+1 issues until you have a verified perf issue in your app

if you have paging and N is relatively small, the additional complexity may not be worth it (although ActiveRecord makes it very easy to tweak locally as needed, which is the right thing)

what is important in my opinion is a server side framework is logging of perf related details out of the box:

- show how long each request takes and flag slow ones - show what queries occur in each request

The more humane formatting of these logs, the better...

I think datomic is the only DB I'm aware of that can actually make the promises in the title.

Why would you spend your time dealing with annoying N+1 problems when you can enable auto-preloading and forget about it for good? Are you surprised it's possible? Check it out!