Hacker News Reader

Comment by listenallyall | Hacker News Reader

>> That ended up fixing the problem

Interesting take, since you're still stuck with "simple reports" that take hours to run. What about writing more efficient queries, adding indexes, normalizing the DB... ?

View on HN

Wouldn't be the first developer to just SELECT * with a thousand joins and then sort data in app(!).

Or even worst/better, thousand of SELECT with joins returning one result and then sort in app.

This sounds like the Hibernate Execution plan I found one time when it wasn't configured properly.

It amazes me the complexity of solutions these days, when 20 years ago almost everything ran in relational databases and query tuning was usually the solution to performance problems.

Because we got rid of DBAs in favor of “big data” developers that never learned much about SQL in the first place.

Because 64gb of ram is really cheap these days.

It no longer makes sense to tune your queries, or to wait weeks\months for the vendor to tune their queries, when you can just slap a few sticks in and call it a day.

Tuning queries still makes plenty of sense. Slow/expensive queries need extra infrastructure like a memory cache server or more app servers to paper over the inefficiency.

That's a bunch of added effort that might as well be spent on understanding your database. Even something simple like using materialized views can significantly increase performance of expensive queries.

If you're like a company I was at before, you'd pay $10k+ for DB consultants to tune some queries and your prod database, and when migrating your DB to new hardware forget to re-tune it and waste the extra 64GB and even SSDs installed. There should still be a bare minimum floor of competence for actually developing against and maintaining databases organizationally whether it's a DBA, better engineers, etc. Throwing hardware at a problem is fine when you're sure that you are actually throwing it in the first place which I have seen surprisingly few places do well.

But we aren't doing something simple like that, we are building monstrosities based around the theory of micro-services in the cloud. Kubernetes. It takes hours to get a development environment put together to try and reproduce / debug a problem. We are adding complexity instead of keeping things simple.

And query tuning isn't that difficult. Spend a few hours on this site and you will be better than 90% of devs out there: https://use-the-index-luke.com/

> we are building monstrosities based around the theory of micro-services in the cloud. Kubernetes

This isn't something I'm doing and none of the people I personally know are doing this (disclaimer: im not in SV or the "startup scene")

> And query tuning isn't that difficult. Spend a few hours on this site and you will be better than 90% of devs out there: https://use-the-index-luke.com/

It used to be that companies had DBAs and could call out their developers\vendors on their sloppy queries. They have been replaced by 64gb of ram

The technology inept organizationally can’t even figure out how to use that 64GB of RAM properly is my point and more RAM won’t fix a full blown table scan on every other query coming in because people don’t have a way to stop queries without limits. However, you can add modern SSDs with millions of IOPS to paper over complete ineptitude at using a free query analyzer in your database. But to myself, inability to bring oneself to use that is about as incompetent as not being able to use a debugger or profiler. I can certainly understand a level of inability when hiring only junior devs or having one’s hands tied when dealing with production systems locked behind compliance and bureaucracy, but all these things are problems that hardware nor more people will fix honestly.

And TBF, I am mostly familiar with the guts of technology laggards and they’re adopting K8S at frighteningly fast rates compared to anything else I’ve seen in almost 2 decades. Very common to see pretty smart people familiar with the latest tech yet lacking some serious holes in fundamentals familiar to those with experience.

So, FWIW. The database I mentioned in my original post had 2TB of RAM. Food for thought!

> What about writing more efficient queries, adding indexes, normalizing the DB... ?