Hacker News Reader

Comment by mozumder | Hacker News Reader

Its always amazing to see web pages take 2-4 seconds of back-end processing. On a modern CPU, that's about 10 billion instructions.

10 billion instructions to send a few kilobytes of data.

There's such a waste in back-end server design. If you're measuring response times are in seconds, and not in microseconds, you're doing something seriously wrong.

View on HN

And that's exactly how client-side rendering became popular again.

"It's taking too long on the server, let's offload everything to every single client out there."

Now we have two problems.

In my experience this seems to be a problem of people using frameworks without getting to understand them in any depth (probably because they are off chasing the next hyped up thing). Django is easy to get started with. Its also easy to produce a shitload of queries unless you spend some time getting to understand the ORM and how to optimize queries with it.

In my previous work we were generating over a 1000 queries for the front page (we were showing maybe 20-25 products with different options). Everything was done in nested loops, with new queries to the database each iteration.

The kid who had written that was apparently building his own framework when I looked up his webpage. Learn to use Django properly before you go building your own crap versions please.

Modern frontends take even longer to completely load.

Too much CPU and too much available memory will get you there. When in time of abundance, we humans are not good at optimizing.

It's more of an economics issue. It does not make sense to optimize, because bandwidth and CPU power are so abundant. Optimizing will take time away from adding new features.

Except you can’t add so many features because you have sucked up all resources already ^^

Failure to optimize in these situations creates a performance bug.

From my experience, most of my waiting time is IO, not CPU load. Even simple stuff, such as SELECT COUNT(*) to display total count in grid on big table can take seconds. It does not have to be complicated.

You may be confused about how databases work. COUNT(*) is not "simple stuff". It almost always involves an index _scan_. For tables with millions of rows, it takes a while.

You have a problem. And it’s probably related to your db architecture, which was probably designed by devs with ORMs.

What kind of database architecture could cause SELECT * to be slow?

Any. "Architecture" has nothing to do with that. SELECT COUNT(*) is usually "slow".

Makes sense: devs with ORMs don't know about indexes, which are proper DB architecture.

>Even simple stuff, such as SELECT COUNT(*) edit seems like my old performance book was outdated or wrong :(

SELECT COUNT(*) should be the one which is resolved by index/metadata

Nope. Not even close.

on the other hand there are savings in development time, which might or might not be more expensive than cpu cycles in question. Especially for smaller scale projects. I'm not saying that 4 seconds to load a page is something to be happy about, but sometimes it's cheaper to "waste" cpu cycles than to spend more time developing faster solution.

Although its really difficult to fathom how loading a webpage takes billions of cycles anyhow. It never used to. Remember the Internet in 2000? Pages are getting slower despite broadband speeds going through the roof and processors getting faster and there being so much more RAM etc.

Yep, I remember trying to optimize my 1-pixel spacer GIF from 53 bytes down to 47 bytes to lighten up the load of a 200KB page of random interwoven table, font, and center tags that took 30-60 seconds to download.

It only saves development time for the first version in my experience unless it's being used by an highly capable team.

The months lost to the clueless developers massaging their horrible ORM code dominates in time.

In the long term properly factored code wins; and lightweight ORMs have a place. Heavy ORMs (Hibernate, old Entity Framework) should only be used if you absolutely must have specific features and are incapable of building that into a more logical place in your code.

Use a light weight ORM for simple CRUD actions (one with a small dependency graph). For performance critical create/update operations use stored procedures (just be careful with complex business logic; if you implement it, remember to test it). Use light weight ORM for mapping efficient SQL views to api/frontend. Use a workflow/async/job processing for anything which has to be long running or crosses system bounds (think credit card processing).

you could start with an EASY query that's guaranteed to work, then AI could figure out a fast way as your app keeps running. it could be a module for postgresql for example.

This. I am building an app and the listing screen is taking 4-5 seconds to fetch and display the database records. Why? The SQL query is complicated (7 joins due to filtering and permission restrictions), and on its own takes 1.7s to run. The ORM to enable pagination [0] runs this query twice plus a third query, adding up to over 3s. Plus other queries and rendering output (all times in debug mode).

Does it matter? No - this screen will be accessed 30 times a day max. Better to have it work inefficiently and spend time on other tasks, than make it efficient - however much I want to.

0. https://www.doctrine-project.org/projects/doctrine-orm/en/2....

You are being downvoted, but it seems perfectly acceptable to many businesses. I wanted to improve the front page on my previous page (generating over 100 queries, taking between 1 and 4 seconds to load). In the end every other crappy new feature was given a higher priority.

Reply on news.ycombinator.com