Hacker News Reader

Comment by didgetmaster | Hacker News Reader

As someone who thrives on tracking down rare but annoying bugs in a debugger, I love stories like this. It is not just bugs that cause real failures which can be headaches; but also bugs that just slow things down unexpectantly. They can sometimes go undetected for decades like this one.

I wrote an article this past year that talks about silent bugs that slowly eat resources and collectively can be very expensive in terms of wasted time and energy: https://didgets.substack.com/p/finding-and-fixing-a-billion-...

View on HN

> but also bugs that just slow things down unexpectantly. They can sometimes go undetected for decades like this one.

Reminds me of the GTA Online quadratic time JSON parsing bug

Okay but where's the bug story? Did I miss the story?

I wrote the article right after I fixed a huge inefficiency problem in a function within my own project. I neglected to give the specifics in the article, but here they are since you asked.

My Didgets tool lets you create pivot tables against relational database tables, even very large ones. For the pivot values, you can choose to just count the occurrence of each value or if it is a number type you can add them up. You can also add up the values in a separate number column. Here is a quick demo video: https://www.youtube.com/watch?v=2ScBd-71OLQ

When adding up numbers in a separate column, I had just a few lines of unnecessary code that ended up being called exponentially. For smaller tables it was barely noticeable, but for tables with 30 million+ rows it really bogged down.

A simple fix to the affected lines caused a certain test against a large table to go from over 10 minutes down to under 20 seconds. The effects of just a few lines of code when applied to a big enough data set can really impact performance. It is the old Einstein equation E=mc2 in effect which is discussed here: https://didgets.substack.com/p/musings-from-an-old-programme...

I guess there is a lost art of writing for optimal code/memory/execution time, especially as our resources increase.

I think the idea here is to write code quickly that's inefficient, and re-write it to be efficient if the performance is required down the line. For companies where there's bigger fish to fry, i.e. customer acquisition, it's more useful to pump out more features (even at the expense of bugs) because that draws customers.

But in places where performance is important, you do see developers squeeze out more cycles/memory. I.e. kernel/OS development, database servers, video games. It's just that most developers aren't in those areas of specialty anymore.

Btw, have you heard of https://handmade.network/ and https://en.wikipedia.org/wiki/Demoscene ? Wondering what your thoughts are in those areas. There are probably more communities like the ones I mentioned, where developers are interested in writing the kind of code that you are talking about.

> As someone who thrives on tracking down rare but annoying bugs in a debugger,

As someone that is cursed to inevitably find some obscure bug the second I start using some piece of software I'm happy I'm not the only one

> I wrote an article this past year that talks about silent bugs that slowly eat resources and collectively can be very expensive in terms of wasted time and energy

"Using JS for backend is ecoterrorism" lmao

Even for frontend, repurposing on the frontend stuff that were done in the past on the backend can be ecoterrorism squared.

For instance, a small team of 40 people found no issue sending 4MB of json english to chinese string localisation to each website visitor for angular to translate. 1 million visitor a month in Hong Kong alone, 4 million MB + a few second of mapping per user per month completely pointless...

Reply on news.ycombinator.com