Hacker News Reader

Comment by dijit | Hacker News Reader

The issue with performance is that junior sysadmins and developers start flipping knobs thinking that the defaults are somehow holding them back.

The truth is usually there are tradeoffs and the defaults fit a broad general case.

If you want throughput there are tunables for that, if you want low latency then usually those are inversely correlated. Same for tuning for low data loss after failure and so on.

You have to spend time learning the tradeoffs, which sysadmins used to do- now nobody has time as they have been munged into one role at many places.

View on HN

This broad case is much broader than the typical servers that Linux is often used for. A simple example is file access times on a (database) server. These are largely unnecessary. It's even rare for a desktop user to actually look at these.

In the past I've been a sr dev (but a jr sysadmin) and was tasked with improving the performance of an upgraded database server. The problem turned out to be with NUMA on the larger server which a combination of reading and semi-random config fiddling of both Linux and MySQL parameters (plus a bit of BIOS/CMOS tweaking) brought up to expected levels.

There's often no better way than learning on the job as there is so much to know that you can't simply learn them up front for when you will need it. What we can do is learn what there is to know and remember to look into those if it seems relevant. I mean everyone who's well experienced now probably started config twiddling somewhere to get there.

> A simple example is file access times on a (database) server.

This isn't a issue if you use `relatime` which is the default.

To give context for those who don't know: relatime is a relatively (I'm a dinosaur) newer introduction into the Linux kernel; mounting partitions noatime used to be the only alternative to the default which was updating the atime on every access, thus fiddling with that setting used to be important. Not any more.

BTW I used to continue to mount everything noatime anyway, since having the atime field be set upon file creation and not anymore afterwards was a way to get file creation time, and I found that more useful than the access time. This isn't necessary either anymore since the introduction of an actual file creation time.

There's a right and a wrong way to learn though. The wrong way, which is the one I most often see is: smash keys until it appears to work w/o every bothering to try to understand the deeper problem or why some particular combination of key smashing appears to work. This becomes cargo culting over time. Stack Overflow and ChatGPT make this oh so much worse. Often the change then get committed with a useless message such as "make thing work" w/o any explanation of what was broken, what the change fixed, and why or how. It becomes instant tech debt.

The right way is to follow the scientific method. Collect data. Make a hypothesis with a plausible mechanism of action. Test the hypothesis. Arrive at a solution. Record how you arrived at your new found knowledge so that those who follow you understand why you made the change. The person who follows you is your future self as often as not.

The truth is that often nobody thinks too much of defaults, unless they are horribly wrong. So there are two good things about defaults: they're probably not horribly wrong and they don't require any additional work.

Some defaults are just historical curiosities, some defaults were configured 20 years ago and nobody took the crusade to update them, some might be bad, but changing them would break too much stuff in the wild.

Now I'm not suggesting that everyone should change everything. I almost never change defaults, myself. But I just don't agree than defaults are good. They're probably not bad, and that's about it.

That may be, but Brendan Gregg is not a junior flipping switches, and a resource like this can help others do smart tuning.

I didnt expect anyone would read my comment that way.

No, Brendan Gregg is certainly no junior. The issue is that people take his advice as law and do not read further.

I, myself, remember flipping random switches because a (book) resource said that it would unlock performance.

I’m suggesting that taking the time to understand performance properly is ideal and I am attempting to urge people to properly invest the time.

Then I am lamenting the fact that we do not actually have time over because there’s so much smushing of responsibilities.

Ah fair enough, sorry if I misunderstood!

> Brendan Gregg is not a junior flipping switches

That's because Brendan Gregg doesn't flip switches, he shouts at hard drives[1]. ;-)

[1] https://www.youtube.com/watch?v=tDacjrSCeq4

I do the same thing to my kids to change their performance.

Does it work?

Not GP, but no, it doesn't. It appears to work but then side effects get much worse. Taking the time to understand and to tweak the right settings, I mean, explain, is much better in the long run.

though i dont know much about it, i run linux and most of the documents i hear about tuning edge cases are now either avoided or boilerplated with a strong word of caution. For example: 10Gb ethernet used to require a litany of sysctl.conf chicanery to even approach half the line speed. not the case anymore, and most of the old kernel 2.4 optimisations are either nonsensical in 2023 or actively worsen the performance of the interface.

> The truth is usually there are tradeoffs and the defaults fit a broad general case.

Part of the problem is that reasonable defaults for performance is a somewhat new phenomenon. It used to be that the defaults for Linux kernel settings, Apache, MySQL, etc... were terrible for production use.

So there's a lot of history that "you have to change them" burned into people minds, documents, etc.

> defaults fit a broad general case.

Yeah, they fit the general case of a MIPS R4400 with a 1mbps network adapter, a situation that nobody faces today. I think the most glaring example is `rmem_max` which is never sufficient to support a coast-to-coast 1gbps flow, and every individual Linux user in history has needed to independently discover this stupid sysctl.

The problem you're explaining can be generalized:

"Junior X starts flipping knobs thinking Y is somehow holding him back"

This is sadly how many of us have to learn. If you don't have a mentor or someone to check your work, it's a way to learn.

Experience will tell you not to make bad decision. You get experience by making the wrong decisions.

Most of the time I senior folks say benchmarking is the only way to know. So I'm not sure that this piece of advice is good.

I wouldn't say that that's a performance tuning, seniority, or even smart people problem. I've seen smart people blindy tuning knobs. Similarly, switching to an expensive O(1) algorithm for a tiny space. It beats me, but I think the problems are closer to lazyness or trying to discredit something.

Reply on news.ycombinator.com