Hacker News Reader

Comment by jiggawatts | Hacker News Reader

Tight feedback loops are great for software development, but terrible for systems administration. In my career, I regularly have these conversations:

"We did A, it was slow, then we did B, and it was fast." == "Congratulations, you discovered that the cache works, not that B is faster than A."

"We ran this test on AWS image size such-and-such, and it performed 5% better than the competing product that we tested previously." == "That's just noisy neighbours on a multi-tenant platform, you've learned nothing."

"We ran a test and it succeeded, hence it is now working!" == "The system fails nine out of ten times randomly, you just stopped when you had your first success. Try it again."

This ability to spin up a test environment quickly and run it through a scenario can be intoxicating. People try all sorts of random stuff, see a "win", and then call it a day without really doing the legwork to prove that they achieved something meaningful.

View on HN

We focus far too much time teaching calculus and trigonometry in school. Applied statistics would be far more relevant for nearly everyone.

It's not easy to learn statistics without calculus.

Basic level doesn't need calc

Any part of statistics is completely dependent on calculus. You don't get to choose how features are distributed over the set of possible samples. It's always some scary looking function without exact solutions.

But you don't need to teach people all the fundamentals of calculus before they can apply it. Definitions and a small set of pre-made answers are enough to start using it and focusing on the statistics. When (and if) those people decide to learn it more in depth, they can go back to calculus fundamentals, and will have an easier time doing so, because they will know why those things exist.

I guess that parent poster was thinking about probability and mixed it up with statistics.

If yes, then I agree with him. People should have more probability lessons at school than trigonometry or calculus.

Basic probability (coin flipping, events over trials) by itself is just as useless as basic trigonometry.

It only becomes useful as you move from simple combinatorics to continuous distributions, at which point you need differentiation and integration to switch between CDF and PDF. Much in the same way trigonometry becomes useful in physics once you need to deal with vectors and waves.

We teach maths too much, we should just give gold bars to every student so they already win at life.

Trigonometry in particular is so ingrained in many aspect of applied science that not learning its fundamentals and then pushing to its higher level concepts is going to seriously reduce adaptation capability in students.

And if you somehow never need to rotate things, it's a good training in applying theoretical rules to concrete problems. In my university, after 2 years of increasingly complex trigonometry, we were left with 3D engine programming and to the fact that opened my mind forever that theoretical mathematics have a purpose.

What % of students struggle and never really get the maths? What % do fine with the maths but never need them in their adult lives? I would argue this is far more than 75% of all people. Same can be argued probably with higher %s for foreign language in the us. I think learning advanced maths and foreign languages are great but I also think we waste insane amounts of human potential and $$$ with the way we go about it.

Do you have good resources on how to do "the legwork"?

This guy has had a few talks at CppCon over the years, and covers the basics of how to attempt benchmarking with actual computer science instead of myth and voodoo rituals: https://www.youtube.com/watch?v=koTf7u0v41o

In practice, it's not critical to go that far. In my experience, huge performance wins are trivial to benchmark with just a modicum of effort, and tiny performance wins are typically not worth any effort at all. At FAANG scale they might be, but I don't work at a FAANG, so not my problem! :)

Simple methods I use and can recommend are:

- Ideally, collect data from production, where the change behaves exactly as it would in production because it is production!

- Interleave tests on the same machine. E.g.: A, B, A, B, A, B, etc...

- Run tests on every machine you can get your hands on over an extended period and draw a histogram or a scatter chart of some sort. You don't have to do this at 100% load, it's actually better to mix this in with normal workloads. This gives you an idea of the noise versus signal, platform variance, variation due to time of day, etc...

- Run tests after a cold boot to get a sense of worst-case performance instead of idealised and often unrealistic "hot loop" performance.

- Run tests with different input sizes and draw a graph. Try find the explanation for every feature of the graph.

- Be aware of every cache layer in the system. Don't forget about the potentially huge RAM or SSD cache of the storage subsystem. Typical SAN arrays now have 256 GB or more. The previous graph above can help, but only if your input size goes waaaay past the biggest cache sizes in the system.

As a real-world example: I recently reduced the size of a database platform by a few vCPUs to save on licensing costs, but increased the per-core performance and simultaneously did some network tuning. Some benchmarks showed the same performance, others showed an improvement. Both were extremely noisy before and after the change.

I would like to take credit and say that I improved performance while saving money. However, the scientifically honest result is that any performance change is statistically insignificant.

The cost saving is significant, because it has very little variance. That's good enough, and I don't have to make a statement about performance that the numbers can't really back up.

I would add always (for network servers) run “open” loads tests with each request being independent of the fates of the other requests, probably from a Poisson distribution (generated by having the delays be pulled from exponential random distribution ( https://en.wikipedia.org/wiki/Exponential_distribution ).

So many load tools send a request then wait for the reply then send the next. That is basically useless for understanding variance under spiky load and for assessing when a system is below its capacity how stable it is.

For real world performance, the variance matters a lot. The code will make some things be cached some not (so the data variability in the test load should follow the same sort of power law that your real traffic does), and even if you get 10% faster at the median if the variance goes up the quality is lower.

The tools I used had a parallel requests option.

Reply on news.ycombinator.com