Hacker News Reader

Comment by onetimeuse92304 | Hacker News Reader

Personally, I dislike configuration parameters that let future admins of the system change parameters like concurrency of certain processes, etc.

A lot of the time even I can't tell what is going to be the optimum setting.

So for a number of years, rather than expose a configuration parameter, I am building in a performance test that establishes the values of these parameters.

This is a little bit of search and a hill climb through the multidimensional space of all relevant parameters. It is not perfect, but in my experience it is almost always better than I can do manually and always better than an operator without intimate understanding of the system.

The results are cached in a database and can be re-executed if change to configuration is found (just take a number of parameters from your environment, version of the application, etc. and hash the value and check if you already have parameters in the database).

View on HN

This works well for expensive long-running things, but works less well for more short-lived programs, and even for expensive long-running things there may be more considerations than "what's the fastest value?" I intentionally set the default jobs of "make" to "4" rather than "8" because I don't want to utilize my full system when compiling stuff, so I can still use it for other things at a reasonable speed. There's lots of reasons you might want to do this: on a web platform you don't really want to use up all resources by the one expensive thing, but what's "reasonable" and "desired" often differs – often there is no objectively best "optimum setting".

You do not have to target best absolute performance. You can choose your own fitness functions to balance absolute performance and other aspects of the system like resource usage, latency, or any other aspects of the application. Think about what logic the operator would use to establish the parameter and capture it as code or as fitness function.

As usual, there is no single universal solution to optimising performance (and in fact it can be trivially proven to not exist in general case). But you do not have to find perfect solution, you just have to be able to find as good or better solution than a human operator would.

When you have a large system you can be working with thousands or tens of thousands of parameters. I usually see that even if the application is migrated to different hardware, the configuration parameters like buffer sizes, concurrency, connection pools, etc. are not really revisited. The effect is that usually the application will work suboptimally for the environment it is in.

My solution is an attempt to keep this part of application configuration sufficiently close enough to optimal to be usable and do it automatically without operator involvement (and associated cost and possibility for defects).

Yeah I’m in agreement with you. It can be hard for performance things (vs tuning parameters for numerical algorithm performance) because such tests can be inherently difficult to reproduce and writing the hill climbing can be tricky. It’s a goal to aspire to for sure. Have you found any tooling to do the hill climb search for you in a multidimensional space?

If you're using python `scipy.optimize` has several good options for the minimization procedure.

I guess you could even use those if you're not using python. But it'd require slightly more plumbing.

consider running make with nice, let the scheduler work for you

Actually I do both:

  % alias make
  make='nice -n 20 make -j4'

But I found that even with a nice level of 20 using all CPUs doesn't quite give me the responsiveness I want for other tasks.

Put it in a container where you can explicitly limit CPU / memory usage?

I run all builds in containers/vms anyway as I am too tired to keep my personal machine able to build hundreds of different things. It is better to have a dedicated development environment frozen for each of the projects I work on.

I just want to run make and not slow down Firefox or maybe some other things I'm doing. This simple alias I've had for over 20 years works, on all (Unix-y) systems. Why make everything a hypercomplex story?

Make also has an option to only spawn new jobs when the load average is below a given value, which can be used to get make to reduce the number of jobs running in parallel when there are plenty of other processes using CPU time.

Sounds similar to ATLAS, an automatically tuned BLAS library from the 1990s https://softlib.rice.edu/pub/CRPC-TRs/reports/CRPC-TR98751.p...

I am not surprised. I like to say that it is hard to have an original thought. Even if you think you are original, it is more likely somebody else has already done it in the past. Most of the computing novelties I am seeing are really old ideas, revisited and potentially improved upon.

Thanks for the link, I will take a look when I have a moment:)

Eventually the pendulum swung back for BLAS libraries, the currently popular strategy is hand-tuned assembly kernel in libraries where the most important kernels have been identified.

But BLAS is a very weird case, there are only a couple kernels that you care about (and the most important, GEMM, has been studied like crazy). And BLAS is used under the hood by pretty much the whole HPC community, so there’s incentive for tons of expert human time investment.

You can look at ilbflame/BLIS for the modern strategy.

ATLAS is still neat, perhaps one can even derive more generally applicable lessons from that library, not sure…

I thought the point of ATLAS wasn't the idea that automagical tuning was going to necessarily be best. Rather that it was expensive to [edit hand tune] for all the hardware variations that were around and this was a practical approach to get to pretty damn good even if nobody had spent the money to produce a hand optimized library for your hardware (or, they had, but you didn't want to pay what they were asking).

e.g. I remember vaguely people using it on AMD cpus because they couldn't use icc/mkl effectively...

> Rather that it was expensive to automate for all the hardware variations that were around

Is the “automate” that I’ve italicized supposed to be hand-tune or something along those lines? If so, I think I agree. I think this was the justification for it, and it made sense at the time, and it might make sense for future architectures or exotic ones now.

But eventually Kazushige Gotō wrote gotoBLAS, which really emphasized the importance of the matrix-multiplication kernel and showed how much performance could come from getting it right. Since it is only a handful of routines, it is possible to hand-tune them. BLIS went on to formalize this in a nice open-source library, so any vendor can drop their kernels in.

What is special about a particular CPU from this point of view is the SIMD extension and the cache hierarchy/memory system. SIMD extensions, somebody will write a matrix-multiply kernel in assembly or intrinsics before optimizing compilers get really good at using that extension (if they ever get to it!). Cache hierarchy is more abstractable I guess, but there are lots of rectangles and other nice shapes that dedicated tuners can think about, i.e, (PDF warning)

https://www.cs.utexas.edu/~flame/pubs/GotoTOMS_revision.pdf

I’m sure one could search across all the possible sizes, but it is easier to just have a human identify the right ones.

Maybe someone can train an ML model to identify good sizes. Machine learning uses lots of linear algebra, so I guess we’ll have a nice feedback loop, maybe that will kick off the singularity, haha.

> Is the “automate” that I’ve italicized supposed to be hand-tune or something along those lines? I

Gah, yes, typing faster than coffee. Fixed now, thanks.

Agree on your points about more recent work.

Also worth observing BLAS is an even weirder case as odds are the CPU architecture designed to ensure they could hit a high % of peak FMA throughput for that kernel, so there's a lot of SW/HW co-design going on there too.

The Fourier transform library FFTW does this too.

Its "wisdom" system will plan/measure the best way to do transforms of a particular size/type and cache it for future use.

More here: https://www.fftw.org/fftw3_doc/Words-of-Wisdom_002dSaving-Pl...

FFTW [1] does something like this, not just for parallelism but also for selecting optimal algorithms for a given data set and architecture. It uses "wisdom" to store known-good combinations, falling back on quick heuristics and brief benchmarking when needed. If you're going to do a lot of something you can have it update the wisdom from full benchmarks.

[1] https://www.fftw.org/

As a devops type person, I've been advocating for basic performance testing to be part of microservice unit tests. It'd be really wonderful to get rough numbers for peak and idle cpu/memory usage along with other test output.

This is interesting - how long does it take to write? Is it reusable?

I'd be really interested in a blog post about this.

I am not big about writing blog posts.

But now that I think about it, I could write a small library to automate most of it while delegating some tasks to the application (providing functionality to be tested, parameters and possible ranges to be searched, a way to save/restore parameters to/from the database, provide fitness function, etc.) It is essentially a small benchmarking framework with some added functionality. I think it could be quite useful to some people.

Reply on news.ycombinator.com