Comment by tialaramex | Hacker News Reader

Comment by tialaramex | Hacker News Reader

tl;dr the saying is that "premature optimisation is the root of all evil", and Casey burrows into contemporary data to show that really although the claim was 3% of the code takes up 90% of the runtime even then it was more likely 4% takes 50%.

The best thing you could take away from this lecture is something a reasonable person should take away from the original "root of all evil" saying anyway. Measure. Measure. Measure. If you aren't measuring that's not optimization it's masturbation.

Ironically in the process of measuring for a third time yesterday I tripped a bug in Bill's language for which I opened an issue. This is not the goal of measuring three times but merely a happy accident.

Along the way Casey discovers (?) that Structured Programming means just what we today call programming†, that software was a lot smaller in the days when 4096 bytes of RAM was a good entry level option and that loads of these famous people from 1970s computer science knew each other.

† And knowing about this is one reason the Structured Concurrency people want that everywhere. Very possibly there's a future where it seems silly that people once wrote programs which did not use structured concurrency.

In other videos Casey argues against the profile–fix–repeat workflow (I’m not saying that you necessarily meant this by measuring), instead arguing for estimating the theoretical maximum, then trying to get close enough to it. His argument is that the former might push you towards a local minimum without realising that you could do much better

When "Estimating the theoretical maximum" you're going to bring your assumptions to that estimate and so you're equally likely to get stuck in some local optimal point because you didn't see what else was possible, so that doesn't persuade me much. Worse, now it's a local optimal which might be entirely in your head.

There is some space for this "Estimate the theoretical maximum" as a high level insight, if the software needs to do a thing which your estimate says is impossible that's important to address up front but I think for practical software engineering you're much more likely to need profile-fix-repeat

Actually measuring is crucial and an estimate is not a measurement. I think one of the reasons many in the Handmade Community don't like measuring is that it too often ruins their "estimate" of how the crap they've written is achieving a "theoretical maximum" in favour of a boring fact that it's much worse than alternatives.

Usually the ideas it that the theoretical maximum is not a local optima though, but some ideal global optima which you know you realistically cannot reach.

That is you know the data is some size, and memory throughput is some rate, and clock cycles is some rate etc...

You are entirely correct about the line of thinking if your assumptions are down a different line of thinking such as algorithmic estimates though, where you might miss some better way of doing things.

I've always used this method of working and can stand by it. For instance let's say you are working on a high-speed real-time image processing system. If you know that each image is 1080p for instance, that's 2 million pixels, and we will have 3 bytes per pixel for colour, then that's like 6MB per image (without compression). If you have 25 GB/s of RAM bandwidth, then there is a hard limit of ~4 thousand frames per second if you aren't even doing any processing. Therefore if you strip out your processing and are only getting say 400 frames per second, you know something is going wrong.

This can be applied roughly to anything. I find it very handy when thinking about CPU/GPU performance as well. For instance knowing that at 5GHz you have 5 billion clock cycles per second, which depending on instructions and pipe lining etc... can be roughly 5-30 billion operations per second per core tells you how long some process should take if you know roughly how many operations are required and the data size you are operating on.

Obviously as you drill down things get much more complicated, but they give you some rough idea about how fast things could be under some set of assumptions. Very similar to how a physicist works with models and assumptions.

Sure, but there are performance issues no profiler will catch in a straightforward flamegraph reading. Some examples:

1. Your hottest loop is spilling registers which only shows up as non-local cache thrashing (i.e. some other random code becomes slow) or randomly slow instructions i.e. "why is this xorps to initialise this float suddenly slow" due to pipeline stalls

2. Your code stops fitting into cache due to the code bloat, there's no one method which is slow, everything is slowed down by a percentage factor

3. A lot of useless work being done like temporary strings being copied everywhere

4. Your code is "I/O bound" because all the data you're accessing is scattered all across memory, leading to completely predictable TLB stalls

It's very easy to make a large program, quite a bit harder to make a small one...

How do you even measure those? Will that show up under something like VTune?

Yes, VTune is good.

I'm on AMD and uProf isn't that good (it often fails to match PDB to source, it's slow / crashy, stuff like that) so I mostly use Superluminal myself (has godly UX) and only drop down to uProf when I have a suspicion and want to gather hardware stats.

You can do all that with perf on Linux, but everything is manual.

> Measure. Measure. Measure.

The problem is knowing what to measure. There's another saying

"When a measure becomes a target, it ceases to be a good measure."

As an example from memory, there was a game dev company that celebrated they had maxed out the cores on the PS3. That didn't mean anything though, anyone can max out the cores by filing them with bad code. But hey, their "measurement" told them they had maxed out the machine

> The problem is knowing what to measure.

This can be a problem, but much less so because so often we're doing "easy mode" where we don't need a proxy. The "it ceases to be a good measure" is because you're measuring a proxy. You wanted to deliver happiness, you measured wealth because it was easier to measure but seemed correlated and now you've got rich miserable people, oops. But software engineers can often measure the actual thing they want to improve directly, not a proxy and so it cannot cease to be a good measure.

With a few caveats, though they do tend to get fixed over time. For example, frame rate. Higher frame rate is better, the end, right? Wellll...

Latency and hitching are annoying to measure, so for a long time, they were pretty much ignored. That has improved and hopefully will continue to improve. There is the bufferbloat initiative, gaming magazines take frame time histograms now, input devices and screens commonly have their latency measured. But latency is still under-measured - for GPUs / GPU drivers, for all software, for games and particularly for websites.

Yeah, frame stutter is definitely one of the cases where somebody may end up targeting a proxy and that saying about targets happens as expected. A game which typically hits 50-60fps but has occasional half second stutters may not fix the stutters if your proxy was "time to render 100 frames" and you throw out outliers - since every stutter will be an outlier.

That quote mostly applies to human organizations where the connection between outcome and metric is questionable. Or where incentives unexpectedly change behavior.

If your goal is fast software it can be measured quantively and you’re likely to improve the actual thing using those metrics.

The goal above was to write fast software. The developers made the mistake of seeing that they had saturated all the cores as "We did a good job of using the entire machine" = "our code is fast". They had a measurement and misinterpreted what it meant.

This is just dumb. You wanted to improve performance but measured memory instead? Whoops.

Why not measure performance for which there are many good metrics.

They didn't measure memory, they measured how busy each processing unit was. It's like going into Activity Monitor, Tops, Task Manager, seeing all CPUs at 100% from your app and thinking "We've reach peak perf by utilizing all the processing power"

People still do that today. They run some LLM or gen AI type app. See all the numbers reach high percentages, and wrongly think "this app is using all of my machine's potential"

Yes it was an analogy about measuring two unrelated things.

Casey generally, across all his material, advocates for non-pessimisation. Measurement takes too long to apply it to your entire program. He advocates for thinking about how much work the computer should actually have to do, then not making it do much more than that, at all times. This means avoiding serial dependency chains on the network, and huge towers of abstractions, and redundant work. He allows for writing lazy slow code as an intentional tradeoff that you may revise later if it becomes a bottleneck. He does not allow for inherently slow architecture.

Reply on news.ycombinator.com