Hacker News Reader

Comment by jwmerrill | Hacker News Reader

"Modern machines are a huge pile of opaque and unreliable heuristics and the current trend is to add more and more layers on top. The vast majority of systems are built this way and it is by all accounts a successful strategy. That doesn’t mean I have to like it."

This is a really valuable observation.

"Smart" compilers seem great for letting you write code without thinking too hard when performance requirements are loose, but they make it difficult to achieve peak performance in two ways:

1. The fact that details of the machine are abstracted away from you means that you may never learn them well.

2. It ends up being insufficient just to know the details of the machine, because you also need to know how to coax the compiler into producing the low level result that you want, and then you need to be vigilant that later changes to the compiler don't break your assumptions.

View on HN

I think the opposite is true. Modern hardware is so complex[1], made even more so by its constant interaction with a complex OS, that any sense of familiarity with the actual performance model is illusory, unless you're doing something very controlled and very specific (like, say, DSP). Modern hardware itself is an abstraction, hiding its operation away from you. We can no longer hope to tame hardware with meticulous control over instructions as we were able to up until the nineties.

Forget about clever compilers; forget even about smart JITs; even if you look at such a big abstraction as GCs and only consider large pauses (say anything over a few tens of milliseconds), it is now the case that in a well-tuned application using good a GC, most large pauses aren't even due to GC, but to the OS stopping your program to perform some bookkeeping. Careful control over the instruction stream doesn't even let you avoid 100ms pauses, let alone trying to control nanosecond level effects.

[1]: http://www.infoq.com/presentations/click-crash-course-modern...

Most of us don't have the time for meticulous control over instructions but those who do can certainly use it to good effect eg http://www.reddit.com/r/programming/comments/hkzg8/author_of...

My aversion to piles of opaque heuristics is not because I'm against smart compilers, just that for certain projects I want to be form a mental model of what code I should write to get a certain effect. The trend of modern languages with heavy heuristic optimisations or complex JITs is towards less certainty and less stable optimisations, so that a program that runs fine today might be unusably slow tomorrow.

Staging and compiler-as-a-library is a promising compromise for projects that really care about stable performance eg http://data.epfl.ch/legobase . You can still have an LLVM-smart compiler underneath but you get to make the first pass.

Rust is actually very predictable in some respects eg generic functions will be monomorphised. I prefer it to wrangling GHC or the V8 JIT.

> I want to form a mental model of what code I should write to get a certain effect

And how do you do that with hyperthreading, virtual memory, power management that may decide to power down your core because what you're doing doesn't seem important enough (and that differs greatly from one processor to another) and cache effects on code, data and TLB (all are strongly affected by other threads and processes running on your machine[1])?

While those effects didn't exist much before the 90s, and they don't exist today in GPUs and small embedded devices, on desktops and servers those effects may be much greater in magnitude than any difference you're able to get by better control over generated code. Not running a hypervisor, turning off virtual memory, pinning threads and isolating cores have a much more profound effect on predictability than which language or compiler you're using. Focusing on compilation before taking care of those much more powerful sources of unpredictability is like trying to get a faster car by reducing the weight of the upholstery fabric.

> so that a program that runs fine today might be unusably slow tomorrow.

I think that slowdown actually applies to assembly programs much more than to, say, Java. As CPU architecture changes, it's actually easier to keep higher-level code performant. I mean, why do you assume that compiler changes will hurt your code performance more than CPU changes?

> You can still have an LLVM-smart compiler underneath but you get to make the first pass.

There are many ways to produce good machine code (my favorite is Graal, HotSpot's next-gen JIT), but none of them really give you a good mental model of what's going on. You may like one approach over another for personal aesthetic reasons, one approach may actually produce better results for some workloads than others, and some approaches really are more predictable -- but no approach produces categorically predictable results, and more predictability doesn't buy you better performance (though it still requires more effort).

It used to be that if you knew what instructions your compiler would emit, you knew how your program would perform. That is just no longer the case (well, it is to some degree, but other effects are stronger). A single instruction may perform anywhere within 7 orders of magnitude (L1 cache hit to virtual memory miss) depending on effects outside the program's control! (of course, those high-volatility costs are usually amortized, but so is a less unpredictable compiler output).

[1]: That is the key to cryptographic attacks that let a process sense what a cryptography algorithm running in another process is doing by the way the cryptographic computation affects the performance of the first process.

I think you are taking a very black and white point of view. Yes, hardware is complex and unpredictable. That doesn't mean that we can't reason at all about performance.

I take a program, measure it's performance on a wide range of real-world workloads across multiple different machines. Then I change some numeric routine to use unboxed integers instead of boxed integers. I measure it again on a wide range of real-world workloads across multiple different machines and find that it is significantly faster in all cases. My approximate mental model of how the machine works allowed me to make a change that empirically improved performance. My model is not perfect so I do have to measure carefully, but it is what allows me to make sensible decisions about which changes to measure rather than just changing things at random.

In a language where the compiler controls unboxing, my mental model is much more approximate. I have to figure out how to influence the heuristics to lead them into making the correct choice, and the solutions tend to be hacks that are highly sensitive to small changes to the heuristics, leading to conversations like https://groups.google.com/forum/#!topic/clojure/GvNLOrN3lGA .

Performance for non-tuned code may be better on average but my ability to tune important areas is reduced. If the compiler was more predictable, or had a interface that allowed me to add information, or if I could make my own passes then that trade-off would go away. I'm not against smart compilers, I'm against smart compilers that don't talk to me.

> I'm not against smart compilers, I'm against smart compilers that don't talk to me.

There are some extremely interesting advances in that area in OpenJDK. Java 9 will contain two relevant changes. The first, JEP 165[1] (fine-grained and method-context dependent control of the JVM compilers), lets you control compilation with metadata depending on context (e.g. inline method foo when called from bar); a much more interesting and powerful enhancement targeted for Java 9 is JEP 243[2] (Java-Level JVM Compiler Interface). It will do the following:

* Allow the JVM to load Java plug-in code to examine and intercept JVM JIT activity.

* Record events related to compilation, including counter overflow, compilation requests, speculation failure, and deoptimization.

* Allow queries to relevant metadata, including loaded classes, method definitions, profile data, dependencies (speculative assertions), and compiled code cache.

* Allow an external module to capture compilation requests and produce code to be used for compiled methods.

This opens the door to what I think is the most impressive compiler of the last decade, and a true breakthrough in (JIT) compiler design: Graal[3]. Graal supports languages of any level (it already has frontends for Java, C, Ruby, Python, R and JavaScript), and then allows complete control over code generation and optimization decisions at runtime. E.g. you tell it what kind of speculations to make, and it tells you which speculations failed. Unlike LLVM, you compile your language into a semantic AST (that may or may not match the language's AST) and feed it to Graal, but each node may contain not just semantics but instructions on speculation and code-gen control at any level you wish. During compilation, Graal interacts with the node and the node gives further instructions. As I understand it, JEP 243 will allow to plug Graal into the standard OpenJDK HotSpot (though at reduced speed), until Graal matures enough to become HotSpot's default compiler.

So what Graal will do is let the developer (if the language designer allows), write simple, high-level code, but tell the compiler, "listen, compile however you like, but when you get to this function, talk to me because I have some ideas on how to compile it just right".

[1]: http://openjdk.java.net/jeps/165

[2]: http://openjdk.java.net/jeps/243

[3]: https://wiki.openjdk.java.net/display/Graal/Publications+and...

Thanks, that is really interesting. I'll have to look into it.

And yet I consistently find that checksum tools, compression libraries, and things like video decoders (such as H264 decoders) written in assembly consistently outperform all other implementations I've had to deal with. "Sufficiently smart compiler" is a tired meme at this point. There are few programs that benefit from being entirely written in assembly, but quite a lot who do having parts of them hand optimized. Some, like game emulators, particularly one man job like No$GBA, are still fully written in assembly and its performance is a sight to behold. No$GBA would lose a lot if it were rewritten into a high level language.

> and things like video decoders

That's precisely the example I gave. Although many modern decoders use GPUs, which are much simpler than CPUs (simpler even than 90s era CPUs). The GPU performance model is very simple to comprehend.

> No$GBA would lose a lot if it were rewritten into a high level language.

That's a nice sentiment, but I don't think it is supported by the facts. You could probably write a JIT in Python that would perform much, much better (but that would be overkill, given that you're emulating a very slow, very small machine), and a trivial implementation in Java would probably perform just as well.

The ability to achieve significantly better performance for general-purpose tasks (let's call that "branchy code") with low-level languages today is more myth than reality. What is true that some high-level languages consciously give up on some performance to make development easier, but that's a design choice. That's not to say that optimizing JIT and AOT compilers get everything right -- they don't -- but they get it right often enough that they're very hard to beat.

Reply on news.ycombinator.com