Trivial Artificial Neural Network in Assembly Language
syprog.blogspot.com
syprog.blogspot.com
During the implementation, if I recall correctly, the main desire was to convert it entirely to integer math, but the biggest problem with that was making a fast integer approximation for the sigmoid function. The final version, IIRC, was part integer and part float for this reason.
NNEDI originally chose from one of 64 neural network contexts for its calculations, but the current version performs a weighted average between a smaller number (context mixing). Since this eliminates the branchiness, it would likely be very suitable for a GPU implementation. Making a shader-based or OpenCL-based GPU implementation that would work in a video renderer (e.g. for realtime video playback) would likely be extremely appreciated.
GPUs (and plain CPU SIMD) would help very much yes. Any serious NN implementation will be using it.
If you write straight-line asm code where every output is immediately consumed in the next 2-3 instructions, you'll find it runs exactly the same speed as the compiler's output. Probably worse, because the compiler would at least attempt this. This is where a human can do better than the compiler, but this isn't an example of it.
Beating the compiler usually has more to do with having more information. Say, "this register will never have integers greater than X", or "the carry flag will be set to Y here", or whatever. Even compiled intrinsic SIMD beats most hand-coded kernels these days, unless you're doing something very exotic (e.g., mixing integer and floating-point logic to save cycles).
Hand-written assembly is something that will probably go away nearly completely (can't really beat optimal schedulers, superoptimizers), and I say this as someone who's spent many years optimizing assembly.
Intrinsics basically are hand-coded assembly: you're specifying every single instruction manually, just in an obfuscated syntax, and without a proper assembler preprocessor.
At that point, it's way easier to just write it straight by hand.
Also, there are still plenty of things compilers are terrible at with intrinsics. This includes things like:
1) Register allocation. Compilers, even "good" ones, will throw things on the stack willy-nilly. The biggest problem here is that the compiler hides register utilization from the programmer, preventing him or her from seeing how many registers are used. If you're aware of register usage, you can tweak the algorithm to avoid the problem.
2) Compilers are truly terrible at making up calling conventions for calls within assembly (e.g. to a subfunction). Humans can make whatever calling convention they like, which can give great performance improvements in functions like FFTs (see: libav split radix fft) that are inherently recursive. In the case of the aforementioned FFT, Loren was able to make a handwritten-asm Altivec version around ~40% faster than intrinsics, IIRC.
3) Compilers are not very good at organizing code; in many cases, it is useful to split up code and reuse blocks to save on code cache (see: x264 trellis asm). This avoids inlining it and creating a massive mess -- which, by the way, trashes the compiler's register allocator too. When using the compiler to do this, you run into the problem of 2); it can't allocate registers efficiently between code blocks, or come up with a decent calling convention, so it's inefficient.
4) There are some things that no compiler in existence is currently able to do, like computed jumps without tables to a templated set of functions (see: x264 cacheline SAD asm).
Anyway, I do not disagree with you. What I was trying to convey was not so much the quality of current intrinsic code, but the progress that has been made in the last ~5 years in this respect. I remember writing some intrinsic code that became a series of useless stores and loads all over the place. When that is your baseline, current tech is impressive, modulo the issues you've quite correctly pointed out.
Dynamic optimization by the core involves deeper pipelines and/or speculative execution and/or many more gates, all of which takes up more area and power. You end up spending more energy than you save, and pretty much everything is power consumption (or cooling, which is essentially the same thing) limited these days.
Super-optimization is intractable for large sequences. It's like trying to solve a chess game by brute force. Likewise for other compile-time optimizations - they're polynomial or worse. You at least need a human to design the structures with knowledge of how they're represented low level and how they'll end up being pumped. It needs someone to come up with better compiler algorithms before we can say for sure that hand-optimization will go away. Increase in CPU speed alone doesn't make it inevitable.