Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
docs.google.com
docs.google.com
I'm not joking. While there are exceptional situations where you might get to work on interesting projects, by far the most reliable way to do it is through an academic degree.
Eventually, it boils down to two things. Find a problem that you want to work on and engage in the community as one-more-minute points out. Second, find a way to support that - PhD, an interested employer, a university, a grant, or perhaps you have are just well off and can do whatever you want!
Don't mind a pay cut but I failed at my first attempt to land a (rare) Julia job in Sydney. Willing to relocate!
How many of these prolific undergrad contributors end up working full time on interesting aspects of the project though?
I do wish more slides are hosted elsewhere not google docs. Because some companies are worried about the use of google docs especially if the data is sensitive e.g. banking and insurance.
Blocking Google Docs is by far the cheapest way to prevent its use, even considering the downsides of a complete block.
Why would you bother to look for some awkward way to work with customer's sensitive data over the internet, if you're going to get fired for finding that way? The main goal is to prevent random people from storing data in unapproved locations "because it was easier that way". You can easily make it not be easier, that cuts 95% of the sharing, and then the only practical reasons to do so are clearly malicious.
Dropbox, drive, icloud, you can easily firewall all of that.
I do wonder, however, how full differentiable programming languages will be supported by compilers in the near future.
Reproducing it here: The Compilers for Machine Learning workshop was recently held at CGO 2019. Since compiler techniques affect a large part of the machine learning stack, this workshop aimed to highlight research that incorporates compiler techniques and algorithms in optimizing machine learning workloads. The workshop included talks from various projects - Julia (Julia Computing), TVM (UW), Glow (Facebook), XLA (Google), nGraph (Intel), TensorRT (Nvidia), and the soon to release MLIR (Google).
Our talk introduced the abstractions in the Julia language and the kind of compiler transforms involved in implementing them. We then had a deep dive into dynamic semantics + static analysis - our JAOT (Just-Ahead-Of-Time) analysis. Building on these capabilities, the Zygote system implements automatic differentiation, effectively treating it as a compiler problem, giving us differentiable programming for free. Finally, compiler backends for GPUs and TPUs give us high performance execution. All this comes together beautifully in Neural ODEs, which we had to show off as our first slide!
https://venturebeat.com/2019/02/18/facebooks-chief-ai-scient...
Of course, our view is that Julia is one such language that people should consider seriously. The talk linked here is a peek under the hood and shows that differentiable programming in Julia is not a special add-on, but something that fits naturally within the language.
> Lush is an object-oriented programming language designed for researchers, experimenters, and engineers interested in large-scale numerical and graphic applications. Lush is designed to be used in situations where one would want to combine the flexibility of a high-level, weakly-typed interpreted language, with the efficiency of a strongly-typed, natively-compiled language, and with the easy integration of code written in C, C++, or other languages.
http://lush.sourceforge.net/index.html
Is there more detail about his proposal? It must be well thought-out.
Parts of Julia are even implemented in Lisp [2], although they tend to be ported to Julia now IIUC. But it's clear the Julia developers are well aware of Lisp and its strong points.
[1] https://docs.julialang.org/en/v1/manual/metaprogramming/ [2] https://discourse.julialang.org/t/the-role-of-femtolisp-in-j...
Do people also think you need a timecard-tracking-specific compiler?
As the slides say, you need the program to be differentiable. Do general purpose compilers make it easy to automatically differentiate a program? No, not until very recent work. So people thought a better idea until we figured out how to do that was ML-specific compilers and frameworks.
> Do people also think you need a timecard-tracking-specific compiler?
No, nobody thinks this, because timecard-tracking does not need any special properties such as being differentiable.
The answer isn't as crazy as your extremely snarky question makes it out to be, is it?
Every functional unit in a net needs to be differentiable and so needs to carry around either dual numbers (forward mode) or adjoints (reverse mode). These requirements necessitate new languages and compilers that current imperative representations don’t currently support.