Is the field still in demand? Are there any opportunities? What will I be working on? What tips can you give me?
At the moment -- and I expect for the foreseeable future -- demand quite significantly outstrips supply. Because of that, organisations are often quite flexible in how they view potential hires in this area. Simplifying a bit, there are two main ways to get in to the field: obtain specific training in the field (e.g. a PhD); start / contribute to a project (not necessarily OSS) that shows expertise (e.g. if you start filing high quality PRs against Rust or LLVM, people will fairly quickly realise you know what you're doing).
Best of luck -- it's a really fun field, with lots of interesting challenges, and a good group of people!
Tangentially, I think the supply is small because compiler engineering is not a fashionable field of study. Everyone wants to do "AI" these days.
Good point. The biggest factor that determines an engineer's salary is location, followed by years of experience. Not whether they work on compilers/embedded/web/whatever.
Not true! See: salaries for people who know kdb+, that embedded engineers almost always get screwed on comp.
If you are suggestingn that kdb+ salaries are extremely high, like I often read here, I disagree. I did some research in this area, and despite the rumors, found lots of kbd+ jobs posted with very mediocre salaries. It seems unlikely that such jobs would been filled if salaries are as high as people have suggested on HN.
For example, here's a senior kbd+ deb job in Manhattan (high cost of living) that pays 165-185,000/year. I make almost that much as a senior dev in a low cost of living area. I'd expect to make at least $250,000/year in NYC or California.
Every type of job site will have postings that are way below market rate. The market rate for kdb+ devs is well above that, I can assure you.
And I don't mean quants. I mean software developers.
And yes, it's true that every job category will have outliers (high and low), but after spending some time several years ago looking at jobs and speaking with recruiters, I'm skeptical. I've enjoyed learning about array languages, and had heard the rumors of high kbd+ salaries, which is why I was curious.
And I get that kbd+ is just a technology, and much more depends on individual qualifications. So let's say, an experienced, highly-qualified senior dev who has taught themselves kbd+, just for arguments sake.
Then I worked on database (query) compilers. I managed Apache Hive at Hortonworks, and that has a compiler. These are very different beasts.
Now, my startup is working on code analysis and cross compilers for data engineering, including Spark.
Some of my colleagues are working in AI chip startups.
It’s hard to find high quality engineers in both kinds of compilers. Just like database internals engineers there is high demand.
Plug - I’m hiring at Prophecy.io for compiler engineers.
What are the real-world motivations behind reimplementing the noted languages on top of Truffle? These projects are frankly incredible - hundreds (thousands) of total man-years of work - and all to _re_build something that already exists. I'm missing context that I don't know where to look for.
Is there anywhere I might look online to immerse myself in the relevant discussion of the business use cases that drive these types of huge projects?
Most of the Truffle languages have small teams, like 2-3 engineers tops, and that's sufficient to catch up with and then keep pace with even a well funded ecosystem like JavaScript. For instance the GraalJS team keep adding proposed JS language features that aren't even fully rolled out yet.
As for why, well, Truffle languages have a bunch of advantages over implementing engines in other ways:
- They all interop with each other, with JVM bytecode languages, with anything that compiles to LLVM, and anything that runs on WebAssembly. All in the same VM with the same heap (i.e. they can all use pointers to each others constructs).
- You get access to HotSpot and in particular its state of the art GCs.
- Graal is an excellent compiler. V8 is better but only for JavaScript and only in warmup time; once code is warmed up (for servers) GraalJS produces code just as good as V8.
- You get a lot of sophisticated visualisation and debugging tools, both to help you develop and optimise the language (IGV) but also to analyse programs in production (VisualVM, JFR).
- You can support native extensions using [Safe] Sulong.
- You get debugger support for free, because Truffle languages all support the Chrome DevTools debugger protocol.
- They just added support for the language server protocol so you get some automatic support for IDEs.
- You get a consistent sandboxing API for free.
- Probably a lot of other things I forgot.
So you get a better engine than most languages can afford, for way less cost, with way more features and access to a huge ecosystem of libraries from all the different languages. Pretty snazzy.
Also eventually I'm assuming there's some plans to compile these other languages with native-image. In my eyes native-image is Java's response to golang: reasonably sized small binaries that are perfect for containerization and quick to elastically scale up/down (startup time).
JIT compiled scripting languages and DSLs (domain specific languages) are all over the place, GPUs and the APIs that run on GPSs all need special compilers, embedded processors and custom chips often need custom compilers and/or custom optimizations.
There’s certainly a ton of optimization work out there, so if you like designing optimizations on top of existing compilers, you can pretty much pick what kinds of API/software/hardware you want to work on.
Instead, think about languages in general. There is a myriad of opportunities between the js eco system, JVM and .NET, things like Rust and even low-level languages like LLVM. It should be straightforward to pick features from one eco system and port them to another. Say linear types from Rust to the JVM. Pick one or two such features, port them to your favorite eco system and provide a prototype compiler/macro library on github. With that kind of experience under your belt, you should be the goto guy for any software company that does tooling in that ecosystem. From that vantage point you could start looking for ever more interesting work.
LookML in Looker is one example I can think of - imo it has enabled Looker to solve the self-service data issue much better via a low-code approach than Tableau and Qlikview (both "no-code").
(or use “goto woman” to subvert so-called typical practitioner expectations)
thank you for considering
While not a list or positions as the URL suggests, ands to help remind people that compilers are in way more companies than you think!
How would you go in the adjacent field of language design? I would be super interested in doing research on the mental load of different programming languages (and different representations of information). I.e. put a hacker in a fMRI and let her solve tasks in different languages.
I wonder how many people you'd need to measure to produce a usable stereotype model. (Hmm, I wonder if studies of other types of problem-solving challenges have asked the same question and published any useful data?)
I fear you'd need to parametrically plot developers, experience, and language together to get a useful result.
And then there's the "I get it" threshold point at which the brain suddenly finds a particular subject much easier to reason about, ie, fundamental learning. Everyone's on a spectrum there - one that seems to be very hard to self-assess.
https://blog.tensorflow.org/2019/04/mlir-new-intermediate-re...
In general we can afford to treat the problem as a search problem since our kernels are small enough. Here there is a lot of stuff that ML aided stochastic search is amazing at compared to normal compilers and even in some cases human hand optimized assembly (!).
By the way, are there any people here (or, perhaps, you could recommend someone) who is very capable and would be interested in doing some potential contract work on a DSL (I'm still pondering about embedded vs. external - obviously, each category has their own pros and cons, but platform stack selection adds an additional complexity dimension)? Several people who I have approached so far, are definitely very knowledgeable, but they all work in academia and, thus, practically, have no time for consulting or contract work. :-( Not to mention potential issues arising from IP clauses in some experts' university contracts, hence the question above.
My current day job, at a research nonprofit, is working on a system whose behavior is driven by an external DSL. I get to work on the parser/compiler type thing that generates executable units and the interpreter that actually runs them. This is certainly not the “big leagues” as far as compilers are concerned, we’re not executing anything directly on anyone’s CPU, but I do find it tremendously enjoyable.
0. https://github.com/openwdl/wdl
I worked there for awhile but compilers aren't really my thing. They're building cool stuff, though. Their tech converts database queries between dialects, and can even run as a translating proxy (for example, run Oracle code on Postgres) and provide performance insights into your code. Their clients are big name companies who need to maintain millions of lines of DB code.
e.g user write documents with something like markdown and then editor renders document with things like auto numeration, displaying other referenced elements by syntax e.g '@element2'
Does there exist job in writing parsers?
Compsci degree? Working compilers? Opensource commits? Github repo full of parsers?
How you demonstrate that can be in a variety of ways. Any of those would probably be helpful if you can get the right person to notice them.
Definitely not that one.
A Survey on Compiler Autotuning using Machine Learning
I've seen jobs advertised by teams developing Cardano, Tezos, and Ethereum. It's worth looking at their project pages, seeing which teams work on it, and then checking their careers pages. I know IOHK have been hiring in the space although that's specifically functional compilers - however nothings up on their careers pages right now.
Not sure if I am being sarcastic or serious.
I also often see job postings for compiler developers in the embedded field.
We always have a great deal of trouble finding qualified candidates to fill open reqs. We find few are interested in this sort of technology, everyone want to either work on cool new applications (like self-driving automobiles) or high-profile web-based stuff. Our entire department is either waiting for retirement or a heart attack (whichever strikes first, and retirement is not a realistic expectation).
tl;dr compiler engineers are in demand and they work on compilers.
But if you're considering an education choice, don't overspecialise. You will most likely change specialisation in 5-10 years (different market, different technology), so if all you know is just compiler engineering, it will be way tougher for you.
Also, make sure you learn a lot about algorithms / computer science in general. If you know that, switching engineering fields will be easy.
And/or get in touch.
It is very sad how the engineer term has been corrupted.
You are an engineer only if you pass the FE exam and are a registered member of the state association of engineers