First of all make sure you are very comfortable with these concepts:
- SSA
- graph coloring, linear scan, and priority coloring approaches to register allocation. You really have to know all of them otherwise you’ll have incorrect ideas about which is best and when.
- sea of nodes. Not because you will necessarily implement it (it’s not that great IMO) but because you will definitely use some ideas from it. It’s a very inspiring concept.
- abstract interpretation
- types, points to sets, abstract heaps, and the ways that these things are the same
- instruction selection. This one is tricky because the literature doesn’t say smart things about it. Gotta read code or talk to people. I learned how to do it by word of mouth.
You need to strike a balance between these two activities to become good:
- Write your own compiler and get it to beat other compilers on some benchmark. It can be a simple benchmark or a simple language. You need to be comfortable with overall compiler architecture and there is no substitute to seeing the whole thing fall together. Then, after you do this, do it again because if you’re like me then your first attempt will be shit.
- Learn a major, mature compiler architecture like JSC, llvm, V8, GC, or whatever. Write some measurable improvement to such a compiler. Every major compiler has brilliant nuggets of awesomeness that you will only come to understand if you jump in there and try to make it better.
Hope this helps and good luck! You picked a fun profession.