On maybe slightly off topic point. I really want to learn more about compiler engineering, what people would suggest for someone with some background in programming but main background is physics (if that matters at all)?
Then you should get a copy of Modern Compiler implementation in ML. I love the Crafting Interpreters book, and also the Writing A Compiler In Go book; but they will not satisfy any person with undergraduate CS education.
From memory (because I left my copy of the Dragon book in another country) the book describes very well the first stages of compiler construction, describing the fundamentals of what Lex/Flex and Yacc/Bison do.
Right now for a compiler I'm writing I have used the Crafting Interpreters book and find it very useful. It's basic but can carry you through a project, and a decent reference for someone who took a Compiler Construction course about 20 years ago.
All of the front-end parsing stuff is good, but heavily biased toward LALR parsing, which is rarely used in production compilers these days because it’s really hard to get good error messaging. There’s also a lot of tooling that can help these days.
But where it really falls down is the backend part. Parsing is just not where you’ll spend most of your time. Once you parse you usually need to translate the parse tree (often an AST but not always) into some other representation.
I found the Appel books (Modern Compiler Implementation in <language>) to be much better at that part. I still own a Dragon book (an older one), but tend to refer to my Appel book a lot more when working on compiler-y things.
A somewhat more playful approach might be to complete the 2019 edition of Advent of Code [2], where you'll learn some preliminaries while implementing the Intcode interpreter.
[1] https://online.stanford.edu/courses/soe-ycscs1-compilers
# Learning Resources
## Theory
Take a look at these two books:
- Crafting Interpreters[2]. Compilers and interpreters have a lot in common and the book is exceptionally beginner-friendly. Basically, this is the perfect one to get started with.
- Introduction to Compilers and Language Design[3]. Doesn’t assume any preexisting compilers knowledge, and teaches all the basics necessary to build a compiler using a hands-down approach. The examples are in C.
Ignore the classics textbooks like the Dragon Book or "Modern Compiler Implementation" for the time. You can always come back to them later if you want to.
## Source code
Explore the sources of
- 8cc[4]. A compiler for the C programming language. It's intended to support all C11 language features while keeping the code as small and simple as possible.
- chibicc[5]. The successor of 8cc from the same author.
If you want a bit of background on these two compilers, check out the author's blog post "How I wrote a self-hosting C compiler in 40 days"[6]. Personally, I find it to be quite a fascinating read.
# Input language
Don’t try to come up with your own language just yet. Go with an existing educational language instead, and focus on learning about compilers. ChocoPy[7] is specifically designed for classroom use in compiler courses and by extension is great for a hobby compiler project.
# Target language
Many educational compilers emit MIPS assembly. Although it's possible to run it using an emulator, running a native executable produced by your own compiler feels much more rewarding. So I'd suggest your compiler emits x86-64 assembly.
For educational purposes, I'd avoid targeting languages such as LLVM IR or C. Emitting assembly helps understand many important concepts better:
- Managing a function's stack frame
- Calling conventions
- Finding the memory address of an identifier
- Converting expressions into assembly
- And so on and so forth.
[1]: https://mykolam.net/posts/toy-compiler-of-scala-subset/2-loo...
[2]: https://craftinginterpreters.com/contents.html
[3]: https://www3.nd.edu/~dthain/compilerbook/
[4]: https://github.com/rui314/8cc
[5]: https://github.com/rui314/chibicc
[6]: https://www.sigbus.info/how-i-wrote-a-self-hosting-c-compile...
[7]: https://chocopy.org/