Tools for Learning LLVM TableGen | Hacker News Reader

Tools for Learning LLVM TableGen

blog.llvm.org

150 pointsmatt_d23 comments

TableGen factors out a lot of detail, out of the code and into a collection of tables of records which are then processed into .inc files by TableGen backends.

Emphasis on factors out as in even if the result is eye wateringly complex, it's much better that that complexity is in another declarative file. This factoring also allows reuse by other tools like debuggers and linkers.

Emphasis on a lot of detail because 500,000 lines of instruction, register, scheduling, ... information is encoding a lot of detail. Processed, --print-records generates 87M bytes of text just for the X86 backend.

Take the simple BPF backend. It has 5 TableGen files:

  BPF.td
  BPFInstrFormats.td
  BPFInstrInfo.td
  BPFRegisterInfo.td
  BPFCallingConv.td

TableGen streams these files and processes the unloveable TableGen language over them producing a collection of records. The backend's CMakeFile.txt tells TableGen to generate .inc include files. Each of these -gen flags invokes a TableGen backend to generate a specific include.

  tablegen(LLVM BPFGenAsmMatcher.inc -gen-asm-matcher)
  tablegen(LLVM BPFGenAsmWriter.inc -gen-asm-writer)
  tablegen(LLVM BPFGenCallingConv.inc -gen-callingconv)
  tablegen(LLVM BPFGenDAGISel.inc -gen-dag-isel)
  tablegen(LLVM BPFGenDisassemblerTables.inc -gen-disassembler)
  tablegen(LLVM BPFGenInstrInfo.inc -gen-instr-info)
  tablegen(LLVM BPFGenMCCodeEmitter.inc -gen-emitter)
  tablegen(LLVM BPFGenRegisterInfo.inc -gen-register-info)
  tablegen(LLVM BPFGenSubtargetInfo.inc -gen-subtarget)

So here's the deal. Yeah, TableGen is ungainly but then the problem that it solves is truly massive. If you were to redo it now with what we know now, the architecture would be very similar and the language would be only slightly better. TableGen solves a really hard problem, maybe not perfectly but well.

I wonder if that could have been achieved without an actual DSL. TableGen ultimately produces definitions for record types and a list of fully expanded records in those types, but those "types" don't have to be actual types during the generation phase---you can instead have a generic associative array that gets validated when every record has been generated. A distinction between `class` and `multiclass` can be removed by using an ordinary function that can generate one or more records. So I feel that everything can be done without the DSL, except for a bit nicer syntax.

> If you were to redo it now with what we know now, the architecture would be very similar and the language would be only slightly better.

Some parts are, of course, a matter of taste (compiling in backend code rather than having them be separate programs consuming an easy-to-parse canonical form; admittedly that’s in line with the LLVM ethos). But also, honestly, the language—late binding and all—seems a bit like a more ad hoc version of CUE. If I were to look for a tool for a TableGen-shaped problem, that’d probably be the direction I would be thinking in.

Kind of surprised to see an article about TableGen hit #1 on HN. TableGen is pretty arcane. I had some dealings with it in about 2016 while creating a backend for compiler targeting a new processor architecture. It was very poorly documented then - mostly you looked at what other people had done with other architectures and tried to fit that into what was needed for your new architecture. Anyway, I guess I'd be surprised if even 1,000 people in the world needed to actually do much with TableGen.

FWIW, its use has grown over time, with new applications most significantly including MLIR (Multi-Level IR compiler framework, which allows you to define your own intermediate representations as well as compiler passes operating on them), https://mlir.llvm.org/docs/DefiningDialects/Operations/

The docs have been improving, https://llvm.org/docs/TableGen/

I also like these resources:

Lessons in TableGen FOSDEM 2019; Nicolai Hähnle https://fosdem.org/2019/schedule/event/llvm_tablegen/

- What has TableGen ever done for us?, http://nhaehnle.blogspot.com/2018/02/tablegen-1-what-has-tab... - Functional Programming, http://nhaehnle.blogspot.com/2018/02/tablegen-2-functional-p... - Bits, http://nhaehnle.blogspot.com/2018/02/tablegen-3-bits.html - Resolving variables, http://nhaehnle.blogspot.com/2018/03/tablegen-4-resolving-va... - DAGs, http://nhaehnle.blogspot.com/2018/03/tablegen-5-dags.html

I found TableGen frustrating when I first started working with MLIR, in part because I expected it to be a layer of abstraction, but it very much is not. It is (in MLIR) mainly convenience tool for autogenerating boilerplate code. Once I realized that and got used to looking at the generated code, it became much easier.

TableGen has been the most annoying part of working with LLVM, it is unfortunate thy doubled down on it for MLIR.

If you want to generate code, define a better language and write a good compiler.

TableGen tries to be similar to C++, but is different enough that it is annoying.

And god help you if something goes wrong, error messages are straight up misleading.

> TableGen tries to be similar to C++, but is different enough that it is annoying.

It seemed to me that TableGen was trying to be a pure functional language more like Haskell than C++.

trying is the keyword here!

It's not entirely doubled down on. There is work on IRDL, a domain-specific language for dialect definitions, and PDL, a language for pattern definitions. In theory, those could replace tablegen one day. I haven't looked into either too deeply, though.

These are research projects you don't want to touch for anything other than publishing papers until they are adopted by $BIGCO and has ~5 man years of engineering effort.

They are upstreamed in LLVM and work today, though they are feature incomplete. From what I understand, the main reason you shouldn't use them is that development is stalled because the people who were working on them went to work at Modular.

They don't work today, give it a try.

The core dialects are upstreamed so that people don't reinvent them, but all tools to utilize any of it aren't. Some of it isn't even open source. For example, nothing else in MLIR uses PDL right now.

TableGen isn't trying to be similar to C++, it's trying to be a language that makes it easy to specify highly redundant data tables.

Now, when you're writing an LLVM backend, these tables are immediately consumed by another tool that tries to write a very complex state machine based on these tables, and the formats of the tables that said tool expects is essentially undocumented and confusing, and the error messages tend to range from confusing to, well, straight up misleading. But technically that's not the fault of TableGen, that's the fault of that TableGen-based tool.

I don't doubt the 'what', clearly it's needed.

My point was about the 'how'. See the following example code from the manual and tell me it doesn't resemble C++ if you squint.

class PersonName<string name> { assert ... string Name = name; }

class Person<string name, int age> : PersonName<name> { assert ... int Age = age; }

def Rec20 : Person<"Donald Knuth", 60> { ... }

Much better than other even more arcane tools, like DSLs based on C macros.

It has been ages since I delved into GCC internals, think GCC vs egcs days, but if I am not mistaken it was either DSL based on C macros, or some variation of GIMPLE.

GIMPLE is a joy compared to Tablegen in my opinion! It tries to do one thing well.

Sure, I only don't remember if the backend part wasn't a pile of C macros instead, for the code generation configuration.

Looking at https://llvm.org/docs/TableGen/ProgRef.html I've just added a partially working grammar/parser for tablegen in https://mingodad.github.io/parsertl-playground/playground/ an Yacc/Lex compatible online editor/tester (select "Tablegen-LLVM parser (partially working)" from "Examples" then click "Parse" to see a parser tree for the content in "Input source").

Any feedback is welcome !

As I know nothing about TableGen, can someone explain when you would reach for this tool? Perhaps even links to simple examples?

Unless you're making an LLVM/MLIR compiler backend you're not going to be reaching for this tool.

Basically this is how you configure a compiler backend, in LLVM's case.

It specificies how the whole set of internal data structures for instructions actually map into real CPU instructions, and also guides the register allocation algorithms.

Good all-round post, I know TG is not easy, but it’s quite powerful and I’d rather use it instead of something else that lacks.

Reply on news.ycombinator.com