A simple x86 assembler C++ template metaprogram | Hacker News Reader

A simple x86 assembler C++ template metaprogram

blog.mattbierner.com

88 pointschesterfield22 comments

Years ago I made an x86 assembler using Common Lisp macros to expand instruction descriptions: https://github.com/rayiner/amd64-asm. Meta programming is incredibly powerful, more so when your macro language is just your host language.

The tests suite is neat (imho). At runtime it walks a data structure built at compile time to randomly exercise each of the possible encoding clauses of the instruction and compare the output to Nasm. Writing that was a very "lisp is for building organisms" moment for me.

As an aside, x86 makes a lot more sense than people give it credit for.

And I asked about this just a few days ago! https://news.ycombinator.com/item?id=10239000

Do you use it? Is anyone aware of a python way of this?

Just a weekend hack, alas. But it generates identical code to NASM for the subset of x86 it handles, so the basic mechanics should be correct.

I wrote a programmable assembler in Python on a consulting project (I needed to write a debugger that used only direct memory access and no other debugger hooks). It wasn't difficult.

Later, we did one in Ruby, which allowed our assembler syntax to almost mirror AT&T assembler while still being pure Ruby.

These are easy libraries to write; people should just write them themselves.

These are easy libraries to write; people should just write them themselves.

Well, if you say so! Seriously though, isn't x86/64 opcode set gargantuan in size? Not to mention the maintenance and what to do with parallel constructs. It doesn't sound easy to me, apart from trivial text emit experiments. I haven't tried writing one though, so I wouldn't know. Also, what's easy for someone isn't easy for others. I find image processing and rendering relatively easy for example.

(a) No.

(b) You don't need even most of the instructions to get to a useful, fun place.

I was surprised by how easy it was to get to that place. Take my word for this. Read this:

http://reocities.com/SiliconValley/heights/7052/opcode.txt

and just start coding.

Ok, thanks for the tip and link. Weekend here I come!

I've done a similar thing in Scheme to learn Intel assembly (it only supports a small subset of it, and produces standard .as output files instead of byte code, and I have not used it for anything in production so far):

https://github.com/pflanze/hasm

I implemented a number of higher level features for structured programming. Example input file: https://github.com/pflanze/hasm/blob/master/examples/fact.sc...

It doesn't use Scheme macros but expands assembly macros using custom code (because the bottom layer isn't Scheme anyway, just S-expressions, and so that I could let the macro expanders access their context).

Of course the difficult part would be to grow the higher level features in a way that they are safe, and I'm not sure how useful they really are when allocating registers manually. Once you build in a register allocator, you'd probably call it a compiler.

I did something similar. I needed the C and C++ libraries plus compiler but hated the languages. Used BASIC-like 4GL's for their power and prototyping speed. So, I just made a knockoff of a BASIC in LISP with naive compilation to C or C++. In dev mode, it just executed functions instantly as LISP. Separate mode generated code from same LISP-BASIC. Added 4GL generators as macros. Got extra benefit of interactive development and incremental compilation. Aside from first integration of libraries, I got the benefits of C/C++ without any of its pain in most cases. Ultra-fast development, too, with few lines of code for common application types.

I miss that tool. Lost it in a triple HD failure along with the others. Closest thing I've seen to my approach is iMatix's combo of DSL's and mini-4GL-for-C for more productive, correct, C-language applications. Racket has potential to exceed anything I did and for more languages if applied wisely. It's main option if I try to rebuild my tool.

Btw, what would I call a tool that combines LISP macros, 4GL features (eg DSL's, generators), C or C++ data-types, and auto-generation of C code? It doesn't necessarily need LISP syntax: one prototype used Tcl style & final looked like BASIC w/ compatibility for BASIC tools. It fits as a systems language, a 4GL, a LISP, and a C/C++ superset all at once.

I made something very similar ages ago: [link redacted] -- I remember struggling quite a bit with machine code generation until I discovered some deeply buried forum message emphasizing that the x86 architecture was originally octal-based.

until I discovered some deeply buried forum message emphasizing that the x86 architecture was originally octal-based.

To "unbury" that fact some more, I'll again link to the message I think you're referring to...

http://reocities.com/SiliconValley/heights/7052/opcode.txt

(Google still refuses to index that link for some reason. IIRC it used to have a very high ranking, but disappeared from search results within the last year or so.)

Yes, that's the one :-)

There is an updated version of this text:

> http://www.dabo.de/ccc99/www.camp.ccc.de/radio/help.txt

  Some bugs were corrected June, the 20th, 1997 by S.Klose (sven@devcon.net)
  (minor bugs in 32bit effective addresses and opcode typoes)

The syntax vaguely reminds me of HLA:

https://en.wikipedia.org/wiki/High_Level_Assembly

I'd be cool if one could leave runtime elements in there, for example leaving an intermediate value as x, with the C++ template magic reducing everything down to producing the sequence of bytes with just one integer to be filled in.

Then this could be used for on-the-fly code generation like you'd see in a jit-emulator like qemu.

For use of a subset of C++ as the assembly language for your architecture, see https://www.youtube.com/watch?v=LitwrElTCLs&t=11m33s

This type of approach works well in image processing, where you can use C++ for the repeatable, hierarchical patterns and assembly for optimizing some key loops.

Expected to see brainfuck code. Actually saw a pretty neat collection of tricks! Well done :)

fork(2) me, this is one cool application of templates. I wish it were as comprehensible as Lisp macros :-)

i used to use that same function pointer cast trick to execute bytes from memory a long time ago.

you should find that it fails in most modern execution environments...

in legacy/desktop windows you can virtualalloc things to be executable but the same is not true in the general case, and this code doesn't do that.

even a long time ago (before windows 8, ios etc.) i wasn't able to just execute things on the stack or heap reliably and had to write code that explicitly allocated executable memory.

The assembler strings are not allocated at run time, but stored in a read only section of the program (which apparently is marked as executable, if this works)

yeah, but the fact that the static data area is marked as executable is luck rather than reliability.

Reply on news.ycombinator.com