Show HN: T2b – A wicked-powerful text macro language for building binary files

Interesting. I have several suggestions though:

- the hex modifier does both turning mode on and off, which will be confusing in a longer file

- it would be good if prefixes such as 0x or bx were handled to temporarily override the current setting

- using "get var" to actually output something is weird, i'd use "put"

- add a way to handle endianness

Trying not to be a downer, but I wouldn't think of writing such a tool. I'd use Perl's pack or Python's struct.

Of course there might be some value in something less than a full language.

Cool idea! Clear and nicely organized too :)

Wouldn’t it be more readable and less error-prone to have separate commands for turning hex mode on and off? Such as `hex on` and `hex off`?

In addition to "pack" and "struct", for prior art (or inspiration), look at any number of structured fuzzing frameworks (Peach might be a good example); constructing arbitrary binary formats from a recipe is how you build a structured fuzzer. I built one a few years ago that pursued a sort of DOM model, with id's and classes and path strings.

In the other direction, given the ability to spit out one binary character from the shell, you can bootstrap all the rest of this in pure shell script. I cheated and gave myself int{8..64} and did a whole ASN.1/DER in shell script. The shell has better control flow than custom language (but the custom language is a little simpler).

Length-prefixed strings would be useful for some file formats. You'd need to be able to pick a size for the length-prefix, such as u8,u16,etc.

Also some ability to length-prefix the output of a macro might be useful for tag-length-value-like formats.

For Unicode, ability to write out UTF-16 or UTF-32 would be useful, also CESU-8 and Java's Modified UTF-8.

Also, for a challenge, think about how you could write macros to generate ASN.1 formats such as BER. You'd probably need some more smarts in your macro language to handle such a task gracefully.

Also, how does one control the endianness of output? Maybe you need u16le and u16be, or some kind of general endianness modifier.

Output of IEEE floating point might also be useful in some applications.

> t2b always writes to stdout. To output to a file, simply use a pipe (|).

Shouldn't this be use output redirection (ie. >)?

I use fasm for this very use case. It has a wonderful macro language that can be used to programmatically generate binary data on top of being able to emit instructions. It's really cool to see other languages for generating binary files as its an interesting niche to fill.

Nice. I once did something similar for parsing and generating binary files from a single specification: https://abau.org/hannah

Not sure if I will use it, but I like it. This is as close to raw bytes as it gets despite some HLL like features.

This to me is real hacking, saying what the heck and writing a tool for oneself instead of looking around and getting lost in the multitude of 'mature' options that exist. I am sure hacking this was quicker, and more fun, than browsing all the available options, picking one, installing the tools, learning to use them and getting the desired result.

Now, how long till someone decides to bootstrap this - use it to hack a binary that compiles it?

Looks good. I can see this being a great intermediate format for hacks etc.

Alternatively you could use some kind of fluent API to write binary data. Depends on your use case of course.

Neat! I could used this for generating mixed binary/ASCII payloads for network protocol testing. One issue, though -- it seems like trailing whitespace in the input isn't handled correctly. It seems to be picked up by the command processor and treated as a duplicate of the previous command. Amusingly, I was able to diagnose this using t2b by piping its output back through itself:

   c:\mingw\MSYS64\usr\bin\echo -n hex u32 20786568 u32 36203875 u8 31 u8 0a u32 34203875 u8 31 | t2b

   Output: hex u8 61[LF]u8 41

   c:\mingw\MSYS64\usr\bin\echo -n hex u32 20786568 u32 36203875 u8 31 u8 0a u32 34203875 u8 31 | t2b | t2b

   Output: aA

   c:\mingw\MSYS64\usr\bin\echo -n hex u32 20786568 u32 36203875 u8 31 u8 0a u32 34203875 u8 31 u8 0a | t2b

   Output: hex u8 61[LF]u8 41[LF]

   c:\mingw\MSYS64\usr\bin\echo -n hex u32 20786568 u32 36203875 u8 31 u8 0a u32 34203875 u8 31 u8 0a | t2b | t2b

   Output: aAA

I don't know very much C++ yet, so I'm not sure of the best way to fix this.

"It's now feasible to write a machine code compiler in shell. Hooray. Not sure why you would ever do that to yourself, though."

For more trustworthy bootstrapping. It's why I included shell compilers on this site:

https://bootstrapping.miraheze.org/wiki/Main_Page

Excellent idea. Throw in elf support and you’ll never need a linker for c code again.

I really wanted something like this when embedding serial numbers and keys in ROMs. At the time I was mostly working in C and PHP, so I was imagining something closer to a template language, but something like this would have given me a good intermediate step where I could have used PHP to render this text format and then ran that to produce the necessary bits on disk.

A decade late for me, but still looks cool.

It would be great if I could specify the number of bits per field like in Elixir/Erlang/C++ bit fields and then also specify what the extra bits should be set to if the total number of bits isn't divisible by 8. I know that document strings are helpful but writing it into the code would be much more robust.

There is already https://flatassembler.net/ which uses macros to emit ELF and PE headers.

> .idea

What's the purpose of checking this stuff into version control?

How does it compare to https://kaitai.io ?

If you use the prebuilt binaries, does the GPL apply?

Not as good as https://github.com/oriansj/mescc-tools But certainly a good start