Hacker News Reader

Comment by ghoward | Hacker News Reader

The answer to this is easy. At least it was for me; I didn't know it was such a problem.

My solution is [1]. It generates a C file with a specific array name passed in through the command-line. It also has a few other niceties that I need.

It works on Windows, Mac OSX, Linux, and the BSD's, no matter the compiler or linker.

I use it to generate the arrays for the help texts ([2] and [3]), as well as two math libraries ([4] and [5]).

People are welcome to adopt and adapt it. Just follow the license, as per usual. I've even adapted to my other software. [6]

[1]: https://git.yzena.com/gavin/bc/src/branch/master/gen/strgen....

[2]: https://git.yzena.com/gavin/bc/src/branch/master/gen/bc_help...

[3]: https://git.yzena.com/gavin/bc/src/branch/master/gen/dc_help...

[4]: https://git.yzena.com/gavin/bc/src/branch/master/gen/lib.bc

[5]: https://git.yzena.com/gavin/bc/src/branch/master/gen/lib2.bc

[6]: https://git.yzena.com/Yzena/Yc/src/branch/master/tests/strge...

View on HN

I use xxd for this and the problem is far from solved. Around 5MB or so, the compile times explode. Somewhere around 7MB I could no longer self-host on 32-bit platforms and had to cross-compile from a 64-bit machine with more memory. Around 10MB the compilers start crashing. These are not very big files. The time added for the couple of resource files far, far exceeds the time it takes to compile the whole rest of the project. I had to add special compile-time options to exclude resources for development builds so I wouldn't drive myself crazy with the long compile times. I previously used all the platform-specific methods mentioned in the article and they don't have this problem, but it's an absurd amount of complexity to handle every platform's little way of doing it in my Makefile.

I wrote a Lua script to do the same thing once.

Mentioned it to an old Unix hand, she replied "hmm, what about `xxd -i < file`"?

Live and learn.

Considering I knew I had to make it work everywhere, and that included not depending on tools that are not standard in POSIX, I still think I made the right choice.

But yes, xxd is an easy solution.

I still use the Lua script, why not?

Granted I can replace it with xxd and two lines of awk, but that would have taken longer, and what's done's done, it's not a task with shifting requirements.

I don't know who needs to know this, but if you pull xxd.c out of the vim codebase, you can compile it by itself without building vim. It doesn't depend on any other part of vim. I just vendored that one file and ditched the rest of vim.

How do you feel about that problem the parent blog post mentioned, of this being slow for large blobs particularly when compiling with Clang?

You can split it up into several files, then concatenate the arrays at runtime.

This can actually be solved by writing out to assembly and using an assembler to make the .o file to link in. That can deal with huge files with constant memory.

I've just run a benchmark on my code.

It did 13875457.34 bytes per second, which means my program would have done the 82 MiB file he had in 6.20 seconds, faster than hexdump, GCC, and Clang.

It used a max of 3223848 Kbytes, which is about the size of the file it was processing. (The file was 3300000020 bytes exactly.)

I also tried with a file as close to 82 MiB as I could get. It used 85168 Kbytes max, and it took 6.77 seconds.

My code could probably be optimized too. It tries to skip a header comment. It also reads all of the input file in at once, when it could probably stream it on demand. It is also checking for stuff to exclude, which takes time.

If I take out the if statement that begins with:

    if (!strncmp(in + i, bc_gen_ex_start, strlen(bc_gen_ex_start)))

That should remove most of everything else that does not matter.

If I run that on the 82 MiB file I made, it uses 85124 Kbytes and 5.40 seconds.

This is still far slower than objcopy and incbin, but maybe it's good enough, right?

I think I could optimize it more, but I still think that's a pretty good showing against the competition, especially for portability.

Edit: I forgot to mention that I did these tests while running a fuzzer (AFL++) on 15 of my 16 cores and while watching YouTube. I didn't want to stop the fuzzer just for this (it's been running for more than 24 hours).

But that’s just to generate the C file, right? You still need to compile that C file, which is the biggest bottleneck.

Argh! I feel dumb. You're absolutely correct. My bad.

Reply on news.ycombinator.com