Huh, you went a little further than I did then, cool. I also vaguely recall playing with some example code, but IIRC I wound up stuck with something that cycled my laptop's backlight every time I loaded it, and the documentation (and my poor attempts at googling) all pointed at that only being an issue if I was using code that was different, or something. Yeah I just gave up lol.
As for what I did get working... uhh... I did get graphics on the screen in the end, but the actual hardware I was testing with proved a tad flaky at the time and I never had the first clue how to figure out what was going on, so I ended up moving on to other things. I did make reasonable progress on the QEMU yak-shaving front before getting to that stage though.
- Compartmentalizing yak-shaving QEMU distinctly from everything else might be accelerated slightly by borrowing your distro kernel for a bit (because they generally always have the modules necessary to reach an initramfs compiled-in).
- You can skip out on needing a disk image, partition scheme and conventional bootloader by passing `-kernel` and `-initrd` (and `-append` to set the boot arguments). This neatly solves an entire dimension of issues at once, and is awesome.
- Specifying `-serial mon:stdio` (and `-append 'console=ttyS0,115200'`) writes serial port output to QEMU's stdout *while* giving you a way to fall-through to the QEMU monitor using ^A c, and insta-kill the VM with ^A k (orrr it might be ^A q or ^A x or something, I don't remember lol). This is awesome for bringup and furthermore irreplaceably practical as a way to get guest-side stdout (aka printf() \o/) onto a consistently-located spot on the screen (ie no pesky windows opening in random locations etc). (As a point of comparison, the -curses option does a fancy thing where 80x25 VGA textmode is translated into a curses window - yup, booting MS-DOS this way is fun - but in this mode terminal scrollback doesn't work; -serial mon:stdio doesn't initialize curses, so terminal scrollback works normally/correctly.) All this is complimented well by `-nographic` which disables VGA output, likely markedly irrelevant beyond bringup, but probably quite useful to begin with.
- QMP, the QEMU Monitor Protocol, is a JSON-based just-verbose-enough-to-be-annoying control protocol that lets you do nice tricks like telling running QEMU instances to reset/reboot (fully restarting QEMU will definitely slow things down a bit). QMP can listen via TCP or a domain socket. Thankfully the initialization handshake is effectively one-way, so IIRC you can just stuff the "yes hello version 1 blah blah" down the wire along with the reboot request as a fixed payload crammed through netcat.
- "Oh but of course" surprising-but-not-surprising thing: in order for the text console to have any practical purpose, writes to it must be synchronous so that the "about to load X" has fully made it (to the VGA text region|to video memory|out the serial port) before X happens. That synchronicity is a truly significant source of boot-time slowdown, and (maybe second to jiggling what kernel code is compiled in, what's in modules, and what's left out), completely disabling boot messages is one of the secrets to kernels that load virtually instantaneously. This is straightforward - `-append 'loglevel=0'` - aaaaand naturally happens after bringup :). Once you get there (and have commandline reboots working), with all the noise gone (QEMU prints nothing itself) you can make decisions about things like whether you'd like to preserve the output from previous boots in the terminal scrollback (incidentally `tput clear` resets that).
- Playing with an initramfs-less kernel (or the distro stock initramfs) can be useful to figure things out, but gets boring after approximately 2 minutes. One frequently-used approach to generating initramfs images is `find | cpio --create --format=newc > initrd.img`, but the cpio method has the caveat that the character and device files (/dev/console etc) must exist in the source directory that gets scanned from (and be of exactly the correct type/major/minor) for the resulting initramfs to work correctly. An alternative, even faster and IMHO honestly easier-to-manage way can be found in the `usr/gen_initramfs.sh` script that comes with the kernel, which basically just passes `usr/gen_init_cpio` a specially-formatted file list (see `default_cpio_list` for a (small) example - incidentally the cpio generated by that file is hiding in every kernel that doesn't have an initramfs embedded in it). This program has the benefit/advantage that you pass a virtual file list and can create character/device files and directories just by describing them in the file list. I would wager that taking the shell script apart and figuring out how to wrangle wrangling this approach is honestly less wall-of-text than it may initially look; you just describe what you want put where. (One note: the cpio format is cute - if you store /dev/console, but don't store a directory reference for /dev before it, /dev/console won't exist as far as the kernel is concerned >:D you have to spell out the directory entries!)
- Just in case... if you get the idea to unpack an existing initramfs with cpio perhaps to visualize how it works (and maybe start by repacking an existing known-working initramfs), you'll need to unpack the initramfs as root so its contents are given the correct ownership and permissions and cpio can create device and character files and whatnot (so that the initramfs gets re-packed with files that have the correct permissions and types and etc). In this scenario I point you at cpio's `--no-absolute-filenames` option; it is the negation/opposite of the `--absolute-filenames` option whose description is: "Do not strip file system prefix components from the file names. This is the default." This obtuse documentation is telling you that, by default, cpio will extract the image to /, which it will do successfully because it must be run as root; and you will presently find you have a very, very broken system :) - I had to reinstall Arch a decade ago because of this lol, not even completely reinstalling every single package on the system fixed it. Basically, if using cpio in extract/"copy-out" mode, forget `--no-absolute-filenames` at your computer's peril. With `--no-absolute-filenames` cpio will extract very happily into the current directory.
- HI I AM VERY LOUD AND I AM HERE TO TELL YOU TO NOTICE AND REMEMBER THE ABOVE POINT IF YOU TAKE NOTHING ELSE IN
- The kernel and initramfs are typically compressed somehow. You might want to do some experimenting to find what works best in QEMU - given that the input files will likely be sitting in the page cache, and QEMU's -kernel and -initrd options work by mapping the files directly into the guest's memory space in one go, skipping initramfs compression (on regen) and decompression (on boot), along with disabling kernel compression, theoretically has virtually no downsides. I forget whether I found gzip to (for whatever reason) be slightly faster despite all of this.
- Copying all the source files to, and generating the initramfs into, a tmpfs will basically mean that the process of re-cutting the image boils down to memcpy() with more steps. IIRC this is what took things from "cool! 5 seconds!" to ^S "...wat." ^S ^S "...how..." ^S ^S ^S ^S ^S "wat how is this possible" ^S ^S ^S :) QEMU can pretend to be Firecracker partially successfully :P
- In terms of actually doing anything at all in the initramfs, a first-class startpoint would probably be BusyBox, not just because it'll be very helpful, but because its (entirely approachable) build system wants to compile a static binary out-of-the-box by design. You'll need to make sure all its little symlinks are in place in the resulting initramfs (but you can also copy the busybox binary to /bin/sh in a pinch), but beyond that, the generated binary should work both on your host and inside the VM.
- Going beyond BusyBox involves selecting a libc. There's nothing stopping you simply just copying your distro copy of glibc into the initramfs, at which point everything you compile normally with gcc will immediately work (and this also sets up the playing field to be able to "argh what file does it need *now*" other stuff into the system, like for example gdb). The convenience of this approach is kind of hard to beat, but pursuing other options is generally how you cut down on initramfs size. I never moved on to properly playing around with alternative libcs, but I did note that diet libc includes an invocation wrapper that handles calling gcc for you. (And on a sidenote, getting remote host-side gdb debugging working would probably be quite the fun nightmare, but is probably the correct route to go down... because gdb will probably want to pull in python... :v)
- Under certain circumstances you might want to transfer data in and out of the running VM. Rewriting the initramfs and rebooting stays pretty fast up to ~20MB or so (!) so that works for getting even largeish amounts of data in (where rebooting is okay), but beyond that, and for two-way I/O, 9P protocol support is built into the kernel as a straightforwardly lightweight alternative to NFS, and `diod` works very well on the host/serving side. (The side project/quest is that the kernel must now support networking and you get to play with busybox's `ip` (or `dhcpcd` if you're lazy). Welp.)
- A lot (not all) of distro kernel configs enable the option to cram a compressed copy of .config into /proc/config.gz, which could work as a known-functional starting point if it's available. (Or you could just `make allnoconfig` then headdesk for the next few hours... this option does admittedly have a meditative component to it, and you learn where all the important stuff is :3)
- On real (Intel) hardware, I (very belatedly) learned about `i915.fastboot=1` only the other day, which skips the backlight cycle on boot when transitioning from the VESA or EFI framebuffer to properly accelerated graphics.
- A footnote that I'm mentioning for completeness: you'll likely use an approach where /sbin/init is a shell script that runs whatever tasks you want at boot time, with a `while :; do /bin/sh; done` at the end (if nothing else this saves you 100% CPU usage of one core from Linux's panic routine spinning the CPU in an infinite loop). This init script would likely stay open in your primary editing session and change frequently instead of having to hammer away repeatedly at the shell on each reboot.
- I mention ^S throughout - just to clarify, I use inotifywait to react to file save events, and in this situation I'd probably do something like `while true; do ./build.sh; inotifywait -qq -e moved_to .; done` (moved_to might need changing, see `inotifywait --help` and experiment with `-m`), where build.sh might call GCC, re-cut the initramfs, then hit the QEMU reboot endpoint. (Yeah, the workflow I ended up with had one terminal for build and another for QEMU/stdout. If I were revisiting this today I'd probably use `printf "\e[1t\e[5t"` to unminimize+raise the build terminal on errors and otherwise keep it minimized, but that's just my own workflow. (Sidenote, searching "man dtterm(5)" FTW. The terminal in question is long gone but everything else supports the same escape sequences. Haven't found a better reference yet.))
- Oh, look into `openvt`. BusyBox comes with a copy.
All of the above basically pretends Buildroot doesn't exist and reinvents it, arguably poorly :). (I should probably look into it sometime and make a proper comparison.) In all fairness Buildroot is legitimately probably slower than this approach.
At the point I got bored with all this and moved onto something else I was bikeshedding the filelist generator for gen_init_cpio (and debating just writing the cpio archive myself, although that would have been of marginal help). This was a moderate-difficulty devops problem: I had virtual files I wanted to always be present along with piles of real files from all over the place in different source directories that I wanted brought together into one image, I needed to ensure directory entries existed before their contents, and then I wanted to be able to toggle groups of things on and off depending on different requirements. This will realistically probably surface as a small "build tooling" concern at some point, because the flexibility afforded by gen_init_cpio (compared to cpio and being stuck with moving things in and out of source trees) affords sufficient complexity to encounter this sort of scaling problem. I guess it's subjective whether that complexity affordance is a bug or feature. All this is objectively a small trivial detail, I'm probably just remembering it as a bit of a saga because this is what I was working on when my attention span went out to lunch.