How Fat Does a Fat Binary Need to Be?
justine.lol
justine.lol
* on x86.
There was a time period when it was reasonable to just assume everything that mattered was x86. It started sometime in the early 2000s, as non-x86 servers died out and Macs switched to x86. And it ended last November, when Apple announced Macs would switch to ARM.
The author has suggested sticking with x86 and handling other architectures using emulation. But x86 is the worst possible architecture to emulate, because its strong memory ordering model means that emulators on other architectures essentially must limit the emulated process to a single core. (Unless you have special hardware support like M1 Macs do… for now.) Also, any hardware CPU architecture is going to be slower to emulate than a purpose-built bytecode – and we already have a good enough bytecode, WebAssembly.
Emulating x86 might make sense anyway if you assume that x86 is and will be the norm, and non-x86 an edge case that can afford to take a performance hit. But that sure doesn't seem to be the way the wind is blowing.
It's a fun project, but I don't think the assumption that everything is x86 is as safe as the author proposes.
I think improving performance on other architectures is great. There's nothing that inherently limits the Actually Portable Executable format from baking in native support for other architectures. It's simply a question of focus.
What Cosmopolitan does, it does really well. No other C library is able to target multiple operating systems and all the x86 microarchitectures. You can now easily distribute software that can be easily run on the vast majority of PCs and servers. Want it to do more? Consider becoming a sponsor. The impact the project is already having should be all the proof we need that it could be doing much more. In order for that to happen it needs your support.
Keep in mind this compiler doesn't optimize. So if you're OK taking a 2x performance hit, you're going to find that your code compiles significantly faster than GCC or Clang and you're going to have the convenience of the fact that the whole toolchain will be a single .com file that you can just scp on any server for a fully deterministic reproducible build experience. It's also the most readable and hackable compiler codebase I've ever seen, so if you've ever wanted to pull out the dragon book at try your hand at optimizing, you'll find that chibicc is fertile ground to do it.
git clone https://github.com/jart/cosmopolitan
cd cosmopolitan
make -j8
o/third_party/chibicc/chibicc.com -include libc/integral/normalize.inc -o hello.com.dbg examples/hello.c o/cosmopolitan.a
o/third_party/gcc/bin/x86_64-linux-musl-objcopy -SO binary hello.com.dbg hello.com
./hello.comHow?
Thank you Hacker News for your support!
By "baking in native support for other architectures", I guess you're thinking of an approach similar to Apple's "fat binaries", which are just multiple independent binaries built for different architectures bundled together? That's a good pragmatic option, and definitely the best choice size-wise for small programs. On the other hand, it lacks forwards-compatibility: you're limited to the set of architectures the original developer decided on. And for large programs, the cost of N copies of the program can add up.
Personally, I'm a fan of bytecode VMs such as WebAssembly. That way the binary can be truly architecture-independent, capable of supporting many different architectures or special architecture variants without bloating the size of the executable. On the other hand, if you wanted the binary to run out of the box with no dependencies, you would have to bundle an entire VM implementation into the binary for at least some architectures, and then you'd have your own share of "hello world" bloat. (I'm assuming there would be an option for the user to supply their own VM for less common architectures.)
I do wonder how small one can make a WebAssembly VM, if its design prioritizes minimal size over maximal speed, while still aiming to get reasonable speed (i.e. not an interpreter)…
Optionally you could consider a binary that on first run (&prompt?) removes the binaries irrelevant to the current host.
This is revisionism. It ended with the smartphone and netbook era, both of which put more ARM devices in homes than Macs ever will.
The EeePC was x86, the Chromebook Pixel was x86 along with many that followed with balance being ARM variants.
More recently you have convertibles/ultrabooks and their stripped down cousins (e.g. Microsoft Surface Pro vs. Surface 3/Go/Go 2 and Surface 2, Surface RT, and Surface Pro X the latter 3 being ARM).
Ultimately the netbook niche has been squeeze by ultrabooks and tablets to few offerings if anyone is even bothering to market computers that way anymore.
By "everything that mattered" I guess I should have said "everything that mattered that you have a shell on".
For some of those non-desktop-operating-system devices, getting a shell is impossible without bypassing lockdown measures. For others, it's possible, but not particularly common as far as I know, for a variety of reasons. They tend to run stripped-down OSes; they don't have great keyboards; they were pretty slow compared to desktops until recently.
Yes, this is a huge overgeneralization. Plenty of people do run shells on their phones or routers. But I do believe that a major desktop platform switching to ARM is qualitatively different from what we'd seen before.
EDIT: Alright, I found something to say. Everything is stunningly perfect about this project except for one thing: the disturbing "mock greek" (using d as o and m as u, etc). Is it a sort of modern utf8 version of 1337-speek? Maybe it is too ironical to understand?
EDIT2: OK, now I get it. The "actually portable executables" use an amazing hack where the PE header itself gets executed as machine code, it turns out to be mostly harmless, and execution is captured from that forward, using other data parts of the header as code. This is the same thing your brain does when reading letters in mixed alphabets. It is confusing when you know both alfabets and try to read on each, but if you squint your eyes, then only the actual shape of the characters matters, and you can read it for its intended meaning. Thus it is a sort of self-referential title which is perfectly appropriate here.
I'd probably be most worried about the bourne shell part. It seems like it'll be easy for this to break on any "non-standard" shell that Linux might execute.
I'm thinking stuff that requires other dependencies, e .g. build a universal Python interpreter by using Cosmopolitan. Would that be possible?
Pure computation and file/socket based IO is enough to solve many problems, but I'm curious what the plan is down the road to support more exotic forms of IO like high performance graphics, audio, human input devices, etc. in a portable way?
It seems like the fully-statically-linked approach can only get as far as the common subset of all supported platforms, which is probably a tautology, but means that the cosmopolitan library will have to not only provide APIs for any possible type of IO someone might want to do, but also bake in all the userspace code to implement it on every platform. For a lot of things that seems currently very difficult, as many hardware vendors only provide dynamic libraries as their interface. I suppose statically linking isn't mutually exclusive with dynamically loading, though?
The other route seems to be targeting a virtual machine of some sort, whether it's running the code and providing a consistent ABI like qemu, or communicating over files/sockets like rendering graphics and sound in a web browser. I get the impression that's the less interesting route to go down though.
I appreciate the holistic nature of the project, and it seems like you've approached it with a pragmatic mindset. Myself thinking pragmatically, it just seems like you'll eventually want to at least support opengl or a GUI library or something, to appeal to the types of programs that really benefit most from easy portability.
Cosmopolitan can be doing more if game studios are willing to back the project. Remember how John Carmack dipped his toes in the water with Quake on Linux decades ago and was promptly scared off by the chaos and complexity of the open source development process? Cosmopolitan brings the kind of order to Unix interfaces that game developers need. Games can't rely on people like distro maintainers to patch and recompile their codebases every time there's a change to some dynamic library. They need to be able to compile their programs once and have game enthusiasts be able to enjoy those programs for years to come. Right now anything you build with Cosmo will run on every Linux distro since 2007. That means there's a good chance your programs will continue to work perfectly and natively on every Linux distro in the future. You might not be able to run shaders on Nvidia chips yet. But what we've accomplished so far is a big leap forward compared to the status quo.
Can you make a program these days that runs natively on M1, x86, PowerPC, and 68k Macs?
Edit: Just realized this all might have been happening at the same time. The ‘90s are a blur!
.kkrieger may be the most notorious in the game category but 96k is actually on the heavy side for such productions. One other example of a really small game is BootChess, a chess engine in less than 512 bytes to fit in a BIOS boot sector.
And you should look what people are fitting into 4k intros nowadays.
Is there some kind of trickery to make a file both PE and ELF valid? Doesn't magic numbers conflict?
https://en.wikipedia.org/wiki/Common_Intermediate_Language
https://www.i-programmer.info/programming/other-languages/93...
Can I use it to make a Rust executable? Maybe it can post-process the executable built by Rust somehow? (I guess Rust needs to link against the cosmopolitan libc and otherwise not make any syscalls)
Maybe it's possible to make a cosmopolitan Rust target, like, x86-unknown-cosmopolitan?
Also, does it support x86_64? (and maybe a fat x86/x86_64 binary as well?)
On the other hand, Docker images is literally the only thing taking up space on my hard drive by now.
[0](https://www.cosmopolitan.com/sex-love/positions/a26874/passi...)