Hacker News Reader

Comment by Gaelan | Hacker News Reader

Windows, macOS, Linux, and BSD all (usually) run on Intel processors, so in theory it should be possible to write a program that runs on all three. However, despite ultimately using the same instruction sets, the four OSes use different formats to store metadata about the programs, and have different ways for programs to communicate with the operating system. Justine Tunney created two things:

1) Actually Portable Executables, a clever way of formatting a program so that all four OSes interpret it a valid program in their own format.

2) Cosmopolitan libc, a library for communicating with the OS that handles each OS's interface, allowing programs to work on all four.

View on HN

Thank you (and thanks to everybody else for their helpful answers). Your note on intel processors meaning in theory programs should run on all operating systems was most helpful and got me across the line - now I get it!

That's pretty much it. Cosmopolitan Libc is the simplest project that could have happened decades ago. Anyone could have built it. There's never been a technical reason why an x86 binary can't run on all x86 operating systems. It's just that traditionally only operating systems publish C libraries, and they have no incentive to spend money supporting their competitors. So it's the kind of project that could only happen if it was done by an indie developer just trying to have fun.

Another reason is that it requires fairly thorough knowledge of 7 different programming environments. Most people don't understand 1.

The only programming environment I used to build this was Emacs on Linux. The rest was simply researching the magic numbers needed to use the binary interface of each system. I figured life's too short to be pulling my hair out with MSVC/Xcode warnings.

s/programming environments/systems/

How does one unzip the .com file? Is that even possible or have i misunderstood how it works?

I recommend using InfoZIP on the Linux or Mac command line. For example, to list the files that are in your .com executables you can run a command like:

    $ wget https://justine.lol/redbean/redbean-2021-02-27.com
    $ unzip -vl redbean-2021-02-27.com
    Archive:  redbean-2021-02-27.com
     Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
    --------  ------  ------- ---- ---------- ----- --------  ----
         426  Defl:N      215  50% 02-09-2021 09:24 f9cc9464  tool/net/redbean.css
         896  Defl:N      509  43% 02-09-2021 09:24 eb74f8a0  tool/net/redbean.html
       16958  Defl:N     6093  64% 02-09-2021 09:24 6a76bd39  tool/net/redbean.ico
        5073  Stored     5073   0% 02-09-2021 09:24 86c2afe6  tool/net/redbean.png
         554  Defl:N      285  49% 02-09-2021 09:24 0aa6870a  usr/share/zoneinfo/Beijing
        2335  Defl:N     1093  53% 02-09-2021 09:24 6ccd4450  usr/share/zoneinfo/Berlin
        2453  Defl:N     1170  52% 02-09-2021 09:24 7fed6d80  usr/share/zoneinfo/Boulder
    ...

Another ZIP tool that works well with APE is Windows 10 file explorer. But you have to change the extension from .com to .zip temporarily.

Does cosmolibc allow you to run on bare metal? If so, I've got an old project in mind that you may have just made happen, and I'm very very excited to know!

"Actually Portable Executable"s can boot from BIOS if that's what you're wondering.

https://justine.lol/ape.html

Cool, thanks! I missed that before. It makes a project idea I had way more realistic for me to implement.

What happens when intel goes bust and we all have to switch to new processors

A little optimistic about Apples SoCs aren't you?

Also...given that AMD exists and the x86 instruction set is not some magic potion ingredients...

They also run on AMD ;)

So it's like a compatibly layer like wine except built into the binary. Does this bloat the binary as well?

The only code that really needs to be duplicated would be the lowest level of userspace code that actually calls into the various operating systems. Things like the guts of "open", "read", "write", etc. Regardless of any particular OS's flavor, they're all going to roughly take in the same arguments and return the same results via some flavor of system call.

Each individual function is probably on the order of a few dozen machine instructions. Let's say there's 30 such functions that are implemented and 7 OSes to support. 36307 is about 7500 instructions. The average instruction length on x86-64 is 2-3 bytes. So maybe it ends up around 20KB of code space for a program that uses every possible operating system feature that's implemented?

I haven't looked at the implementation. I wonder how it selects between OS implementations. Does it switch on every system call, use a function pointer table, or do some sort of clever in-memory code rewriting?

Will this rush an optimized version of Electron or even deprecate it?

Technically, I guess you could use this to compile a universal browser binary that would run on all platforms. However, this wouldn't solve any of the problems caused by Electron.

Or really any of the problems Electron attempts to solve; the GUI toolkits are still not (generally) not cross-platform in the way Electron is.

I think his point was that with Cosmopolitan native binaries one would no longer need Electron to provide cross-platform binaries. There is a lack of good GUI frameworks to build this native Electron replacement with though, nor is it certain the effort would not be too high.

Note that packaged electron apps are not portable across OS, so it's orthogonal. What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary.

Meanwhile Cosmopolitan doesn't provide a portable GUI toolkit so it can't replace electron on its own. Maybe if you could package Cosmopolitan + Qt or something like that you could end up with something very interesting though.

> What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary.

Likewise for non-GUI apps, like command-line utilities that are implemented with NodeJS. As prior art, Fabrice Bellard's QuickJS is able to compile JS programs to spit out a native binary that requires no external qjs runtime. If you modified it to output Actual Portable Executable binaries instead, it would be even more portable than the programs installed with `npm install`.

I've done some experiments and progressed to writing tooling that approaches that I-want-ridiculous-portability problem differently. It doesn't use APE, so it still requires a separate interpreter (such as NodeJS) capable of executing the compiled output, but the object files it outputs are such that even if you don't have a copy of NodeJS installed on your system (or the right copy of NodeJS installed), then you can piggy back off the interpreter built in to your browser. You get a similar double-click-to-run experience, but when you do double-click to run, it launches in the browser, and you can go from object file to modifiable source trivially, since the object file and the Git repo from which it is built are automorphisms. The downside, if you consider it one, is that it doesn't work with arbitrary NPM modules. You have to really buy in to the scripting language dialect and the development philosophy to get anything out of it.

It would be interesting the use the QuickJS+APE strategy above so that there is _no_ reliance on an external interpreter, at the cost of marrying yourself to x86 and introducing some small hurdles in the path to modification and also giving up the safety of the browser sandbox, since you're back to running arbitrary executables.

Well, the main problem with Electron is that every app ships its own browser using tremendous resources at runtime, having huge attack surfaces and lots of unused functionality. So just giving one optimized electron version to all users, certainly does not solve the problem as APE does nothing that changes these problems.

Speaking of deprecating it, it might be one scenario, but developers probably don't just choose Electron because it is platform independent. I mean, there were other frameworks that supported that use-case even before Electron (e.g. QT). However, what is specific to Electron, is that you can use web technologies to build Apps. But that is also nothing you can do with APE.

So as much as I like to see one binary format for many operating systems, I doubt that it will change anything regarding the Electron problem.

There certainly multiple ways of solving it, but in my opinion, Apple and Microsoft must come to a common ground and support some kind of web-view that is built-in to the OS and extended with a platform independent API for the things that normal browsers do not support/allow, like filesystems access.

At some point both supported the Progressive Web App (PWA) movement for a while, but afaik Apple is the one holding it back at the moment. My guess is that they saw their App-Store business at risk. And even if PWAs were a thing, they are still some use-cases they can not cover in the current state (e.g. missing filesystem access).

the main problem with electron is js

Reply on news.ycombinator.com