A proof-of-concept Python executable built on Cosmopolitan Libc (2021) | Hacker News Reader

A proof-of-concept Python executable built on Cosmopolitan Libc (2021)

ahgamut.github.io

151 pointsthunderbong39 comments

By "Actually Portable", this article means that one can build a portable executable of `python` (compiler and runtime) which can run on many systems, including Windows and Linux.

But the reality is a bit different . The article states that their result is slow and passes one third of the regression tests. So it's a proof of concept, not a portable executable one can use for real-world projects.

The reality is a bit different, the work on Python 3.6 was checked into the Cosmopolitan repo and I have been able to use it for production workloads that are in pure python. [0]

As Cosmopolitan Libc has evolved, it has been possible to compile more software without modifications, and that includes latest Python through a project called superconfigure[1].

Last person who tried to reproduce it from scratch did it last week (granted it too them a few days of solid work) but in the end they ended with a portable binary with Python 3.11.9, brotli, ssl and asyncio for their work related project.[2]

[0] https://github.com/jart/cosmopolitan/tree/master/third_party... [1] https://github.com/ahgamut/superconfigure/ [2] https://github.com/croqaz/cpython/

I find the use case where you have a system in production but could not possibly use the official Python for that system... unusual?

Double so since Python code (.py) is itself portable, and I would assume that the amount of work to make all the necessary extension be built into your APE vs just using the standard pip/pipenv/poetry/etc to be in favor of the latter.

Not even talking about the maintainability of the resulting system by other people who'd have to learn how to handle building an APE.

The use case was a client creating electrification plans based on structures from satellite data. We were able to get rid of gdal/pandas/networkx and several other dependencies and ended up with a fast python based process that could be given to clients for them to reproduce on their own machines (windows workstations).

In my use case, the niche is not having WSL/Docker available and letting end users repeat studies or re-run configuration scripts.

Python isn't built into Windows 10 by default and its easier for users to just run myape.exe than installing Python for Windows, pip install dependencies, etc.

Hey, this is my post from 2021, when I was testing Python2.7 and Python3.6 with Cosmopolitan Libc on an old 4-thread Haswell. It's now a lot easier to build Python (and gcc, gnu coreutils, curl etc.), and the binaries are faster, multi-threaded, and quite convenient to use. There are lots of interesting directions to explore when it comes to building software with Cosmopolitan Libc.

Thanks for updating the blog post too with the datasette screen recording, the speed difference is quite noticeable.

Adding it here since a few people were wondering about it in the comments, but feel free to check the original article for the 2024 update:

https://ahgamut.github.io/images/ape-datasette.gif

This article was written when cosmopolitan: https://justine.lol/cosmopolitan/index.html was v2. v3 has since been released: https://justine.lol/cosmo3/ .

https://cosmo.zip/ Also contains a python build https://cosmo.zip/pub/cosmos/bin/python , though I'm not sure if the problems mentioned in the article are solved.

most likely before v2 as well. The original post is from July 2021, which is a bit after Cosmopolitan Libc's v1 IIRC.

Yes, you're correct. I had in mind the 2022 date of the update. It also looks like you've updated the article again today! Thanks

For those wondering, the submission is a library / tooling usage that generates a single binary that executes across Linux, Mac, Windows, and BSDs. The author defines that as Actually Portable (which was the confusing part to me as we all know that python runs on just about everything)

fwiw the definition doesn't come from the author, it's how Cosmopolitan/APE describes itself (APE being Actually Portable Executable - as opposed to a mere Portable Executable, aka EXE)

Justine's αcτµαlly pδrταblε εxεcµταblε website: https://justine.lol/ape.html

Couldn't the same result be achieved by just copying an entire virtualenv? Though that is rather annoying to source every time and in every terminal.

Any binary dependency you have would need to be present for all target architectures in your virtualenv for this to work. But when you pip install a package only the binary for your current platform gets installed.

I presume you'd have the same problem with the binary, there'd have to be separate builds for arm, x64, etc.

There are separate static builds for aarch64 and x86_64, those are put into a "fat binary" that runs on current generation hardware/software combos (osx with aarch64, win with x86_64 and linux with both). Just building for two things is better than figuring out the whole matrix of OS vs architectures which was the other option.

For other architectures like Power/armv7/i686 the software can run using the Blink project [0] [0] https://github.com/jart/blink

If you are distributing software to an audience who want to use your thing without having to build it, then something like this is much more appealing than virtualenv.

This is a good place to mention https://nuitka.net/ which aims to compile python programs into standalone binaries.

Not binaries, but I've had success with Shiv[1] which builds a Python application into a single-file package that can be run on any machine provided it has the Python binary installed (but not much else). We use it to ship products that run as-is on both Linux (including WSL2) and macos.

[1] https://shiv.readthedocs.io/en/latest/

I built a Win10 binary with Nuitka just the other day, and was surprised to find the pyinstaller binary had higher performance at runtime. Pyinstaller also had several other advantages, such as producing smaller binaries, and building faster, and Win7 support.

For reference, I kept notes on the exact commands I used: https://github.com/9001/copyparty/blob/hovudstraum/docs/nuit...

I've had better luck with pyinstaller! Nuitka is buggy when you use multiprocessing

Same here.

Wrote a simple program to which I added Fire to parse arguments as a CLI, and Gooey/Tkinter for a GUI on top of it. To get a working standalone .exe file which does not require folders on the side, Pyinstaller did the job. Unfortunately it also triggers antivirus scans for some AVs..

I've tried nuitka before, and a recent question that occurred to me was: does nuitka have an option to output just C files? Something like:

  python -m nuitka example.py --no-compile

Might be interesting to see if the above is possible. We could get things like a completely-statically-compiled Python stdlib within the APE.

Based on the screen recording demo it looks unexpectedly slow. What's the bottleneck?

Software linked with Cosmopolitan Libc usually goes 2x faster than Musl.

- https://justine.lol/cosmo3/

- https://twitter.com/JustineTunney/status/1726141024597324189...

What exactly looked slow to you?

In the screen recorded demo gif in the article, it takes several seconds for the "/python" and "/cosmopolitan" requests to load (although the index page seemed to load pretty quick).

It's probably due to Chrome penalizing the connection for not being HTTPS

The interpreter is linked to a cross-platform libc-like library, and compiled to a funky executable format, supporting multiple OS targets simultaneously. The initialisation and startup are probably computationally-intensive.

We can see the initial startup speed from the time it takes the command in the terminal to print something, which isn't so bad (looks like under a second).

Maybe Flask is forking on each request, which I could imagine being slow, but not slower than the initial process startup.

ahgamut is a college student. He doesn't have $15,000 workstations like us.

My 4-core computer's probably building Cosmopolitan Libc in the background + the Python binary is 2.7 unoptimized. This blog post is originally from 2021.

The "Python", the "Actually" and the "Portable" parts.

I'm aware of both Python and Cosmopolitan implementation details and neither of these things explain a multi-second page load.

Python is Actually Portable - https://news.ycombinator.com/item?id=32245430 - July 2022 (120 comments)

https://gregoryszorc.com/docs/python-build-standalone/main/

HaHa! 2 weeks too late for April Fools...

The article's ambitious promise of an "Actually Portable" Python executable across diverse systems such as Windows and Linux ends up revealing a rather melancholic truth. What we encounter instead is a slow, underperforming proof of concept that passes merely a third of its tests.

Is this not, in some perverse way, reflective of our contemporary technological striving, where the ideal of universality often crumbles under the weight of practical limitations?

Are we not, in our pursuit of perfect tooling, always chasing a mirage? The reality, starkly exposed by these tools, might be that our pursuit is as flawed and fragmented as the tools themselves.

The good news is that it's 2024, this article is a few years old and many of the problems mentioned have been (or are being) ironed out.

Reply on news.ycombinator.com