Cosmopolitan Libc v2.0 | Hacker News Reader

Cosmopolitan Libc v2.0

github.com

210 pointsOrochikaku34 comments

> You should use /c/... paths on Windows rather than C:/...

This is the start of the realization that building C/C++ on Windows is only the beginning of the pain. Once you are past that, there is a multitude of semantic differences like paths, CRLF, the fact that file move is not atomic, the fact that an opened file cannot be removed (e.g. because Windows Defender or some other BLODA is scanning it), etc, etc. It's a death by a thousand papercuts. My suggestion is to avoid supporting Windows at any cost.

Yes we actually need to use exponential backoff when calling a lot of WIN32 APIs in our polyfills, due to the way Windows is. There's also layered service providers like Komedia which need to be worked around by a C library implementation. Supporting Windows also means that being falsely accused of being malware is the norm rather than the exception. It got so bad once that Google started delisting our websites. Microsoft even considered our Hello World executables a "severe threat". https://twitter.com/JustineTunney/status/1541801499210092544 The way I fixed it is every time I compile a program I upload it to Microsoft Security Intelligence and ask them to whitelist it. Sometimes I'll just zip up the build output directory into a giant zip file and send them hundreds of executables at once, and they'll whitelist them all. Needless to say, WIN32 related features aren't the highest on our list of priorities right now. We're actively working on getting Microsoft to address their biases regarding open source executables.

Regarding the C:/ vs. /c/ thing, we got away with not caring for the longest time, because our programs are generally designed to (a) always use relative paths, and (b) never call chdir(). However porting Python changed that, since Python likes to be clever with paths. So I'd say it's really the end of a realization, not the start.

Without supporting Windows, the whole thing makes little sense.

I would prefer recommending an openat-style solution where possible.

Having a single global root is in fact bad. Having paths be relative some origin is good. DOS was, in a funny way, ahead of the curve.

> We've polyfilled the OpenBSD pledge() and unveil() system calls for Linux. You can now use these APIs in your C/C++/Python/Redbean programs, or you can use the new pledge.com command to sandbox your existing Linux userspace.

> This change boosts SSL handshake performance from 2,627 to ~10,000 per second which is the same level of performance as NGINX at establishing secure connections.

I'm continuously impressed by how much functionality is baked in and the quality. Fantastic work.

If you're here and maybe learning about this space for the first time (or not), you might like to scan:

https://github.com/shmup/awesome-cosmopolitan

Congrats!

From "Getting started":

build/bootstrap/make.com -j8

Lol, do I take it that make.com is built with Cosmopolitan? If so, that's a nice demo of its purpose! Just download and run.

Correct. It's Landlock Make. It does pledge() and unveil() sandboxing. See https://justine.lol/make/ and https://github.com/jart/landlock-make and https://justine.lol/pledge/

I love it when I see this project here. Congratulations on the 2.0 release!

> You can now use Actually Portable Python by building o//third_party/python/python.com

Okay that is seriously amazing.

There is a strong need for a good, modern and well maintained cross-platform C/C++ libc.

I am not sure about the APE executable format, it is a nice and technically impressive feature but I would not mind if the lib dropped the support for it in the future.

This might be a minor issue for some, but I do not like having to distribute a .com file, it look suspicious.

Overall, this is fantastic work, very useful and inspiring, the kind of project that could help to fuel a C based apps renaissance.

We support that. If you want a freedom-optimized build:

    make -j16 MODE=tinylinuxbsd

Will produce ELF executables that run on Linux, FreeBSD, NetBSD, and OpenBSD. You'll never have to worry about being red-flagged by Microsoft. You won't have worry about Apple breaking your binaries. You won't have to worry about Glibc's license or wine binfmt_misc issues. Your binaries will be smaller, starting at 8kb in size. It's real nice and still supports 4x more platforms than the existing tools. If you use it, please note that "tiny" modes disable a lot of cool features like --ftrace and --strace. You can turn them back on by putting:

    ifeq ($(MODE), linuxbsd)
    CONFIG_CPPFLAGS += \
      $(BACKTRACES) \
      $(FTRACE) \
      -DSYSDEBUG \
      -DSUPPORT_VECTOR=113 \
      -O2
    TARGET_ARCH ?= \
      -msse3
    endif

In either ~/.cosmo.mk or build/config.mk. Then you'd say `make MODE=linuxbsd`. We also ship a `tinylinux` prebuilt amalgamation on https://justine.lol/cosmopolitan/download.html

Lastly, you can always change the .com extension to .exe. I would have chosen the latter, but I think UNIX has more in common with DOS than it does with Windows. I know people in the Microsoft camp hold negative associations with .com. But I didn't want people in other camps to think APE is just a Windows program.

Are you sure that a .com executable can be renamed as .exe and still work on Windows?

Unfortunately Windows might not be that smart.

I really like the tiny ELF executable running on any unix, very neat and not scary.

Having the same style of release for macOS and Win32 would be perfect.

Yes I'm sure. On Windows .com and .exe are the same thing. If you want just a Windows binary without any APE, you can use -DSUPPORT_VECTOR=4 and just change the .com extension to .exe. If you want just a MacOS binary, then use -DSUPPORT_VECTOR=8 and then run the assimilate.com program with the -m flag to convert it to Mach-O.

It seems that it should work: https://web.archive.org/web/20180223071736/https://blogs.msd...

Is there is there a good deep dive explaining how the actual portable executable works? The original blog post is kind of light on the details.

Not _exactly_ what you're asking for, because certainly you're looking for a write-up/blog, however, if you haven't peeked at ape.S you absolutely must:

https://github.com/jart/cosmopolitan/blob/master/ape/ape.S

It is very commented, in a good way.

Don't forget the linker script:

https://github.com/jart/cosmopolitan/blob/master/ape/ape.lds

It has a longform explanation in the comment at the top.

Sadly not. Everything I've read assumes you are already very familiar with the obsolete executable formats it uses (COM, 6th edition Unix shell scripts).

I presume it uses tricks similar to those files that are both ZIPs and PDFs simultaneously or whatever - basically they've chosen formats where the parser starts by reading different bits of the file.

But that's an educated guess because I don't think there's a clear explanation anywhere.

It can't be easily explained. There's no school for it. You have to learn by fiddling around. Try this "Tiny Portable ELF" tutorial I wrote a few months ago: https://justine.lol/sizetricks/#elf It shows you how to build your own executable from scratch that runs on four UNIX systems. If you build that code and try to improve upon it, then that should give you a pretty rock solid intuition and background for appreciating APE, since it takes that to the next level, by supporting Mac and Windows too.

It can definitely be explained! For instant your tiny ELF tutorial is currently missing any explanation at all - it says what you're trying to do, and a dump of code to do it, and a demo of it working but nothing about why it works.

But it's easy to add. From skimming the code I think the explanation should be something like this:

All four operating systems support the ELF executable format, and we're using the same x86 hardware in all four cases. When you compile a function like `int fib(...` the exact same x86 instructions end up running on all OSes. So why doesn't a Linux binary run out of the box on OpenBSD?

There are a few reasons.

First, the ELF format itself requires us to specify the OS ABI (conventions for how to functions at a binary/register level). Fortunately we can get around this by just setting it to the FreeBSD ABI because (insert explanation here).

There are a few other minor differences in how the OSes interpret ELF fields, like (insert explanation here) But they are broadly compatible and the following ELF header will allow us to start executing x86 instructions on all four OSes.

At that point we can run our `fib()` code and it will work! But the second issue occurs when we actually want to print the result.

The print a string on all four OSes you can use their `write()` syscalls to tell the kernel to write a string to stdout (file descriptor 1 on all the OSes).

Unfortunately although they all have the same ABI for making syscalls (maybe this is standard on x86??) they gave the write syscall different ID numbers - on Linux it is 1, but on the BSDs it is 4. If we were to try to execute Linux's `write()` on BSD it would actually call `exit()` - not what we want!

So we need some way of detecting the OS at runtime. There are probably many ways to do this but I have done it by detecting differences in how the OSes set up the registers and program arguments before calling `start()`. Specifically... (insert explanation here).

Totally doable! I'm sure some of that is wrong, but hopefully you get the point.

It definitely is harder to explain something when you are an expert on it, but it's still possible. Probably the best way to do it is to work with someone who is interested but doesn't understand it and then they'll ask questions that you assumed they new the answer to.

Anyway I'm not saying you need do this - it does take some time. I just wanted to impart that it could be explained if you wanted to.

Please take a look. https://justine.lol/sizetricks/#elf

That's a fantastic explanation and very interesting - nice work!

I'd love to hear how you do the full APE system if you ever get time.

Very cool! Is there a way to make a single executable file with the python build and our own python script files?

Indeed! Try this:

    # git clone https://github.com/cosmopolitan
    # make -j8 o//third_party/python/python.com
    wget https://justine.lol/ftrace/python.com
    mkdir .python
    printf 'print("hello world")\n' >.python/hello.py
    printf -- '-m\nhello\n' >.args
    zip -r python.com .python .args
    chmod +x ./python.com
    ./python.com

This is a great way for white-labeling the APE Python interpreter as your own program. You just put your source code inside the executable using the zip command. Then you add a .args file that specifies the default argv[] array if the user didn't specify any arguments.

Does it support ARM as well as x86? I don't see anything about that.

APE binaries work out of the box on ARM machines running Windows. https://youtu.be/CQ7Npip88Yw APE binaries work on Apple M1 with Rosetta, but it's not fully tested yet. We're still working on Linux + ARM.

Cool. ARM+Linux is relevant, that would support Raspberry Pis. Good luck!

Hoping for someone to implement the missing %f scanf directive.

> Binfmt_misc support for Linux users

You love to see it.

Dupe https://news.ycombinator.com/item?id=32518089

No, it isn't. Seven upvotes and zero comments don't count as having had attention. Please read the FAQ linked at the bottom.

> Seven upvotes and zero comments don't count as having had attention

Nothing about that in the FAQ or guidelines (please link if I missed it) and it was just one day after the linked submission.

> It's just bad form to reprimand others when it's not clearly rooted in the group consensus.

I am not reprimanding anyone. Indicating a dupe allows mods to merge the posts (and no, I'm not setting up a new e-mail account just for that).

> Doubly so when it's about my own prior submission that didn't get attention.

If you did post it within the past year, that submission is not showing up in search (unless you're operating multiple accounts to facilitate your high post rate..?).

Personally I'm glad OP did a repost that did get traction attention but I don't get the assumed hostility here at all.

Usually, even at zero comments, a story that has been posted the day before is considered a dupe.

That said, I'm happy this time around it got more attention.

That has never been the case.

But sure, everyone can have their opinions and interpretations. It's just bad form to reprimand others when it's not clearly rooted in the group consensus. Doubly so when it's about my own prior submission that didn't get attention.

Reply on news.ycombinator.com