Actually Portable Perl | Hacker News Reader

Actually Portable Perl

computoid.com

136 pointstosh71 comments

Shameless plug: my earlier project StaticPerl, statically linked Perl 5.10 for i386 Linux and FreeBSD: https://github.com/pts/staticperl

Differences:

* Actually Portable Perl runs on more operating systems, StaticPerl runs on Linux i386 (requiring i686 or newer processor), Linux amd64 and FreeBSD with Linux emulation.

* Actually Portable Perl has more of the standard Perl modules and C extensions bundled. StaticPerl contains only a few modules, simplified, with documentation and non-Linux support removed. StaticPerl is unlikely to work with existing Perl scripts with many use calls, but it is useful to write new Perl scripts from scratch, and these scripts will work with both StaticPerl and regular Perl.

* Actually Portable Perl executable programs are larger (5 MB minimum, StaticPerl is ~1 MB) because they are compiled for amd64 rather than i386, and they contain more Perl and C code.

* The first time an Actually Portable Perl executable program is run, it modifies a few bytes at the beginning of the program file (except on Windows). Thus these programs don't work on read-only filesystems unless they have been run on a read-write filesystem first. StaticPerl doesn't have this limitation.

Author here. Nice comparison. By default (and the released) APPerl binaries do not self-modify (they use the non-self modifying ape loader) however on unix-like they install the APE loader to`$HOME/.ape`and use that ELF executable to run the APE. To skip the fork+exec to the ape loader, they can be assimilated with `--assimilate`, that does turn the APE executable into ELF.

There is a kernel patch to add APE (Actually Portable Executable) support to Linux removing the need for the the apeloader or assimilating. I'm not sure what the status is on that.

I forgot to mention APPerl and StaticPerl have many overlapping use cases, but APPerl is more tailored to application packaging. APPerl is built with a patch to Perl adding argv[0] script execution, allowing the APPerl binary to turn into a binary for any of the embedded scripts, with a rename or symlink rather then needing to pass flags the Perl executable to load modules.

Could you suggest a way to do electron-like apps in APPerl?

I'm thinking of serving content on 127.0.0.2. There is no need for a high performance server like nginx, but serving files and having some CGI capability would be a plus.

I'd be interested in suggestions for the HTTP server part and maybe an example for how do serve an index.pl file that reads payload files packaged in the APPerl binary (ex: .js, .jpg)

I would first create a script that runs a web server that performs as expected (serving files, and handling dynamic functionality in the script or in a perl module, or by paths to cgi scripts as needed). I rolled by own for packing my personal http media server (MHFS), but there's plenty of HTTP servers on CPAN. I'm not sure which one's are best.

Then, I would create a custom build of APPerl using Perl::Dist::APPerl 's apperlm and modifying the apperl-project.json to pack in the script, CPAN modules, cgi files, and the static resources into a APPerl binary. When bundled into APPerl, the script can read the files embedded in APPerl from the /zip path.

Perl::Dist::APPerl (with tutorial on building apps): https://metacpan.org/pod/Perl::Dist::APPerl

MHFS's APPerl packaging config: https://github.com/G4Vi/MHFS/blob/master/apperl/apperl-proje...

> there's plenty of HTTP servers on CPAN. I'm not sure which one's are best.

Same. I know about HTTP::Daemon, HTTP::Tiny and HTTP::Server::Simple

If you have tried one of the 3 and can confirm it works with APPerl, it might save time?

If not, first with HTTP::Daemon (since it shows how to bind to a given IP in https://gist.github.com/mikkun/6388508, unlike HTTP::Tiny) I will try to release something similar to redbean but based on perl and DBD::SQLite

> When bundled into APPerl, the script can read the files embedded in APPerl from the /zip path.

That much I knew, but I was afraid there might be extra things needed

> MHFS's APPerl packaging config: https://github.com/G4Vi/MHFS/blob/master/apperl/apperl-proje...

Thank you! Then I might go with MHFS at first (tweaking its index.pl), if only because it's been tested by you with APPerl :)

> HTTP::Daemon, HTTP::Tiny and HTTP::Server::Simple

I haven't tested any. Pure Perl servers should be the easiest to get working. The Cosmopolitan Libc doesn't support fcntl F_SETFL O_NONBLOCK on Windows yet, and so non-blocking / event loop servers may be a no-go for now.

> I will try to release something similar to redbean but based on perl and DBD::SQLite

I'd like to see this.

> Then I might go with MHFS at first

I wouldn't consider MHFS's HTTP server "production ready" and suffers from the above mentioned O_NONBLOCK issue, but I'm curious how it would go.

If you haven't already, feel free to join the Redbean/Cosmopolitan Libc discord to discuss. (There's a link on the APPerl webpage).

FYI, building on windows make it detect Trojan:Win32/Wacatac.H!ml

I used your perl.com as an input. Fake positive? Or have you been infected?

Also, even if I cpan install the modules and have:

         "perl_repo_files" : {"cpan" : [
            "HTML-Template",
            "URI",
            "Class-Inspector",
            "HTTP-Daemon",
            "HTTP-Status",
            "IPC-Open2"
         }

I get:

C:\test> .\test.com Can't locate HTTP/Daemon.pm in @INC (you may need to install the HTTP::Daemon module) (@INC contains: . /zip/lib/perl5/site_perl/5.36.0/x86_64-cosmo /zip/lib/perl5/site_perl/5.36.0 /zip/lib/perl5/5.36.0/x86_64-cosmo /zip/lib/perl5/5.36.0) at /zip/bin/test line 9. BEGIN failed--compilation aborted at /zip/bin/test line 9.

Do I need to use https://github.com/G4Vi/MHFS/blob/master/apperl/download_pac... or something else to put the dependencies pm files in the zip?

It's mostly likely a false positive, but you're welcome to build from source with apperlm. The polygot binary format including the compressed ZIP archive probably sets off the scanners. The official APPerl binaries are compiled and uploaded by Github Actions: https://github.com/G4Vi/Perl-Dist-APPerl/blob/master/.github...

apperlm does not download perl module dependencies right now, but download_package.pl can be used to download dependencies. apperlm mostly just acts as a glorified front-end to Info-ZIP when doing a build of a "nobuild" config (building off of existing apperl)

download_package.pl example https://github.com/G4Vi/MHFS/blob/6db6ad24886ca0ea335a050228...

Edit: You can see what files were packed into it by renaming with .zip extension and opening with Windows Explorer.

Edit 2: "perl_repo_files" just specifies where to copy files to inside of the perl source tree. It is only used if you are building APPerl from source (it is unused if you are building a "nobuild" config). IPC-Open2 should already be bundled in the full perl.com as it's included in the perl5 repo: https://github.com/G4Vi/perl5/tree/cosmo-apperl/ext/IPC-Open...

Thank you! I've started actually playing with it besides the first tests, and it's great!

I'll see if I can have something akin to redbean ready this weekend.

Update on the APE Linux kernel support (for fork/ --assimilate)less APE execution. jart informed me: "It's in the process of being emailed out to the kernel developers. I anticipate it'll be merged within the next three months."

Perl is really one of the best languages and in modern times is underrated. I've been using it in production for most of my critical services since about 2012. I've used many other languages since then but always keep coming back to perl. It's a true gem of a language.

It's grand child, Raku, also seems to be an impressive language actually. Hated by Perlers and non-Perlers alike, it actually would be a great environment to build powerful programs in:

https://www.oreilly.com/library/view/learning-perl-6/9781491...

https://raku.org

>>Hated by Perlers and non-Perlers alike

Wrong. Perlers still cheer for it.

But for some reasons it hasn't wide spread adoption yet. We hope to use it some day in production.

Yeah I wish Raku would pick up some steam. It’s the only newish language I’ve looked at that’s made me say wow and it cleans up some of the thorny Perl5 sigil issues.

I always liked the sigils :-)

When I was first learning to program and picked up Perl 5 (I was interested in Web programming, so... that was the thing to learn at the time) and then later read some books (yes, kids, we used to read books) on Java and C++, I wondered how the hell anyone could make sense of them. And hacking up early Javascript snippets (you know, for image rollovers, the only thing anyone used it for for the first few years) was so painful. It's just words! There's nothing to indicate what anything is! The examples in the books were so hard to read.

The answer, of course, was "they use syntax highlighting", but I'd not made that discovery yet.

Oh Sigils aren’t bad. It’s just they’re more consistent in Raku, particularly with regard to arrays and hashes.

I don't think anyone hates it, but there were many people who were looking for a solid release that ran fast-enough. There was also... drama, which I feel no one cares for.

I messed around with Raku and it is seriously underrated. If anyone hasn't yet I would recommend giving it a try. It still does all the cool stuff Perl 5 can do but with the features you would expect from a modern language baked in. Raku and the core modules are solid and can be used in production but unfortunately the overall ecosystem is still struggling. 3rd party modules can be hit or miss. I found several of them weren't well maintained or seemed abandoned.

What do you typically use perl for? I thought it was most commonly used for scripting and running cgi scripts. I find bash to be much more readable and tend to use it or python for most of my scripts

> I find bash to be much more readable

Lol, you find bash readable? With all its weird ad-hoc constructs? I don’t pretend Perl looks pretty but at least its constructs seem to be consciously designed and do not look bolted on.

At my work place we still maintain a huge web application written in Perl 5. Most of the time if we are in trouble it‘s due to bad architectural designs that are not specific to Perl as a language or ecosystem. Only rarely I stumble over issues like implicit returns, or context sensitive sub routines. Also I hate Perl 5 allowing circular includes. The worst offenders among in-house modules are Perl modules written in a Java-esque style which mostly would have benefitted well from using native Perl features instead of trying to mimick Java programs.

What does Perl do in the web app?

Everything basically: Generating HTML, serving AJAX requests, communicating with other services, processing files, running DB queries etc etc. It even has a (questionable) feature embedding LuaScript using Lua::Inline and, hence, allows our clients to write little Lua scripts against our Perl API. Our application continuously runs over 20 years already and its user base kept growing ever since.

When I arrived I tried to move as much of the front end logic to JS but the backend remained in Perl as most business logic still lives there (and not in the DB). These days we try to implement new features in separate JS or TypeScript services and SPAs that, however, communicate with our REST API that is implemented in Perl/Mojolicious. This way we can hire cheaper JS devs (that never learned Perl) and people can still enjoy React and modern frameworks if they so wish. The most talented ones I try to sucker into little refactoring projects in Perl. Most JS developers that tried got fluent in Perl 5 only after a couple of weeks.

In the future we will try to move more business logic of the "Kernel" into the DB system since Postgres grew much more powerful compared to the time back when the application was invented. I believe this will be our only hope to actually shrink our Perl 5 code base eventually.

I thought it was just running some scripts. But that's interesting. Are there any performance differences between TS services and Perl ones?

What about develop time? how long does it take to deploy a feature?

Adding new features to a Mojolicious web service is quite painless, at least compared to CGI, Mason and other Perl stuff. Also a lot of code reuse is possible. Now we also have Mojo.js, brought to you by the same authors. It's basically the Perl framework with the same name rewritten in TypeScript.

https://mojojs.org/

Yeah I started to play with mojojs after your comment. Looks cool, thanks!

Well, the heavy lifting is usually done by the Perl backend. So you cannot directly compare the performance between Perl and JS/TS services in this case. From my web dev experience so far I believe the language's execution environment rarely matters at all unless you're doing something stupid. That's because most webapps wait for their DB most of the time anyway. Native compilation like in golang or rust does not pay off unless you're writing a streaming service delivering videos.

"I thought it was just running some scripts."

Ah, how times change.

Perl was the original practical back-end development language for the web. There's a bit of opinion in that, but not a lot; there's not a lot of other contenders, unless you count writing raw C CGI programs which was never that popular.

Perl performance isn't great, but then I don't consider TS/JS performance all that good either. Perl is a cut below JS nowadays because of the JIT that JS has but it's not impractically slow. If it wasn't impractically slow in literally the late 1990s, when it was powering nearly every dynamic site that existed, it sure isn't today.

I think you'll find that most older, popular (or once-popular) languages perform better than you might think. Perl couldn't perform too badly or it'd have been unusable on 1990s hardware. Early webservers often had single cores clocked at sub-100Mhz, mid-tens megabytes of memory, and tiny, spinning rust hard drives—a bunch of modern web server stacks would struggle to run at all on that hardware, let alone to serve even tens of requests per minute.

Go look up some real-world-(ish) performance benchmarks for various web frameworks if you want a real surprise. You might find it'd be hard to sell any scripting language other than PHP(!!!) for a project, if performance actually matters and for some reason you can't use a compiled language. Don't assume the new thing is better, or faster, even if that's its reputation. It may not be entirely earned.

>> I find bash to be much more readable

> Lol, you find bash readable?

Note, they didn't say it was readable, just more readable.

Depending on the task to be solved, a Bash script (using AWK, Coreutils etc.) executes much slower than the correspinding Perl or Python script, and having to run slow scripts often interactively can have a strong negative impact on developer productivity, especially on flow state (of mind).

BSD grep vs GNU grep has burned me enough times that when I do anything moderately complex in a bash script, I quickly move to Perl. And since Perl is installed most everywhere (or can be bundled up easily with PAR::Packer), it's an easy decision for me.

The sad thing is that once, a while ago, there was an attempt to compete with the new market of scripting languages (Perl, Tcl) from a shell perspective. The result being Korn Shell, especially ksh93, and tools like dtksh (which allowed Motif UI apps to be written in shell).

Compared to that, bash is really anemic.

Python took Perl's money a bit when it came to scripting, Ruby – clearly influenced a lot by Perl – made big grounds in web apps, once they outgrew CGIs, and of course then we collectively got Javascript Stockholm syndrome.

But seriously, they're all good dogs. We all revel in the Narcicissm of Small Differences, but in the end it wouldn't really matter if we used Perl instead of Python or any other algol-ish imperative OO language with some functional bits.

I learned Perl in the mid-1990's. I'm in semiconductor design and most of our CAD tools use Tcl so I write a lot of that but for any text report processing or sysadmin type tasks I write in Perl. I've got nothing against Python. I learned enough of it and it is fine. If I was writing larger systems I would probably use Python.

I admit the only time I reach for Perl is when I need a really strong and dependable regex service. Still one of the best ways to parse text to this day. Perl text parsing capabilities are nearly unrivaled still.

APPerl creator here. I appreciate all the feedback and hope it comes in handy.

It's linked from the page, but thought I'd point out if you're curious what went into making APPerl, check out the blogpost (self-plug): https://computoid.com/posts/Perl-is-Actually-Portable.html

If it does what I think it does, then you've found the holy grail.

> a distribution of Perl the runs on several x86_64 operating systems (most Unix-like and Windows) via the same binary

Unix and Windows in the same binary?? Sacrilege! Haven't we still laws against witchcraft?

God I must have written 200k lines of Perl5, and I hate it as only an experienced user can.

But the fact is: it gets shit done. It’s pretty good at dealing with strings and byte sequences, and even more importantly, it’s always fucking installed,

The “best” software helps you nothing if you’re machine isn’t running it.

> It’s pretty good at dealing with strings and byte sequences,

Ugh. Close to half the bugs I’ve chased in Perl in the last 10 years are encode/decode string bugs (or more specifically, the lack of doing encode/decode where necessary). After just one ‘downgrade_utf8()’ use, you get sick of Perl pretty fast.

This to me is where Perl ends up super short.

Basically try using it outside of the anglosphere in the world of Unicode and support falls apart. Sure is excellent if you need to do something like extract a bunch of 1980s database files encoded in LATIN-1 though!

> Sure is excellent if you need to do something like extract a bunch of 1980s database files encoded in LATIN-1 though

Or, y'know, a bunch of 2010s database files encoded in latin1. Fun fact: Did you know that MySQL 5.7 is still a supported version in 2022?

If you don’t (need to) care about Unicode, maybe. If you think pythons package management is not disastrous enough, maybe. If you do not (need to) consider Windows, maybe.

Even then, I’d rather choose Python.

> If you don’t (need to) care about Unicode, maybe

Perl has had strong unicode support since Perl v5.12 (~2010).

references: 1. http://xahlee.info/perl/perl_unicode.html 2. https://stackoverflow.com/a/6163129/12458

> If you think pythons package management is not disastrous enough, maybe.

Huh? I've used PAR::Packer with great success since 2005.

> If you do not (need to) consider Windows, maybe.

Microsoft used to ship Perl scripts to work with COM and WMI with their NT4 and Windows 2000 resource kits. Perl works *extremely* well with Windows.

I mean, there are lots of reasons to choose Python over Perl, but IMHO, these are not those reasons.

I use Perl as a better-bash. If I was writing a modern website, a large application, or any ML related work...I'd choose Python. But for complex system administration, I'm picking Perl. YMMV

Nice to see Cosmopolitan libc starting to be used more widely!

Indeed! Cheers to jart!

・^_^・

Maybe a little off topic, but I'm curious -- why are all of the Actually Portable programs always named with ".com" at the end? Does this have any special purpose?

A .com file is an old form of DOS executables with only code and data. No fancy headers or anything. Only thing specified is that execution will start at a specific address (0100h) and if you put a JMP you got enough space to put other headers around it.

A com file is different from a typical dos/Windows exe, which requires a specific file header. Each exe begins with the two bytes MZ followed by specific information on page sizes etc. and there is no way around.

The challenge a program like APP has is that it should be recognized by all operating systems, so to Linux it should look like an ELF executable, on Windows like a windows executable, etc.

Now if you take this APP perl.com file and load it with Linux, Linux will read the ELF header and be happy. If you load it into Windows, it will see the .com ending and thus just put the code into memory, jump to address 0100h and start executing. The program then can ignore the elf header. And if you open it with some (unzip) Programm such a Programm will look at the end of the file, where it will find a zip file header info and will think it's an zip archive. (Zip files have the header at the end mostly for historic reasons, as all information was only available once compression was done, but as you span multiple disks you can't reach the front anymore)

In the end it is a trick to make it look like different kind of file, depending on who looks at it.

It's a legacy type of windows executable that it uses as part of its multi-OS trick, and other OSes don't care about that extension.

I wonder why microperl or miniperl didn't really take off. Would be a nice thing to have -- just a single-executable-perl with the basics and file i/o, alongside awk, Lua or any other tiny language.

An interesting effort and article from around 2001: https://www.foo.be/docs/tpj/issues/vol5_3/tpj0503-0003.html

And an old Perlmonks discussion on a 850kb Perl: https://www.perlmonks.org/?node_id=222465

There is a newer github project as well -- it did work for a simple project with core modules for me: https://github.com/bentxt/microperl-standalone

I was waiting for this, but it doesn't mention which version of Perl 5 this is.

    $ bash ./perl.com --version

    This is perl 5, version 36, subversion 0 (v5.36.0) built for x86_64-cosmo
    (with 3 registered patches, see perl -V for more detail)

Cool, thanks!

This is really interesting. I had to give up on Perl on Windows because Windows Defender was forever flagging anything compiled with Par::Packer as a virus and deleting it.

Shame there isn't a portable windowing toolkit you could use with this. I'm pretty sure it's not possible to get the Qt or GTK bindings to work across all platforms from a single install, even if you are using the toolkit's widgets instead of trying to use the platform native widgets.

Has anyone done one of these (or wants to) for PHP?

Yeah, I’ve wanted this for a long time. I’m going to try but can’t promise anything.

Same but for Python. I'll probably look into it when I'm sufficiently bored with my current projects.

To quote the blogpost:

  After stumbling upon Gautham's APE Python port (https://ahgamut.github.io/2021/07/13/ape-python/) [...]

5 months ago on HN, "Python is Actually Portable", https://news.ycombinator.com/item?id=32245430 (120 comments)

We managed to get it to build I think last year. https://github.com/ahgamut/php-src/tree/cosmo_php73 See also https://github.com/shmup/awesome-cosmopolitan

Adding on to jart's answer -- I ported PHP 7.3 back in July 2021 (building on alganet's work), and got up to a simple "hello-world". Have never used PHP, so I didn't know what else could be done with it at the time.

However, the libc has grown a lot over the last year (Cosmo has a pthreads API now!) -- the updated libc enabled a quick experimental port of Python 3.11.0rc1 back in August (https://github.com/ahgamut/cpython/tree/cosmo_py311), so I would expect that a newer version of PHP can be ported similarly. Let's see....

ah, here we are: https://github.com/ahgamut/php-src/tree/cosmo_php81

(note it's an experimental port, I just put it together).

really cool, thanks!!

One file is nice, but I like being able to run on Alpine Linux. A lot of stuff like Deno depends on glibc. That's one thing I think about when I think "Actually Portable".

Does this support musl?

It doesn't depend on anything, completely static. Here you can read more about Actually Portable Executables: https://justine.lol/ape.html

Cosmopolitan Libc uses a musl-cross-make toolchain. https://github.com/jart/cosmopolitan/tree/master/third_party... Its primary developer also runs Alpine. Cosmopolitan is closely related to Musl as far as C libraries go. You could think of Cosmo as a Musl that's twice as fast on x86 and runs on more than just Linux.

Reply on news.ycombinator.com