Warp – self-contained, single binary applications | Hacker News Reader

Warp – self-contained, single binary applications

github.com

197 pointsGordonS155 comments

It's fascinating how self-contained binaries are now coveted as such a desirable feature. Dozens of languages (if not more) were capable of this decades ago. In fact, some of them could generate self-contained binaries that allowed you to bundle app assets like icons and images inside the executable - no additional add-ons needed. And they were native executables, not complicated wrappers with layer upon layer of different technologies.

This is not to take anything away from this project, just making an observation. The programming field often feels like it's in a giant loop constantly re-discovering what went before.

> The programming field often feels like it's in a giant loop constantly re-discovering what went before.

It might feel that way, but this is driven by economics. Basically disk, memory, and bandwidth is far cheaper than it used to be so who cares if you waste a GB or so copying the same libraries all over if you don't have to solve for dependency hell?

> disk is far cheaper

Not in the cloud.

To be clear, you're asserting that the price of disk in the cloud is comparable to what it was decades ago?

Things are more complicated than that. (Depending on the pricing scheme, you might end up paying per gigabyte per hour, for example; which may or may not include transfer/storage of actual data; a faster "drive" could be much more expensive than a slower one, etc.)

Sorry, I'm not following. Granted pricing schemes have changed, but adjusting for that, are you claiming that disk is not dramatically cheaper than it was ~20 years ago? Not trying to be a dick, just not following your point.

Point is, "disk" in the cloud is not the same thing as the disk in your PC.

No, the point is it doesn't matter where the disk is, the overall price has gone down.

Of course it's not, but the cost of both has declined.

Disagree, cloud storage is cheap - bandwidth is expensive.

1) It is

2) Application space on the server is vanishingly tiny compared to the data you're processing, so it is a far more generous trade-off.

Vendorizing libraries and static linking has become a security hell.

Just an example: https://blog.acolyer.org/2017/04/03/a-study-of-security-vuln...

Your link cites "little thought to the inclusion of dependencies", _not_ static linkage, as the cause of this "hell".

The implication is that statically linked binaries make it easier to throw in dependencies because you neither need to worry about how to ship them nor bother with re-using the shared libraries on the system.

If there's a bug in Go's TLS implementation, you need to identify, recompile, and redistribute every Go application you use. Whereas if it were using the system's OpenSSL library everything would magically be fixed by a simple `apt-get upgrade` or `yum update`.

Is this always true? If the shared library you linked to uses symbol versions and the bug fix requires a new version of a symbol, the application has to be relinked anyway. Same as static linking, except now it is not obvious you need to relink to get the bug fix.

Only if the fix involves breaking the old semantics entirely. It's possible in some cases to support old semantics in a safe way at the cost of performance, or in the case that it's not you can disable that call in such a way that the program aborts if it is called, thus indicating that the user should relink that program. This is why integration projects such as Linux distributions are hard, and useful.

> The implication is that statically linked binaries make it easier to throw in dependencies because you neither need to worry about how to ship them nor bother with re-using the shared libraries on the system.

I don't buy this argument; it's every bit as easy to `pip install` something as it is to `go get` something.

> If there's a bug in Go's TLS implementation, you need to identify, recompile, and redistribute every Go application you use. Whereas if it were using the system's OpenSSL library everything would magically be fixed by a simple `apt-get upgrade` or `yum update`.

I think the fix is fairly trivial; package maintainers just need to introduce vuln scanning into their automation. As soon as a security issue pops up for one of their dependencies, their automation automatically compiles a new version and publishes it to the package registry so the end user can pull it down with their next `apt-get upgrade` or `yum update` or whathaveyou.

> I think the fix is fairly trivial

Not at all. Package maintainers and the Security Team in Debian do plenty of manual work to backport and test security fixes.

> As soon as a security issue pops up for one of their dependencies, their automation automatically compiles a new version

That pulls down new releases for the dependency instead of backporting a security fix. Now you have no guarantee that the new binary will behave like the old one (minus the vuln). On the contrary, it would be practically a new release.

A lot of companies have security policies allowing security updates on live production systems. A complete rebuild against new libraries is not that.

That doesn't mention linking at all.

> so who cares if you waste a GB or so

I wonder if that sounds arrogant to just me. As a user I DO CARE. Its frustrating to find applications shipping their own copy of libraries wasting precious disk space. When I was using a Mac, I had at least 4 applications shipping their own copy of Qt5.4 ... aaaarrgh

Back on the Linux desktop, package managers handle this beautifully. And for the cases where you need to skip the distro, Flatpak still lets you use system dependencies. Everyone wins.

And for whoever claims storage is cheap, please link me the magical place you're buying yours from. Thanks.

Package managers come with their own limitations. You can't have multiple versions of the same application, packages are often out of date, packages sometimes conflict, you can't choose where applications are installed, etc.

And storage is ridiculously cheap. You can get an 8TB HDD for $150[0]. Or a 1TB SSD for the same price[1].

[0]https://www.amazon.com/Seagate-Expansion-Desktop-External-ST...

[1]https://www.amazon.com/Blue-NAND-1TB-SSD-WDS100T2B0A/dp/B073...

> You can't have multiple versions of the same application

In many package managers you can have multiple versions of the same application without a big deal -- in most cases you don't want to SUPPORT multiple versions of the same application, so there's always a trade-off.

> In many package managers you can have multiple versions of the same application without a big deal

In my experience, only if the distro has specifically set things up to allow that (Python2/3), or you find a PPA maintained by someone. Otherwise, time to compile from source I guess.

> This is not take anything away from this project, just making an observation. The programming field often feels like it's in a giant loop constantly re-discovering what went before.

I think part of this comes from various tradeoffs becoming more or less important at different times. Processing power goes up so you move to fat clients, then people move to battery powered devices and heavy lifting makes more sense on servers again.

It's not really processing power so much as storage and RAM that used to be seen as the disadvantages of static binaries. The advantages of the static binary model are that it avoids "DLL hell" and makes sourcing dependencies a bit better, which are indeed laudable aims.

But nowadays it has terrible security (and to some extent reliability) trade-offs - if everything is a static binary and there's a security issue in a library, you need to upgrade every single binary depending on it, which is much less reliable and more prone to error/oversight.

As security and reliability have increased in importance, so the "cost" of the static binary route has actually gone up significantly in my view compared to 20-30 years ago. So it's a bit odd to me to see this model (and closely related ideas like flatpak etc) gaining currency. There must be better ways to manage the "DLL hell" issues than that.

From the POV of folks working around massive server farms, all the servers are disposable, so having one binary is convenient, and the short lifespans mean they avoid the downsides.

The concern has more relevance to client applications, but then the cost becomes related to update strategy, and if every app is tied to an update manager process(whether it's an app store type of thing or a custom one) then the thinking becomes similar to the cloud server setup: Why bother, I'm going to blow it all away in the next update.

When the dependency management is exposed to users, there's a much bigger surface area for troubleshooting and support. It makes sense that devs favor static if they expect the whole environment to be disposable. It isn't like it was in the 90's, where patching was occasional but major new versions were essentially new products.

You can also see it as a security gain because you now can upgrade things independently without fear of how that may propagate to other applications. Being able to upgrade rapidly is the single best way to improve security. With DLLs people take a lot more time to upgrade because the risk is so high, meaning systems sit with older less-secure versions for longer.

Most compilers that produce native, static binaries for a platform also use static type systems that can be used to determine exactly where and how any given piece of code is used, including libraries. Also, if one can't get their internal builds correct, then they've got problems that go beyond security concerns.

Also, if you're talking about native, highly-distributed desktop/server applications, then the problem with using shared libraries is even worse. The only exception is system-level libraries that are updated automatically by the OS/distribution.

Finally, on Windows, with static binaries you can be sure that the binary is a particular version/build and is signed (this ensures that the binary version/build is valid). You don't get much better peace of mind than that. We've been distributing our installations and other binaries like that for over a decade, and it works great.

It's understandable due to RAM, disk, and network being way less of a constraint than it used to be (which necessitated dynamic linking and shared libraries), and the modern need to ship around binaries to hundreds or thousands of servers and have them run predictably everywhere.

Maybe it better to use DLL and ship to tens of thousands of servers instead.

That trend suggests that previous efforts are either not discoverable or not sufficiently extensible.

Yes, I remember when it was such a big deal that we could finally use dynamic libraries on Amiga OS, MS-DOS protected mode and Windows 3.x.

Even GNU/Linux made such a big deal of moving from a.out to ELF binary format, due to easier support for dynamic linking, which quite cumbersome with a.out.

Now almost 30 years later it is a big deal to support static linking, go figure.

Technology goes in cycles, like humans do :)

Yeah, like renaissance rediscovering Roman and Greek tech.

it's not really a loop, but there are really two categories of users :

* those who want applications that take the least space on the drive, and share most of their data / libraries across themselves - shared electron runtime, steam runtime shared with the os * those who want applications to conform exactly to what the developper used since the developer can guarantee a "working" configuration

Ideally it should be up to the user to choose what configuration he wants, but the problem is that it's up to the developer to make that choice. Thus, the eternal loop :

    applications released as a blob -> this takes too much disk space, we should use shared libraries ! -> applications released as shared libraries -> DLL hell -> applications released as a blob -> ...

Typing me some Nim and shaking my head. Passing -static to the linker, I get neat, robust, non-dynamic executables, typically in the low MB or even kB size-range.

const varname = staticRead"filename" embeds a given resource at compile time - to make the project completely selfcontained if need be, and just for the hell of it.

I know you can do this in several other languages, but the Nim way is - unsurprisingly - the most ready-made, obvious and transparent one I've come across.

Tcl beats it in terms of both transparency and ready-made-ness. In Tcl, all filesystem access goes through the Tcl Virtual Filesystem (VFS) layer. This is a layer of indirection that allows for the interpreter, including at the script-level, to intercept filesystem calls and replace them with their own data. The default (and catch-all) driver is called "native" that just passes things to the native OS, but you can write other drivers and over the decades people have done so.

The most popular one is Starkit/Starpack which uses the Metakit database (which is a heirarchial database) and appends that archive to a loader (either a script, as a stub, or to an executable which includes the stub with the ability to open itself). This database can be modified externally using the starkit tools (sdx), and when you pack all your resources into it, it appears from within the interpreter that all the files are just normal files even though only the database is being accessed.

Tcl is still a breath of fresh air, here 30 years after it first appeared. Such a pity all the minshare it has lost.

And it can now be compiled to machine code (TclQuadCode), an amazing feat which should have further information presented at this year's Tcl conference (already underway).

Seriously? (well, yes, I did a search). Clearly, I haven't been paying the attention I should, so thank you, this is super exciting.

We passed through a period where shared libraries were a great way to save storage and memory, but now dependency management has become such an incredibly headache that it's driving us back to monolithic binaries.

The root of all problems are incompatible file system layouts, library version, and package formats across Linuxes. I don't see how adding yet another package format is solving this.

Latest .net core 3 apparently supports single-bibary packaging.

It has been a often-requested feature for at least 2 years.

https://github.com/dotnet/corefx/issues/13329

(The last comment referring to warp is indicative)

.Net Core 3 wont be out until next year.

The current .Net Core 2.x runtime has had this feature since it came out though (last year?).

(Hello world + runtime is like 100 MB and several hundred files)

For what it's worth, the IlLinker can get Hello World down to 20MB and as many files:

https://github.com/dotnet/core/blob/master/samples/linker-in...

If you use some extra options to disable crossgen and link System.Private.CoreLib, you can get the size down to 10MB, at the expense of startup time:

https://github.com/AustinWise/IlLinkerExample

You can do this in regular .NET already using Costura Fody. It puts the DLLs (native or managed) as resources and load them at runtime.

https://github.com/Fody/Costura

Very neat. Would like to see more languages targeting single binaries. This is one reason (I believe) that has made Go so popular.

I wish <my favorite language> also had this. However, containers work. Not sure why they do java though, i thought java has been happy with jars so far.

If anybody is thinking about Java, check Graal out. I was able to successfully create fast, completely static ELF binaries of real JVM applications from fat JARs (not in the fake self-executable archive way).

It's still early days in terms of what's available in the runtime (e.g. the AWT subsystem is missing), but it's pretty damn impressive anyway.

JARs are okay, but they still require you to have the java runtime installed in order for them to execute.

You can bundle everything together, or use a commercial JDK with AOT compilation support.

We're struggling using containers for our Python and Javascript applications. I think we're using venvs and node_modules and we oughtn't (or we shouldn't keep them in our project directory where they sometimes--but not always--get overwritten by source code volume mounts).

For Python, this works well for my team: https://www.dharmab.com/programming/python/docker/2017/07/24...

Sorry if I'm a little dense; that doesn't seem like it solves any problems. It just says "use virtualenv and/or docker". I guess I was hoping for "how to manage dependencies in a localdev-friendly way for a Python/Docker app" or something.

Have you tried pipenv? It creates a lock file that hashes all your dependencies so it makes sure you get the right one when you deploy.. And it automatically creates the virtualenv if its nog there when you install them.

We do use pipenv. I like the idea of it, but I think we ran into issues with where the pipenv lives. We used to use `--system`, but that seemed buggy and/or idiosyncratic, so we started using a venv and then we started using PIPENV_VENV_IN_PROJECT=true (for some reason) and now it's causing other issues. It seems unclear how it's supposed to be used with Docker containers, but maybe we're just overcomplicating things?

I'm not sure if I understand correctly. Are you saying that you run venv commands as part of the build step for your containers (i.e. a RUN line in you Dockerfile) and then pip installing modules in another build step?

No, we use pipenv instead of pip, and pipenv manages a venv. There are `--system` flags, but I guess support for installing things to the system is a little buggy or idiosyncratic or something so we use the venv behavior. I'm not sure why we tell pipenv to install the venv in the project directory though. We do the pipenv install as a RUN line in our Dockerfile. Does this answer your question?

Yeah, I think I have an understanding now. We use tagged Docker base images with the app dependencies baked in so you can just use a FROM your Dockerfile and know you have the dependencies in the image and you just add and run your app code.

Yeah, we do the same thing. This seems to work pretty well, except right now we build these images out of band of our normal build process (if you change these base images, you're expected to kick off another CI job that builds/pushes new versions and updates the versions in the FROM statements in our production Dockerfiles (and our docker-compose.ymls). I'm not sure why we're doing this, since the build cache is actually pretty good at avoiding unnecessary base image builds.

That's how we do it also. You need to integrate a new process that is triggered on base image rebuild that redeploys your container with the new base image (which has security fixes, etc). This pipeline needs to start in Dev and be applied in all your environments, so that you know you are promoting a good base image once you get to production.

We use Kubernetes to trigger rolling restarts with new images when we release a change to the base image, and so far it's been painless, but a lot of work went into it. We use Gitlab, but any CI/CD should allow you to do it.

Plop a requirements.txt file down in each repo. Use direnv or a similar tool to automatically manage virtualenvs on the local dev machines. In your container, don't use virtualenv; just add requirements.txt to the container and run pip install -r requirements.txt

See the linked code at the bottom of the page

There are so many places where this will never work. For example, if you're using openssl and you're system is missing the header files.

Which is why I run and debug locally with an interactive container; installing the packages locally is just for my editor autocompletion.

Not sure if that helps, but here is what I do:

We just use a requirements.txt [1] for each service and run pip with the -r flag in the dockerfile. No Virtualenv in the container.

Most of the time we run these containers with docker-compose locally, but sometimes we want to run the service outside the container. For that we create a virtualenv outside the folder where we keep the source for the service. The reason for that is that we don't want to accidental copy the virtualenv in to the container. It wouldn't do much there but it would increase the container size.

You can do this easily with just "python3 -m venv", but there are some tools that help with that as well. I personally use pyenv-virtualenv [2] which just keeps all virtualenvs in ~/.pyenv/versions/<env>, but there is also conda [3] and virtualenvwrapper [4] which also store the virtualenvs in a central directory.

I am not sure if there really is any more to it.

[1]: https://pip.pypa.io/en/stable/user_guide/#requirements-files

[2]: https://github.com/pyenv/pyenv-virtualenv

[3]: https://conda.io/docs/user-guide/getting-started.html#managi...

[4]: https://virtualenvwrapper.readthedocs.io/en/latest/

It can be a little tricky to do things right here.

There are a few things of note:

1. The .dockerignore file can be used to prevent use of your `node_modules` folder during `docker build`, even if you have something like `COPY . .` in your Dockerfile.. This can let you create a new `node_modules` folder from your lockfile as part of the docker build process to create an image for testing/deployment.

2. You can maintain separate Dockerfiles and such for development and for release, so e.g. for development you might use volumes and not copy things in, but for release you wouldn't use volumes for source code.

I can't tell from what you said, but it seems like one of those two tips might be relevant.

It's okay to have a development setup which doesn't use containers and then use containers for deployment (so long as you have an integ or staging environment that uses containers) as well. It's quite reasonable to have a venv during development, but inside the docker image to not use a venv at all since things will already be reasonably isolated inside the container's fs.

Thanks for the suggestions! I had to check, but we do both of those things. To make sure that node_modules aren't overwritten, we mount an empty named volume to the node_modules directory. I guess mounting the named volume directory somehow causes the node_modules directory from the base image to appear inside of the bind-mounted directory. We do the same thing for venvs as well.

We use venvs inside of the Docker container because we use pipenv and apparently pipenv's support for installing to the system is buggy and/or idiosyncratic.

Named volumes are hacky to no end and you shouldn't be using them to make sure it's empty; you should be using `.dockerignore` or being careful about what you copy in.

The pipenv bit is probably more reasonable, and I'm afraid I haven't used it enough to be sure what rough edges are likely there.

I've been using shiv recently for my django deploys.

You might wanna check it out.

Here is a small readme I put together...

https://github.com/devxpy/shiv/blob/8d8298d21380dcf0b1970856...

JARs are fine until you need to get someone to install a JRE. Not a problem for server apps, but definitely not painless for client distributed apps.

You can bundle everything together, or use a commercial JDK with AOT compilation support.

Neither of those options are "painless".

Is it? Often you would distribute an installer anyway, right?

It is nice to have the option to go with either path. I think it's a bit less relevant in the modern development world but it can also be nice to leverage dynamic linking to present several executable entry points that share a large pool of common logic between them.

Tcl is still the king of this kind of distribution with Starkits/Starpacks. Since you can virtualize the filesystem within Tcl, you can pack in all your resources as regular files and generally scripts will work without knowing that this has happened (and without hacky extracting of the archive to disk before running, or while running).

This is an awesome design concept and I'm surprised other languages aren't copying it.

It generally takes other languages a long time to become more like Tcl ;-)

Python can do something similar albeit far simpler via packaging as an executable zipfile. Just structure everything you need in a directory and zip up with __main__.py in the root. All resources are available as an in-memory binary stream, including individual source .py files. Of course if you want to package something like a C library, you need to unpack that to disk to be able to access it with ctypes.

Are you referring to the zipapp library? I had no idea this even existed until now but that seems pretty clever. Huh. Why can't it also dynamically unzip a shared object library?

He's talking about pex. [0]

[0] https://github.com/pantsbuild/pex

You don't need to use any libraries, you can do something as simple as:

  $ zip -r mymodule.zip mymodule/
  $ echo '#!/usr/bin/env python3' > myapp
  $ cat mymodule.zip >> myapp
  $ chmod 755 myapp

CPython will automatically detect it's a zipfile, unzip it in-memory, and import the __main__.py in the root of the module and any nested modules.

The reason why you have to handle shared libraries and any other non-.py data yourself is because Python doesn't know what to do with it. You can access it as binary data via pkgutil.get_data(), but linking shared libraries is system-dependent and Python doesn't load them itself. As the dynamic linker can't find a shared library in a zip file, the only thing you can do is extract it separately. This is documented in https://docs.python.org/3/library/zipimport.html

Any files may be present in the ZIP archive, but only files .py and .pyc are available for import. ZIP import of dynamic modules (.pyd, .so) is disallowed.

You can simulate a single file exe using py2exe and NSIS. That doesn't have the no library limitation.

PHP has something similar, called .phar https://en.wikipedia.org/wiki/PHAR_(file_format)

By the way, if you are looking for a tool to embed resource files in Go binaries you might find these interesting:

- https://github.com/benbjohnson/genesis

- https://github.com/go-bindata/go-bindata

We use go-bindata a lot - love that package

I've not used those two but have always had success with

- https://github.com/gobuffalo/packr

I like that it allows you to `go run` without changing any code.

I'm ambivalent about this. It is cool to distribute a single binary, but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here).

It was used in multiple programs in both Windows and Linux. On Windows there was a rush of individual updates as each application was fixed... on Linux, simply upgrading that library fixed all applications.

I'd rather be in the latter than the former situation when the next vulnerability happens.

I think this represents a fundamental (and rather annoying) difference between Linux and Windows.

On Windows, the only platform-supported global library format is COM, and that requires a very specific programming style--HRESULTs and all the rest. True, registered COM DLLs can be linked to without separate header/symbol files, and installing new programs/libraries never requires local compilation, but the extra overhead of COM itself, and the fact that it's a walled garden code-wise, makes it rather unpopular. It's also nigh on impossible to port COM code to other apps without resorting to something like WINE (which can't compete performance-wise when running highly parallel COM apps/services). Not to mention how ugly C++ COM source files are.

It would be so nice for us all if Windows shipped with compilers, straight-up C libraries with headers, a package manager, a better shell, ...

Windows has C-style, shared libraries just like Linux - DLLs. The entire Win32 API that is the backbone of Windows is implemented in system DLLs.

Every compiler that produces native binaries for Windows links with these DLLs (C, C++, Go, FreePascal/Delphi, etc.), either statically or dynamically, using C calling conventions.

COM is the Component Object Model, which is implemented in DLLs, but is something different:

https://en.wikipedia.org/wiki/Component_Object_Model

COM, or some form of it, is used heavily with .NET and the new UWP runtime, but those are newer runtimes that sort of sit on top of the Win32 API (to some extent, UWP is actually integrated into the OS itself, but, AFAIK, still uses the underlying Win32 APIs).

Thanks for the corrections.

I think. You can have global DLLs in Windows. They just need to be in PATH.

You sound extremely confused. COM has not been the only global dependency format on Windows since.. longer than I can remember. Heck, even when .NET first shipped in 2000 or 2001 you could register .NET binaries in the GAC. And any DLL in system32 is automatically available for global consumption.

I think the inverse argument can be made against shared libraries: if an update introduces a vulnerability, now all programs which depend on that library become vulnerable.

I would like to think that projects overall tend to fix more bugs than introduce them. It's not like projects go from orderly to disorderly over time.

But some are high value targets (crypto ones, etc).

If I write malware, I guess it’s possibly easier to infect one library than thousands of executables.

Security problems often exist for years at a time.

Rather than making all of those applications vulnerable at the same time, they slowly become vulnerable as the release binaries are linked against bugged code. If it's not linked at runtime, or recompiled, it'll be vulnerable forever.

> but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here).

I believe you're referring to an OpenSSH vulnerability?

...though, I did a search just now to find when it was from (I remember there being a major one around the time you're thinking of), and instead found posts from only two months ago about another one that has existed for at least 18 years.

Here's one for Lua: https://github.com/ers35/luastatic

I always wonder to what extent this is true for Go. How does it handle depending on C libraries?

This sounds like a much less general binctr [0].

I don't see how warp handles the most trivial case of a dynamically linked executable; I see no references to patchelf or other such tricks to ensure it looks within the application's cache directory for dynamically linked libraries, nor do I see any mount namespaces.

Does warp do that? Why would I use warp if it doesn't when there are more generic solutions that can handle such a case, namely binctr, docker, guix pack, nix....

[0]: https://github.com/genuinetools/binctr

> This sounds like a much less general binctr.

binctr is Linux-only, and appears to need Docker as a runtime requirement? This has neither constraint, apparently?

> appears to need Docker as a runtime requirement

binctr does not require any runtime component other than a modern linux kernel with unprivileged usernamespace support.

Targetting OSX / Windows does make warp distinct from what I mentioned, though I don't use those OSs so I admit I didn't think about that possible issue.

Anyway, I thought this was a solved problem on both Windows and OSX since they both have conventions for self-contained applications, while linux does not.

Anyway, I thought this was a solved problem on both Windows and OSX since they both have conventions for self-contained applications, while linux does not.

This is mistaken for Windows. Even if programs are run from Program Files, they will often source their DLLs from other locations.

I wrote stowage.org to do something similar, having a Docker runtime (which stowage _does_ have) is actually pretty neat in a couple of ways because you also get a kind of basic package management for free. It's pretty nice for sharing dev tools amongst a team. Maybe these things don't have broad use cases, but I think there are a lot of solvable pains here.

check out whalebrew it is similar to your project

  tVQhhsFFlGGD3oWV4lEPST8I8FEPP54IM0q7daes4E1y3p2U2wlJRYmWmjPYfkhZ0PlT14Ls0j8fdDkoj33f2BlRJavLj3mWGibJsGt5uLAtrCDtvxikZ8UX2mQDCrgE

Anyone know what this magic is all about?

Honestly not feeling warp-packer downloading executable blobs during runtime either.

It's a placeholder in the final binary that gets replaced by the target application executable file name so Warp knows what to run.

This is a known and established technique. I don't have the link right now (mobile) but .NET Core does something similar for their native deployment, for example.

Good point about the downloading. Warp-packer should have all the warp-runners installed locally with it. Or at the very least they should be packages you install depending on your target platforms.

From the README, it looks like this unpacks the application and dependencies into a local cache in the user's home directory on first run.

Is it possible to instead execute the compressed application code directly and present to the resulting process a virtualized file system that decompresses the dependencies on the fly too? Then you could run the single binary without giving it any filesystem access. One way to do it would be to have have the parent process provide some routine that the kernel can call for reading a particular "file". Or perhaps just have standard embedding and compression methods that the kernel supports, so you would just provide the offsets to the kernel.

I guess what I mean is: do any platforms provide the necessary syscalls to pull that off?

This is in fact what Tcl Starkits/Starpacks do -- all access to the filesystem goes through the Tcl VFS, which can be redirected to the script level.

Yes, on linux you could do this with FUSE, in principle you would still need one empty temporary directory as mountpoint, but that can be placed in a tmpfs, e.g. under /run/user/ by convention.

What you describe can be accomplished on linux using mount namespaces. It gives a namespaced view of the filesystem which the parent process can control via picking its root location and what is bindmounted where.

You've basically described a container where the filesystem is virtualized to a safe location on disk.

Disallowing reading/writing data entirely is pretty easy with seccomp or ptrace, though that's not required with namespacing.

I asked this question just yesterday. The challenge (the one I was interested in) was if this can be done in a general purpose way for arbitrary programming languages so long as they used libc to go through the file system. I actually think it might be theoretically possible using LD_PRELOAD and some equivalent on OSX. I don't know why I'm compelled to want the executable to leave no trace on disk, but it seems much cleaner. However, if you _write_ to the virtual file system, that will need to change the packed binary itself which might result in problems (the binary would grow/shrink as you pack more / less files into it at runtime by writing to the "disk").

https://twitter.com/jordwalke/status/1050277143085608960

Maybe it's just not worth it and a temp directory is not so bad. I just worry about little changes to the way `mktmp` works across OS updates etc. A totally self contained executable that leaves no trace, with virtual file system support is about as isolated and reliable as you could ever hope. I just don't think anyone has been determined enough to achieve it.

That’s a good idea, doing it by shimming libc, but it feels kind of fragile. Go, for example, does not use libc and makes syscalls directly instead, so this would not work for any fs-accessing Go code either in the application or its dependencies. Even for things that do go through libc, wouldn’t you end up in a cat and mouse game to add shims for any new file operations added to the kernel? I suspect even shimming the existing interfaces is less straightforward than at first thought.

I don’t think you can have a truly robust solution without kernel support. Maybe the FUSE kernel-side code already supports, or could be modified to support, per-process ad-hoc virtual file systems? If done right, you might even get existing fs caching and other optimizations for (closed to) “free”.

I also can’t quite explain why I find this so desirable. It just feels intuitively cleaner and more robust to not have to spray files elsewhere into some cache.

It's far from easy, but you can create virtual filesystems on Windows using a minifilter driver. There are some wrappers, like Dokan[1], that make this a lot easier though.

[1] https://github.com/dokan-dev/dokan-dotnet/blob/master/README...

> From the README, it looks like this unpacks the application and dependencies into a local cache in the user's home directory on first run.

Isn't this a huge security hole? The user should not have write access to the application binaries.

Based on the readme, this is basically an SFX archive, which runs a binary upon completion of extraction to temporary folder, much like winzip & the like have created for ages.

That's a bit odd, with weird system configuration you'd still be able to inject a bad .dll or .so as a priority shared library. I don't know why you wouldn't just prefer to bundle all the function references into a single binary blob.

It was me that posted this link, and I must confess I did so before reading the full readme. I had assumed this was some kind of clever AoT tool... but after reading more, I was rather disappointed :(

I'm sure this still has some uses, but they are rather limited, and very dull.

Haven't read the link, but that's quite terrible, and the title is misleading. No idea why it's getting buzz in this thread. It means if tmpfs/equivalent isn't available, it will cause disk writes each application launch. Or if it caches the unpacking, it's susceptible to injection. And suppose the binary doesn't have write access to the filesystem?

It's the latter - it caches and is susceptible to injection.

The best of breed here is zeroinstall - https://0install.net/. The software catalog is here - http://0install.de/catalog/

comes with its own SAT solver for dependency resolution. zeroinstall is what Canonical was considering before it created Snap Packages.

Any idea why they went with snap instead? There seems to have been a recent proliferation of new packaging systems -- which is fine. But, from an Ubuntu perspective I don't think they should be included if they're not integrated with apt (perhaps it is and I missed the memo?).

Digikam moved to AppImage, which sounds cool but adds a further system one needs to update, and the automatic update never worked for me (I think their package didn't include the necessary metadata?).

Having all application updates available in one place, and official packages be recent, is the 'killer feature' for me. I used to be a Slackware user, but when I migrated to slapt-get I realised I was done with rolling my own packages and checkinstall-ing them and wanted system where things were more managed - not having half the apps installed ad-hoc.

When you have to install a new system to get just a few apps it feels like I might as well go back to ./configure;make;[check|make ]install.

Seems we need an xdg install record with a unified app that calls the sub-system (pip, 0install, AppImage, shell script, dpkg, whatever) to abstract all that away.

AppFS ( http://appfs.rkeene.org/ ) is an extension of the idea proposed originally by 0install (in their original LazyFS approach).

Its funny. Im creating a multi-os application platform based on Chrome runtime, and thats how im packaging my multi-target binaries.

Im using a SQLite btree file as backend, working basically as a key-value store. On the header i define what target triples are supported (the ones the dev built the binaries), and on the runtime open, see if the current host is supported and unpack the bynary payload always checking the hash before running it. (If its on the filesystem already see of the hash matches with the one saved before on the DB as a record)

Giving its a db the owner of the public-key signature(the creator of the file), can modify and add others binaries latter.

Im not disclosing it now, because its just part of a bigger platform, and you dont distribute only the apps, but a whole "container"(not in a Docker sense, but as a multiplatform container more focused on final users instead of cloud backends).

I hope i can launch it soon here on HN.

I hope it can use downloaded executable for warp-runner. Downloading executable from windows console doesn't work too well in my corporate environment. Currently placing windows-x64.warp-runner.exe to same directory as warp-packer doesn't help. I really like the idea though. Simplest 3rd party solution I've encountered so far for the purpose.

Reminds me of https://github.com/solidsnack/arx/ which I came across recently. I've not used it, but it sounds like it does a similar decompress-then-run approach, but using a shell script as the runner rather than a native binary.

And for Python there is Subpar: https://github.com/google/subpar

I don't understand a thing, is this for any kind of application or Rust? How will this thing know how to run my bizarre app in a bizarre language?

It downloads the runtime for your bizarre language, assuming it's supported.

It doesn’t seem to package my app correctly. I didn’t use dynamic loading, but it still say can’t open dependencies.

so bash self extracting tarball of the 2000s??

https://www.linuxjournal.com/node/1005818

What problem does this solve? How does it differ from Docker?

Docker isn't really used for desktop applications. Docker also requires a daemon (and on OSX a full virtual machine) to run.

A better analogy for this system would be jar files that allow more languages than just java.

This solves some of the app packaging pain that goes into building desktop apps. Not sure how this works but it probably bundles dependencies into the binary a la static linking. Docker mocks the OS and virtualizes, its a very different approach

I'm stoked for anything that eases the pain of desktop deployment that isn't Electron. Right now the cross-platform story is... complicated

But it seems like it continues one of the pain point of Electron: Shipping hundreds of MBs in the same dependencies for every single app.

Indeed, but it's either that or you deal with the dependencies at deploy, or you write your software to not have any dependencies.

For starters you don't need docker installed, or any container at all. This is largely targeted to end-user deployments where you're providing a binary to users to run, and the current defacto is to make a self extracting exe which dumps all the dependencies along with your executable to a temp directory before executing. Rather messy.

Edit: just looked at the warp implementation, it's a self extractor and executor

I think it’s funny to use java as an example when jar files have been around forever. Still there are some use cases for other things where statically linking is not possible.

I wonder what benefits this has over other solutions? Originally when it said multi-platform i thought it meant that one binary could run on multiple platforms like a fat binary. But i think all of the examples require you to have a specific target in mind.

There's something to be said for the fact that these are native binaries. Like under windows you'd be able just call CreateProcess with the file. Jars never did that and intrinsically needed an interpreter to run them.

This. A .jar file could be anything. Is it a library or an executable? When I run it, do I need to make sure its dependencies are on my classpath? What're the java command line flags I need? Do I have the right version of java on this machine?

There's something to be said for an entirely self-contained native binary, at least as an easy on-ramp for users.

Unfortunately, it turns out this is not a completely self contained binary - it's more of a packer. So if you pack a Java app, you're still going to need the JRE, and if you pack a .NET Framework app, you're still going the .NET Framework, etc.

So I don't know what this is doing. But I know that microsoft is developing WPF to run in a fulled contained .NET Core binary. Meaning that there is no requirement of having any preinstalled software when running it. I would assume this technology to be translated into most any .NET core app.

Yes, any .NET Core app can already be 'published' as a self-contained app. That basically means it ships along with the .NET Core files (there are a lot of them), so you don't need to have a globally installed .NET Core to run it.

What Warp does is act like a packer, creating a thin, native executable that compresses and embeds all the files required to run your app, then decompresses them to disk the first time the app is executed.

You can build "fat jars" that contain all dependencies. IMO, this is how most Java apps should be deployed.

You still have issues of needing a JVM. The JVM is not a small runtime considation. I'm in the process of converting to GO since I have more time than money. 1 JVM in Docker uses 512 MB to sit still. A Go version is around 16.

Here is what Google did internally:

- Bundle all C/C++ native libraries, and link them all together with the java launcher - e.g you produce your own java.exe that has your C/C++ libraries linked in, then you slap the .jar file - for extra bonus you can slap at the back of your .exe and make it load from there

  Voila - you have single executable (almost) with all you need, and you don't need to unpack .dll's anywhere

  But... you must compile java.exe yourself...

Java is somewhat like chrome in that it pre-emptively gets some RAM, because it's managing memory allocation to programs itself. (and because the speed increase of not doing system-calls offsets some of the performance hits from running a JIT)

Is that 512 the lowest you can get it with command line options?

That was with Xmx set to 64 MB and a stack of 256k

Mostly FUD. You can easily tweak your min and max heap sizes. You can have an app that takes less that 512 megs of RAM.

> I think it’s funny to use java as an example when jar files have been around forever.

Jar files are not a self-contained, single binary (executable).

I can't

    curl -o binary http://example.org/binary
    ./binary

In reality, you can't do that with these either. You have to have the runtime installed before your example works.

It's really not much better than running a fat jar that contains all of its dependencies.

    curl -o binary.jar http://example.org/binary.jar
    java -jar binary.jar

This is more like a self-extracting shell script, but more opaque.

You might be right, but documentation says "self-contained".

How is it self-contained if it's not self-contained?

Beautiful

Reply on news.ycombinator.com