Hacker News Reader

Comment by w8rbt | Hacker News Reader

Very neat. Would like to see more languages targeting single binaries. This is one reason (I believe) that has made Go so popular.

View on HN

Tcl is still the king of this kind of distribution with Starkits/Starpacks. Since you can virtualize the filesystem within Tcl, you can pack in all your resources as regular files and generally scripts will work without knowing that this has happened (and without hacky extracting of the archive to disk before running, or while running).

This is an awesome design concept and I'm surprised other languages aren't copying it.

It generally takes other languages a long time to become more like Tcl ;-)

Python can do something similar albeit far simpler via packaging as an executable zipfile. Just structure everything you need in a directory and zip up with __main__.py in the root. All resources are available as an in-memory binary stream, including individual source .py files. Of course if you want to package something like a C library, you need to unpack that to disk to be able to access it with ctypes.

Are you referring to the zipapp library? I had no idea this even existed until now but that seems pretty clever. Huh. Why can't it also dynamically unzip a shared object library?

He's talking about pex. [0]

[0] https://github.com/pantsbuild/pex

You don't need to use any libraries, you can do something as simple as:

  $ zip -r mymodule.zip mymodule/
  $ echo '#!/usr/bin/env python3' > myapp
  $ cat mymodule.zip >> myapp
  $ chmod 755 myapp

CPython will automatically detect it's a zipfile, unzip it in-memory, and import the __main__.py in the root of the module and any nested modules.

The reason why you have to handle shared libraries and any other non-.py data yourself is because Python doesn't know what to do with it. You can access it as binary data via pkgutil.get_data(), but linking shared libraries is system-dependent and Python doesn't load them itself. As the dynamic linker can't find a shared library in a zip file, the only thing you can do is extract it separately. This is documented in https://docs.python.org/3/library/zipimport.html

Any files may be present in the ZIP archive, but only files .py and .pyc are available for import. ZIP import of dynamic modules (.pyd, .so) is disallowed.

You can simulate a single file exe using py2exe and NSIS. That doesn't have the no library limitation.

PHP has something similar, called .phar https://en.wikipedia.org/wiki/PHAR_(file_format)

I'm ambivalent about this. It is cool to distribute a single binary, but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here).

It was used in multiple programs in both Windows and Linux. On Windows there was a rush of individual updates as each application was fixed... on Linux, simply upgrading that library fixed all applications.

I'd rather be in the latter than the former situation when the next vulnerability happens.

I think this represents a fundamental (and rather annoying) difference between Linux and Windows.

On Windows, the only platform-supported global library format is COM, and that requires a very specific programming style--HRESULTs and all the rest. True, registered COM DLLs can be linked to without separate header/symbol files, and installing new programs/libraries never requires local compilation, but the extra overhead of COM itself, and the fact that it's a walled garden code-wise, makes it rather unpopular. It's also nigh on impossible to port COM code to other apps without resorting to something like WINE (which can't compete performance-wise when running highly parallel COM apps/services). Not to mention how ugly C++ COM source files are.

It would be so nice for us all if Windows shipped with compilers, straight-up C libraries with headers, a package manager, a better shell, ...

Windows has C-style, shared libraries just like Linux - DLLs. The entire Win32 API that is the backbone of Windows is implemented in system DLLs.

Every compiler that produces native binaries for Windows links with these DLLs (C, C++, Go, FreePascal/Delphi, etc.), either statically or dynamically, using C calling conventions.

COM is the Component Object Model, which is implemented in DLLs, but is something different:

https://en.wikipedia.org/wiki/Component_Object_Model

COM, or some form of it, is used heavily with .NET and the new UWP runtime, but those are newer runtimes that sort of sit on top of the Win32 API (to some extent, UWP is actually integrated into the OS itself, but, AFAIK, still uses the underlying Win32 APIs).

Thanks for the corrections.

I think. You can have global DLLs in Windows. They just need to be in PATH.

You sound extremely confused. COM has not been the only global dependency format on Windows since.. longer than I can remember. Heck, even when .NET first shipped in 2000 or 2001 you could register .NET binaries in the GAC. And any DLL in system32 is automatically available for global consumption.

I think the inverse argument can be made against shared libraries: if an update introduces a vulnerability, now all programs which depend on that library become vulnerable.

I would like to think that projects overall tend to fix more bugs than introduce them. It's not like projects go from orderly to disorderly over time.

But some are high value targets (crypto ones, etc).

If I write malware, I guess it’s possibly easier to infect one library than thousands of executables.

Security problems often exist for years at a time.

Rather than making all of those applications vulnerable at the same time, they slowly become vulnerable as the release binaries are linked against bugged code. If it's not linked at runtime, or recompiled, it'll be vulnerable forever.

> but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here).

I believe you're referring to an OpenSSH vulnerability?

...though, I did a search just now to find when it was from (I remember there being a major one around the time you're thinking of), and instead found posts from only two months ago about another one that has existed for at least 18 years.

By the way, if you are looking for a tool to embed resource files in Go binaries you might find these interesting:

- https://github.com/benbjohnson/genesis

- https://github.com/go-bindata/go-bindata

We use go-bindata a lot - love that package

I've not used those two but have always had success with

- https://github.com/gobuffalo/packr

I like that it allows you to `go run` without changing any code.

It is nice to have the option to go with either path. I think it's a bit less relevant in the modern development world but it can also be nice to leverage dynamic linking to present several executable entry points that share a large pool of common logic between them.

Here's one for Lua: https://github.com/ers35/luastatic

I wish <my favorite language> also had this. However, containers work. Not sure why they do java though, i thought java has been happy with jars so far.

If anybody is thinking about Java, check Graal out. I was able to successfully create fast, completely static ELF binaries of real JVM applications from fat JARs (not in the fake self-executable archive way).

It's still early days in terms of what's available in the runtime (e.g. the AWT subsystem is missing), but it's pretty damn impressive anyway.

JARs are okay, but they still require you to have the java runtime installed in order for them to execute.

You can bundle everything together, or use a commercial JDK with AOT compilation support.

We're struggling using containers for our Python and Javascript applications. I think we're using venvs and node_modules and we oughtn't (or we shouldn't keep them in our project directory where they sometimes--but not always--get overwritten by source code volume mounts).

For Python, this works well for my team: https://www.dharmab.com/programming/python/docker/2017/07/24...

Sorry if I'm a little dense; that doesn't seem like it solves any problems. It just says "use virtualenv and/or docker". I guess I was hoping for "how to manage dependencies in a localdev-friendly way for a Python/Docker app" or something.

Have you tried pipenv? It creates a lock file that hashes all your dependencies so it makes sure you get the right one when you deploy.. And it automatically creates the virtualenv if its nog there when you install them.

We do use pipenv. I like the idea of it, but I think we ran into issues with where the pipenv lives. We used to use `--system`, but that seemed buggy and/or idiosyncratic, so we started using a venv and then we started using PIPENV_VENV_IN_PROJECT=true (for some reason) and now it's causing other issues. It seems unclear how it's supposed to be used with Docker containers, but maybe we're just overcomplicating things?

I'm not sure if I understand correctly. Are you saying that you run venv commands as part of the build step for your containers (i.e. a RUN line in you Dockerfile) and then pip installing modules in another build step?

No, we use pipenv instead of pip, and pipenv manages a venv. There are `--system` flags, but I guess support for installing things to the system is a little buggy or idiosyncratic or something so we use the venv behavior. I'm not sure why we tell pipenv to install the venv in the project directory though. We do the pipenv install as a RUN line in our Dockerfile. Does this answer your question?

Yeah, I think I have an understanding now. We use tagged Docker base images with the app dependencies baked in so you can just use a FROM your Dockerfile and know you have the dependencies in the image and you just add and run your app code.

Yeah, we do the same thing. This seems to work pretty well, except right now we build these images out of band of our normal build process (if you change these base images, you're expected to kick off another CI job that builds/pushes new versions and updates the versions in the FROM statements in our production Dockerfiles (and our docker-compose.ymls). I'm not sure why we're doing this, since the build cache is actually pretty good at avoiding unnecessary base image builds.

That's how we do it also. You need to integrate a new process that is triggered on base image rebuild that redeploys your container with the new base image (which has security fixes, etc). This pipeline needs to start in Dev and be applied in all your environments, so that you know you are promoting a good base image once you get to production.

We use Kubernetes to trigger rolling restarts with new images when we release a change to the base image, and so far it's been painless, but a lot of work went into it. We use Gitlab, but any CI/CD should allow you to do it.

Plop a requirements.txt file down in each repo. Use direnv or a similar tool to automatically manage virtualenvs on the local dev machines. In your container, don't use virtualenv; just add requirements.txt to the container and run pip install -r requirements.txt

See the linked code at the bottom of the page

There are so many places where this will never work. For example, if you're using openssl and you're system is missing the header files.

Which is why I run and debug locally with an interactive container; installing the packages locally is just for my editor autocompletion.

Not sure if that helps, but here is what I do:

We just use a requirements.txt [1] for each service and run pip with the -r flag in the dockerfile. No Virtualenv in the container.

Most of the time we run these containers with docker-compose locally, but sometimes we want to run the service outside the container. For that we create a virtualenv outside the folder where we keep the source for the service. The reason for that is that we don't want to accidental copy the virtualenv in to the container. It wouldn't do much there but it would increase the container size.

You can do this easily with just "python3 -m venv", but there are some tools that help with that as well. I personally use pyenv-virtualenv [2] which just keeps all virtualenvs in ~/.pyenv/versions/<env>, but there is also conda [3] and virtualenvwrapper [4] which also store the virtualenvs in a central directory.

I am not sure if there really is any more to it.

[1]: https://pip.pypa.io/en/stable/user_guide/#requirements-files

[2]: https://github.com/pyenv/pyenv-virtualenv

[3]: https://conda.io/docs/user-guide/getting-started.html#managi...

[4]: https://virtualenvwrapper.readthedocs.io/en/latest/

It can be a little tricky to do things right here.

There are a few things of note:

1. The .dockerignore file can be used to prevent use of your `node_modules` folder during `docker build`, even if you have something like `COPY . .` in your Dockerfile.. This can let you create a new `node_modules` folder from your lockfile as part of the docker build process to create an image for testing/deployment.

2. You can maintain separate Dockerfiles and such for development and for release, so e.g. for development you might use volumes and not copy things in, but for release you wouldn't use volumes for source code.

I can't tell from what you said, but it seems like one of those two tips might be relevant.

It's okay to have a development setup which doesn't use containers and then use containers for deployment (so long as you have an integ or staging environment that uses containers) as well. It's quite reasonable to have a venv during development, but inside the docker image to not use a venv at all since things will already be reasonably isolated inside the container's fs.

Thanks for the suggestions! I had to check, but we do both of those things. To make sure that node_modules aren't overwritten, we mount an empty named volume to the node_modules directory. I guess mounting the named volume directory somehow causes the node_modules directory from the base image to appear inside of the bind-mounted directory. We do the same thing for venvs as well.

We use venvs inside of the Docker container because we use pipenv and apparently pipenv's support for installing to the system is buggy and/or idiosyncratic.

Named volumes are hacky to no end and you shouldn't be using them to make sure it's empty; you should be using `.dockerignore` or being careful about what you copy in.

The pipenv bit is probably more reasonable, and I'm afraid I haven't used it enough to be sure what rough edges are likely there.

I've been using shiv recently for my django deploys.

You might wanna check it out.

Here is a small readme I put together...

https://github.com/devxpy/shiv/blob/8d8298d21380dcf0b1970856...

JARs are fine until you need to get someone to install a JRE. Not a problem for server apps, but definitely not painless for client distributed apps.

You can bundle everything together, or use a commercial JDK with AOT compilation support.

Neither of those options are "painless".

Is it? Often you would distribute an installer anyway, right?

I always wonder to what extent this is true for Go. How does it handle depending on C libraries?

Reply on news.ycombinator.com