Hacker News Reader

Comment by reacharavindh | Hacker News Reader

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging.

Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to adopt. Python misses out on a lot of use because of the inability to do this. And No! I do not want to pip install a lot of stuff on the servers just to be able to run this script once. Heck, some of these servers don't even have access to public internet to be able to pip install whatever.

Yes, I have looked into pyinstaller and Nuitka. They threw up some errors that were indicative of deeper issues that I didnt feel like a good use of my time to debug. I'd rather choose a language that has this as a priority/design goal instead.

View on HN

Yeah, I was writing a Docker image built around Prometheus' jmx_exporter intended to be used as a mixin for our Java apps, and part of what I needed to do was provide a simple script to preprocess separate config files to produce the config used by jmx_exporter.

My initial thought was Python, but it needed a couple of 3rd party dependencies, and there wasn't an overly clean _and_ simple way to copy the script in from the mixin and run locally.

So I shrugged, rewrote my script in Go, and then used a multi-stage build to copy in the binary and nothing else.

Ending up with a single statically linked binary was cool, even if Go does some stuff that made my eyebrows quirk a tad (I still can't believe that an idiomatic set in Go is map[T]struct{}...)

> I was writing a Docker image built around Prometheus' jmx_exporter intended to be used as a mixin for our Java apps, and part of what I needed to do was provide a simple script to preprocess separate config files to produce the config used by jmx_exporter

Did you consider, and stop me if this suggestion is completely wild, Java?

It's a script that merges YAML objects, there's an overhead to Java for this use case. I did consider Kotlin, but that same overhead of a POM, plugins to build capsules/fat jars etc.

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

This is becoming a parody of software engineering.

People are now irrationally scared of using dynamic libraries, OS packages and even directories, like virtualenvs.

Instead, a simple solution is replaced with containers, or by rewriting tons of code in the new hyped language.

Are we trying to create job security through unnecessary complexity?

> Are we trying to create job security through unnecessary complexity?

As a python/go dev, I can assure you that go is not much more complex. And as a server admin, it's much easier to deploy.

It does have drawbacks when compared to python, but it can often be the optimal solution.

Also a Python dev. I do use VirtualEnv but the other thing is a huge performance boost for small performance intensive projects.

Isolation is merely one utility of containers.

This is the weirdest straw man I've seen in a while. Python, not Go, benefits most from containerization. Installing "Dynamic libraries, OS packages, and even directories like virtualenvs" on target machines is far more complicated than sending a single file. I say all this as the DevOps guy in a Python shop.

Installing a full virtualenv with all related libraries just to run one script is kind of ridiculous.

The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera.

Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.

> The server might have the wrong Python version available

That's the point of using virtual environments, so that you can run the Python version and libraries that you need. Also, as of 3.3, Python ships with venv which means you don't need to separately install virtualenv anymore. It's all very portable.

You'd need to install the right Python version though, and even with something like pyenv you might need to install system packages.

It really isn't as portable as one might wish.

This is my exact situation right now. Our website is made in Django, and we use Pyenv to manage our python versions. We'll likely be going to route of a docker image/ pyenv install script. It takes trial/error to figure out some of the packages you need if they aren't installed already, a single binary would be far easier for sure.

Portable means something else. Portable would mean that I could ship the virtualenv. Python missed the forest for the trees.

I still need to have python installed on the machine and download all of the libraries needed.

Not all operating systems carry Python >3.3 by default, that would mean installing backports or unofficial repositories on production machines.

Or you can use a static binary that just works.

Why is it ridiculous? You'll need the related libraries either case, and by default virtualenv uses symlinks. The "full virtualenv" part makes it sound like it's heavy, or something like that.

If you do it in a language that provides static binaries or an equivalent, you don't need to pull dozens of libraries with possibly hundreds of files just to run a single script.

Imagine a situation where you'd need to perform an operation on a hundred servers, would you either transfer one static exacutable or build an virtualenv in every machine and download correct versions all of the related libraries?

If I controlled the environment, I'd probably build a Docker image and deploy that. Otherwise yes, This kind of extreme example isn't very representative of the initial situation though. And you're also talking about switching up languages, which isn't always possible or easy, especially in large teams or codebases.

So a virtualenv for every script I want to run? each of that downloading a full Python interpreter and whatever modules that Python script uses? How is this any better than a statically compiled binary? Other than useless additional work and bloat on the servers?

virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.

I am an utter noob at Python packaging, but i would like to learn more, so excuse the basic question, but: why doesn't it work? And is there anything in the ecosystem which is like that, but does work?

I had heard that the Python packaging and deployment story had got a lot better in recent years, but this sounds like it still falls far short of table stakes.

I'm not sure why the Python community can't get it sorted out. We've been using Pipenv because it's one of a few tools that supports lockfiles/reproducible builds, and at the time we chose it, the Python packaging authority was advertising it as the official solution for Python packaging; however, after we were already invested in it, they backpedaled becaused the community realized it was very buggy and super slow (as in "it takes literally half an hour to add or remove or update any dependency").

We're finally getting around to migrating away, and we've settled on Twitter's github.com/pantsbuild/pants which is like Google's Bazel except that it's jankier in every way _except_ that Pants supports Python while Bazel doesn't (Bazel advertises Python support, but it doesn't work as the many Python 3 issues in their issue tracker can attest). The nicest thing about Pants is that it builds pex files (github.com/pantsbuild/pex) which are executable zip files that ship all of your dependencies except the Python interpreter and shared objects.

I'm still not very satisfied with this solution, and it's still far, far worse than Go's dependency management and packaging model, but it's a dramatic improvement over Pipenv, virtualenv, etc (gives us both reproducibility and performance).

Its not that Python packaging doesn't work, or that it is impossible to automate, but whatever you do it requires nontrivial time and effort to get right.

The point is that the static binary created by go is a single file that you copy into place and run. It is a lot simpler.

You just need a venv with the right architecture, which is fairly trivial. You can then copy the venv directly over.

See this repo: https://github.com/unixtreme/D3Edit. It has Linux/Mac/Windows virtualenvs and it works without any additional setup

I think what people are arguing is deploying a single binary is even more trivial than a "venv with the right architecture". Deploying one file is going to be easier than deploying a set of files.

I've done things in a way that it doesn't.

Basically build a virtualenv and then pack that into a .deb, install the .deb on the target machine and you have a self-contained package that requires no external resources to install.

That said, using golang is WAAAAY simpler. The virtualenv/deb solution, while it works, is very sketch.

Fully agree about Go; however, if you can't move away from Python, github.com/pantsbuild/pex is a pretty good alternative. It's an executable zip file that contains all of your python dependencies; you still need the Python interpreter and any shared objects that don't come with pip (e.g., for whatever reason, the Python graphviz package depends on the graphviz-devel system package and the latter doesn't ship via Pypi). It's still an order of magnitude worse than Go, but it's an order of magnitude better than pip or pipenv. :)

> The single binary deployment makes it a lot easier for users to adopt.

I've had some success making a Python file an executable. Though, I do understand your gripe with needing to install dependencies.

You didnt mention virtualenv. That is what I use. I even made a deb installer that forces itself to use virtualenv and fetches system python files for those packages available to Debian but not pip (kind of a hack but it worked! Look mah!). I agree though I like that with D and Go deployment seems simpler.

zipapp, which comes bundled with python since 3.5, is your friend.

It can produce a single .pyz file which has all dependencies except the interpreter itself inside. Actually this format is compatible all the way back to Python 2.6, it's only the zipapp convenience scripts that are new.

https://docs.python.org/3/library/zipapp.html#creating-stand...

Reply on news.ycombinator.com