Hacker News Reader

Comment by parhamn | Hacker News Reader

I think people often don't realize that depending on the language runtime, micro-services can easily be a must.

Most service boundaries at organizations are "I need a different version of a pinned package we can't upgrade.". This is common in languages where there is support for only using one version of a given package, and it's worse if there isn't a culture of preserving function APIs. E.g. any python company with pandas/numpy in the stack will need to split the environments at some point in the future, no ifs ands or buts!

View on HN

I have heard that the reason Docker (and containers in general) took off was that they solved the problem of Python's awful package management. I didn't believe it until I saw people put Python in production and have to deal with this. At this point, I would rather have a physical snake in my server racks than any Python code.

Do we live in different worlds? Virtual environments solve 99% of all python packaging and installation use cases.

I guess so. Virtual environments doesn't solve any of the problems as python authors either don't specify the version or their dependency author doesn't specify it. Try running any of the program which is >3 years old. I remember in one of the programs I needed to pin 10 dependency versions manually to make it run.

> I needed to pin 10 dependency versions manually to make it run

um... You should have been pinning all your dependencies from the very start

It wasn't my code. I pin all my dependencies, but lot of python code has been written by people without software engineering experience like university students and to an extent ML engineers.

OK but your occasional contention was that

> Virtual environments doesn't solve any of the problems as python authors either don't specify the version or their dependency author doesn't specify it.

This is a problem of lack of training or willful poor practice, not an issue with venvs themselves which absolutely do solve this issue.

Agreed, the problem would not exist at all if an approach like that of say, node development, existed, but here we are.

But only in what you deploy. For the love of God don’t pin versions in libraries unless you’re really sure it has a completely different api. That’s how you get a dependency graph with no solutions.

> But only in what you deploy

You mean only your direct dependencies, right? And yes, of course

I mean the whole pip freeze output, if you’re still using that (this works like a lock file).

Agreed, or if you dont, you should run pip-compile during CI/merge in order to pin every dependency in the entire tree.

These are very solveable (and have been solved for a while) problems. It really pains me that most people are either unaware or do not dive deep enough below the surface to find them.

Dependency pinning is one of those things where I do see valid use cases, but "making it run" isn't one of them. It should primarily be use to deal with incompatible version until you can make the necessary changes.

If you depend on dependency pinning due to unmaintained code then you should go deal with the problem directly. Say you pin 10 external libraries to three year old versions, how many security holes does that expose you to?

That's really my issue with dependency pinning, you end up with software that are just allowed to rot, making upgrades more difficult with every passing year.

> Say you pin 10 external libraries to three year old versions, how many security holes does that expose you to?

> That's really my issue with dependency pinning, you end up with software that are just allowed to rot, making upgrades more difficult with every passing year

You seem to be suggesting not pinning and just pulling in the latest (minor, I presume?) versions every time you redeploy.

A better way is to pin _all_ your direct dependencies and check for minor version upgrades on a regular basis, especially for publically available services. A proper CI system should allow you to do this with confidence.

Occasionally you will be forced into major version upgrades, but in my experience this is rare.

This should be a regular and essential part of software maintenance and security vigilance.

> A better way is to pin _all_ your direct dependencies and check for minor version upgrades on a regular basis

I'd agree with that, I just don't believe that to be happening on any significant scale. Reasonably I think you should be able to do pinning like:

  library>=3.0,<4

Depending on the versioning scheme of the library, but yes, pulling in minor release. We currently do that using Debian packages and just rely on the OS to provide security updates to underlying Python packages.

Pinning is fine is you manage updates, but if you're then pulling in three year old libraries, which may come with their own pinned third party dependencies, I still think you're doing it wrong. In the example from GP it appears that they are pinning versions as a way to avoid forking and patching a deprecated and unmaintained package.

I'm not oppose to version pinning as such, but if you don't have a plan to stay on top of security updates, then you're better off pulling in any minor update. It's not just at redeploy, you may reasonably need to pull dependency updates more frequently than you redeploy.

If you deploy using Docker e.g. you're CI system needs to be constantly be pulling in updates, rebuilding and testing container images. People just don't do that. Most developers I worked with don't even care to update their container images if they aren't updating their own code. I've on multiple occasions had to deal with developers who absolutely lost their mind because we didn't automatically pull in OS updates, yet they themselves shipped outdated Java libraries or at one point even relied on an out date version of an alpha release of Tomcat that hadn't been updated for three years. The only different was that their dependencies where in containers and therefor, in their mind, "safe".

> Depending on the versioning scheme of the library, but yes, pulling in minor release. We currently do that using Debian packages and just rely on the OS to provide security updates to underlying Python packages.

Why not use a venv or Docker? It's a world of difference. There are multiple issues with relying on the OS package manager for Python dependency management.

Overall it sounds like you know good practice and you know the value of venvs, pinning and Docker but you've had to deal with some workplaces with really shoddy practices. Still, that's no reason to say "venvs don't work" or to give up altogether on the notion of good practice by saying "People just don't do that".

I've worked in shoddy shops, I've worked in shops with excellent practices. In some places I've been the one that made them transition from the former to the latter.

In this case your accidentally upgrading your dependencies only when rebuilt. Especially with micro services some things can be ran for years without being rebuilt.

I have python service with does some AI job but it just doesn't build anymore. I have "golden image" which I carefully back up, because losing it would mean the catastrophe.

Had the same problem with node 16->18 recently. No recent OS version would build some extension. Had to rearchitect a good bit of an app to get back to something working.

Python has a real problem with version incompatibilities for the interpreter, and a few packages require C libraries of specific versions. But outside of that, vitualenvs solve all of the issues of "how do I run those two programs together".

After the Py2 vs. Py3 thing settled down, almost all of the operations issues got away.

That said, Python has a really bad situation about dependencies upgrade on the development side. But Docker won't help you there anyway. Personally, at this point I just assume any old Python program won't work anymore.

I have used python in production for years, multiple servers, multiple racks, and deployment has always been as simple as

./deploy.sh pull sync migrate seed restart

pull calls git pull, sync runs pipenv sync, migrate runs django migrate, seed runs django management command seed, restart calls systemctl --user restart

This was not my experience building infra for a startup heavily leveraging a Python monolith. It was painful AF (both when developing locally and deploying to VMs) and Docker made the deployment story palatable (build, push to hundreds of VMs, run).

Any details as to why?

Virtual envs, dependency hell, etc. I just want a binary to build and push (previously to VMs, now to k8). Docker does that for Python.

Target simplicity, it’s the ultimate sophistication.

I target simplicity by having one virtual environment instead of many and synchronizing dependencies instead with the main branch instead of... hell?

Have you tried deploying to hundreds or thousands of VMs? In my experience, managing container deployment and state is much easier, vs wrestling with inconsistent env state on compute for whatever reason.

For the life of me I don't understand how deploying to 10 machines is functionally different from deploying to a billion machines as the process is exactly the same. Unless you sabotaged your deployment machines with some manual meddling, it's the same with Docker images and Ansible.

When you run a single, simple Python service, it's fantastic. When you scale your Python, it's an awful dependency hell.

Can you please explain how this dependency hell manifests itself if you are using a dependency resolver such as poetry or pipenv which locates the appropriate maximum version? Once you've locked the appropriate versions, you just do pipenv sync in prod like I said.

Yes - you pull in two third party libraries that need two incompatible versions of a mutual dependency. That library (the common dependency) is not backwards compatible due to deprecated functions or whatnot. That dependency could also be Python itself.

This is an incredibly common occurrence, especially with ML systems which are not designed by people with an engineering mindset.

Not getting the package versions you want is a local dev problem and not specific for python at all. I will remind you that this thread is about deploying to prod and dispelling vague assertions of nebulous problems that are supposedly solved by docker.

That only works until you have a conflicting dependency (same codebase: a.py imports libraryA which needs dep>3.0.2, and b.py imports libraryB that needs dep==1.8.3). Then you're screwed.

no, pipenv install will resolve that and use the appropriate max version.

It will use the max version, but that version will likely have deprecated things that the other dependency relies on in its chosen version. Specifying that you must use an old version of a library is how a lot of Python maintainers resolve deprecations in their dependencies.

“Microservices can easily be a must” please explain.

Your example talks of packaging issues.

Microservices (or really services in general) solve some of these packaging issues. If I have my application that depends on package A that pulls in dependency C version 1.x, and also on package B that pulls in dependency C version 2.x, this just doesn't work in Python, and many other languages. The only way to make it work is either rectify my dependencies so all my versions match (by running one of your dependencies out of date) or to split them up so my application is composed of one service that pulls in package A and another that pulls in package B, and have them talk over some IPC.

Did you read the same article I did? How is this relevant?

I meant to reply to "good lord is this what modern microservices are like?".

Reply on news.ycombinator.com