Hacker News Reader

Package management was so frustrating coming from Ruby to Python. There are so many programs, and version conflicts that I run into all the time. With gems and bundler, it was rare if things didn't work.

Really? Pip should be pretty straight forward. Maybe you use different libraries than I do.

But be sure to use Pip into this year's flavor of virtual environments rather than globally, since you might have multiple projects that need different versions of the same package. And be sure to run the right version, I've seen situations where a computer had python 2 and 3 installed, somehow python3 became the default when running "python" but if you ran "pip" you'd get dumped into the old version.

And if your editor has a "Run script" function, it also needs to deal with "Activate the correct environment" for whichever flavor of virtual environment you've used.

Any what beginner would be confused by tutorials using all sorts of different setups (virtualenv, venv, pyenv, pipenv, virtualenvwrapper, pyenv-virtualenvwrapper)? Most of them, I think.

"venv" is the newest one - shipped in the python3 package except when it's not (Ubuntu, maybe others) and doesn't make copies of the python binaries for each environment. That always seemed wasteful to me, but I suppose it was the easy way around path problems with the ecosystem not having a solution built in.

I love python when I'm only using the standard libraries, but if I have dependencies it's messier to set up than I want it to be.

Those all seem like pretty basic developer best practices to me.

Of course each project needs it's own virtualenv. In fact, each project should have it's own docker image. Why install an application library to the entire system? That is not a good idea in ANY language.

And of course scripts need to use the right virtualenv. Or better yet - why not put each app in it's own docker? That bundles up everything into a single "file" you can run on command.

All of these things you are listing as downsides seem like things that one normally moves away from, when they put a codebase in git and stop deploying to prod by copying a few files to a shared network drive and restarting the server?????

> Those all seem like pretty basic developer best practices to me.

Those are all bleeding edge developer best practices that didn't exist five years ago. Folks coming from matlab (an example I read above,) probably aren't in the industry and many likely don't even know docker exists OR why they would want to use it.

requirements.txt is NOT bleeding edge.

And the GP was saying that the ruby package manager is more magical then the python package manager... when pip solved all of the complaints mentioned.

Odd.

Honestly, virtualenv works. Every single time. I use virtualenvwrapper because I find it and the aliases it provides useful.

I'll stop using virutalenv when it actually becomes a hindrance. I know venv exists, but I have precisely zero incentive to use it when virtualenv works and allows me to get on with what I actually want to do.

virtualenvwrapper, virtualenv, venv...

This is kinda discouraging :)

virtualenv is python2, venv is python3. virtualenvwrapper is as described, a wrapper around them to provide some convenience functions, but they're perfectly functional without it.

I do mostly Ruby professionally and dabble in Python, and I agree with heydonovan. Pip works fine for installing things, but near as I can tell, Python has no equivalent of bundler - a tool to install specific versions of a bunch of libraries, generate a lockfile ensuring that your production server and all of the devs are all running it with the same version of everything, and letting you have as many versions of a gem installed as you want, and still running a project with specific versions other than the latest.

You can do this, but it's not as easy. The pipfile project is an effort to improve this.

The current standard way to do this is with a virtualenv and a requirements.txt file in which you specify the version of your dependencies, which you can then install with pip install -r requirements.txt.

Format is "packagename==0.1.0" (>=, < etc work too). VCS urls work too btw, e.g. "git+protocol://site/repo@tag_or_commit"

To get something similar (but inferior) to lockfiles you can use "pip freeze > filename"

Look at Pipenv: http://docs.pipenv.org/en/latest/

Packaging/distribution is certainly a mess.

That's not true. A requirements file is the canonical way to do this. Just use a requirements file and pip install -r requirements.txt. Then version control that file and other Devs do the same. Just like a gemfile.

pip install -r requirements.txt

Pretty simple, no? Most any python project I have seen in the last 5-6 years uses it.

requirements.txt don't track transitive dependencies. Your project could break if a dependency decided to swap out its own dependencies. This isn't theoretical either - it happened to us in production.

We are using `pip-tools` to manage that: https://github.com/jazzband/pip-tools

What do you mean? `pip freeze > requirements.txt.lock` will freeze all your current deps, including transitive.

This is only practical if you only have one set of requirements.

If you have multiple sets of requirements (for dev vs testing vs production, eg), this becomes impractical real quick. You'll essentially have to tear down and rebuild your dependencies multiple times to update your reqs.

> Python has no equivalent of bundler - a tool to install specific versions of a bunch of libraries

Zope and plone, the project(s) that came up with python eggs, used to have the concept of "known good sets"[1], which were massaged and used with zc.buildout :

http://www.buildout.org/en/latest/

I'm not sure I'd recommend it for greenfield projects - splitting things up in sane-sized chunks, and using the now in-standard-lib vevn (python3 -m venv) is probably a better idea.

But just for the record, there exists a canonical(ish) solution to some of these issues in python-land.

[1] http://grok.zope.org/doc/community/life_cycle/known_good_set...

Sorry to nitpick, but Zope and Plone were not the originators of Python eggs, although they may have popularized them.

The design was done by Phillip Eby, with funding from OSAF for the development of Chandler, not Zope.

Much shade has been thrown at eggs over the years, and people bemoan the warts, but eggs and setuptools were better than what we had at the time in Python.

https://pypi.python.org/pypi/setuptools/0.6c11

pip (py2) on windows is an absolute disaster. pip install numpy ....

if you are using windows, using python without this website is nearly impossible: http://www.lfd.uci.edu/~gohlke/pythonlibs/

Things are a lot better with Python 3 on Windows, and especially so with Python 3.5+.

"pip install numpy" will actually work, among other things. But even for packages that don't have binary wheels and have to compile things, it's much easier now to install a compiler toolchain (it's actually a single download!), and have your Python installation just pick it up automatically.

> Things are a lot better with Python 3 on Windows, and especially so with Python 3.5+.

For one of my main applications py3 proved to be too slow. For my application I use: numpy, pil, tkinter, and subprocess (and process/produce/draw on screen images (at a rate of 120/sec)). I don't know WHY but with python2 it runs acceptably fast, on python3 it's 20% slower. This has been something common for me actually.

I do tend to use py3 when doing any kind of data science though.

If you get into the habit of making a virtual environment and requirements.txt in every repo then it's smooth. Just activate the env when you're using the repo and use vanilla pip install to install dependencies. Then pip freeze them into the requirements file. It's a very similar workflow to a gemfile and bundler.

I worked on python for a few months last year, and although I didn't find anything representing a canonical description of that workflow, I did eventually come to this conclusion. The only thing was that pip freeze > requirements.txt didn't really give the same power as I was used to with Gemfile.lock and the various permutations of bundle update. I forget the specifics, but I remember being unable to get the finer points of optimistic version locking to work in a way I found acceptable.

That said, I have been doing ruby since before Bundler, and I really have to take my hat off to what Yehuda and company accomplished with Bundler. It was both a technical and open source community triumph to get Bundler done, stable and covering the breadth of use cases it applies to.

You're going to have a hard time supporting multiple versions of Python that way. If I started a new package today I would target 3.4, 3.5, 3.6, 3.7-dev, and likely, 2.7. A lot of well maintained packages have different dependencies based on the exact version of Python. If you're crazy enough to support 2.6, then there will be a lot of additional packages in pip freeze.

Ideally, packages would have just one set of dependencies and the packages would be version locked, but that's just not the case in the Python community.

Just target 3.4+ then...

Why would you sort 2.6 in 2017?

Supporting 2.7 these days in a new package is already a bit of a wtf - Django for example drops support for it next year.

For the industry I work in, because of this: http://www.vfxplatform.com

We're still maintaining stacks on py 2.6 which predate the reference platform specs

Ok. It looks like they also drop python 2 next year?

They plan to release the spec like that, but I'm certain on this update most studios will be lagging due to the vast amount of 2 to 3 migration work that needs to be done.

Oh well knuckle down and use six for another couple of years then.

At least you're not trying to write Fortran compatible with the 77 and 90 specs....

If you're using Windows, then WinPython (http://winpython.sourceforge.net/) is an absolute necessity. I have two separate installations, one for 2.x and one for 3.x, and I have zero issues. Works amazingly well and comes with just about everything you need already installed and working. Can't say enough good things about it.

> it's the newer ... Java

Python (1991) is older than Java (1995).

But Java was an overnight success, while it took Python years to get traction. I was exposed to both around the same time, probably 1995, but I ignored Python until the second time I encountered it in 2003. It just didn't have the hype.

Java was by no means an overnight success. It was way too slow for several years until they really started to do anything more than run the bytecode in the JVM.

McNealy freely admitted in an interview that:

"we were able to hook Java runtime onto the free, open source Netscape browser. And it was a collision of two things that accelerated and launched each other together. It was fascinatingly lucky timing for both companies."

While it may have taken time to mature at that time it was an "overnight success".

Sun was considering dropping it and considered it an expense before that.

Did Sun ever make any money on it?

It was an overnight success in the hype department.

I'm curious, what issues do you run into with python3 incompatibility?

Most scientific stuff I've done works in python3. In fact, its now possible to run a tensorflow/opencv/scipy environment entirely from pip in a virtualenv, though it won't be the fastest. which is amazing. Doing that kind of thing in any other language would require docker.

In general, if you're sticking to python3, the answer will always be "use pip", everything should work with pip, and everything should install as a binary without needing to build source.

Edit:

There's exactly one context where I use python2, and its for a robotics project where some transitive dependencies are python2 only.

Why won't it be the fastest? I didn't think there would be any difference between "native" in the system paths and in a virtualenv.

So, for things like tensorflow, you have to build from source to enable certain optimizations. Wheel's aren't source builds essentially.

I think GP is talking about virtualenv, not pip. Using an optimized build in a virtualenv environment should not be any slower than using it outside the environment, right?

Ah yes, that's true as far as I know.

Actually I always complain about Python packaging when I'm only using Python. But the moment I start using it in other languages I dream about the comfort and professionalism of tools in Python.

Of course in scientific Py you need compiled C attachments, which makes it more complex. But in normal Py usage you only need pip. Easy Install is only needed when you don't have pip installed. Pip is a drop in replacement with additional features.

And at least from what I heard for everything with C compiled stuff inside ("binary packages") you use conda.

So not that hard actually.

Easy install hasn't been needed for years, consider it deprecated.

Well, as said before sometimes you need it because pip isn't installed by default.

Just had such a situation yesterday at work. A guy from QA (not a dev) needs to setup an environment to install our tool for testing. He doesn't know python, so explaining to him what a virtualenv is would take way too much time. He probably also doesn't know how to use the OS's package manager to install pip. So, what to do? Give him the oneliner "$ sudo easy_install whatever". It works, he doesn't need to understand it, and actual system maintenance doesn't matter because it's just a VM that gets deleted after he's done anyways.

You don't. First, if your QA peeps can't figure out an OS package manager, they need training. It's also a one-liner.

Without a package manager it is only a two-step process. 1) download get-pip.py from the source 2) run it with sudo

I highly recommend switching to the conda distributions and package ecosystem. A lot of those issues have gotten a lot better for me since I did that.

> "Ahhh device developer only provides example code for Labview and Matlab!", "Ahhh python isn't even supported for this device/application!").

What kind of devices or libraries do you use from LabVIEW but there isn't Python support or examples?

If its an NI product, some are starting to get Python support

DAQmx https://github.com/ni/nidaqmx-python/blob/master/nidaqmx_exa...

XNET https://github.com/ni/nixnet-python/blob/master/nixnet_examp...

DMM https://github.com/ni/nimi-python/blob/master/src/nidmm/exam...

FPGA Interface https://nifpga-python.readthedocs.io/en/latest/examples/basi...

EDIT: All with Python3 support.

Andor cameras: http://www.andor.com/scientific-software/software-developmen... Labview, Matlab, lots of other frameworks, no python

PI instruments: https://www.physikinstrumente.com/en/products/linear-stages-... Matlab, Labview, etc. yes. No python

Thorlabs equipment: https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=81... Same story, no python.

In fact, it's always a _SURPRISE_ when I see Python being supported. I would say after having gone through about a hundred major lab devices, less than 2% I found to be supporting Python.

Importantly, example code should be made available (as it generally is for matlab, and labview) because creating things from scratch in a move-fast environment is a little too much.

I think you're right, the biggest pain for a general developer for Python is determining which package manager to use. I'm thankfully insulated from it a bit due to using Docker for a lot of my development, but I hate the fact that Pip won't install to your home/project directory unless you coax it to (and sometimes still won't even if you give it all the proper environment variables), and I think virtual environments are not the solution (I shouldn't have to think about how to install a package or work on a project). Especially coming from Node-land, and before that Ruby-land, Python seemed like a wonderfully designed language but with terrible package managers. From what I remember Pip is going to change the default to install on the home directory sometime in the future, but it shouldn't have taken years of discussion to come to the conclusion that it's the right thing to do.

I like python3 -m venv myfolder; source myfolder/bin/activate;

It is the easiest way to use python3 because even fedora (one of the more bleeding edge) likely won't finish migrating 2 to 3 in current year.

Why do you not like venv? What could be done better?

I don't really like virtualenv because you have to activate it to do anything and deactivate it if you want to go to another project or work on something unrelated. I'd much rather it work by the project directory so I don't have to worry about the activation part (this is why Node's package manager, for all its issues, is great: you install in a local folder unless you explicitly supply the -g flag). At least I should be able to install packages to my home directory.

Ah yes. Local by default would be nicer.

I still think that the "project" directory is best, but I'm glad that Pip is moving to local-by-default in any case. It won't take away from the necessity of virtualenv for cases of different dependency versions, e.g. Django 1.4 and 1.10, I think, unfortunately, but it's definitely better than global by default.

The problem is not for people like you who already decided, the problem is for beginners who aren't aware of some of the 'envs' and are looking for recommended practices.

I've moved to Anaconda and never looked back. Worst case you use pip within your Anaconda environment. Best and usual case everything just works.

    pip install --user pkgname

Does that not work for you?

The user flag doesn't always work correctly, but my main annoyance is that I shouldn't have to supply a flag to do this. It should be default.

I agree, and they are working towards it. In the meantime, there are shell aliases. That's how I ask rm to confirm deletion as well.

>I think the one thing we are all learning from this experience is: JUST USE UBUNTU, doing python with Windows is a headache.

Meh. I use python on Windows all the time - it's not a problem. Mostly pip just works.

Of course, I have over a decade of experience with Python - albeit only the last few years in Windows.

the #1 issue i have with py on win is having to download a million different C++ compilers esp for packages not distributed on pip, and esp the more obscure ones with no support whatsoever (e.g. data processing or math packages that were put together by some chap who was working on it for a specific set of research papers and has since moved on).

chris gohlke's website was a real lifesaver back then, although now pip seems much better. nowadays it can be a bit difficult replicating environments across different machines especially since conda is moving rapidly from python ver to ver while some obscure packages i rely on are still stuck on 3.4 or earlier... or if c++ 10.0 or 9.0 or 14.0 is not set up correctly. then it becomes an issue of crossing fingers and hoping "python setup.py install" doesn't fail.

What's the point of using Windows as a Dev or data science environment anyway? Everything is built and works well in Linux, scripting is powerful, it's free, etc.. seriously asking.

When you work for a company that gives you Windows PC's, and have a horrible Linux environment (through a network, not on your laptop). From a Python standpoint, there is no down side to developing in Windows.

It's easier for me to install the Python libraries I need in Windows than to install them on Linux where I don't have root privileges.

Yeah use Ubuntu, your life will get way easier.