Hacker News Reader

Comment by andrewstuart | Hacker News Reader

Python 2 advocates were quite firm in telling us that Python 3 and its incompatibility with Python 2 had killed Python and people would abandon Python altogether in droves for other languages.

This post is suggesting the opposite, that Python is more healthy than ever and growing incredibly fast.

View on HN

Three of us (including me), in an academic lab, are making the jump from matlab to python. We use py3 (of Anaconda, on Windows10).

We like it, but we are constantly screaming and bitching about the complications of 2<->3 ("Ahhh X and Y module only works on py2!", "Ahhh device developer only provides example code for Labview and Matlab!", "Ahhh python isn't even supported for this device/application!").

Package management ("Do I use conda? pip? easy_install? compile?") is annoying, I hope that something like hatch (https://github.com/ofek/hatch) maybe takes over everything -- because I think the mess that is package management is THE biggest deterrent for newcomers.

I think the one thing we are all learning from this experience is: JUST USE UBUNTU, doing python with Windows is a headache.

All of that said -- python is nice, it's amazing, it's the newer, funner Java. It is incredibly approachable, it can do pretty much EVERYTHING because of the expansive availability of libraries, it's finally a programming language that is great to get my 7 year old niece started on.

Package management was so frustrating coming from Ruby to Python. There are so many programs, and version conflicts that I run into all the time. With gems and bundler, it was rare if things didn't work.

If you get into the habit of making a virtual environment and requirements.txt in every repo then it's smooth. Just activate the env when you're using the repo and use vanilla pip install to install dependencies. Then pip freeze them into the requirements file. It's a very similar workflow to a gemfile and bundler.

I worked on python for a few months last year, and although I didn't find anything representing a canonical description of that workflow, I did eventually come to this conclusion. The only thing was that pip freeze > requirements.txt didn't really give the same power as I was used to with Gemfile.lock and the various permutations of bundle update. I forget the specifics, but I remember being unable to get the finer points of optimistic version locking to work in a way I found acceptable.

That said, I have been doing ruby since before Bundler, and I really have to take my hat off to what Yehuda and company accomplished with Bundler. It was both a technical and open source community triumph to get Bundler done, stable and covering the breadth of use cases it applies to.

You're going to have a hard time supporting multiple versions of Python that way. If I started a new package today I would target 3.4, 3.5, 3.6, 3.7-dev, and likely, 2.7. A lot of well maintained packages have different dependencies based on the exact version of Python. If you're crazy enough to support 2.6, then there will be a lot of additional packages in pip freeze.

Ideally, packages would have just one set of dependencies and the packages would be version locked, but that's just not the case in the Python community.

Just target 3.4+ then...

Why would you sort 2.6 in 2017?

Supporting 2.7 these days in a new package is already a bit of a wtf - Django for example drops support for it next year.

For the industry I work in, because of this: http://www.vfxplatform.com

We're still maintaining stacks on py 2.6 which predate the reference platform specs

Ok. It looks like they also drop python 2 next year?

They plan to release the spec like that, but I'm certain on this update most studios will be lagging due to the vast amount of 2 to 3 migration work that needs to be done.

Oh well knuckle down and use six for another couple of years then.

At least you're not trying to write Fortran compatible with the 77 and 90 specs....

Really? Pip should be pretty straight forward. Maybe you use different libraries than I do.

But be sure to use Pip into this year's flavor of virtual environments rather than globally, since you might have multiple projects that need different versions of the same package. And be sure to run the right version, I've seen situations where a computer had python 2 and 3 installed, somehow python3 became the default when running "python" but if you ran "pip" you'd get dumped into the old version.

And if your editor has a "Run script" function, it also needs to deal with "Activate the correct environment" for whichever flavor of virtual environment you've used.

Any what beginner would be confused by tutorials using all sorts of different setups (virtualenv, venv, pyenv, pipenv, virtualenvwrapper, pyenv-virtualenvwrapper)? Most of them, I think.

"venv" is the newest one - shipped in the python3 package except when it's not (Ubuntu, maybe others) and doesn't make copies of the python binaries for each environment. That always seemed wasteful to me, but I suppose it was the easy way around path problems with the ecosystem not having a solution built in.

I love python when I'm only using the standard libraries, but if I have dependencies it's messier to set up than I want it to be.

Honestly, virtualenv works. Every single time. I use virtualenvwrapper because I find it and the aliases it provides useful.

I'll stop using virutalenv when it actually becomes a hindrance. I know venv exists, but I have precisely zero incentive to use it when virtualenv works and allows me to get on with what I actually want to do.

virtualenvwrapper, virtualenv, venv...

This is kinda discouraging :)

virtualenv is python2, venv is python3. virtualenvwrapper is as described, a wrapper around them to provide some convenience functions, but they're perfectly functional without it.

Those all seem like pretty basic developer best practices to me.

Of course each project needs it's own virtualenv. In fact, each project should have it's own docker image. Why install an application library to the entire system? That is not a good idea in ANY language.

And of course scripts need to use the right virtualenv. Or better yet - why not put each app in it's own docker? That bundles up everything into a single "file" you can run on command.

All of these things you are listing as downsides seem like things that one normally moves away from, when they put a codebase in git and stop deploying to prod by copying a few files to a shared network drive and restarting the server?????

> Those all seem like pretty basic developer best practices to me.

Those are all bleeding edge developer best practices that didn't exist five years ago. Folks coming from matlab (an example I read above,) probably aren't in the industry and many likely don't even know docker exists OR why they would want to use it.

requirements.txt is NOT bleeding edge.

And the GP was saying that the ruby package manager is more magical then the python package manager... when pip solved all of the complaints mentioned.

Odd.

I do mostly Ruby professionally and dabble in Python, and I agree with heydonovan. Pip works fine for installing things, but near as I can tell, Python has no equivalent of bundler - a tool to install specific versions of a bunch of libraries, generate a lockfile ensuring that your production server and all of the devs are all running it with the same version of everything, and letting you have as many versions of a gem installed as you want, and still running a project with specific versions other than the latest.

You can do this, but it's not as easy. The pipfile project is an effort to improve this.

The current standard way to do this is with a virtualenv and a requirements.txt file in which you specify the version of your dependencies, which you can then install with pip install -r requirements.txt.

Format is "packagename==0.1.0" (>=, < etc work too). VCS urls work too btw, e.g. "git+protocol://site/repo@tag_or_commit"

To get something similar (but inferior) to lockfiles you can use "pip freeze > filename"

Look at Pipenv: http://docs.pipenv.org/en/latest/

Packaging/distribution is certainly a mess.

That's not true. A requirements file is the canonical way to do this. Just use a requirements file and pip install -r requirements.txt. Then version control that file and other Devs do the same. Just like a gemfile.

> Python has no equivalent of bundler - a tool to install specific versions of a bunch of libraries

Zope and plone, the project(s) that came up with python eggs, used to have the concept of "known good sets"[1], which were massaged and used with zc.buildout :

http://www.buildout.org/en/latest/

I'm not sure I'd recommend it for greenfield projects - splitting things up in sane-sized chunks, and using the now in-standard-lib vevn (python3 -m venv) is probably a better idea.

But just for the record, there exists a canonical(ish) solution to some of these issues in python-land.

[1] http://grok.zope.org/doc/community/life_cycle/known_good_set...

Sorry to nitpick, but Zope and Plone were not the originators of Python eggs, although they may have popularized them.

The design was done by Phillip Eby, with funding from OSAF for the development of Chandler, not Zope.

Much shade has been thrown at eggs over the years, and people bemoan the warts, but eggs and setuptools were better than what we had at the time in Python.

https://pypi.python.org/pypi/setuptools/0.6c11

pip install -r requirements.txt

Pretty simple, no? Most any python project I have seen in the last 5-6 years uses it.

requirements.txt don't track transitive dependencies. Your project could break if a dependency decided to swap out its own dependencies. This isn't theoretical either - it happened to us in production.

We are using `pip-tools` to manage that: https://github.com/jazzband/pip-tools

What do you mean? `pip freeze > requirements.txt.lock` will freeze all your current deps, including transitive.

This is only practical if you only have one set of requirements.

If you have multiple sets of requirements (for dev vs testing vs production, eg), this becomes impractical real quick. You'll essentially have to tear down and rebuild your dependencies multiple times to update your reqs.

pip (py2) on windows is an absolute disaster. pip install numpy ....

if you are using windows, using python without this website is nearly impossible: http://www.lfd.uci.edu/~gohlke/pythonlibs/

Things are a lot better with Python 3 on Windows, and especially so with Python 3.5+.

"pip install numpy" will actually work, among other things. But even for packages that don't have binary wheels and have to compile things, it's much easier now to install a compiler toolchain (it's actually a single download!), and have your Python installation just pick it up automatically.

> Things are a lot better with Python 3 on Windows, and especially so with Python 3.5+.

For one of my main applications py3 proved to be too slow. For my application I use: numpy, pil, tkinter, and subprocess (and process/produce/draw on screen images (at a rate of 120/sec)). I don't know WHY but with python2 it runs acceptably fast, on python3 it's 20% slower. This has been something common for me actually.

I do tend to use py3 when doing any kind of data science though.

If you're using Windows, then WinPython (http://winpython.sourceforge.net/) is an absolute necessity. I have two separate installations, one for 2.x and one for 3.x, and I have zero issues. Works amazingly well and comes with just about everything you need already installed and working. Can't say enough good things about it.

> it's the newer ... Java

Python (1991) is older than Java (1995).

But Java was an overnight success, while it took Python years to get traction. I was exposed to both around the same time, probably 1995, but I ignored Python until the second time I encountered it in 2003. It just didn't have the hype.

Java was by no means an overnight success. It was way too slow for several years until they really started to do anything more than run the bytecode in the JVM.

McNealy freely admitted in an interview that:

"we were able to hook Java runtime onto the free, open source Netscape browser. And it was a collision of two things that accelerated and launched each other together. It was fascinatingly lucky timing for both companies."

While it may have taken time to mature at that time it was an "overnight success".

Sun was considering dropping it and considered it an expense before that.

Did Sun ever make any money on it?

It was an overnight success in the hype department.

I'm curious, what issues do you run into with python3 incompatibility?

Most scientific stuff I've done works in python3. In fact, its now possible to run a tensorflow/opencv/scipy environment entirely from pip in a virtualenv, though it won't be the fastest. which is amazing. Doing that kind of thing in any other language would require docker.

In general, if you're sticking to python3, the answer will always be "use pip", everything should work with pip, and everything should install as a binary without needing to build source.

Edit:

There's exactly one context where I use python2, and its for a robotics project where some transitive dependencies are python2 only.

Why won't it be the fastest? I didn't think there would be any difference between "native" in the system paths and in a virtualenv.

So, for things like tensorflow, you have to build from source to enable certain optimizations. Wheel's aren't source builds essentially.

I think GP is talking about virtualenv, not pip. Using an optimized build in a virtualenv environment should not be any slower than using it outside the environment, right?

Ah yes, that's true as far as I know.

Actually I always complain about Python packaging when I'm only using Python. But the moment I start using it in other languages I dream about the comfort and professionalism of tools in Python.

Of course in scientific Py you need compiled C attachments, which makes it more complex. But in normal Py usage you only need pip. Easy Install is only needed when you don't have pip installed. Pip is a drop in replacement with additional features.

And at least from what I heard for everything with C compiled stuff inside ("binary packages") you use conda.

So not that hard actually.

Easy install hasn't been needed for years, consider it deprecated.

Well, as said before sometimes you need it because pip isn't installed by default.

Just had such a situation yesterday at work. A guy from QA (not a dev) needs to setup an environment to install our tool for testing. He doesn't know python, so explaining to him what a virtualenv is would take way too much time. He probably also doesn't know how to use the OS's package manager to install pip. So, what to do? Give him the oneliner "$ sudo easy_install whatever". It works, he doesn't need to understand it, and actual system maintenance doesn't matter because it's just a VM that gets deleted after he's done anyways.

You don't. First, if your QA peeps can't figure out an OS package manager, they need training. It's also a one-liner.

Without a package manager it is only a two-step process. 1) download get-pip.py from the source 2) run it with sudo

I highly recommend switching to the conda distributions and package ecosystem. A lot of those issues have gotten a lot better for me since I did that.

> "Ahhh device developer only provides example code for Labview and Matlab!", "Ahhh python isn't even supported for this device/application!").

What kind of devices or libraries do you use from LabVIEW but there isn't Python support or examples?

If its an NI product, some are starting to get Python support

DAQmx https://github.com/ni/nidaqmx-python/blob/master/nidaqmx_exa...

XNET https://github.com/ni/nixnet-python/blob/master/nixnet_examp...

DMM https://github.com/ni/nimi-python/blob/master/src/nidmm/exam...

FPGA Interface https://nifpga-python.readthedocs.io/en/latest/examples/basi...

EDIT: All with Python3 support.

Andor cameras: http://www.andor.com/scientific-software/software-developmen... Labview, Matlab, lots of other frameworks, no python

PI instruments: https://www.physikinstrumente.com/en/products/linear-stages-... Matlab, Labview, etc. yes. No python

Thorlabs equipment: https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=81... Same story, no python.

In fact, it's always a _SURPRISE_ when I see Python being supported. I would say after having gone through about a hundred major lab devices, less than 2% I found to be supporting Python.

Importantly, example code should be made available (as it generally is for matlab, and labview) because creating things from scratch in a move-fast environment is a little too much.

I think you're right, the biggest pain for a general developer for Python is determining which package manager to use. I'm thankfully insulated from it a bit due to using Docker for a lot of my development, but I hate the fact that Pip won't install to your home/project directory unless you coax it to (and sometimes still won't even if you give it all the proper environment variables), and I think virtual environments are not the solution (I shouldn't have to think about how to install a package or work on a project). Especially coming from Node-land, and before that Ruby-land, Python seemed like a wonderfully designed language but with terrible package managers. From what I remember Pip is going to change the default to install on the home directory sometime in the future, but it shouldn't have taken years of discussion to come to the conclusion that it's the right thing to do.

I like python3 -m venv myfolder; source myfolder/bin/activate;

It is the easiest way to use python3 because even fedora (one of the more bleeding edge) likely won't finish migrating 2 to 3 in current year.

Why do you not like venv? What could be done better?

I don't really like virtualenv because you have to activate it to do anything and deactivate it if you want to go to another project or work on something unrelated. I'd much rather it work by the project directory so I don't have to worry about the activation part (this is why Node's package manager, for all its issues, is great: you install in a local folder unless you explicitly supply the -g flag). At least I should be able to install packages to my home directory.

Ah yes. Local by default would be nicer.

I still think that the "project" directory is best, but I'm glad that Pip is moving to local-by-default in any case. It won't take away from the necessity of virtualenv for cases of different dependency versions, e.g. Django 1.4 and 1.10, I think, unfortunately, but it's definitely better than global by default.

The problem is not for people like you who already decided, the problem is for beginners who aren't aware of some of the 'envs' and are looking for recommended practices.

I've moved to Anaconda and never looked back. Worst case you use pip within your Anaconda environment. Best and usual case everything just works.

    pip install --user pkgname

Does that not work for you?

The user flag doesn't always work correctly, but my main annoyance is that I shouldn't have to supply a flag to do this. It should be default.

I agree, and they are working towards it. In the meantime, there are shell aliases. That's how I ask rm to confirm deletion as well.

>I think the one thing we are all learning from this experience is: JUST USE UBUNTU, doing python with Windows is a headache.

Meh. I use python on Windows all the time - it's not a problem. Mostly pip just works.

Of course, I have over a decade of experience with Python - albeit only the last few years in Windows.

the #1 issue i have with py on win is having to download a million different C++ compilers esp for packages not distributed on pip, and esp the more obscure ones with no support whatsoever (e.g. data processing or math packages that were put together by some chap who was working on it for a specific set of research papers and has since moved on).

chris gohlke's website was a real lifesaver back then, although now pip seems much better. nowadays it can be a bit difficult replicating environments across different machines especially since conda is moving rapidly from python ver to ver while some obscure packages i rely on are still stuck on 3.4 or earlier... or if c++ 10.0 or 9.0 or 14.0 is not set up correctly. then it becomes an issue of crossing fingers and hoping "python setup.py install" doesn't fail.

What's the point of using Windows as a Dev or data science environment anyway? Everything is built and works well in Linux, scripting is powerful, it's free, etc.. seriously asking.

When you work for a company that gives you Windows PC's, and have a horrible Linux environment (through a network, not on your laptop). From a Python standpoint, there is no down side to developing in Windows.

It's easier for me to install the Python libraries I need in Windows than to install them on Linux where I don't have root privileges.

Yeah use Ubuntu, your life will get way easier.

I really don't know where this is coming from. The only people I hear talking about a 2v3 divide are my Javascript developer friends, who are saying that as outside observers.

As a Python dev, I (and all the other Python devs I know) were just waiting for (and helping) libraries to start working with 3 before we can switch over. These days, all my new projects are in 3, 2 is pretty much for legacy code only.

Same experience here -- got back into dev earlier this year and decided to go with Python 3 everywhere. The only libraries I've needed that were Python 2 only were ones that weren't being maintained anyways. It's been a pretty good recency filter for me. I know that wasn't the case when Python 3 was younger, but the ecosystem is pretty dang good today.

Those people were advocating for stability, not necessarily for P2. The 2->3 was a pretty big screw up, and it definitely had an impact on the language and ecosystem.

Python seems to have overcome that (keyword: seems), thanks to a strong offering in data science, but 2->3 is still a headache. I never really used Python for application development, but rather for utility/scripts and I am still on 2, because I don't need more.

2 is available out of the box on all OSes and at work we also use 2 for building tools.

I have the feeling that Python is still losing people to Golang due to its weak performance and Golang's reputation in that area.

So instead of being arrogant about 2->3, I'd say Python devs should be thankful to data science for saving their butts. :)

I doubt Python is losing people to Golang in scientific computing.

>>Python is still losing people to Golang

Any source (other than Rob Pike) ?

Anecdata I'm afraid, hence "feeling". This can only be solved through an actual market study.

The SO post is not the study we need, because it uses a proxy variable (SO tags).

That's not the way to look at this. The control you would need to establish whether what you are saying is true is a world in which the 2/3 split never happened. And in that world Python is most likely doing even better than in this one...

They fucked up. That Python is resilient enough to absorb such a blow and that no other language is a viable alternative for that particular niche is what saved the day.

And even now there is still plenty to improve on, fortunately Anaconda at least makes package management more sane.

Jan 1 2018 is the day I abandon Python 2 for all greenfield projects.

Why are you deliberately making things harder for yourself? You're going to need to upgrade these projects within three years. Just seems like a wasted opportunity not to get off on a strong footing.

My Python 2 code already converts cleanly to Py3 with 2to3. There will be zero effort for me to port anything when 2.7 is EOLed. I just want to be able to use the new features without dancing around the forward compatibility game.

We did that in 2014 in production already :)

I abandoned Python 2 around April 2017 for greenfield.

I feel like it's still a better idea to migrate back to python 2.7 and update it to match python 3 features in a backwards compatible way. At the very least there needs to be a way to interop between python 2 and python 3 code without modifying the python 2 code.

I haven't used python for a while though. Maybe they found a solution to the python 2/3 problem already.

Maybe the confusion over python 2 vs. 3 is why people have so many questions about it..

</snark, mostly>

Are most developers finally making the transition from Python 2 to 3?

Reply on news.ycombinator.com