just spent about 4 hrs getting Python 2 and pip setup on my Mac. Any time I have the misfortune of needing to use python I get cold sweats at the thought of the environment stuff.
Why is this still such a massive problem?
Why is this still such a massive problem?
Use pyenv to install any python version and pyenv-virtualenv to manage virutal environments.
brew install pyenvI've had nothing but pain with conda in mixed linux/mac/win environment at work, and actively worked to get it deprecated. We're on plain venvs now, with the occasional setup.py
Curious if I missed some useful case for it.
- very slow dependency resolvers
- using conda in docker is annoying
- the worst thing: inconsistencies in downloading binaries and other resources between win/mac/linux when installing packages
- conda insisting on messing with bashrc to run their 'activate' crap
- beware the fool who runs 'sudo conda whatever' on a multiuser system
Conda will package and version things like compilers and C libraries etc in a sane way. I can 'conda install' a package and start using it, but if i 'pip install' the same package I'm still left having to hunt down and install a bunch of additional dependencies. For example 'conda install cython' gives me a fully working and consistent cython on all platforms in a way that 'pip install cython' won't.
There are still several computational science and data science packages where 'pip install X' on windows simply doesn't work (and 'conda install X' does) due to missing dependencies and other weirdness.
Conda can version and manage R as well as Python if you work on projects that use both then you only need one tool.
Conda can also install dev tools like Spyder, VSCode or Jupyter(labs) making it easier to offer a single point of entry for an entire dev environment.
But basically the big one is simply that Conda will install a lot more numeric packages with a lot fewer headaches on win, linux and mac than pip. Although in pip's defense it has gotten a lot better over the past 2-3 years. And I do agree that the dependency resolver can take a ridiculously long time in some situations.
Some people really just want Python+extras and don't want to worry about getting a fortran compiler. Conda is great for this.
Other people really don't want a big meta-install system that bundles a bunch of dependencies. Maybe they're on Linux, they target a single platform, or they want to use a package not in conda that works with a more recent version of the dependencies that conda bundles.
For some people Python is really more of an end user application. They don't want to develop and distribute and deploy software, they want to write scripts and analyze and transform data. They expect the script a colleague emailed them to just work and expect their colleague to be able to open and run their Jupyter notebook as easily as they can open and run an Excel document. For those people Conda comes much closer to solving their problems.
https://docs.cupy.dev/en/stable/install.html
With conda this installation tends to just work, conda will also install the non-python cudatoolkit for you. With pip you have to either make all your python developers set up their c++ environment the same way as well, to install from source, or set a fixed cuda version that all users have to be on.
Now `cupy` is just one python library that has non-python dependencies. If your project has several dependencies like this, where conda is a one line install and pip means you have to mess around with your c++ environment, conda is probably the right choice for you.
All of the pain you mention with conda is totally correct, though, it's just a question of which sort of pain happens to be worse for the packages you are going to use.
conda create -n myEnv python=2.7
conda activate myEnv
But Python works just a little bit differently in that it advocates for the use of its own environments to isolate dependencies. I'll have to admit it is not super intuitive (I already have a system package manager (apt, brew, etc.) and I have Docker containers -- why do I need another yet another environment?) but over time I've just accepted it for what it is. It's a Python convention.
1) docker! If you only plan on tinkering with something and don't want to install a bunch of random things on your machine, this command will start a python 2 image with access to the current directory: `docker run --rm -it -v "$PWD:/app" python:2.7 bash` (IIRC, on my phone atm!) (Windows users will need to write `"/$PWD://app"` if using mingw/etc). Once inside, you can pip install, etc, and it'll all go away once you exit the terminal! If you want it to stick around, don't use --rm and add a --name to give it a name you can remember for later. It does require learning some docker, but it's a good investment :) I use the same technique to try out/play with other languages, versions, packages, etc. without making complicated mods to my actual computer. And there might even be docker images that come with certain packages/etc pre-installed for certain use cases!
2) Pycharm! A good python IDE, with virtualenv support baked in if you don't feel like playing around with a bunch of commands at all. Although disclaimer, I can't fully remember how hard it was to get things setup (or how easy it is to work with multiple python versions), so YMMV.
The right menu is File -> Settings -> Project: <your project name> -> Project Interpreter, and then you pick whatever you want by Python Interpreter.
In Windows I use Chocolatey to manage Python versions, in Mac idk though.
I do not see a way to install a whole new python version (only the ability to add to the list, which requires you to select the interpreter from an open file dialog)
virtualenv -p interpreter venv
source venv/bin/activate
pip install . . .
python3 -m venv venv
Now compare that with cargo or npm and the canonical way makes newbie upbringing SL much easier.
I’d almost guarantee there was something simple I was missing but for someone not familiar with the ecosystem getting python set up to do more than just hello world is a shitshow.
It is a standard library module and serves as the base functionality, most the other things you list are third party tools that build on the venv concept in various ways.
Once you grok venv, you will be in a position to understand if those tools can provide value to your particular situation.
2. Python was originally popular with old-school sysadmins, Debian types, and a lot of its package management is based around that philosophy of carefully hand-tended servers shared by multiple users.
Pip is a recipe for disaster, indicated by the huge amount of churn in the Python packaging sphere. It's constantly the worst part of my day whenever I pick a Python project up. Conflicting dependencies, non-deterministic installs, etc.
I used to cope with this crap fest until I tried Elixir and experienced the beauty of modern package management and build tools. One tool, Mix, that handles task running and dependency management with a proper resolver.
I honestly think even Node has a better package management and tooling story.
Also: virtual environments are a hack and a pain. Python is moving forward in this aspect with the recent PEP for the pypackages folder, but we're still a long way from adoption.
All of this stuff is painful for beginners. Virtual environments might seem easy to us, but I've had ridiculous amounts of trouble explaining why they're necessary to beginner developers. Then you have to explain `poetry` or `pip freeze`.
Even I have trouble coming back to older projects of mine and I'm an experienced Pythonista: I usually waste an hour or two trying to sort the pip dependencies out with whatever conflicts the resolver comes up with this time.
Python package management is not okay, we're all just used to coping with it. Other languages put it to shame.
Things continue to improve. For example, Pip's resolver has been deterministic since around November 2020. Bringing something minimal into Python that provides deterministic builds (e.g. pip-tools) would help packaging a lot.
- What you get when you import a library depends on state that's scattered all over the system: system-managed packages, pip-managed system-global packages, pip-managed per-user packages, which virtualenv is currently active, which directory you're currently in, which directory the program you're running is in, whatever it is that conda does....
- There's no concept of reproducible builds or dependency pinning. There's "pip freeze" but that's a one-time operation that you can't then reverse, so it's only usable for leaf applications. If you're developing a library, you'd better get used to having your transitive dependencies changed on you all the time. And since the whole ecosystem is built that way, even if you use some tool that lets you make stable releases of your library, that doesn't help you develop at all.
- Virtualenvs are stateful and attached to whatever terminal you were in at the time. This interacts hilariously with the previous point: if you accidentally run "cd myproject && pip install -r requirements.txt" in the wrong terminal, you permanently, irreversibly fuck up that virtualenv. All you can do is wipe it out and try to recreate it - but, per the previous point, it probably won't come out the same as before.
- You're supposed to use pip to manage which python version each project is using. But you're supposed to use the installer for it that's distributed with the python runtime. But only certain versions of the python runtime...
- There's only one global repository. If you want to build some libraries and reuse them the same way you'd use a normal library dependency, you have to publish them to the global PyPi. I think there might be an expensive service that works around this, but there's no repository program that you can just spin up on your own servers.
It's really a lot worse than other languages. If you build a real system (like, a couple of libraries and applications) in another language (not, like, C/C++ - but even Perl or TCL will prove the point) and then come back to Python, you'll find yourself hating it all the time.
You can setup proxy such as Nexus, which allows hosting your own librarires: https://help.sonatype.com/repomanager3/formats/pypi-reposito...
What do you mean by reverse? You can certainly upgrade all packages locally and then run pip freeze again if you want to set up newer versions, or like manually change versions in your requirements.txt and `pip install --upgrade` to update them. For stronger reproducibility guarantees there's also `--require-hashes`, although admittedly freeze doesn't appear to support easily writing hashes to a requirements.txt.
> per the previous point, it probably won't come out the same as before.
I don't see how this follows. If you have frozen dependencies, it will.
> You're supposed to use pip to manage which python version each project is using. But you're supposed to use the installer for it that's distributed with the python runtime. But only certain versions of the python runtime...
No you use venv for that. Venv which has been available since python 3.3, in 2012, and on pip, ensurepip has been since 3.4, in 2014. If you're using a version of python that's 10 years old, I don't really know what to say.
> There's only one global repository. If you want to build some libraries and reuse them the same way you'd use a normal library dependency, you have to publish them to the global PyPi. I think there might be an expensive service that works around this, but there's no repository program that you can just spin up on your own servers.
I mean you can install from git directly: https://pip.pypa.io/en/stable/topics/vcs-support/#vcs-suppor..., and there is documentation on spinning up a local repository (which for packages, is just files in a known directory structure): https://packaging.python.org/guides/hosting-your-own-index/
You need to know, and maintain, both what you intended to depend on and what you physically ended up depending on. So in more sensible ecosystems you will have, e.g., Gemfile and Gemfile.lock. pip freeze is, effectively, the way you create Gemfile.lock, but it forces you to destroy Gemfile to do it. And so you can't really use it.
No, it doesn't.
pip freeze -r requirements.txt > requirements.lock.txt
https://pypi.org/project/pypiserver/
Also, pip will install from a git repository.
Is it? I'd love for apt to support virtual environments or `apt install --user` like pip does, let alone populating venvs with requirements.txt.
Second, brew and pyenv “just work” if you need any other version. I’ve been developing in Python for decades, and it is one of the cleanest and easier approaches to running multiple runtimes I’ve used in any language.
the problem is apple. they are (understandably) refusing to ship an up-to-date global python interpreter with macOS. not only that, they aren't removing the default 2.7 that they ship.
macOS Big Sur ships Python 2.7. ha. ha. ha.
none of the proposed "virtual environments" solutions are going to rescue you when you operate globally at the OS-level and not in a sandbox.
been there.
What is the understandable reasoning?
For anyone that's still using python2 (as a nearby comment mentioned), there is now at least a warning to avoid using the default Python 2.7 (/usr/bin/python). For an OS vendor, this transition takes a long time... but at least it's in process.
Is that the case? If so, I still don't understand the reasoning.
> the problem is apple. they are (understandably) refusing to ship an up-to-date global python interpreter with macOS
Ie the claim is that it's causing a problem for users. I don't have an opinion about this point, but it is the premise of my question.
Yes they are:
$ python
WARNING: Python 2.7 is not recommended.
This version is included in macOS for compatibility with legacy software.
Future versions of macOS will not include Python 2.7.
Instead, it is recommended that you transition to using 'python3' from within Terminal.
Python 2.7.16 (default, Aug 30 2021, 14:43:11)
[GCC Apple LLVM 12.0.5 (clang-1205.0.19.59.6) [+internal-os, ptrauth-isa=deploy on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>2. /usr/bin/python being a symlink for python2.7 is entirely consistent with PEP 394 recommendations, especially the version before June 2019: https://raw.githubusercontent.com/python/peps/214736457f6d61...
* Unix-like software distributions (including systems like Mac OS X and
Cygwin) should install the ``python2`` command into the default path
whenever a version of the Python 2 interpreter is installed, and the same
for ``python3`` and the Python 3 interpreter.
* When invoked, ``python2`` should run some version of the Python 2
interpreter, and ``python3`` should run some version of the Python 3
interpreter.
* If the ``python`` command is installed, it should invoke the same version of
Python as the ``python2`` command (however, note that some distributions
have already chosen to have ``python`` implement the ``python3``
command; see the `Rationale`_ and `Migration Notes`_ below).
https://www.ibm.com/support/pages/work-around-frustrating-py...
https://unix.stackexchange.com/questions/468620/how-to-chang...
OP's problem was not configuring default Python version as in your second link. Your first link is about some IBM stuff, and IBM is proprietary, so no comments there.
Neither was OP's problem about setting up Python on an old version of the Mac OS.
In any case, on old RHEL, you can just do something like this: https://www.2daygeek.com/install-python-3-on-centos-6/
Installing Python on Linux is much easier than you're trying to say.
For the the love of apples.
The process in the supplied link, which is the same as the process you weirdly called IBM proprietary even though it has nothing to do with IBM, is essentially the same as the process on MacOS. Realize that the OS python is not what you want, enable a non-default package repository with a few shell commands, install Python.
Prejudice about anything to do with Apple aside, glad to see we agree that RHEL and MacOS have similar install processes.
I use Anaconda for package management on Ubuntu, which helps a lot, but it's still not my preferred programming language.
To eliminate the need for braces and semicolons, of course. It's part of what makes python so easy to read and write. Is it really such a problem?
I honestly can't imagine disliking python. It's my goto for anything where performance doesn't matter due to its unmatched ergonomics and vast libs. Golang comes close but for some reason having to implement really common and basic things yourself is part of the language's design philosophy. Plus it's compiled so you can't just drop into the interpreter for a slick one-liner.
What's your preferred quick and dirty scripting/programming langage?
It's definitely a preference, and Python has a lot of library support that the Ruby community doesn't (yet, hopefully), but grandparent's comment about headaches setting up Python with a simple project are pretty common.
Rarely do I encounter a Python project that "just works" - If your setup isn't exactly the same as the repo owner, and often even they don't even know why their setup is the way it is, then there's often a lot of fiddling and package adjustments needed. Not a deal breaker normally, but enough to make it more painful to quickly drop in and start experimenting with.