Do you handle deployment of this Python application? For me, that's where the pain points arise. I love writing Python, but deploying it does not spark joy at all, at all.
- `curl -L https://app.example.com/install | sh` that downloads installer and runs for instance: apt/yum install <your-package>
- in CI environment on a VM: `git checkout` & `pipenv install --deploy`
- `pipx install glances` on a home computer
- just `pip install` e.g., in a docker container [possibly in a virtualenv]. For pure Python packages, it can work even in Pythonista for iOS (iphone/ipad)
- just copy a python module/archive (PyInstaller and the like)
- give a link to a web app (deployed somewhere via e.g., git push)
- for education: there are python in the browser options e.g., brython, repl.it, trinket.io, pythontutor.com
- just write a snippet in my favourite editor for literate devops tasks/research (jupyter-emacs + tramp + Org Babel) or give a link to a Jupyter notebook
- a useful work can be done even in a REPL (e.g., Python as a powerful calculator)
I dare you. Do mention any tool/any language that handles all the above use cases without sacrificing the requirements for each use-case.
No. But if I talked about how I used 9 different word-processing programs, you'd see that as a problem, or at least an indictment of those programs. Deployment isn't that complicated.
> I dare you. Do mention any tool/any language that handles all the above use cases without sacrificing the requirements for each use-case.
I use Maven/Scala and as far as I can see it covers all of them other than "give a link to a web app" which isn't actually deploying at all (and I'd still have used maven to deploy the webapp wherever I was deploying it).
I don't think there's any legitimate case for curl|sh, and I don't think there's any real reason for separate pip/pipenv/pipx (did you make that one up? Have I fallen for an elaborate troll?) - rather pipenv exists to work around only being able to install one version of a library at a time. Nothing's gained by having "just copy a module/archive" be different from what the tool does. Running in browser, notebook, or REPL can and should still use the same dependency management tooling as anything else.
If I want to deploy my code, I use maven. You can use curl (since maven repositories use standard HTTP(S)) or copy files around by hand, if you have a use case where you need to, but I can't think what that would be. If you want to bundle up your app as a single file, you can configure things to do that when publishing, but the dependency resolution, repository infrastructure, and deployment still look the same. Even if you want to build a platform-level executable, it's the same story, all the tooling just works the same. If I want a REPL or worksheet, I can start one from maven (and use the same dependency management etc. as always), or my IDE (where it's still hooked up to my maven configuration). If I want to use a Zeppelin notebook then there's maven integration there too.
Ever wonder why you don't hear endlessly about different ways of doing dependency management in non-Python ecosystems? Because we have tools that actually work, and get on with actually writing programs. It baffles me that Python keeps making new tools and keeps repeating the same mistakes over and over: non-reproducible dependency resolution, excessively tight integration between the language and the build tools, and tools and infrastructure that can't be reused locally.
- system packages (deb/rpm/etc)
- binary wheels (manylinux)
- building from source
To take your examples in order:
1) system packages: almost always out of date for my needs
2) Binary wheels: I actually haven't investigated this much, maybe it will work (and if it does, I'll buy you a drink if we ever meet in person).
3) Building from source: this kinda proves my point about Python having poor dependency management tools if this is a serious response. In general, this would be much further down the rabbit hole than I want to go.
That said, I do run into trouble when I have a dependency that requires compilation on Windows (i.e. like the popular turbodbc) because say, a wheel isn't available for a particular Python version. Any time a compilation is needed, it's a headache. Windows machines don't come with compilers, so one has to download and install a multigigabyte Visual Studio Build Essentials package just to compile. Sometimes the compilation fails for various reasons.
Require gcc compilation is headache for installing dependencies inside Docker containers too -- you have to install gcc in order to install Python dependencies and then remove gcc after.
I think requiring local compilation (instead of just delivering the binary) is a UNIX-mindset that is holding back many packaging solutions. I think a lot of pain would be alleviated if we could somehow mandate centralized wheel creation for all Python versions, otherwise the package manager marks a package as broken or unavailable and defaults to the last available wheel.
Also if only we applied some standards like R's CRAN repo does -- ie. if it doesn't pass error checks or doesn't build on certain architectures (institute a centralized CI/CD build pipeline in the package repo), it doesn't get published -- the Python packaging experience would be much improved.
For those who don't realise, when there's a new version of R, anything that doesn't build without errors/warnings is removed from the archive.
This is really annoying if you want something to keep running, but it prevents the kind of dependency rot common to Python (recently I found a dependency that was four years out of date).
Once a pip install needs to start compiling C, things do go way south very quickly. At that point you can install the union of all common C development tools, kernel headers and prepare for hours of header hunting.
I've done that too much to like python anymore.
Additionally, they are part of a larger application, which is mostly managed by pip, which means that I need both pip and conda which is where things get really, really hairy.
I actually blame Google and FB here, as neither of them use standard python dependency management tools, and many of their frameworks bring in the world, thus increasing the risk of breakage.
And putting them into a common shared directory.
Try doing that without writing convoluted code in your setup.py.
Deployment for development is just pyenv and virtualenv.