It would be nice to see the ML community move on to something that's actually easily reproducible and buildable without "oh install this version of conda", "run pip install for this package", "edit this line in this python script".
Is it possible that creativity and innovation actually require a level of chaos to succeed? Or alternatively, that chaos is an inevitable byproduct of creativity and innovation, and any ecosystem where these are heavily frowned on deters the type of people who actually drive the cutting edge forward?
Just putting it out there as food for thought. It does translate back to quite often, that it acutally does make a lot of sense to do your prototyping and experimenting in one ecosystem but then leave that behind to deploy your production workloads where possible.
Both are easy and reliable with a few months of experience. Both are terrible if you rarely ever use them.
The forks are weird mashups of bits of repos, and running on a M1 GPU is something that barely works itself.
Give it maybe 3 months and it will be much smoother.
I have a suspicion that that something written in Julia, Go, Rust, or possibly even C wouldn't have nearly this many issues. I'm not talking about debugging the actual functionality of the software, but rather the environment and tooling surrounding the language and software built with it.
This project in particular should be an easy case because you know the hardware you'll be running on ahead of time.
I'm ranting a bit, but I've tried so many tools based on python and almost _none_ of them built/installed/ran correctly on the happy path laid out in each project's readme.
Anyway, sorry, rant over.
I do think that experience helps here. I have a recipe for installing Python that works on most python projects most of the time.
git clone <project>
python3 -m venv ./venv
source ./venv/bin/activate
pip install -r requirements.txt
deactiviate # need to do this to include the correct command line tools in path (eg Jupyter)
source ./venv/bin/activate
On a Linux or Intel Mac system this works with pretty much every reasonable Python project.
On M1 Macs the situation isn't great at the moment, though.
People also tend to forget that these ML packages are ridiculously complicated, and have a lot of dependencies not just on other libraries but on particularities of your system.
That, and ML researchers can't also be expected to be good at everything. They are busy doing ML research and waiting to be given a recipe to follow.
Meanwhile I can put together a Python package in my sleep that works perfectly on pretty much any system, but I don't know a damn thing about Autotools and would probably make a total mess if I tried use it. Or CMake. Or whatever Java uses.
This "look Python bad!!" stuff has some merit (if only historical), but it mostly amounts to FUD and does a big disservice to the people who have worked hard over the past few years to get everything fixed up.
Plus at least one of the issues I see here is because people disregarded the instructions and used Python 3.9 even though it says to use 3.10 because the code assumes it's running under 3.10.
Firstly, non-experiences Python users (the kind who need recipes to follow) will also blindly follow installation instructions on websites. These all say "pip install xxx" and users will inevitably forget they need to change the path to make that work for their environment. By getting into the habit of activating the environment this problem goes away.
Secondly, you do need to activate the venv so that command line tools in ./venv/bin are used (instead of whatever is in your default path).
This includes both the correct version of python (if you set it when you setup the virtual environment) and importantly jupyter. From personal experience there's a whole set of pain when you don't realize your jupyter isn't the one inside the virtual environment and so is using other libraries.
Because many shells cache the location of executables, if you don't deactivate and reactivate your virtual environment after installing Jupyter into it then it may (in some circumstances) use a globally installed version.
…it’s still awesome that people put in the effort to do these things at all, but the tools often have a tendency to make me feel like an archaeologist trying to piece what ancient artifacts are missing and how they were supposed to all fit together.
In a university, ten steps to replicate the PI's personal environment is perfectly fine. How would releasing a single binary make their life any easier? Why would they bother?
Worked out fine in the end, though. Highly recommended if you're on an Nvidia rig: https://github.com/AbdBarho/stable-diffusion-webui-docker
Python is pretty much innocent here.
Could not possibly disagree more. https://xkcd.com/1987/
The problem with dependencies is that for very bad reasons people don’t ship them.
Someone needs to package SD with a full copy of the Python runtime and every dependency. This should be the default method of distribution.
#ShipYourDamnDependencies
That XKCD is more about messing up your system by not knowing what you're doing but randomly following shitty tutorials which suggest stuff that collide with each other... Python's only fault in this is that it's a simple language and thus attracts people who aren't software engineers (students and math majors) and thus mostly don't know or care how to keep your system clean but love writing tutorials.
It's pretty easy to keep your pythons clean, don't use conda, never run pip with sudo, never run pip with --user..., never run pip outside a virtualenv (a good safety measure for that is to have pip point to nothing in your user shell, you can access system python with python3/pip3 if needed)... To check your python is clean, create a new environment and run pip freeze, it should output nothing.
pyenv is a non-destructive system for managing multiple pythons and virtualenvs (all pythons and envs get installed into ~/.pyenv), pip is a good system for distributing dependencies (when library authors don't skip out on providing binary wheels, and software authors use pip freeze to generate requirements.txt files).
You know what’s radically simpler? Shipping your damn dependencies so all any user needs to do is double-click the launcher and it will always work no matter what weird and bad things your user as done. The fact that “keep your system clean” is a thing anyone thinks about ever is a catastrophic failure.
you see it here on hacker news, because it's open source and thus hackers can and do already play with it..
as it's an interesting piece of software, I'm pretty sure someone will eventually make an easy to use GUI app based on it that ships all dependencies, but it's not yet at that stage,
this article is literally about the first easy (for a software engineer who knows python) way to run it on M1 macs, these people literally just discovered which versions of their dependencies work and share their work to invite other people to help them build something on top of it....