I've had the exact same issues with the JS ecosystem (ran into a problem where npm wouldn't work but yarn did, still haven't figured out why).
Both are easy and reliable with a few months of experience. Both are terrible if you rarely ever use them.
That's probably fair. I've found pyenv/venv/poetry/pipenv/pip/conda/etc more frustrating than ESM/CJS/yarn/npm/PNP/etc, but then I just do a lot less Python stuff than JS these days.
Js libs don't need to care for things like system packages and drivers as much as Python ML does.
Ever tried running JS libs with C bindings? You're going to run into the exact same problems as you have running python code with C bindings.
CommonJS vs ES6 module loading is already a nightmare.