Hacker News Reader

Comment by navels | Hacker News Reader

And chaos ensued. Unable to install aws-cli via pip: https://github.com/aws/aws-cli/issues/8036

Mostly because of PyYAML: https://github.com/yaml/pyyaml/issues/724

I was pretty annoyed to discover that it had a separate build-time dependency on cython that you can't control from your requirements.txt.

That's unfortunately a pip limitation. You can either disable build isolation so that the build requirements are using your current environment or you can use the PIP_CONSTRAINTS env var which will be passed through to the isolated build invocation allowing you to constrain the build requirements. In this case PyYAML did release a 6.0.1 version which contains an upper bound constraint on Cython.

That's what I ended up doing just for the pyyaml installation.

Wonder if Cython didn't think of checking with their biggest "customers" for compatibility

PyYAML knew about the breakage since january 2022[0], and nothing really happened. After a year and a half with lots of alphas and betas, I don't think there is much cython could do, short of fixing PyYAML themselves.

[0]: https://github.com/yaml/pyyaml/issues/601

In Spack [1] we can express all these constraints for the dependency solver, and we also try to always re-cythonize sources [2]. The latter is because bundled cythonized files are sometimes forward incompatible with Python, so it's better to just regenerate those with an up to date cython.

[1] https://github.com/spack/spack/ [2] https://github.com/spack/spack/pull/35995

Oh great, yet _another_ Python package manager.

Spack is a lot more than just python package manager :)

It’s not a python package manager. It is a generalized package manager written in python.

Neat! I had no idea that Spack can be used for cythonize dependencies. I use it to manage my local compiler/CUDA/Trilinos stack but never considered it for non C++ things.

That's what PyYAML does as well. It uses PEP 518 [1] to specify the dependencies for the build which PyYAML has included Cython [2]. It's just that for previous releases there was no upper bound here so pip and other tools just selected the latest version which was incompatible. In the past PyYAML included the cythonised .c files in the sdist but as of 5.4.0 they went the PEP 518 route and ensures the client will cythonise them if installing from the sdist.

[1] - https://peps.python.org/pep-0518/ [2] - https://github.com/yaml/pyyaml/blob/release/6.0/pyproject.to...

My impression is that the Python ecosystem rarely specifies upperbounds on dependencies even if they follow semver.

In the Julia ecosystem it's the default, and you basically have to release a new patch version for updated compat bounds with your dependencies. This is much more stable. It works because the process is mostly automated: dependency releases new version => a bot opens a PR on your repo updating the compat bound, you just merge it.

I like how the log in the issue also has another warning for breaking changes in it

    The license_file parameter is deprecated, use license_files instead.
  
    By 2023-Oct-30, you need to update your project and remove deprecated calls
    or your builds will no longer be supported.
  
    See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.

The level of churn and breakage in these basal python packaging packages is honestly insane.

I am reluctant to refer you to aws-cli V2 because it is way more involved to install. No idea why aws moved away from their original distribution method... However, the V2 is going to be the way to go moving forwards :/

>No idea why aws moved away from their original distribution method

Didn't they move away exactly because of issues like this? As I understand it, aws-cli v2 is still written in Python but ships everything it needs self-contained so it can avoid Python dependency hell[1]

1: https://www.youtube.com/watch?v=U5y7JI_mHk8#t=3m40

Indeed!

Funnily enough, I didn’t have issues with Python’s dependency hell before working for AWS, not because we have more dependencies, but because… reasons.

Is it special files that begin with a capital letter, version sets, version filter instances, and massive symlink farms?

If you already know that much then you don’t need a response ;)

    cat -n shell.nix
    { pkgs ? import <nixpkgs> {}
    }:
    pkgs.mkShell {
        name  = "dev-env";
        buildInputs = with pkgs; [
            awscli2
        ];
   }

And then you can have it available when entering shell with nix-shell.

And this is why you pin your dependencies + never install the latest version of anything.

The lesson here is different: package developers must put upperbounds on major versions of dependencies that follow semver.

Should this be the take away? It looks to me that people are building pipelines without taking into account that they change roles every time they have to build a package, from a package consumer to a package packager. Even if the package developers set an upper bound on the build dependency it is the packagers responsibility to provide a deterministic build environment.

Reply on news.ycombinator.com