Hacker News Reader

Comment by kbar13 | Hacker News Reader

http://pythonwheels.com/ solves the problem of building c extensions on installation.

Pair this with virtualenvs in separate directories (so that "rollback" is just a ssh mv and a reload for whatever supervisor process) and you get to skip the mess of building system packages.

Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems or pay for a hosted PyPI?

> Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems or pay for a hosted PyPI?

Yes. I've seen them, and they've been huge shops.

Can you point me to your recommended PyPI-in-a-box system?

I currently use this one: https://localshop.readthedocs.org/en/latest/installing.html

It works. It's django based and you can setup s3-backed storage. It also has a docker-compose script.

Do I understand this correctly? It only mirrors the packages that are requested from it? So I won't need to download 100GB+ of packages that I am not interested in?

Correct. My team has two main use cases: private packages and guaranteed access to packages we've built with. It's extremely frustrating to come into a codebase after several months or years to find it using a library that no longer seems to exist on the public internet.

We migrated off of localshop and onto devpi. Devpi is a much better product and much more actively maintained. localshop was nothing but headaches and constantly breaking.

Maybe you use more esoteric features. The only thing I've done in the last 18 months is patch a bug that prevented uploads from Python 2.7.4-2.7.10. We just run it under circus with chaussette and front it with an elb.

localshop has significantly improved in the past few months. Before I could barely consider it production-ready.

Author here: I created it to solve an issue I was running into a couple of years ago. I've only recently started using it again myself. I think the development version (not on pypi) is in much better shape with things like multiple repositories and better user management (teams).

I have had good success with http://doc.devpi.net/latest/

That looks good. Might work exceptionally well for my needs. I want to have a PyPI mirror in an environment that doesn't have public internet access.

Running devpi in another environment and syncing the resulting repository should allow me to achieve what I want.

As a first step, what about trying pip's `-f` option, combined with dumping your wheels in a dumb folder served by apache/nginx:

    pip --help
    [...]
    Install Options:
      [...]
      -f, --find-links <url>      If a url or path to an html file, then parse for links to archives. If a local path or file:// url that's a directory,then look for archives in
                                  the directory listing.

EDIT: in the context of deploying some app at work, what's the interest of a full-blown hosted cheeseshop? Users of these solutions, what value does it add to a simple `pip install -f INTERNAL_PKG_URL pkg_a==1.2.3 pkg_b==0.1.2` ? Which features do you frequently use?

> you get to skip the mess of building system packages.

Only in cases where you don't have wheels depending on external libraries. If you do, you should still package with the right dependency constraints. Otherwise you can install a wheel which does not work (because of missing .so)

This simply isn't true. You can package .so or .dll (or anything else) with your wheel. An example of doing it with a dll or so is here: http://stackoverflow.com/questions/24071491/how-can-i-make-a...

What you said does not disagree with me. You can have wheels depending on external dynamic libs. Or you can package them together. Both ways have good and bad sides, but if you leave them external, it's very useful to have the right dependencies.

Reply on news.ycombinator.com