Hacker News Reader

Comment by mattikus | Hacker News Reader

How are you hosting the wheels internally? Are you still using an internal pypi instance but with wheels instead of sdists?

I've been looking at doing something similar in our environment but there's so many options I haven't figured out what the best and most straightforward way might be.

View on HN

Shameless plug: I had the exact same question at my last gig, and wrote up a quick open-source tool to build wheels and dump them to an S3 bucket. [0]

Usage is as simple as

    mkwheelhouse mybucket.mycorp.co scipy numpy

which will automatically build and upload those wheels for your current architecture and dump them into mybucket.mycorp.co. It builds a pip-compatible index, too, so you can just tell pip to search your wheelhouse first before falling back to PyPI:

    pip install --find-links mybucket.mycorp.co scipy numpy

If you need to build for several OSes, you can run mkwheelhouse against the same bucket from each OS.

The downside of this approach is you can't host private packages, because you need to enable public website hosting. (Although, VPC endpoints might have changed this!) But the simplicity of this approach plus the massive speedup of not needing to constantly recompile scipy was totally worth it.

[0]: https://github.com/whoopinc/mkwheelhouse

More plugging unashamedly: I wrote a guide for what is probably the quickest and simplest way to get started with a private pypi instance - just Apache with directory autoindex, and wheels that are manually-uploaded to the server:

https://gist.github.com/Jaza/fcea493dd0ba6ebf09d3

Your easiest bet is probably devpi, but you can download the cheeseshop code that powers pypi as well.

I'm using cheeseshop, but some people swear by warehouse, which is supposedly a legacy-free version for running pypi eventually.

If you don't care about the search api, you can also just enable an directory listing index page and use any web server. Pip will do the right thing when given the right incantation of magical arguments and you make a prayer to the pip gods.

Another shameless plug: At my previous job I wrote a basic PyPI server running on App Engine with packages stored in Cloud Storage.

You get the benefits and drawbacks of Google Cloud Platform.

https://github.com/vendasta/cloudpypi/tree/master

Use PyPICloud or DevPI to host your built binaries. Wheels have another advantage that you don't install any dev tools in your production environments (no need for compiler tools). The only annoying thing is that there isn't any way to distinguish between different Linux distributions, so hopefully all your servers are standardized.

Reply on news.ycombinator.com