Hacker News Reader

Comment by totalperspectiv | Hacker News Reader

I think that at this point all the ceremony around setting up and deploying a python project outweighs it's 'easy to read and use' aspects. Unless there is a library you can't live without or rewrite, it seems like a language with better tooling and real benefits of a type system is a better choice.

If you're mainstream: Go or Java. If you're edgy: Nim, Scala, or Crystal

All of those have much more sane type, build, and packaging systems.

@perl-people, was this a solved problem when Perl was big? Or is python walking the same roads?

View on HN

It most definitely was not a solved problem when Perl was big, as far as I know Ruby is the language that finally solved it with bundler, which was released after Rails was, so that's like 2007 or something.

The recipe for the 'solved' deployment:

  - a compiler that ships with a build tool (so building is homogenous in the community)
  - a centralized or at least uniformly accessible package repository (so dependency acquisition is homogenous in the community)
  - a common file format for describing dependencies (so dependency resolution is homogenous in the community)
  - a common file format for locking dependency versions (so deployment can be done reliably without vendoring dependencies)
  - optional but very nice: a tool for managing compiler versions so it's easy to switch/upgrade projects

Any programming language that has all of these boxes ticked is a modern programming language in my book. As far as I know Ruby is the first that ticked all of them, but other ones I've used that have this: Node.js, Go, Rust, Haskell, Python (though it's a bit messy). I'm pretty sure C# checks them as well nowadays, but I haven't used it in over 5 years so I'm not sure. Same for Java.

Somebody needs to tell the Racket people about this. Dependency management in Racket is still C-style no-version-pinning-anything-goes. The third party dependency managers (see below) are all primitive and do not support multiple versions of the same libraries which is a standard feature in Go/Node.js/Rust/Java class loaders

https://github.com/Bogdanp/racksnaps

all should be handled by pip. all efforts should be centralized into pip, maybe using a plugin system, whatever.

I think a language should have its own official tooling that can achieve those without installing any external shit

> Unless there is a library you can't live without or rewrite

There are quite a few of these. I couldn't do any of my ML work in a language other than Python or C++.

Calling what Go has a "type system" is a stretch IMO.

I’ve been toying with Nim lately to wrap C++ ML libraries, which gives a nice Pythonic syntax but compiles to a binary wrapping the ML lib. Seems to work well for Torch. There's a nice wrapper library nimtorch for Nim [1]. It's a bit out of date bit would be easy to update, probably. Well easier than bundling pytorch on an embedded device. Even manually wrapping the needed C++ libraries isn't that hard in Nim, IMHO.

Overall looking at Python deployment story after using Elixir for the past couple of years makes me cringe a bit. Rather I have no idea how to do it. Deterministic versions, lock files, and container/tarball (or binary) support seems a given in 2020.

1: https://medium.com/@giovanni_94706/introducing-nimtorch-b8b0...

> Well easier than bundling pytorch on an embedded device.

I don't follow - wouldn't nimtorch also require bundling pytorch on an embedded device?

Yes and no, Pytorch (like Tensorflow for Python) is just a wrapper around a core C++ library. So you can compile a binary from C++ linking to the pytorch libs without including Python. Technically it's pytorch, but without Python and Python dependency management which is way simpler. Nimtorch let's you wrap the C++ api of pytorch with nice Pythonic looking code, a GC, but using only C++ and possibly statically linked. Win win.

See: https://pytorch.org/tutorials/advanced/cpp_frontend.html

Yes I understand pytorch quite well - I was merely confused because I consider the C++ library to still be pytorch.

I don't see the advantage of this setup over just loading a torchscript model in C++ or any other static language. A full set of bindings seems unnecessary unless you need to train in nim.

True, my terminology was a bit outdated as I keep thinking of pytorch as just the Python wrapper on libtorch. That's not really true.

I prefer to avoid Python nowadays due to the pain of dependency management, a nice as this article is don't care to learn about Poetry. So training in pure C++ (or better Nim) is my preferred setup. Keeping a similar build setup for both training and deployment saves a lot of headaches which is why the nimtorch interface is handy even if it's not as full featured/up to date. Now I'm only deploying a very simple NN without much need for experimentation.

As said in upper comment, use zipapps. Don't ship your entire env in prod.

Deployment then just become:

- build zipapp - upload zipapp - ensure python interpretter - run zipapp

Which is just one more step than go.

Zipapp doesn't support c extensions though? And that is a large chunk of libraries that I use.

It does, if your extensions are provided as wheels. In which case the resulting pyz file will be runnable on any machine using the OS those wheels are compiled for.

Let's try an example with numpy:

    $ py -m venv foo
    $ foo\Scripts\activate
    $ pip install numpy
    $ code hello_numpy.py
    # import numpy as np
    # def main():
    #     print(np.arange(15).reshape(3, 5))
    $ python hello_numpy.py
    [[ 0  1  2  3  4]
    [ 5  6  7  8  9]
    [10 11 12 13 14]]

Now with shiv:

    $ copy hello_numpy.py foo\Lib\site-packages
    $ shiv -e hello_numpy.main --site-packages test\Lib\site-packages\ -o hello_numpy.pyz
    $ python hello_numpy.pyz
    [[ 0  1  2  3  4]
    [ 5  6  7  8  9]
    [10 11 12 13 14]]

So it works fine, but remember:

- it will only run on the system this particular numpy wheel has been designed to run on. In my case cp38-win_amd64.

- it will comes bundle with numpy. Numpy is quite big, which means your hello world pyz will be 14.1 Mo.

- it needs to unzip, so the first run will be slow

Yeah, Shivs work ok. I've run into some stupid issues with the self-extracting directory being have too long path names on Windows and such, so it's not perfect.

if you're long-term focused and investing in a career as a programmer: Rust

Reply on news.ycombinator.com