Hacker News Reader

Comment by jsmeaton | Hacker News Reader

Tried to use graalvm (interpreter) to run a fairly large project at my $dayjob$ and ran into a few issues right away.

  - Maturin doesn't support the graal interpreter, so no Py03 packages
  - uv doesn't seem to run, as `fork` and `execve` are missing from the os package?
  - Graal seems to have a huge number of patches to popular libraries so that they'll run, most seem to be of the form that patch c files to add additional IFDEFs

I don't think Graal is going to be a viable target for large projects with a huge set of dependencies unfortunately, as the risk of not being able to upgrade to different versions or add newer dependencies is going to be too high.

It's impressive what it does seem to support though, and probably worth looking at if you have a smaller scale project.

View on HN

The number of patches is going down with time and many are trivial one liners, e.g. uvloop

https://github.com/oracle/graalpython/blob/b907353de1b72a14e...

    -        self.cython_always = False
    +        self.cython_always = True

That's the entire patch. Others are working around bugs in the C extensions themselves that a different implementation happens to expose, and can be upstreamed:

https://github.com/oracle/graalpython/blob/b907353de1b72a14e...

Still others exist for old module versions, but are now obsolete:

https://github.com/oracle/graalpython/blob/b907353de1b72a14e...

    # None of the patches are needed since 43.0, the pyo3 patches have been upstreamed

And finally, some are just general portability improvements. Fork doesn't exist on Windows. Often it can be replaced with just starting a sub-process.

So the patching situation has been getting much better over time, partly due to the GraalPy team actively getting involved with and improving the Python ecosystem as a whole.

There is basic GraalPy support in Maturin[0] and PyO3[1], the problem is often that packages require older Maturin/PyO3 versions and/or they use CPython-isms, semi-public APIs, etc., but it is getting better, for example [2].

It is fair to say that large projects with a huge set of dependencies will likely face some compatibility issues, but we're working on ironing this out. There is GraalPy support in setup-python GitHub action. GraalPy is supported in the manylinux image [3]. Hopefully soon also in cibuildwheel [4].

[0] https://github.com/PyO3/maturin/pull/1645 (merged)

[1] https://github.com/PyO3/pyo3/pull/3247 (merged)

[2] https://github.com/pydantic/jiter/pull/135 (merged)

[3] https://github.com/pypa/manylinux/pull/1520 (merged)

[4] https://github.com/pypa/cibuildwheel/pull/1538

Appreciate the further details, thanks! This is a huge undertaking, good luck, and I'll be checking back in here and there.

To be fair, this also happened when Graal was released for Java. Give it another go in 3-6 months, the Graal team will have improved interoperability massively.

It is a chicken (interpreter) and egg (dependencies) problem. You cannot fix the dependency problems without the interpreter. Neither can you release an interpreter with full dependency support.

For projects using GraalPy, I'd wager that most would vendor all their dependencies at the start of the project and upgrade along the way. I have shipped a couple products with Jython, and very little 3rd party code was used and almost none of the standard library, it was all driving Java from the same project.

So it does have to do with scale but in the opposite direction. Big long projects will want to adopt something like GraalPy because of how long the project will take.

What I was hoping to be able to do was run our existing cpython project on graal to try and benefit from whatever speedups the jvm (or, if possible, compiling to a native module) would provide, rather than build with the jvm specifically in mind from the get go.

That is a problem with Python as a language and a platform and has nothing to do with Graal. PyPy is in the same boat. If the alternative Python's banded together there would be 10.

Reply on news.ycombinator.com