Pyodide: Python for the Browser | Hacker News Reader

Pyodide: Python for the Browser

lwn.net

173 pointslukastyrychtr81 comments

If you want to play with Pyodide in a web notebook you can try Starboard [1][2].

A sibling comment introduces JupyterLite and Brython, which are Jupyter-but-in-the-browser, whereas with Starboard I'm trying to create what Jupyter would have been if it were designed for the browser first.

As it's all static and in-browser, you can embed a notebook (or multiple) in a blog post, for instance to power interactive examples. The bundle size is a lot smaller than JupyerLite for the initial load - it's more geared towards fitting into existing websites than being a complete IDE like JupyterLab.

[1] https://github.com/gzuidhof/starboard-notebook

[2] https://starboard.gg

I’m concerned with the performance. It’s up to 16x slower than native Python, which we already know is 100x(?) slower than native code.

So roughly 1600x more clock cycles are consumed than is necessary.

I’m already jaded with the extreme bloat of JS frameworks as it is (how many times does my computer need to ramp up fans because of your React blog?). This could potentially be an order of magnitude worse.

I think the main usecase for Python in the browser using Pyodide is to enable scientific computing and visualization in the browser, and also as a way to ship small Python applications to those that don't have Python installed (e.g. drag and drop a CSV file which gets processed with pandas). Lastly it's useful to power interactive coding environments without needing to spawn a pod per user (and fighting off people using them to mine bitcoin).

Building a website or SPA entirely backed by Python seems like the wrong use of this technology. The same goes for crunching large amounts of data.

The quality of an answer is often orders of magnitude more important than the speed at which it's generated. Speed is important, but it is always secondary.

You wouldn't pull this in for a snappy, reactive UI driven by Python. You pull this in so that you can re-use a server-side module to perform some activity that isn't critical and so not worth re-writing in JavaScript. In another 5 years (assuming Python is still of interest and that this solution worked), it will have been refined and optimized where maybe you do reach for it by default.

Genuine question: What are the quality improvements offered by Python? Is it mostly access to the scientific community's Numpy/Scipy/Matplotlib? The language's semantics?

Most ML papers are implemented in Python, so you need it in order to be able to play with the latest and greatest toys, or you can wait for someone to port them to your favorite language, if ever.

And I would say that this is because of the language's syntax & semantics. Pythonic code looks like pseudocode, aka business logic.

There is very little boilerplate, and libraries are designed to hide what little there is.

So this means that mathematicians who hate verbosity can read and write their algorithms with little cognitive overhead.

Then you start to get the network effect where libraries are implemented or wrapped in Python, and people use those and start adding abstractions etc. The two main deep learning libraries (TensorFlow and PyTorch) are there, and those have huge gravity wells.

16x? FTA:

> performance since the 2019 announcement has improved greatly: "Performance ranges between near native to up to 3 to 5 times slower, depending on the benchmark."

Concrete Python applications wouldn't be 100x faster if re-coded in some low level language.

In most places where performance matter Python packages would be using either C extension, Cython or numba to get near native performance. Pyodide is able to build those packages (except for numba). So overall it's currently 3 to 5 slower than native Python (which uses C extensions). See detailed benchmarks in https://hacks.mozilla.org/2021/04/pyodide-spin-out-and-0-17-...

Try it in JupyterLab using JupyterLite- https://jupyterlite.readthedocs.io/en/latest/_static/lab/ind...

(choose the pyolite option for Pyodide)

This project is actually quite impressive, I believe they've even gotten some pip install paths working??

Yes, installing pure Python packages that have wheels from PyPi works. Assuming they don't use functionality that's not supported by the WebAssembly VM such as multiprocessing, threading (for now), and sockets.

Also, Basthon: https://notebook.basthon.fr/

I'm having trouble getting anything plotted using this -- either via plotly or matplotlib (they don't appear to have seaborn available).

Have you (any one else) succeeded?

Not being able to display plots seems a major limitation.

This works for me in Chrome 90 on Windows

  import matplotlib.pyplot as plt
  plt.plot([1,2,3,4,5], [1,4,3,2,5])
  plt.show()

Thanks -- it looks like the issue occurs when the import is in a different cell; or when the plotting cell is being re-run.

Try splitting cells on the first line and re-running the second.

I quickly get Recursion-Depth-Max errors.

I just split the cells and it worked fine. Weird.

The next version of plotly should be more compatible with Pyodide :)

Wow amazing, thanks for sharing.

Getting started: https://pyodide.org/en/stable/usage/quickstart.html

PyOdide on GitHub: https://github.com/pyodide/pyodide

I use this for the isort in browser demo: https://pycqa.github.io/isort/docs/quick_start/0.-try/ it was really awesome to be able to directly use a Python package without any wrangling or modification as is usually required with other Python on the browser solutions.

Has anyone gotten this to run in a wasm runtime on the server? At some point I saw that wasmer had a fork adding server support, but it was out of date. Main challenge seems to be assumptions about running in the browser?

My motivation is that I’d like to run sandboxed Python scripts on the server, in a wasm runtime embedded in C.

Why not just use an OCI container and runtime (docker, containerd, crun, etc) or even systemd-nspawn?

Because it’d be cool if it didn’t require OS level sandboxing.

AFAIU sandboxed embedded Python has been a bit of a Holy Grail for a long time. So if wasm gives an easy answer for it, that would be sweet!

Work on Node.js support is in progress in Pyodide. And as you mentioned there is also the Wasmer build that could be updated.

How does Node play into this? Can I embed a v8 engine in my C program and use pyodide, or do I need to bring Node along too? Does the compatibility layer depend on any Node-specific concepts like the event loop or garbage collector? (I think maybe V8 does its own GC? This just shows how thin my experience is with this topic.)

Neat: "The 0.17 release has a number of interesting features. Support for Python asyncio has been added, so that Python coroutines can run in the browser event loop; a JavaScript Promise can be awaited in Python and vice versa with Python awaitables. Error handling has also been upgraded so that exceptions generated by Python can be caught in JavaScript; that can be done in the other direction, as well."

I tried brython once and wrotesome video/image gallery with it, I was quite happy, but it ended up being quite slow.

I'm super excited about pyodide, although:

* I can't find a discord or IRC channel to ask questions, I guess the project is still young. Last time I've seen this project, it looked like a scientific software suite, which came with a lot of stuff I don't need.

* I wish there was more examples or samples, especially with events and the DOM.

* I've glanced at the doc and howtos, and it seems it requires a server, which I don't understand. Brython seems much more straightforward.

* I've read that memory leaks can happen in multiple places

* This example seems like misleading advertisement, in my view, at least for now:

    from js import document
    x = document.getElementById("myElement")

(Or I might need to read up more doc to understand how to get to that point)

no server is required. check https://pyodide.org/en/stable/usage/quickstart.html

A chat channel is mentioned in the readme: https://gitter.im/pyodide/community

Yes, we are working on adding more examples.

Well you need to be able to serve static files. So if you are building locally you need to start a webserver, otherwise you can use the JsDelivr CDN.

Memory leaks can happen if you translate objects between Python and JS. So of it is unavoidable because it's difficult to know when to destroy Python objects from JS vice versa, but lots of work on it has been done in the 0.17 release with more to come in 0.18

Why do you find the example misleading?

> Well you need to be able to serve static files. So if you are building locally you need to start a webserver, otherwise you can use the JsDelivr CDN.

I don't understand, can't I just use offline, static files ?

I would still expect to use my own webserver, unrelated to pyodine, like flask or whatever.

> Yes, we are working on adding more examples.

Glad to hear this :) !

> Why do you find the example misleading?

I haven't really seen that example in the doc or elsewhere, and I don't understand the steps required to make such example work offline or without a webserver.

EDIT:

Another cool thing of brython is that could "inline" a python script inside a html file, such as:

    <script type="script/python">

    <script src="thing.py"></script>

If you just open an HTML file in your browser via its local path (e.g. /home/user/...) the browser will prevent you from loading other files by JS due to CORS. So we need a web-server to set the appropriate CORS headers. Any web-server would do.

For inline code, yes it would be doable someone would need to continue https://github.com/pyodide/pyodide/pull/692

For examples, I'll respond in the Pyodide issue where you commented.

Any other way to agree with CORS without a webserver? What about embedding some js code, or moving the WASM files locally?

Thanks for the answers.

Cheers and good luck for this project, I can't wait for it to progress!

You can work around it by embedding data into HTML files but it's really not recommended https://github.com/pyodide/pyodide/pull/606

They have different goals, nobody expect people to write a website ui with pyodine.

(although, python is my fav lang, and I wouldnt use brython either, 700ko to pay upfront before writing any code is too much)

Is this substantially better than Pyjs?

https://en.wikipedia.org/wiki/Pyjs

> Running the Python interpreter inside a Javascript virtual machine adds a performance penalty, but that penalty turns out to be surprisingly small — in our benchmarks, around 1x-12x slower than native on Firefox and 1x-16x slower on Chrome. Experience shows that this is very usable for interactive exploration.

I seriously question these benchmarks.

That data is from 2 years ago. Lower down in the post it has an update for the present:

> performance since the 2019 announcement has improved greatly: "Performance ranges between near native to up to 3 to 5 times slower, depending on the benchmark."

This project is amazing, but I would love to see the same effort with Julia so I can show differential equations in browser.

Why not transpile python to JS instead of running an interpreter?

Because many want JS to die in a giant hellfire and then shat on the ashes. So alternatives are being made that don't rely on it.

Is dart your preferred alternative?

The parent is talking about transpiling python to JS. I don't see why people who hate JS would like python. The language itself is not any better IMO (excluding ecosystem here).

edit: following some comments I want to add that the situation is symmetric IMO. I don't see the point in transpiling any of the two to the other.

It's just what you are used to. I dislike Python, I hate PHP, but I really like JS. I've seen people like PHP and hate Python too.

You see more JS hate because it's a more common scenario that people happy with PHP/Python/Whatever would be exposed to JS later, than the other way around.

You see a bit more Python hate lately because ML is all the rage and Python is to ML what JS is to the web.

All mentioned languages having some weird backwards compatibility quirks doesn't help either.

I find it good that people are trying to port their favorite languages, although personally I'd stay on the happy path for the target platform. Especially the browser, as it already has weird APIs, JS or not.

I agree, I was fluent in JS and other languages before learning python and hated it for a while. Eventually, you get used to the bad parts of any language so they become mostly transparent (and every language has bad parts).

Specifically, now that I'm very much OK with both python and JS I find transpiling one to the other kind of pointless in terms of any language property (can make sense in terms of ecosystem).

Aggressive type coercions in js frequently comes up

  16 == [16]

true in js false in python

    >>> a = 5
    >>> b = 5
    >>> a is b
    True
    
    >>> a = -4
    >>> b = -4
    >>> a is b  
    True
    
    >>> a = 300
    >>> b = 300
    >>> a is b
    False

Every language has these, JS a bit more (because mostly impossible to fix thanks to backward compatibility). I don't understand why you would do that anyway.

Most linters would display errors even if you did (in both languages).

Great. Thank you. I'll remove the "is" operator from the supported subset and perhaps rewrite the commonly used "a is None" to idiomatic code in the target language.

You should always use 'is' with None. == Is operator that can be overloaded and 'is' is object identity comparison. It make sense when you understand object system. You will be terribly messed up when use some numpy array == None.

a==b would be true. The is operator is not the same as the equality operator

The is operator checks to see if both point to the same area in memory. Small numbers are preallocated in python so they point to the same memory.

>>>a=300

>>>b=a

>>>a is b

True

Of course there is some internal logic.

The point was that both languages have pitfalls.

In python it completely makes sense because it's an object oriented language and threat everything as object. In fact it's very consistent about that. Even classes are object and instance of metaclass. Difference between '==' and 'is' is came from that. This is not the case with Javascript. It's sad many peoples who are dealing with python don't understand that.

"-In python- it completely makes"

It is a pitfall. It's not about what python developers think about it. It is about what people from other languages experience when they learn the language.

(Though it's not my main point, I want to add as (also) a python developer, that I think this is a terrible design choice, which could easily be avoided. In this case python doesn't "threat everything as object" it treats small int literals differently than bigger int literals)

Small integeres are object in python. They are cached in some implemention. If you only care about shallow syntax, yes it's confusing. But if you care more about semantics, it's completely consistent and doesn't make especial case.

Many programmers do care about the "shallow" syntax.

Regardless, if, as you said, the operational semantics of the language is inconsistent among implementations and depends on internal caching, it only makes it worse.

jsc >> 16 === [16]

false

I really wish it didn't though. I started working with js ~5 years ago and haven't used == a single time, there are eslint rules that error on it[0] and you can reject push to repo if you have them, which many companies do. It's not a problem.

[0] https://eslint.org/docs/rules/eqeqeq

Python is loads better than JS. I have experience with both.

That's https://github.com/QQuick/Transcrypt

I'm looking into a python -> dart -> js approach. Emphasis on static type checking and small feature set that's easy to understand.

https://github.com/adsharma/py2many/

Made a new release earlier today:

https://adsharma.github.io/py2many0.2.1/

Python's object semantics are extremely different from JS. There might be _some_ ways of emulating them with JS thanks to Proxy objects nowadays, but you're looking at a pretty big uphill battle I think.

python has a lot of features. But people use a small subset of it. So instead of supporting all of python, why not stick to the small subset that people actually want to use to solve a problem?

Different Python programs use different subsets, so as you work towards supporting more programs the union of the subsets becomes essentially the full set, I would assume.

The distribution of popularity of the features is not uniform. My intention is to carefully look at the intersection of relatively less used features and features that make python slow/inefficient and exclude them from the supported subset.

are you sure? can you give an example?

personally, i think pythons object model is extremely similar to js -- down to python essentially having prototypal inheritance

I don't think you have that quite right. Objects don't inherit from other objects in Python — you can't do `class Foo(object())`, for example. Python has class-based inheritance.

classes are objects, and inheritance is just the __mro__ on a class -- ie., its prototypal chain

if you want to inherit from an object, just construct a class object from it using type().

Eg., at least, for any given object `me`,

    type('ChildOfMe', (type(me),), vars(me))

But if you wanted a more literal dispatch to `me` you could create a proxy class which dispatched to a reference to `me`.

    type('ChildOfMe', (proxyclass(me),), {})

Where `proxyclass` is a function of the form,

    def proxyclass(obj):
        class Proxy:
            ... dispatch to obj ...


        return Proxy

"If you want to inherit from an object, just create a class based on that object" is a description of how you'd imitate prototype inheritance in a language that only has class-based inheritance. It does not mean you're actually looking at a prototype-based language, and in fact strongly suggests the opposite. You can't get inheritance in Python without engaging with the class system.

I suppose you could just say that classes are a particularly restricted sort of prototypes, and thus any class-based language is effectively prototype-based. But at that point "<OOP language> has prototype inheritance" is just tautological based on the set of definitions you've chosen.

It's not tautological. I don't think C#, C++, Java, etc. can have classes proxy for objects. Classes there aren't objects. You cannot write "return NewClass".

Python is prototypal in just the same way js is: inheritance is a daisy-chain of references between objects.

JS is "maximally prototypal" if you like, where any object may be refernced. In python, only objects of type "type": but that isn't a big restriction, as shown.

The entire VM is different:

- you can replace builtins in python. - you have __dunder__ methods and multiple inheritance. - JS has an ever running event loop and is async by default. - python is more strickly typed.

It's not impossible (see brython), but it's not easy either.

Numbers. Everything is a float in JS. Python has ints that transparently turn into bigints on demand, floats, and if you need it a whole library of C numeric types to do interop (along with complex numbers and all kinds of other great stuff in python's standard library).

My point was more about the object model itself being incredibly similar, rather than the data types.

However, much appreciated.

It's perhaps interesting to note that browsers pretty much already implement this behaviour for js, ie., by spec everything is a float -- but vms don't start with a float type for much of the numeric data stored; and transparently move to it when needed.

That's an interesting way to put it. Python does have a form of prototypical inheritance (given how objects end up being `__dict__`s with behaviour defined through `__class__`, though not only that!).

Python descriptors are extremely flexible though, and I _guess_ you could emulate almost all of it through Javascript proxies, but I can't imagine the performance of that being great. Hash tables would also not be re-usable between JS and Python (since Python hash tables allow many more varieties of objects), and Python has a distinction between attributes and keyed access (which JS lacks).

Python exposes much more of the global environment and internals than JS does, and doesn't have some of the requirements JS has put on itself for security reasons. I'm not super sure you could emulate Python blocking style with a JS transformation statically. Blocking Python for something that is only available in a non-blocking JS environment would require effectively wrapping every Python operation (since almost every Python operation can have arbitrary side-effects).

I think there are people who write sort of syntax-y transpilers, but that's not really going to lead you down to nice code re-use.

Because a large part the Python ecosystem uses packages (and stdlib modules) with C extensions and you can't easily transpile those. The end goal of Pyodide is to be able run existing projects mostly without modification in the browser, and that is not possible if one doesn't support Python C extensions.

(I'm one of the maintainers)

https://brython.info/

It would be helpful if someone could clarify some differences between Brython and Pyodide.

That's totally different. Pyodide is CPython in WebAssembly, and can use existing Python libraries (including those written in or backed by C), like NumPy, Pandas, and MatPlotLib. Brython transpiles Python code to JavaScript, so can handle some pure-Python libraries only: https://stackoverflow.com/questions/57517734/how-to-import-n...

Brython => a JS Python implementation. It will not behave like cPython for somethings, does not have access to the whole stdlib nor the scientific stack. But it is quite small (700ko without their stdlib) and has native bindings to the DOM. Use it if you feel like writing your web page in python (I wouldn't, I love python but 700ko+ with no framework loaded yet is way too much).

Pyodine => the cpython implementation, but compiled to wasm. It's very heavy (10Mo), but it brings the real, whole cpython implementation in the browser so it will behave the same and have the entire stdlib. It also comes with optional NumPy, Matplotlib, pandas, SciPy, and scikit-learn. Use it if you want people to write python in the browser, but don't want to expose your server to their code.

Should the new front-end framework based on this be named Plazor, Plyzor or Blythor?

The site is about 7 MB download.

I feel Blazor WASM is a better choice in every way.

To elaborate : Blazor uses C#, is more performant, .NET 6 is bringing AOT for even better performance, has an ecosystem of tools to build sites with, integrated well into the .NET ecosystem, has a great set of libraries to choose from.

For what use case?

The main use case for pyiodine is running python without having to setup python.

If you don't want python, there is not need for it.

If you do want python, there is no replacement for it.

OK, great, but this is not a new idea; in 30 seconds I was able to find three "Python in the browser" sites. Executing

    import sys
    print(sys.version)

in each produced,

https://www.programiz.com/python-programming/online-compiler...

"A module you have imported isn't available at the moment. It will be available soon." Really? sys?

https://www.online-python.com/

3.8.5 (default, Jul 20 2020, 23:11:29) [GCC 9.3.0]

https://replit.com/languages/python3

3.8.9 (default, May 3 2021, 02:40:41) [GCC 7.5.0]

The latter two, certainly, are adequate for class exercises and demonstrations. No idea if they are good enough for production of course, which Pyodide seems to be claiming.

Reply on news.ycombinator.com