A lightweight (~5000 LOC) Python interpreter for game engines | Hacker News Reader

A lightweight (~5000 LOC) Python interpreter for game engines

github.com

106 pointsvitaut76 comments

I've been waiting a long time for MicroPython to become usable as an embedded script interpreter -- it can't be run asynchronously [1] so scripts can't do waits (aside from that, MicroPython is amazingly full-featured). CPython can't either, which is why the Stackless Python fork was created, which is used in games such as EVE Online. But that's far too heavy for my tastes.

So I was hoping that PocketPy could do that, but after looking around I can't find any suggestion it's possible, and the API looks very sparse. I'm very happy to see a new Python interpreter built for embedding, but I can't use it. Maybe it can be added easily enough. Keep it up!

If you want to see a scripting language designed from the ground up for embedding, Lua and Squirrel IMO have the most complete APIs (though not the most efficient or easy to use) [2][3], and LuaJIT has even better lua<->C bridging. I've looked at the APIs of many other small scripting languages, and haven't found any others that can do everything that these can.

[1] https://github.com/micropython/micropython/issues/3619

[2] In particular https://www.lua.org/manual/5.4/manual.html#4.5

[3] http://squirrel-lang.org/squirreldoc/reference/embedding_squ...

After 40 years of dicking around the C21 has an #embed preprocessor feature.

I think which allows you to write stuff like

   const char python_script_foo [] = 
   {
      #embed <foo.py.wav>
   };

Would be useful for embedded scripting. Especially for firmware.

This was of course available though inline assembly or code generators from the beginning. So C21 is more complicated for the extra looks-like-prepreprocessor-but-probably-isn't syntax with no increase in expressivity.

How could that not be a preprocessor? Or are you suggesting that the compiled binary will now include a Python interpreter?

It is a preprocessor directive, the result must be as if the contents of the file were inserted byte by byte as integer literals.

See [1] for more information (which, btw, also states that #embed is in C23, not C21 (which doesn't exist as far as I know)).

[1]: https://thephd.dev/finally-embed-in-c23

If it's implemented in the preprocessor - e.g. expanding to a comma delimited char array - then the performance justification for adding it to the compiler is lost.

However you might be using "preprocessor directive" in the language semantic phase sense, not in the "preprocessor does it" sense.

It's inserted as a binary blob by the compiler. So just a simple copy of the file contents.

Have you ever looked at newLISP [0]? It's been on my radar for a while, but I've not yet had the chance to try it. Its FAQ's [1] item 18 lists a few projects it's been used as an embedded language in.

[0] http://www.newlisp.org/ [1] http://www.newlisp.org/index.cgi?FAQ

Thanks, I'll give it a look when I have time. There's certainly a lot of embeddable lisps due to their simplicity, but I've never really given any serious consideration.

Nice idea, as python is much easier to write code for than lua so should fit game scripting well.

But, the examples are not the kinds of things that I would care about before trying to put this into a game engine. It needs a different set of examples.

* How hard is it to call into a C function from a hook in the py code?

* How hard is it to wrap / unwrap values from their py representation to C?

If you want people to embed the interpreter into an engine then show that it is clean/easy to do the embedding, rather than showing that the interpreter can execute code. I would assume that it executes python, but need to know how well it embeds.

Here is what I came up with for calling a c function after poking around a little bit:

  #include "pocketpy.h" 
    
  // builtin function that can be called
  PyVar test(VM* vm, const pkpy::Args& args) {
    printf("called!\n");
    return vm->PyStr("hi there");
  }   

  int main(){
    // Create a virtual machine 
    VM* vm = pkpy_new_vm(true);
    vm->bind_builtin_func<1>("test", test);
    
    // Hello world!
    pkpy_vm_exec(vm, "print(test('Hello world!'))");
    
    // Free the resources
    pkpy_delete(vm);
    return 0;
  }

> as python is much easier to write code for than Lua

No, it isn't. And it isn't harder, too. They're just different.

Is assembly harder to write than lua? How about lisp? I mean you must have some kind of rough ranking of language difficulty in your head. Sure, some of it will be cultural, in a world where middle schoolers are taught s expressions and reverse polish notation lisp might be a lot easier. There certainly is some subjectivity involved.

Still, I think it's reasonable to say that python is easier to write code for.

Scripting use is by definition a less demanding type of requirement than being able to write entire apps.

I wrote a fairly complete "C" subset as a scripting language for a programmable regression test tool (self-contained test-cases able to evaluate the test-case responses). To make it more friendly I made it dynamically typed (so no need to declare variables) and also added XMLPath syntax support as a type of built-in l-value since the requests/responses it was dealing with were XML/JSON, and this meant you could access any elements of the test request/response(s) as if they were just variables.

Anyhow, "C" may not be what comes to mind for a scripting language, but targeting a group of developers who knew C/C++, and with these minor language tweaks made it very simple to use. My point is that for this type of application I'm not sure that the choice of language makes a huge difference - choose something your target audience is comfortable with, and extend in DSL fashion to make it tightly integrated.

In my experience I've found it easier to write Python code. The set of basic types is a little larger, and the core builtins cover more of what I need to do. Lua is a bit more fiddly and verbose, which leads to a more difficult development process (in my opinion). Why would you argue that they have the same level of difficulty?

It was pretty hard for me to get used to Lua's 1-based index and everything treated as a table

I think it's easier to mentally map concepts between C and Python than Lua

Lua scripting can be more commonly performed by people without an engineering background, depending on the workplace. Game designers and level artists probably don't have much C experience, so Python's relative proximity to C is a non-factor.

So is this a clean reimplementation? Given how small this is it probably is.

Which means this is not Python but a language that is syntax-compatible ("Python-like"). To prove compatibility something like a (very) extensive set of compatibility tests would have to be written and mentioned in the README. Otherwise it's a minefield of subtle semantic differences.

I don't know if they plan to expand it, but it's a pretty small subset of python now. No bytes or bytearray types, for example. Only string escapes like "\t" are supported, so "\0" raises an error. Trying somestring[1:2:3] raises an error. It's missing list.reverse(), list.sort(), list(some_dictionary) raises an error, and so on. Most areas, if I dug in a little, were missing things.

Still impressive for 5kLOC, of course.

...and there's no official way of finding these limitations.

It's amazing for a personal project but not good enough for a serious option.

What's the performance like compared to Lua? Since Lua is the standard here, this feels like nonstarter unless the performance is similar, even with the friendlier python syntax.

to my mind the differences in syntax between python and lua are like two white, long-sleeved, button-up, collared shirts, but they have different buttons.

and also the buttons on the Lua shirt are all shifted by one

.. while the python shirt has a huge gap that shows a nipple, and if you try to cover that nipple you're going to fall over.

HAHA,good one!

The python shirt has a combo of buttons, zippers, Velcro, and drawstrings in different places and with a long history about why it's there and why it made sense at the time but no one would do it that way again.

Doesn't this apply to every mature and widely used language after a few years of evolution?

"There's more than one way to do it."

> In general, the current performance has not been specially optimized. I estimate that it is about 30% to 50% of the performance of CPython.

> Because stack-based virtual machine is used, the performance is hard to reach Lua, where the latter is register-based.

> With the optimizer in place, I think the performance of PkPy can reach 1-2 times that of CPython in the future.

https://pocketpy.dev/performance/

If the author comes around, first of all congratulations on what seems like a really cool project. Second, I am curious as to whether the lack of 'star unpack' is due to any real blocker, or more of a 'not done yet'/'not prioritized'?

How does this compare to tinypy? [1] http://www.tinypy.org/

TinyPy is a bit larger at 10k LOC, seems not as easily embeddable since source split among numerous files, not really suitable for embedding since it's partially bootstrapped (parser and compiler written in tinypy; although pretty cool), it's dead and stuck at py2.

why does

1. source split among numerous files, and

2. partial bootstrapping

affect embeddability? i'm asking in earnest. i thought to embed we (typically) just need a shared lib and to #include a header api.

The issue for the first is you either have to download the shared lib precompiled or compile yourself and link as extra step. PocketPy is entirely in header file(s) that can be included in the project directly. That's easiness. (Of course header-only libs have their bads too.) It isn't really about the difference in the result but the process. Having part written in the language can contribute to previous but can also impact speed and memory use. That's suitability.

i learned something. thanks for responding.

How is this on the homepage twice with no comments and less than 10 votes?

Because the world rotates and it was a hit while the other half had their eyes closed?

See my comment here for more context: https://news.ycombinator.com/item?id=34707361

There's not only the question of why use this over Lua, but why use this over many compile-to-lua languages, providing a great range of syntax and typing features.

I can confirm that, for example, Fennel[1] is nice if you're looking for something lisp-y.

It's a bit unfortunate that it assumes that you are using Lua's standard library (so you can't use it in cases where you only want to embed Lua-the-language with your own restricted APIs), but other than that it's real nice.

[1]: https://fennel-lang.org/

I wonder how it compares with MicroPython?

MicroPython is 286kloc of C.

That is great information, but perhaps it would be even more interesting with a comparison focusing on supported features, standard library support, and so on.

MicroPython [1] is designed for "real" embedding, i.e. in the "embedded development" sense, on microcontrollers and so on and seems quite successful in that niche. Those are often a smaller target than a game running on either a phone/tablet or PC, which would seem to kind of invert things.

[1]: https://micropython.org/

The other comparison would be memory and disk/ROM usage.

Also, micropython, in its hundreds of thousands of lines of code, supports quite a number of microcontrollers and boards; I suspect the core language is smaller, or counting lines used for a single build target would drop it considerably.

You are correct, the micropython vm/interpreter + the unix target (one of the simplest) combined are ~50k lines of C.

Note that to run correctly micropython also needs its python stdlib which is about 15k lines of Python.

Micropython is made to run on dozens of boards, each with a nontrivial board support package (the platform-specific glue code to talk to hardware). A better comparison would be looking exclusively at micropython's unix/x86 platform specifically.

The demo is fun to play with: https://blueloveth.github.io/pocketpy/

It uses https://blueloveth.github.io/pocketpy/lib/pocketpy.wasm which is 766.66 kB - transferred compressed as 240.72 kB

> The demo is fun to play with: https://blueloveth.github.io/pocketpy/

Type `exit()` and press [Enter].

Wow, this is excellent! I love the fact that it is just in 5K lines. I have a few questions!

1. What were your motivations for building this?

2. Did you eval any similar projects before writing this one?

3. Can this be compiled to WASM and used in the browser? edit: I see that it already supports WASM!

4. I didn't see this on the README, but does it support IO (either disk or network)?

5. I noticed you linked to byterun. Does this also implement a VM similar to Python? Can I run the bytecode produced from CPython in the pocketpy VM? If so, are there any reasons for that? I thought writing an independent implementation would be faster/easier than a compatible one.

6. Any plans on adding type annotations?

Hmm duplicate of https://news.ycombinator.com/item?id=34702386 maybe I should text @dang to merge...

Hmm, wish this had been built as a PocketPy -> LLVM compiler. While the compiler might not have been lightweight, the final emitted binaries would have been superfast.

That kind of defeats the purpose of having an embedded scripting language

https://github.com/Chocopy-LLVM/chocopy-llvm

Your wish is my command: Do it!

You can't just put LLVM on it and expect it to go fast, and I can't put a turbo on a 2CV and expect to use it on the motorway now.

It's _a lot_ of work to make a language like python "superfast", and most of it is really the dynamic typing part. LLVM doesn't do that for you.

> and I can't put a turbo on a 2CV and expect to use it on the motorway now.

Well, actually, I remember being passed on the autobahn by 2CV going well over 200 km/h. That was quite a shocker.

I'm quite sure the owner didn't get there by just adding a turbo :)

I'd love to see it though.

No doubt! As it came past in a blur, it was difficult to see what was left of the original car beyond the external shell..

Option 1 is mandating mypy-style type annotations (with type inference where there’s no ambiguity).[1]

Option 2 is using C++-style vtables, and putting each value (including primitives) inside an object, with each object inheriting from a common object superclass that has stub methods like __add__, __sub__, __mul__, __div__, __str__, __repr__, etc, etc. You do all the dynamic checks at runtime.

With option 2, I’m sure vtable dispatch is still significantly faster than Python (while substantially slower than static typing).

One could also combine approaches 1 and 2 make the static typing optional.

[1] I realize implementation type inference isn’t trivial.

There are indeed lots of options. In this case, 1 is limiting and 2 is slow because of cases like integer operations. A very common way to make it "fast" is specialising at runtime based on usage, so it's kind of like option 1 but dynamic.

Re nr 2, Like stefncb's sibing comment mentions, numeric ops would be horrendeous due to excessive memory allocations (even if you did value like semantics driven you'd still be luggin an entire vtable-pointer along with your numbers and there is usually more value movement than operations in code).

Type-inference depends on the language. Languages with prepared semantics do it just fine(ML family), it's just that languages like Python and JS didn't consider it from day-1 so we're always stuck with explicit subsets (or extensions like TS, Cython/Mypy) to gain back some order. (I did my thesis work on JS type inference).

In practice, there is nothing that hinders the Python language to be as fast as JS (see PyPy or even faster with Cython annotations). The problem is that C-Python sees the simple FFI as an golden egg (for kind-of good reasons at this point) and this will always limit C-Pythons performance because it touches upon internals that are hard to change out without breaking the FFI contracts.

So, even if one would go about and bolt LLVM onto C-Python we'd still be way behind V8/JSC/Spidermonkey, because

1: The value and FFI model still would hold it back

2: LLVM while awesome at static code, would still have a ton of issues with the dynamic nature of Python. There has been a couple of articles on the new JIT of Erlang that is based on a custom "simple" compiler that is working better than many more "advanced" variations has worked because it's more tailored to the dynamic nature of Erlang rather than try to make a regular JIT make something sane out of Erlang code.

If you want to see improvements to Python there's 4 avenues..

1: Nuitka , this is one(?) guy who's doing a lot of manual engineering work together with some JIT ideas, it probably has a performance ceiling but he seems serious about doing it whilst retaining compatibility. (No idea how it'll fare with the regular series C-Python optimizations).

2: PyPy, Jython and GraalPython. These all depart from mainline C-Python and does deprive you of many C/C++-bound libraries whilst retaining the language semantics. The PyPy team has suggested a new better FFI that I do hope gets adopted by C-Python as the way forward for new modules (and ports of older ones) that would allow PyPy and C-Python to share modules as a transition way.

3: Cython, here you have the original source (iirc?) of the MyPy like annotations and an actual compiler that will specialize and should be mostly compatible with C-Python.

4: Shedskin, this is probably the fastest "Python", but it's a specialized subset that's mendable to full type-inference.

There is also Cinder and Pyjion, two JITs built on top of CPython. API, if not ABI, compatible. And now there's S6 too.

Disclaimer: I worked on Cinder.

*interpreter for a small Python subset

Should be expected from the size of codebase but should also be clearly mentioned. And this isn't about just lacking Python's modules. Plain language features are also not there. For example reversing a string with "abcd"[-1::-1] doesn't work. Now there's a features list on README, but not sure whether this is meant to be the comprehensive list this subset has or just a TODO.

This is neat! I assume this is Python 3?

Still looking for a replacement for the excellent Jython project [1] on the JVM, which has been fantastic for adding scripting to Java/Scala based projects, but sadly stuck on Python 2.7. I wonder if this could be bolted onto or run inside a JVM somehow (perhaps the WebAssembly version?)

1: https://www.jython.org/

I think it is python-like rather than perfectly python compatible. Therefore the exact version is indeterminate...

I thought that "weight" generally referred to less resource usage (RAM primarly, cpu second) and loc weren't part of those resources.

Was I wrong?

weight == resources (both in terms of RAM, code space, maintenance overhead, etc).

There is no "standard" terminology but I would say small line count is a definite factor here. Small LOC count implies low complexity of integration and low risk. Risk? Yes, the smaller the codebase, the easier it is to take your own fork and maintain it, even if the original project is no longer maintaining the codebase.

Of these two low complexity is probably the main selling point.

Small also implies generally "an elegant engineering solution to a problem that generally may involve layers of accidental complexity" and hence also incites a sense of beauty. Aesthetics can be important for people that are hand crafting their codebases.

I'd like to give the WASM build a try

Great project!!

I was trying to build my own python interpreter to my project and this saved me a lot of work!

super nice-to-have ask: could you replace the is_prime test on the README from range(2,x) to range(2,int(math.sqrt(x)))?

impressive work! i suppose it implements a subset and not the full language? would be nice to see some benchmarks. everyone claims super fast nowadays. i have no doubt though, since it's cpp.

They say "The speed of PocketPy is approximately 50% of CPython" so apparently the focus is a small implementation rather than performance.

https://pocketpy.dev/performance/

There's a table in the README saying "Star Unpacking" isn't supported.

That's not all that's not supported. Through poking at the live demo, I've found that at least slots, kw-only args, some basic syntax elements, decorators, generators, generator comprehensions and async are additionally not.

Exceptions are marked as supported … but `raise Exception()` is a syntax error? (Re-raise and binding exceptions also fails to parse.)

Why not lua though?

an i for an i is an i in range

Reply on news.ycombinator.com