Python and Lua (2019)
brmmm3.github.io
brmmm3.github.io
As an anecdote, when I was making my first Lua script for Alfred on a horribly underpowered machine, I briefly couldn't figure out if I actually configured the script properly, because the dummy example output indeed appeared right as I typed the last input key. Turns out, Lua really runs that fast.
Alas, the lack of proper ‘null’ bugs me—it may interfere with making middleware for external APIs, receiving and transmitting schema-less data. A bunch of libraries use cjson.null or interoperable lua-null or somesuch, but I'm sure I'll bump into a lib that won't know about those.
Ah, and by the way, from what I vaguely heard, actual native multithreading might be doable in Lua with libraries. At least I've seen a couple of such libs, but haven't tried them.
The JIT effort is great, but projects like mercurial had to hack lazy import because they are so slow in python.
I wish this can be baked in the language some day: loading half the stdlib to run hello world is not very efficient.
Some years ago when Squirrel started to get traction, particularly due to its adoption in a couple of IoT frameworks, I fiddled with it for some months and it almost got its way into a commercial electronics project. However by then uPython was getting more popular, had better tooling support and ARM microcontroller chips had become very fast. Being able to reuse the same code on the host with minimal or no changes, or having an interface or emulation layer that you can also program in Python is a game changer. It took me less than a week to rewrite the Squirrel code to Python, with even better features thanks to the language facilities.
I thought I was never going to touch Lua again but a couple of months ago I found myself hacking some REAPER audio processing scripts, and cursing Ierusalimschy once more for his decision on neglecting Dijkstra's advice and not starting indexes from zero...
However, life is full of unexpected surprises. While Lua small engine size was a deliberate strategy to make it cheap to embed, it also meant it's much easier to optimise than something the size of Python. And ... that's what happened, so LuaJit is the fastest interpreted language - even faster than the best javascript engines when I last looked. That despite javascript engines having had man decades of optimisation effort put into them.
Pallene is Lua with strong types, designed for the purpose of being able to (AOT) generate faster and native code.
It's compiler design also has deep knowledge of the Lua implementation, so it short circuits the formal Lua/C API so crossing the Lua/C language boundary in Pallene is even faster (and Lua's is already good...much less overhead than crossing in Python).
Why is it down? I found other links for those interested:
https://github.com/titan-lang/titan
https://github.com/titan-lang/titan/blob/master/doc/manual.m...
Easily interoperable languages present an interesting proposition: a vertically integrated approach to language design where tools are more specialised, but designed from the get-go to work together. Fascinating, and awesome.
This is a great example of something that's more than the sum of its parts. A GIL-constrained language and a single-threaded language working together to run code in parallel - something that neither language can achieve on its own.
Although I believe there is a plan to add support for this pattern (multiple threads with one interpreter state each) to pure Python in the form of "subinterpreters".
Point of interest, Python already has another embedded language if you have Tkinter available: Tcl.
Larry Wall of Perl fame once said: “Tcl tends to be ported to weird places like routers.”[0]
Routers are just the beginning… see here[1] for calling Tcl scripts directly into a Python-embedded Tcl interpreter. I believe at least one of the Perl/Tk attempts did something similar (embedding Tcl wholesale) too.
[0] https://github.com/thibaultduponchelle/perl-fortunes/blob/ma...
[1] https://stackoverflow.com/questions/7362846/running-tcl-code...
The Tcl system was ported from the Very Large Telescope VLT, in Chile.
So that's a couple of weird places.
>>> import tkinter as tk
>>> tk.Tcl().eval('puts {hello, world}')
hello, world
''
The 2 codes don't do the same thing, if you perform the same computation on the python code (monochrome mandelbrot, inline loop, 49 iterations, etc), you get 0.8 vs 4.8 sec, which seems about right.
I've removed numpy since it's cheating if we compare raw language perfs:
https://0bin.net/paste/nOtLmeMA#0DKGHSvmgoDLPzHlQOhwvVhxxayf...
So yes, Python is about 6 times slower. Significative, but can be worth the compromise if you consider the Python version is much, much easier to read.
And of course, a vectorized algo would be even faster (see pbourke's comment).
What's more, with SharedMemory from 3.8, you could speed that up even in pure Python using multiprocessing by sharing efficiently the array.
The same rewrite in Python would not speed things up, it would slow things down.
It's more than that, though. It looks like the Lua code does only 49 repeats where the Python code does 200. It's clear that this isn't just a Python->Lua rewrite, this is a rewrite that was followed by a lot of optimisation work on the Lua code.
Despite that, I kinda buy it. I can believe that you won't get speeds in pure-Python code for this particular problem that are competitive with a JIT. numpy doesn't help, because the number of iterations varies for each point, so you can't easily parallelise. And speaking of parallelisation, he has a point about threading: Doing the computations in pure-Python would not perform well over multiple threads, you need Cython or something. And that something can apparently be Lua.
(After working on the numpy code posted in this thread, I think Python is pretty competitive, but it requires quite a few contortions - rewriting it using numpy.)
import time
from PIL import Image
import numpy as np
def PyMandelbrot(size):
t1 = time.time()
xPts = np.arange(-1.5, 0.5, 2.0 / size).reshape((1, size))
yPts = (np.arange(-1, 1, 2.0 / size) * 1j).reshape((size, 1))
cs = xPts + yPts
z = np.zeros_like(cs)
red = np.zeros_like(cs, dtype=np.uint8)
mask = np.full_like(cs, True, dtype=np.bool_)
max_idx = 0
while max_idx < 199:
np.putmask(z, mask, np.multiply(z, z) + cs)
np.putmask(red, mask, red + 1)
mask &= np.abs(z) < 2
max_idx += 1
dt = time.time() - t1
print(f"dt={dt:.2f}")
zeros = np.zeros_like(red)
pixels = np.dstack((red, zeros, zeros))
Image.fromarray(pixels).show()
if __name__ == '__main__':
PyMandelbrot(1_280)
> python .\mandelbrot_orig.py
dt=32.01
> python .\mandelbrot.py
dt=5.86
I'm learning as I go here, using numpy functions I've never used before, but we can use more in-place modifying operations like this (np.multiply, np.add with where= and out=) and it decreases runtime of the benchmark by 25%.
Then, after that, this problem is embarassingly parallel and thanks to numpy we can parallelize your code very well, using all cores (improves runtime for me from 3.5 to 0.92 seconds) Using Python3.8.:
import concurrent.futures
import time
from PIL import Image
import numpy as np
def compute_mandelbrot(cs):
"""
cs: complex grid subset
"""
z = np.zeros_like(cs)
red = np.zeros_like(cs, dtype=np.uint8)
mask = np.ones_like(cs, dtype=bool)
for _ in range(199):
# z² + c -> z
np.multiply(z, z, where=mask, out=z)
np.add(z, cs, where=mask, out=z)
np.add(red, 1, where=mask, out=red)
mask &= np.abs(z) < 2
return red
def py_mandelbrot(size):
t1 = time.time()
yPts, xPts = np.ogrid[-1j:1j:size * 1j, -1.5:0.5:size * 1j]
cs = xPts + yPts
# over-split because chunks are not equal in time taken
nsplit = 32
nthreads = 8
css = np.array_split(cs, nsplit)
with concurrent.futures.ThreadPoolExecutor(max_workers=nthreads) as executor:
reds_out = list(executor.map(compute_mandelbrot, css))
red = np.concatenate(reds_out)
dt = time.time() - t1
print(f"dt={dt:.2f}")
Image.fromarray(255 - red).show()
if __name__ == '__main__':
py_mandelbrot(1280)
https://docs.python.org/3/library/multiprocessing.shared_mem...
If you use a data structure that supports the buffer protocol (such as a numpy array), you won't even pay a serialization cost.
Python has made a LOT of progress, concurrency wise. Between asyncio, process pools, threads releasing the GIL and shared memory, you can get much perfs with the stdlib alone.
Then of course you have Dask if you need more later.
Thanks for bringing it further.
import numba
from numba import complex128, boolean
@numba.vectorize([complex128(complex128, complex128)])
def square_add(z, c):
"compute z² + c"
return z * z + c
@numba.vectorize([boolean(complex128)])
def not_escaped(z):
"compute |z|² < 4"
return ((z.conjugate() * z).real < 4.)
def compute_mandelbrot(cs):
"""
cs: complex grid subset
"""
z = np.zeros_like(cs)
red = np.zeros_like(cs, dtype=np.uint8)
mask = np.ones_like(cs, dtype=bool)
for _ in range(199):
# z² + c -> z
square_add(z, cs, where=mask, out=z)
np.add(red, 1, where=mask, out=red)
# mask &= |z| < 2
not_escaped(z, where=mask, out=mask)
return red
Ironically I’m going through this process at work this week to improve the performance of scipy.stats.ks_2samp
# guvectorize also supports target='parallel' for automatic parallelization
@numba.guvectorize([(complex128[:], uint8[:])], "(n)->(n)")
def compute_mandelbrot_gu(cs, out):
"compute number of iterations of z² + c before it escapes"
for i in range(cs.shape[0]):
c = cs[i]
z = 0j
count = 0
while count < 200:
z = z * z + c
if (z * z.conjugate()).real > 4.:
break
count += 1
out[i] = count
One thought here: the number of 100%+ improvements made here just underscores how slow python is to start with :). And yes, everyone in comments here complaining about numpy are sort of right, since the ultimate end here is just a flat loop without numpy, but as a numpy function.
I'd like to think that I'd rather have Lua as a language, with Python's class definitions and Python's ecosystem. What I mean by Lua as a language is the interpreter - and luajit - but also its `local` variables and scoping rules (Python's rules are rather strange, which the nonlocal keyword signals clearly).
I probably disagree with 90% of the criticisms leveled against Python, but it definitely is slow in "loopy" tasks, and the indentation-based syntax can make for relatively verbose code in quick ad-hoc scripts. I'd be plenty happy using Lua instead, if only it had a more thorough standard library. The nil-handling thing is annoying, but fine enough for quick scripts.
Isn't significant indentation less verbose than Lua's use of `do ... end`?
if x < 3 then x = 3 end
The Lua ecosystem was seriously damaged by the 5.1>5.2 split for a while since LuaJIT (hence Torch/nginx) and LÖVE both basically stayed with 5.1 and those were major projects that drew people to the language. The other cool unique thing in the Lua ecosystem, the native-widget (Windows/Gtk) IUP toolkit, remains barely known about and never ported to OS X, possibly due to a lack of Mac users in Brazil. Hence the "language without an ecosystem": it didn't happen for no reason, though the willingness to make breaking changes is a reason the language became so popular in the first place.
NB. IUP is still around, but it still doesn't have a Cocoa port.
> a form for short Python-style lambdas would really help
Interesting - many in the Python community feel that its lambdas (particularly the use of `lambda` as a keyword) are too verbose and would prefer something like `x => x+1`. Maybe such an arrow-function syntax is achievable with Python's new PEG parser - I'm fairly sure it couldn't have been supported by the old LL(1) parser.
About lambda syntax, to me it would be far more important to allow multiline lambdas than to have a shorter keyword.
Even Lua has a built-in `math.max` function that works with integers. Maybe it’s Lua’s way of flexing on Go :)
> About lambda syntax, to me it would be far more important to allow multiline lambdas than to have a shorter keyword.
Never going to happen, unfortunately. It’s just not possible within the fundamental rules of Python’s grammar: that statements use significant indentation and expressions do not.
(lambda x: x + 1)
(function (x) return x + 1 end)
((x) -> x + 1)
But using False for invalid key accesses seems to be a really bad idea. Sometimes you need to store boolean values in tables. When that table access returns False then you don't have a way to tell if it was from a successful key retrieval or not. (Lua solves this dillema by making that storing nil in a table is just as if nothing really happened, which is pretty reasonable)
If Lua had a more fleshed-out exception system (The current one based on pcall() is a bit limited both in syntax and functionality), I think it would have been better to go for the Python route (throw exceptions for invalid table access). Don't know how it will take a toll on performance though, maybe that's the reason why it chose such a simplistic route for implementing tables?
"Absence of value" is different from "data that happens to be some kind of empty data point".
Also part of the issue is function arguments, there's no arity-checking by default.
With all flaws of js, this is where it shines:
var t = undefined
var u = {b:1}
var {a=42, b=true} = {undefined, …t, …u}
// a==42, b==1
table.insert(t, string.gsub(s, “1”, “2”))
-- happy debugging
console.log('foo' in {foo: undefined}); // true!In fact, even global variable accesses are designed to work as table accesses to _G (the global table), so if you make a typo then the interpreter doesn't say that you've mispelled a variable but instead silently gives you nil. This leads to all sorts of weird bugs that are very hard to track down. Probably the number one reason why I ditched the language for something else.
Questions for the Lua critics. How many Python modules, e.g., in C, have you written. Which language is easier to write modules for, Lua or Python. Ever compare the startup time of the Lua interpreter to the Python one. What s the tally of applications that have embedded Lua interpreters versus those that have embdedded Python ones.
Because it is (optionally) embedded in applications I use every day and/or rely on, whether it's a DNS load balancer, a TCP/HTTP proxy server or a fast OS kernel, I always use Lua for "practical purposes".
Python seems to have development team of millions, but would I rather have a smaller interpreter with faster startup that is easier to write modules for. To me, that is an important question. Will I ever write Python modules in C for own use, or will I have to choose someone else's library and become 100% reliant on it. Does that limit the true "DIY" potential of Python vis-a-vis Lua.
At least the parent did not use the "indices start at one" justification for concluding Lua is inferior. In the early days of Lua, that was a red flag to me that this language must be good, if that s the best criticism anyone could come up with.
static int l_foo(lua_State *L) {
size_t len;
const char *s = luaL_checkstring(L, 1, &len);
luaL_newuserdata(…);
return 1;
}
luaL_register(…);
As of nil, “:/.” and others like local…in, table=array, it just becomes an always-itching scratch after years. Lua doesn’t even respect full backwards compatibility (not that it was claimed) but its authors stick to the principles that many folks tried to un-convince them to no avail.
language must be good, if that s the best criticism anyone could come up with
That’s only the shallow surface, the real talk always goes in the mailing list before the release.
And of course you could always just write functions in Lua and call them from C.
If you need C modules for Python, use Cython for the binding, don't write them "from scratch".
#pythran export PyMandelbrot(uint8[:,:,:], int)
Note I passed the array into the function to remove the image creation from the module.
This yields a 18.6 ms vs 1.1s for the original for size 320. As a side note I find it worrying that a blog which is called speed matters uses time instead of timeit.
(*) https://www.linusakesson.net/programming/kernighans-lever/in...
Still heavily used in game development.
For speeding up pure python, the first thing I would do is reach for numpy. It would be interesting to compare the performance of the lua-embedded code with a version using numpy.