[0] http://www.opengroup.org/comsource/techref2/CHP06GDC.HTM
Very much an XKCD 927 story.
[0] https://en.wikipedia.org/wiki/Raft_(computer_science)
[1] https://en.wikipedia.org/wiki/Log-structured_file_system
He was a smart and productive person that designed an ugly programming language.
Lua https://p.thorsen.pm/5ecb8ce026ab
Tcl https://p.thorsen.pm/ee8a792acd52
Results: Lua 5.2: 0m0,027s
LuaJIT 2.1: 0m0,016s
Tcl 8.6: 0m0,630s
I'd pick Lua over Tcl any day.I got an order of magnitude improvement on Tcl just by bracing the expressions:
Bad: expr $i +$j + 2
Good: expr {$i + $j + 2}
... it saves interpretation and evaluation to occurring once, hoisted in the [expr] engine; it’s idiomatic. The lua version still appeared faster, though I’ve only spent 2 minutes looking at this.
Again, strings should play no issue here; values must be able to generate a string-representation, but it’s computed lazily, and in this case, the $i, $j, etc in Tcl are working with native integers.
I’m happy we have things like Tcl and Lua; either one of them makes development enjoyable. I appreciate you taking time to write your code samples.
Keep in mind that while, yes, Tcl lists have a more efficient internal representation when Tcl knows that it's a list, a string that happens to be a valid representation of a list is also a list, just by virtue of existing - but it won't get a special list representation until you try to use it like one! It's just a string like "foo bar baz". Or, say, "objref#1 objref#2 objref#3".
And the semantics of the language is EIAS through and through - all those internal representations are optimizations that are not supposed to change the observed behavior, only performance. A GC cannot treat lists without internal representation differently from those with it without breaking this. So it has to assume that any string is potentially a list that contains object references.
So a proper Tcl GC would have to be incredibly pessimistic about what it considers to be roots of the object graph - basically, any substring that looks like an object reference, has to be considered as one. Now imagine just how much string scanning such a GC would have to do in a non-trivial app for a single sweep!
It sounds like you’re familiar with Tcl internals, so I presume you know that (if we stick w the list as a prototypical “complex” structure) that the management is reference counts, freed when count==0, and may be >1 if this is a shared item (I’m going to refrain from calling it an object, though internally they’re a Tcl_Obj type, which has nothing to do with OO-programming). In a list, the list is an item, and ea. element is an item, with (of course) incr/decr references at when appropriate, as determined by the language rules itself.
Wrt strings vs lists, a string could be considered a single “blob”, but coercion to a proper list (“shimmering” in Tcl parlance) does the tokenization with requisite ref. counting as part of the conversion.
...I think the above may be entertainment more for anybody else who’s following along, but if that illuminating for you too, that’s a bonus.
Perhaps you could show me by example:
What’s a piece of (eg) Ruby and what you think a Tcl workalike would appear as, and where that falls down.
(I'm oversimplifying and ignoring the command part of it, because it doesn't really matter for any of this.)
For the sake of readability, let's assume that those object IDs / namespace names are just numbers, e.g. ::42 (in practice, it's something like ::oo::Obj42). Since that number was generated by TclOO machinery under the hood, said machinery can properly tag it as a reference at that point, and return such tagged value from [... new].
Now, let's say that we have some class, and some code that constructs an instance of that class, and then builds a list that references that instance from one of the elements. One way to do it would be start with an empty list, and build it element by element:
set foo [Foo new]
set lst {}
lappend lst blah
lappend lst $foo
lappend lst blah
unset foo
This produces the list {blah ::42 blah}. Because we built it element-by-element, our Tcl-with-GC tags it as a list, and uses an optimized representation for it, in which ::42 is a separate element that's known to be an object reference. Even after we remove the direct reference by unsetting foo, GC can use that metadata to trace a path to ::42 via lst, and know to keep the object alive. If we want to recover foo later, we can safely do: set foo [lindex lst 1]
So far, so good. But... that wasn't idiomatic Tcl. We're much more likely to build the list thus: set lst "blah $foo blah"
unset foo
It's still the same list {blah ::42 blah}. But, since we haven't performed any list operations on it, Tcl had no reason to tag it as one, and to switch it to the optimized representation - it's really just a string at this point. And because the elements haven't been separated out, we've also lost the "this is an object reference" tag on ::42 - there's no separate element as yet, so there's nothing to tag!So then, how would GC know to look at lst, treat it as a list, and find the object reference ::42 inside? Note that if it doesn't do that, then the object will possibly be garbage collected by the time we get to:
set foo [lindex lst 1]
Tcl finally realizes that lst is a list because of our use of lindex here, and converts it to the efficient representation that separates the elements... but at this point, how would it know that the element ::42 originated as an object reference, rather than a random string that looks like one? And even if it just pessimistically assumes that it's a reference, that doesn't really help - the object is already collected, and the reference is invalid anyway. So, we just made shimmering observable, and thus broke EIAS semantics. Our Tcl-with-GC is no longer Tcl.So, to remain true to EIAS, our GC can't rely on the presence or absence of efficient list representation for a value to decide whether to scan it for references, or not. It has to proactively treat every variable in the program as a potential list with potential object references in it - because, well, it could be; and if it is, then it's also a GC root that must be traced!
And yet this is a trivial case. We actually have to consider nested lists, and dicts, and unevaluated Tcl code stored in a string, and all possible permutations thereof...
This is how the garbage collector in Jim Tcl works. It scans the string representation of all objects [1] for the reference syntax [2]. In practice, it does not cause performance problems. Since Jim Tcl is intended as a smaller, more easily embedded counterpart to Tcl 8, nobody runs it with a large heap.
How slow is it? I have just tried a naive benchmark on a first-gen Raspberry Pi Model B. With a live list of one millions strings and 50 thousand references garbage collection took around 750 ms. On a Core 2 Duo laptop running Linux it took around 120 ms.
. for {set i 0} {$i < 1000000} {incr i} {
set v$i "<reference.<--NOT padding padding $i"
}
. for {set i 0} {$i < 50000} {incr i} {
set r$i [ref $i test]
}
. time collect 5 ;# Average of five runs
765443 microseconds per iteration
The tracing garbage collector only affects references. Normal strings are reference-counted like in Tcl 8. References are the low-level basis of Jim's OO system, which is implemented in pure Tcl.[1] "Objects" in the sense of Jim_Obj (Tcl_Obj). https://github.com/msteveb/jimtcl/blob/da293a1eef2ddd709f10b...
[2] http://jim.tcl.tk/fossil/doc/trunk/Tcl_shipped.html#_garbage...
I should have just said "all values". The name Tcl_Obj/Jim_Obj is a legacy artifact that leads to lots of confusion. Perhaps Tcl 9 will rename Tcl_Obj to Tcl_Value.
Aesthetics matter.
Unfortunately, LuaJIT is tightly-coupled to the Lua language and its code is very complex. I'd be surprised if anyone other than Mike Pall was able to retarget it to support a different language.
Also, is JIT compilation really necessary for a scripting language? Projects like the PyPy JIT for Python have seen little adoption, and many platforms (e.g. iOS and game consoles) don't support JIT at all.
Wow, the understatement of the year.
Not that difficult. See e.g. https://github.com/rochus-keller/Oberon#lua-source-and-bytec..., https://github.com/rochus-keller/Smalltalk#a-smalltalk-80-in..., or here a whole list of other language frontends: https://github.com/hengestone/lua-languages
The projects you link to all either compile to, or are written in, Lua source code or LuaJIT bytecode. That's less efficient than retargeting the LuaJIT VM. For example, LuaJIT includes bytecode instructions specifically tailored to Lua tables. It looks like these projects build, say, their array data structures on top of those existing instructions, rather than adding array-specific support to LuaJIT.
That's like saying LLVM is inefficient because all the frontends compile to LLVM IR.
E.g. the Oberon compiler directly generates efficient LuaJIT bytecode also using FFI and even reuses the VM features for it's source-level debugger. Then benchmark written in Oberon has nearly the same performance like compiled to native when it runs on LuaJIT
1. Decouple the extension mechanism. Use cffi
2. Modularize the batteries
Each alternative Python has to re-implement large swaths of the stdlib. The code is the spec in Python.
Stackless got the PyPy treatment before PyPy even existed. Besides PyPy, we now have graalpython, rustpython and ...
https://github.com/beeware/ouroboros/issues/21 points to a python stdlib I had never heard of, https://github.com/pfalcon/pycopy-lib
The point is, the core CPython devs need stop being so selfish with their toys. The greater Python ecosystem is diverse in spite of their actions not because of it.
I was under the impression that LuaJIT has support for at least one game console because it's used in games. Is that not the case?
"Due to restrictions on consoles, the JIT compiler is disabled and only the fast interpreter is built." [0]
It's 5.1 plus some of 5.2.
I've had an ok using unicode libraries but I haven't had to do much with them so far.
I moved from Matlab to Python and the biggest improvement was zero-based indexes.
Apparently this little detail is never taken into account.
You don't if you're writing low level algorithms. It's difficult for me to articulate the exact problem but I liked 1-based indexing until I needed to implement some algorithms that operated on multi-dimensional arrays element wise. Things just don't compose the right way.
It's actually the primary reason I don't make more use of Julia. I like the syntax and overall design more than Python but it just isn't worth it to me.
(If you're wondering what I'm doing in an Lua comments section, it's because I really like the ideas behind some of the Lua based languages such as Terra.)
One or zero indexing is a question about domain and communication. In signal processing, it is quite normal to need a negative index (autocorrelations f.x.). Ada (and I believe Pascal and Julia) has supported for specifying the indexing type, which can then be selected to match the problem domain.
Maybe you should try it before passing judgement.
In LUA you can use the metatable functionality to implement 0 based indexes.
In Julia you can overload the array access operation too. There a packages which allow zero based indexing. Or even StarWars based indexing including the machete viewing order. https://github.com/giordano/StarWarsArrays.jl Julia's array interface has has functionality what the first/last index is, so you can use it in other peoples code. If they haer coded 1 as first index just make a pull request to fix it.
I tried to do that a while back and kept coming across corner cases that didn’t work. Is there a complete implementation of zero-based arrays for Lua available anywhere?