Lua 5.3: integers, bitwise operators, and a basic UTF-8 library | Hacker News Reader

Lua 5.3: integers, bitwise operators, and a basic UTF-8 library

lua.org

176 pointsdottrap60 comments

I hope luajit will backport the utf-8 library.

A little bit OT here but I always think the situation of lua 5.1/5.2 is just like python2/python3. E.g. lots of applications' embedded lua version is still 5.1 (Lightroom, Max 7, WoW?, Codea on iPad). Some packages like Metalua still only run on 5.1.

They are much closer together though. 5.3 is a bigger change because of integers.

There are a few utf-8 libs for lua, some in pure Lua.

https://github.com/starwing/luautf8

https://github.com/luapower/utf8/blob/master/utf8.lua

I believe the main issue is there are too many such libraries and text is such an elementary piece of exchanged data.

utf-8 is trivial. There aren't major issues with processing code points. Having several libraries is fine.

Sadly though a huge Lua fan I tend to ignore any advancements not also made in LuaJIT for pragmatic reasons. Full-width 64bit integers would be very nice but LuaJIT cannot do them sadly without boxing or widening its current 64bit tagged value approach. I'd still like if LuaJIT implemented boxed 64bit integers that worked properly with its tables (eg by interning the integers, currently box identity comparisons for table lookup are awful).

I think having a world class JIT is somewhat of a distraction for regular Lua which is a world class language with an excellent interpreter.

Lua literally runs everywhere with implementations that run in or on any platform you are currently using (Java, C#, C/C++, Javascript).

It's nice to develop code with the possibility of plugging in LuaJIT though, and the language is indeed excellent - so much in fact that I can do with ignoring post 5.1 features

I can't imagine that Mike Pall won't find a way to make LuaJit fast with the current Lua features, together with full 64-bit integers.

Can you please elaborate (or give some link)?

He can if course make it fast, even very fast but conceptually its hard because he has focused so much work on making floating fast. He uses them as the basic element on the stack and hiddes pointers in the NaN. Ints dont have NaNs so that approche does not scale that well to more primitives.

There are a range of compiler tricks that can make boxed math go fast as well.

I asked Mike about this a while ago. Here's what he said about the challenges of it (this was a year ago, so I don't know if this represents his current thinking or not):

"Using 64 bit integers all-over will definitely hurt tight loops on 32 bit machines -- narrowing number ranges is very tricky for the compiler. Also, the fact that int64_t is neither a subset nor a superset of a double is worrying. This kills various optimizations and may give surprising results even in simple mixed-type expressions."

I'm a bit late to the party, but isn't development of new LuaJIT features pretty much halted? The notice on [1] saying that Mike Pall cannot accept new sponsorships for the project was added one month after the last LuaJIT release. Since then, he's still made many commits [2], but they're all bugfixes (along with a couple of contributed enhancements).

[1] http://wiki.luajit.org/Open-Sponsorships

[2] http://repo.or.cz/w/luajit-2.0.git/shortlog

What's the rationale for adding integers? I thought the rationale for not having them was to keep the language simpler, and integers 2^53 and less can be store exactly as a (64-bit) float anyway. Is the rationale for adding them so Lua can have full support for 64-bit-wide integers and bitwise operations at that width?

Yes. It is hard to interface with lots of C without 64 bit support. You can fudge it and hope it never happens but it is better to have full support.

Its one of the reasons. If you want more details you can check out this talk from Lua's main designer:

video: https://www.youtube.com/watch?v=bjqNK1jA77M

slides: http://www.inf.puc-rio.br/~roberto/talks/ws2014.pdf

The main advantages of adding integers to Lua are:

* For some things you need 2^64 bits, not just 2^53 (file handles, crypto algorithms with bitwise operations, etc)

* Some embedded systems don't have hardware support for floating point (you could compile Lua with integers instead of double but it was very hacky)

* It simplifies the C API. There are some things in the API and various internals in the implementation that benefit from separating integers and floating point numbers.

Thanks for the links -- slides were very helpful. This makes sense.

Another information from someone using Lua for years:

Sometimes the hardware/OS/drivers/something else is buggy with floats...

I once designed a game that needed integers for some things, and implemented it with Lua, on Windows machines it started to sometimes give VERY WRONG results (even in very short operations, like reporting that 5+5 == 11 or that 8+2 == 9, those are actual errors, not just short numbers to make typing easier).

Later I found out it had to do with a infamous directX bug, where it changed the settings of the floating point units without permission, and sometimes this resulted into really, really, really funky floating point math, meaning that all math made with Lua in those cases maybe could be completely wrong.

Back then when I found this (The issue, not the reason for it), many Lua devs kept telling me I as crazy or a troll, thankfully one guy in freenode.net #lua had the brilliance to tell me to check if it was the DX issue (and it was... the issue don't happened on Linux or OSX)

and integers 2^53 and less can be store exactly as a (64-bit) float anyway.

Until you divide them by three.

There is a time and a place for integer arithmetic. You could get there with liberal use of truncation, but it reads better as native integer arithmetic.

The slope is slippery, time to resist or embrace native complex, rationals, and quaternions… I only stop the list here because it is as much as I've ever used in real world code.

When did you use quaternions? (Genuinely curious!)

Quaternions are pretty common in 3D graphics.

Minecraft-like game I wrote with my daughter. I discovered their use the hard way after having suffered through without them. Any time you are freely rotating a viewpoint or an object in 3D space, you will probably want quaternions.

Do write some good test cases though that compare to someone else's implementation. I accidentally wrote my multiplication backwards and didn't catch it for a long time. I just had a vague feeling that something was wrong every time I wrote rotation code.

Quarternions are much easier then the typical vector arithmetic, that is used by OpenGL ES.

Linden Scripting Language even provides quarternions conversion with llEuler2Rot, llRot2Euler and llAxisAngle2Rot. See: http://wiki.secondlife.com/wiki/Quaternion for an introduction.

So quaternions are useful for 3d games, and games are a hot topic for Lua. But its not required, that complex numbers or quarternions are part of base language. See: http://lua-users.org/wiki/ComplexNumbers - thats easy to implement with Lua prototyping.

I wonder's what's been used in real world code more, quaternions or Peano naturals. =)

53 bits (double) is not enough to represent 64-bit integers. It was enough for 32-bit integers, but not 64-bit.

There are two main reasons: - Modern systems are now 64-bit and they sometimes have APIs that demand 64-bit integers. (For example, file APIs that deal in bytes like fread).

- Small, embedded systems that are 32-bit or less, which don't have good floating point hardware and are RAM constrained. Lua 5.3 now means double no longer has to be used to represent 32-bit integers. Instead integers can be directly (avoiding floating point for performance) and floating point sizes can be reduced down to float instead of double (reducing the size of internal objects).

Roberto Ierusalimschy (Lua creator) has given several talks about why bring integers to Lua 5.3. https://www.youtube.com/watch?v=bjqNK1jA77M

One reason would be, as already mentioned, about embedded systems not having floating point support, making number calculations slow(even with fixed point, maybe?).

A somewhat related thing(not addressed by this yet) is about Electric Imp, who had to switch to a different language from lua(which they started with). The reason was that they needed to store values for an LED array, and the only data type is tables, and all integers are 32 bit.

Not sure but they might have used user defined types, with the arrays stuff being done in C, might have been easier.

Wow - surprised it's only just getting bitwise operators ... are they not commonly required in systems that tend to embed Lua?

Thinking about Redis bitmaps, they could be very useful when available in the scripting context/

Well, Lua is only just now getting integers. Bitwise operations don't really make that much sense when you only have access to floats.

Additional clarification. The integer reason was the main reason it wasn't in the language, but various libraries have always been available to do bitwise operations. Lua 5.2 included a bitwise library as part of the standard library (which is now redundant and deprecated).

As a library, there were edge cases such as what if you used the number as floating point or what happens with 64-bit.

Now with integers, bitwise operations can be put directly into the language instead as a library.

There have been a number of bit modules developed by 3rd party developers, including LuaJIT's Mike Pall, in addition to 5.2 which had bit functions. However, the bit functions were slow and a dog to use.

Is it that hard to just make Lua another W3C standard, enable it in the browsers and make this world a bit better?

It would probably be next to impossible at this point. Javascript is too entrenched.

Would you be opposed if all browsers made Lua available for webpages?

It does not seem impossible to me to add functionality to a browser to accept some other language to run. What you're talking about is how hard it would be to convince JS developers to stop using it. But no one is talking about that.

I think that's already a feature of HTML, using the language="..." attribute of the script tag.

Believe it or not, a long time ago, Javascript was actually the less bad option in web scripting. IE used language="vbscript" to allow web scripting in VBScript, which, AFAIK, was the only other language that became popular for scripting web pages.

I would be opposed for one main reason:

The extra overhead that supporting an additional language implies with hybrid websites (even if you disregard JS completely, external code like advertisements would introduce it). People can claim that the overhead is negligible with today's hardware, but this is not always the case, especially when you are trying to get as much performance as possible for a web-based game engine.

We should be working hard to kill Flash and Java completely from the web, substituting them with modern and standard web technologies.

I don't mind an obscure corporate website requiring their employees to install Flash or Java, but I do mind being bombarded with "please install Flash with Mc Affee unless you opt-out", because we are too lazy to display video without Flash.

A better alternative to have Lua in the browser (at least in my opinion) is to support Mozilla's asm.js initiative, and compile Lua to LLVM/JS.

> The extra overhead that supporting an additional language implies with hybrid websites (even if you disregard JS completely, external code like advertisements would introduce it). People can claim that the overhead is negligible with today's hardware, but this is not always the case, especially when you are trying to get as much performance as possible for a web-based game engine.

If you're worried about performance, why are you advocating asm.js? That's always going to be less performant than direct browser support for a language.

> We should be working hard to kill Flash and Java completely from the web, substituting them with modern and standard web technologies.

The JVM (on which Java is only the most prominent of many languages) is standard and modern. So was Dart. So is Lua.

> If you're worried about performance, why are you advocating asm.js? That's always going to be less performant than direct browser support for a language.

TL;DR: You won't get Lua support in your favorite browser if you don't build web projects with Lua today, and you can only use Lua via asm.js today.

non-TL;DR: because of 3 reasons:

1) JS is not going away, and you are not going to convince the main browser vendors to use Lua, this makes Lua via asm.js the only realistic way to use Lua in any browser today.

2) VMs like OdinMonkey use a different strategy when they detect the "use asm" pragma, and this allows Lua via asm.js to run faster than JavaScript, but this only applies to Firefox.

3a) In order to convince browser vendors to take Lua seriously, Lua has to be massively adopted in the web.

3b) In order to help Lua to become massively adopted in the web, it needs to run fast everywhere, not just in Mozilla's browsers.

3c) In order to help Lua to run fast everywhere, the teams behind other JavaScript VMs (V8, JSCore, Chakra, etc.) need to be convinced that the web needs asm.js.

3d) In order to convince the people of other JS VMs, we need to build more projects which use asm.js.

> > We should be working hard to kill Flash and Java completely from the web, substituting them with modern and standard web technologies.

> The JVM (on which Java is only the most prominent of many languages) is standard and modern. So was Dart. So is Lua.

The JVM, nor Dart, nor Lua are web standards, also, I expect from you to not be nitpicking my comment, I was referring to the frontend (ie. applets / flash player).

> The JVM, nor Dart, nor Lua are web standards

What does that even mean? They're standardized in exactly the same way as ECMAScript. Every standard isn't a web standard until it is. Javascript wasn't any kind of standard at all when Netscape introduced it, nor even when Microsoft adopted it too.

I don't think asm.js is a very good fit for garbage collected languages. You really want to integrate with the browser's existing, native GC instead of implementing your own GC on top of a huge typed buffer in JavaScript.

It wouldn't be a big enough improvement to be worth it. It's overall less crazy, but has enough of its own warts.

OH YES IT WOULD BE!!!1

At the very least, maybe JavaScript can take a page from this release of Lua and finally give us integers.

Whether its difficult or not may be besides the point -- Not all languages want to take on the responsibility/burden of being a standard. Not being a standard allows flexibility, in theory.

Well as for simply supporting Lua in browsers, aside from the obvious Emscripten/ASM.js way, which would work for practically all browsers which support JavaScript, Lua can run as a PNaCl app in Chrome/Chromium, see: https://chrome.google.com/webstore/detail/lua/nkcnhkobblkmia...

Of course this is only a proof-of-concept which just runs a Lua REPL in the browser. I wonder if it is possible (and if so, how hard it is) to make the browser run <script language="lua"></script> code.

You need to define a DOM interface to do anything useful, which is a fair amount of work.

lua.vm.js can manipulate javascript objects. the only outstanding issue is tracking garbage (lua closures can leak into javascript).

In regards to actually doing the DOM work, craig barnes is doing great work over with lua-gumbo: https://github.com/craigbarnes/lua-gumbo

It supports script tags: https://kripken.github.io/lua.vm.js/repl.html

Code that does it: https://github.com/kripken/lua.vm.js/blob/master/src/lua.js#...

fwiw, the only reason Lua is a decent language nowadays is that it was able to evolve in a backwards incompatible way. If we had frozen the language back in the 90s like they did with JS then Lua would be super shitty (no lexical scope, coroutines, metatables, etc etc)

The main reason Lua stayed small, is that its developed as an open source, closed shop software. I'm using my own PHP microframework called DataFace for over 20 years. Nearly no two applications share the same DataFace code, each of them is a little bit different. My current DataFace can create dynamic AJAX forms, but its still less then 1000 lines, because its my open source, closed shop code, where I'm able to cut dead branches, to grow new flowers.

I stopped using Turbo Pascal, C++ or TCL, because backwards incompatibility. A language implementation like Turbo Pascal, C++ or TCL is outside of my application scope. Any operating system distribution update could break compilation of my code anytime. It did and I decided for my self to avoid those language.

But its a total different case with Lua, or my own DataFace microframework, because those are inside of my application. The Lua interpreter code is not some library coming from operting system distribution, but a shared lib, I ship with my code, and a source code tree within my own application. My old applications do not need to care, if something in Lua 5.3 breaks them. But I can investigate the new features for next application.

This is great news .. I'm looking forward to getting this integrated into the latest MOAI client. The lack of utf-8 support up to now was definitely a little frustrating .. going to be great to get that integrated and functional.

I wouldn't call it "utf8 support". It's barely a starting point. string.xxx functions still work with single bytes, and there aren't equivalents on utf8.xxx side.

Fairly good news.

Why so many down-votes? The main changes can't be really backported to LuaJIT as far as I know. Otherwise I'd write "great news", then your down-vote warriors would be happy.

The HN community takes a rather strict approach when moderating comments that contribute noise to the conversation. "Nice article!" comments are routinely downvoted. As is sarcasm, witticisms, memes, references and other styles of comments that occur frequently but do not contribute to the discussion. It's a knowingly doomed attempt to hold back the flood of noise that covers Reddit.

Well... I'll make sure to elaborate a little more next time. Edit: Thanks for the explanation.

Wowwwwwwwwwww

Disappointing that Lua is going down the same broken road of having limited integer types. But, then again, Lua was always about a small, simple base than correctness at all costs.

I think you could replace the integer type with an unbounded one fairly easily. Lua is designed to have changes made and it already allows other sizes of integer and float to be used.

I'd love to be proven wrong, but all Lua lets you do is typedef to another built-in type. Swapping in something like Python's long type would not be a simple matter.

I think you would probably want to do it as a userdata and not as a language mod.

Roberto addressed the idea of a infinite number type, but it has a lot of downsides which are very un-Lua like, i.e. complicated, slow (no real hardware), and interoperability problems with the C API (which is very important to Lua).

video: https://www.youtube.com/watch?v=bjqNK1jA77M slides: http://www.inf.puc-rio.br/~roberto/talks/ws2014.pdf

Ah yes the C interface would be a problem.

Reply on news.ycombinator.com