Fast Lua Serialization (2023)
artemis.sh
artemis.sh
Lua itself can load millions of records per second from disk.
https://www.lua.org/history.html covers the early history.
I know there are Fortran shops that make profitable use of Lua as both an embedded extension language and as a glue language. But Fortran users do so in the same way Lua is used with C, C++, and other languages. That merely reflects that the Lua creators had a good understanding of real-world software ecosystems, and successful in how they shaped the design and implementation to minimize impedance mismatches between Lua and other environments, across syntax, semantics, API, and various implementation details that are invariably ignored or overlooked in comparable languages.
[1] For those who like reading, see https://www.lua.org/docs.html#papers
[EDIT] sorry folks! Must have misunderstood something I read a while back. Trying to dig it up now. But I could’ve sworn I’d read somewhere that this at least wasn’t suggested, and I thought it was also removed from the build in a 5.4 patch. Will circle back if I find what I’m looking for.
I know that the Lua team has internal private repositories, and that luac is developed in a separate repo for them. I've seen occasional reports on the Lua mailing list that luac.c is forgotten or not updated in non-official releases. That is because those source drops didn't go through the full official release process which includes merging in from their luac repo. Maybe you are confusing these intermediate source drops with deprecation? If there is deprecation, I would like to see the details on that. I presume they would be introducing some kind of replacement that addresses all the real world use cases that rely on the abilities of pre-compiling to Lua bytecode.
Maybe you’re thinking about the use of untrusted bytecode? Loading it is strongly discouraged because that’s insecure - Lua has nothing like a bytecode verifier.
Yes it's safe as long as you're serializing correctly (which isn't very hard).
Lua has strong sandboxing capabilities (i.e. ability to limit and control the environment visible from a chunk of code), but the Lua authors years ago explicitly disclaimed the ability to sandbox untrusted code. The compiler and runtime are not bug free. They don't have the resources, notwithstanding that compared to any other non-formally verified implementation their track record is pretty decent, even compared to past and current efforts from Sun, Microsoft, Mozilla, and Google. If you want to run untrusted Lua code, Lua sandboxing should be just the first of multiple line of defense, just as modern web browsers rely on various operating system mechanisms to constrain breakouts.
Please read the comment again. They said the person loading the data is the same person that serialized it. The data came from an untrusted source prior to being serialized.
For example, consider a guestbook program. People send it untrusted text and then the program serializes it into a database. Reading the database back is safe.
Lua has strong sandboxing capabilities (i.e. ability to limit and control the environment visible from a chunk of code), but the Lua authors years ago explicitly disclaimed the ability to sandbox untrusted code. The compiler and runtime are not bug free. They don't have the resources, notwithstanding that compared to any other non-formally verified implementation their track record is pretty decent, even compared to past and current efforts from Sun, Microsoft, Mozilla, and Google. If you want to run untrusted Lua code, Lua sandboxing should be just the first of multiple line of defense, just as modern web browsers rely on various operating system mechanisms (process separation, filesystem to constrain breakouts.
But note that this is a tool used for laying out bytes in a certain order (like a C struct), not a general "serialization framework" for traversing data structures and whatnot. But it can be the building block for one.
Also included in the package is a striped down version of my CBOR encoder that doesn't support the CBOR tagging that's about 1/3 the size, so I wonder how that compares against lua_cbor.
New versions are schemaless and I simply stayed in the old one. 10 years using it happily
Had to tweak the java version for managing strings over 32 bytes though, it was failing from flutter now
Can you point me to some information on how that older format worked?
The article doesn’t expand on this - what are they referring to? Are lua tables arrays?
Serializing something common like tables 2x seems like a massive overhead.
I can't comment on the efficiency of whatever this implementation is doing, I haven't read the code. It does sound a little expensive. Also it's not always 100% clear which the correct mapping is, unless the developer explicitly picks one. For example, a serializer has no way to determine whether '{}' is intended to be an empty array or an empty hash.
Another common gotcha, e.g. when translating to JSON, is that Lua does not restrict the types of keys (while JSON objects only support string keys), so boolean true/false are possible keys, for example, which cannot be translated to JSON directly.
for k, v in pairs(t) do … end
for i, v in ipairs(t) do … end
Merged dict and array looks neat on syntactic level but really is pita to deal with. It’s a questionable design in itself, and then you go outside and realize that you’re incompatible with everything else cause {} and [] are two different things.
Abstraction seemed neat at first but completely fell apart in the details. Many such cases. Looks like a true vector type is a good idea after all.
> Merged dict and array looks neat on syntactic level but really is pita to deal with.
It doesn't have to be that way.
An insertion ordered hash table also contains an array part, it's just that it's completely internal to the implementation. It behaves just like a normal hash table, but internally the keys hash to indexes in the array part. The order of the hashes is undefined but the order of the key-value pairs in the array part is defined to be the order they were inserted into the table.
It's really neat and it seems to work really well. It behaves exactly like an array if used like one. It definitely isn't an array though: integer keys are not offsets, they are hashed like every other key type. This doesn't seem like a huge problem to me. The lack of this property seems to be the root cause of the complexity of Lua's tables.
It's tempting to think of these insertion ordered hash tables as some kind of superset of arrays. Just like it's tempting to think of hypergraphs as supersets of graphs, of graphs as supersets of trees, and of trees as supersets of linked lists. I haven't learned of any cracks in this abstraction, at least not yet. The reason to prefer less general data structures always seems to be performance, not correctness. Offsetting a pointer is faster than hashing some bytes, indexing two arrays, comparing keys for equality and then offsetting a pointer.
which in Lua are indistinguishable, both in theory and practice, from a newly created empty array: {}
Explained by Hettinger here:
https://mail.python.org/pipermail/python-dev/2012-December/1...
https://github.com/lone-lang/lone/commit/da13b67a61248b832b2...
I didn't know he had done a presentation about it, thank you for posting that video!
Should accessing an absent record really be an _error_? Shouldn’t it just be a “None” or “Missing” option or something like that instead? It doesn’t seem to me like indexing a set with an absent key is really an error, it’s just not a value.
Ideally yeah you'd return the Nothing side of an Option type but that's not representable in lua's type system. Returning an {:ok, data} tuple erlang-style is a pretty solid middle ground, and that is a common convention in lua with multiple return of ok, result.
But throwing an error has advantages too. There is a potentially important semantic difference between an unused key and a null data. The difference becomes especially important when you go mixing data types IMO. I'm begrudgingly fine with a hashmap returning null for unset keys, but not with an array returning it for an out of bounds index. With lua's approach you can't easily differentiate these things and it does cause serious problems. Everyone hated this in php, I don't see why it's such a popular choice when lua does it.
Looking at python, it’s more often and irritating to convert to .get() after an error than receiving an error and realizing it was helpful. Code that optimistically assumes non-null results rarely survives few next lines after getting nulls anyway.
There’s no difference in “nil” vs “exists but nil”, because nil means “doesn’t exist” by definition. The gotcha is in # operator which when applied to a table may return incorrect “length” of its array part if it has holes in it. For example #{1,2,nil,4} can be 2 or 4.
It happens because Lua arrays are not “just dicts with integer keys” as someone said above. A lua table impl has two parts: one for dict and one for array. # basically returns “impl(t).array.length”. This array part gets populated and depopulated heuristically creating non-determinated outcomes for #.
This has nothing to do with bounds or existence tests. “Really null” is an attempt to have nil in an array, which Lua doesn’t by design.
It's why almost every serialization library will have userdata values to represent "null" and things like empty arrays instead of empty objects.
Still love Lua though.
Apart from names, I find explicit models easier to work with and they allow for better designs. E.g. in Lua if t has mt.__index = {x=1}, then t.x == nil is non-representable, because t.x = nil deletes t.x and now .x proxies into index, which returns 1. In javascript, since existence is a separate thing, you can assign literally any value to t.x and it will not be proxied to a prototype until you delete t.x. This has a whole cascade of consequences that make your meta life easier.
Undefined vs null is still a mess though, but not because the two exist, but because standard APIs use and return them mostly arbitrarily.
I'm fine not having explicit property existence, and prefer throwing out the complexity needed to support it. Maybe I'm underestimating the usefulness, but I think in most situations I'd be happy with either setting to false or using a plain old initial value instead of __index, and in the rest of cases I could get the same effect by having __newindex disable __index for that key.
The best Lua way to do it imo is to not do it at all and use plain tables for data. Reinventing something that was omitted by design, and broken by that same design, is futile. Lua is as it is. You use it as is, or you fight with it, or you choose something else.
Okay, I guess I should have elaborated. I'm talking about the situation where you'd use "in" because you want properties on the prototype to count. HasOwn would not work there. There is no way to default via prototype to "yes it exists" but override that with "no it does not exist".
> The goal is not to find a universally infinite looped superidea, but to have one that is practical and no less.
I find nil plenty practical when I use Lua. Even when working in Javascript, I have never felt the need to override a prototype value with null or undefined.
myTable[1] = "red"
myTable.name = "Color Codes"The issue is not knowing if the table can be encoded as an array, since that's more compact in many encodings.