Hello Lua
haxe.org
haxe.org
Daniel Stenberg is the author of curl: https://daniel.haxx.se/
Nicolas Cannasse is the (main) author of haxe: https://medium.com/@ncannasse
Both are awesome in their own right.
I wrote the Lua target announced here, and am happy to answer any questions.
How much speed?
CloudFlare's firewall system uses LuaJIT to analyse approximately 4 million GET requests per second.
(Small pause, reread)
That much speed. That was the figure in June 2015, at least.
Some of the autogenerated code looks like this: http://i.imgur.com/F5aT0NF.png (snagged via pdfimages from http://www.slideshare.net/jgrahamc/lua-the-worlds-most-infur...)
This video was where I had my major TIL moment: http://www.thedotpost.com/2015/06/john-graham-cumming-i-got-...
I also got the impression from the one other page of autogenerated code briefly visible in this video that the firewall logic is also full of regular expressions. And yet, it completes its analysis in roughly less than 1 millisecond.
It's almost a crime I haven't learned LuaJIT yet. :P
That said, looking at the slideshare link, you can see that Lua as a language has some pretty annoying syntax, so I've sorta been shying away from it.
So maybe I'll learn HaXe instead. :D
While I agree with your premise (LuaJIT is a pretty awesome project), this example means nothing in terms of performance.
— "analyse" doesn't give an indication of the workload -- it could vary from "matching against a precompiled regexp" to running advanced statistical models.
— 4 millions requests per second does not mean anything without knowing the size of the deployment. You can always throw more hardware at a "trivially" parallelizable problem.
— Throughput is only one dimension on the "speed" continuum, latency (and especially tail latency) is also very important. Some languages have very good throughput and fail hard at tail latency (Java, Go until recently). Some others have lower throughput but more predictable latency.
My understanding is that the WAF detects trivial stuff like SQL injection and XSS, and the video also mentions it detected ShellShock and so forth (yeah, this was a while ago).
"Thousands of regular expressions" is mentioned verbally in the video around ~11:25. That's where I got that from.
There are some bits mentioned about the hardware around 19:30:
- E5-2630v3 (16 cores, 32 w/ HT which is turned off), 128GB RAM, 10Gbps
- Custom built by Quanta: 4 nodes per 2U (each node is like half-width and 1U high), redundant 2xPSU in the middle
- 40 machines run Kafka storing 400TB of log data (ingest @ 15Gbps), 50TB disk per node
- 5 machines run CitusDB (sharded PostgreSQL) to power realtime graph generation, 12TB SSD per node
- 100+ machines for realtime analytics running Go (and presumably LuaJIT) - LuaJIT seems to be the realtime "hmm, interesting" flagging system, while Go is used for in-depth intelligent analysis that's (I assume) slightly behind realtime.
Also, the video is actually over here - https://www.youtube.com/watch?v=LA-gNoxSLCE
---
I'm curious to understand what tail latency is, and very interested to learn more about language latency and performance in general. What subjects would you recommend I learn about / read? (I would describe myself as interested from a hobbyist-intermediate standpont, but from a non-academic POV; I unfortunately get dismally lost with formulaic presentations, math is not a strong subject for me.)
Yup, without knowing deployment details etc this number doesn't really mean much.
perhaps you might have heard of the snabb-switch project ? which does line rate 10gbps forwarding of 64b packets on a single x86 core. entire thing is written in lua including the user land 'drivers' for Intel 10g cards :)
Edit: s/shabby/snabb (damn you autocorrect)
from 'user' perspective you are still writing lua code. lua-jit would be an optimizing compiler doing it's magic no ? get the code from here : https://github.com/snabbco/snabb
I'd heard of snabb a little while ago (I think in the context of Erlang), but this only makes it even more interesting. Thanks for the pointer!
However, all those little features add up over time and become addictive. You start really missing them when you go back to other languages. In the back of your mind you think of how you might solve something simply with abstracts or macros, or how you might easily reduce the file size or execution speed with dead code elimination or inlining.
I would choose Haxe over Perl because of Haxe's expressive type extension capabilities. I could add useful methods to string or array types, and make the resulting code short and sweet (although perhaps less readable for others).
I would choose Haxe over OCaml and I would gain Haxe's superior compiler error messages. I would also get to move seamlessly between functional and imperative programming styles, and still get to use ADTs.
On the other hand, Haxe lacks features like higher order modules and signature ascription, which give the programmer very fine-grained control over the scope of abstractions. Haxe's abstract types are more like what ML had in the 70's, before MacQueen invented the ML module system.
Arguing that ML had a precursor to Haxe abstract types is well within reason, but I'm not so sure they're the same. I'll have to do some more reading.
However, we finally don't need to support flash anymore... so now there's no reason to not just write in javascript.
My favourite Haxe feature is compile-time macros. This leads to massive meta-programming and language extension capabilities. My least favourite aspect is that it does not strictly unify or abstract every target language & runtime, so you end up writing in that language's semantics but using Haxe as syntax sugar. (Good for C++, less useful for other targets.)
In day-to-day use, how would you notice this problem? Can you give an example of what problems or hurdles this can cause?
I've only used Haxe briefly for a few small game demos, and loved it, but I've only used the JS backend.
Still orders of magnitude less difficult than rewriting the whole thing in another language, and given that a big target is games, abstracting over the platform semantics would be too costly IMO.
Overflow is normally not handled consistently[2]; it's left up to the target platform to define the overflow behaviour. E.g, JS only has one numeric type (a 64-bit floating point), and can't represent integers over 52-bits. Can be avoided by using Int32 and Int64 classes, but this reduces the code interop if you want to export a library (consumers would need to make use of the Haxe representation of a 64-bit integer).
Arrays aren't fixed sized; if you want this, you need to use Vectors[3], which aren't interface-compatible with Arrays, so you can't use them as a drop-in replacement. (Compare this to Java's Collections API.)
File system access isn't available on every platform, e.g. JS, Flash. If you target Node.js you can use the file system, you just can't use the normal file API, so you need to write stuff specific to each target.
Dates are, as always, a pain to deal with. Timezones aren't supported by the Date class[4]. DateTools.makeUtc() is not available on some platforms[5].
The built-in database API is only accessible on C++, Neko (Haxe-specific bytecode/runtime), and PHP[6]. AND it only seems to support MySQL. This has stymied my ideas about a cross-language multi-DB ORM.
Basically, you have to intimately know the gotchas of every language you target, and if you already know them you may as well write in that language.
Having said all that, Haxe seems to be greatly improving these abstractions with every release; I think given enough time, it will become an amazing tool. I'm starting to use it in place of Python as my "go to" language. For most purposes you can just use a single target; Neko works well for most functionality, but you can always target C++ for "true" cross-platform compatibility and performance, and JS for the browser.
[1] http://haxe.org/manual/types-nullability.html
[2] http://haxe.org/manual/types-overflow.html
[3] http://api.haxe.org/haxe/ds/Vector.html
[4] http://api.haxe.org/Date.html
Even then, this still leaves C/emscripten which can also fulfill those requirements, and with similar output sizes (I've tested both and compared the sizes of the resulting web versions, seems that both have an overhead of 1-2 MB for a small graphical application). It depends on what you're planning to do, but one problem I found with Haxe is that it's lacking a good general-purpose UI library that also works well on the web target. Only lower-level graphics libraries.
[1]: https://en.wikipedia.org/wiki/Curl_(programming_language)
* transpiles
Haxe does everything a typical compiler does, right up until the very last step. Not much point in correcting people about the use of the term.
I don't bother clicking that logo anymore.
I remember seeing a random website the other day, the blog had a homepage link (not in the image, as "Home" text). That was really nice.
I use it for some game development projects and its able to compile and run code on all Windows/Mac/Linux/iOS/Android (all native code) and even Javascript+WebGL.
Obviously the framework I use is targeted to games but there are other frameworks that target webdev (can target Node.js/PHP and lots of other things).
Which one is that, and how well does it do cross-platform?
If you have any questions about it, their Gitter/chat [3] is pretty active.
[1] https://github.com/underscorediscovery/luxe
I haven't used it so I'm not sure how powerful it is
> Its syntax largely follows the ECMAScript standard, but deviates where necessary.
Anyone can tl;dr the main differences to JS?
There's always the Cookbook (http://code.haxe.org/category/beginner/index.html) if you want to see for yourself.
block level scoping (vs. function level scoping)
classic inheritance (vs. prototypes)
distinction between hash and object (vs. nope)
required semicolons (vs. still uncertain on that after 10 years)
Prototypical "inheritance" allows different types of values to be assigned to the same symbol.
E.g. if you have a method bob() on a prototype, you can set object.bob = 'astring'; and it isn't an error.
Personally my favorite feature is its abstract types, which allowed me to define a 0-overhead color abstraction the was just a 32 bit RGBA integer everywhere. Stuff like this can have a sizable performance benefit for games by reducing GC pressure.
for(x in 0 ... 5) {
// increment by +1 only
}
var t = Json.parse("{ x: 1 }");
var tx = t.x; // just might be undefined
// vs working
var tx2 = Reflect.getProperty(t, "x")
myArr.sort(function(a, b) {
// just might crash
});
// vs stable
haxe.ds.ArraySort.sort(myArr, function(a, b) {
});
var v1 = 0;
var v2 = 0;
var v3 = 0;
var v4 = function(a, b) {
// also no 'arguments', that's more verbose too
};
https://github.com/andyli/jQueryExternForHaxe
new JQuery("li").hide(); // $("li").hide()
If your target is only JavaScript / NodeJS you can get a lot of the benefits of Haxe by using TypeScript, with a much larger ecosystem to help you achieve your goals quickly.
I have found the language to be a bit quirky. Its powerful but an odd combination ocaml features and actionscript cruft.
Shouts to Nicolas and all contributors!
UFront is a good example of a server-side usage for Haxe: http://ufront.net/
Any tutorial for a crossplatform app on haxe with using native features ( eg. gps)?
For example, you can use Haxe to build a native Android application using the Java target as you would do with using Java in the first place. Same with iOS (but using the C++ target)
AFAIK, there isn't any active project that offer the same as Xamarin in Haxe. You could, in theory, use Haxe to target C# and Xamarin but I haven't tested it.