Lua/APR - proper standard library for Lua | Hacker News Reader

Lua/APR - proper standard library for Lua

peterodding.com

100 pointspiranha28 comments

Only downside with LuaAPR is you're back to writing Apache style servers, with an OS thread for every connection instead of using non-blocking sockets where possible (Didn't find any support for nonblocking sockets or a select\poll variant in the linked docs)

For my Lua server [1] I currently use the nixio library from the LuCI project and polling instead[2]

Although the downside of both nixio and Lua/APR library is the use of compiled C bindings to manually interact with the virtual machine.

The exciting development with LuaJIT is the high performance FFI that allows you to make system calls and create structs directly from in Lua[3], so I am currently considering switching from nixio to ljsyscall when I do a new version [4]

[1]https://github.com/davidhollander/ox

[2]http://neopallium.github.com/nixio/

[3]http://luajit.org/ext_ffi_tutorial.html

[4]https://github.com/justincormack/ljsyscall

There is a good reason for the thread per connection architecture and there is a good reason for the polling style arch. The latter being better for static content.

I'd contend the opposite, polling is better for dynamic content.

Let's say you have 1000 requests for a landing page, and the landing page has a data request that scrapes the entire database to find the popular-maximum-geo-similarity-reccomendation-zone-geo-user-thing list for the entire site, but only updates once every 10 seconds.

Ideally, you want to only send at most 1 request to your database every 10 seconds, without having to open a new connection. If you are doing a polling server and have all 1000 clients waiting for the data, you can trigger a callback to all of them when the database reports back without having to recopy data, use shared memory, do more ipc, or context switches. And for the 1000 clients to pause to wait for that single combined data callback, no extra blocking reads\connections to anything were necessary.

If by static content you are referring to file transfers, polling does not actually work on regular files in Linux. I believe they will always return ready to read even if reading would block. So usually a thread or forked process is actually required when handling files unless you dont mind disk seeks pausing the entire event loop on your server.

At the risk of feeding the trolls, I couldn't help weighing in here...

First of all: you're just straight-up wrong about file transfers and polling. The select(), epoll(), et. al. syscalls are exactly what you want for large network file transfers. This is not the same as naive "polling", but is what most implementers mean when they talk about an event/polling based server implementations.

Even "green" threading runtimes like MRI and CPython can still multiplex threads in the presence of blocking I/O calls, because the standard C syscalls can easily indicate when the main thread would block while trying to read or write.

Furthermore, a "threaded" server in no way has to allocate a full hardware thread for every client. Single-threaded, callback-based systems may fare nearly as well for I/O-bound workloads, but threads still have real value for many if not most real-world concurrent systems. A pool of worker threads can keep both the CPU and I/O subsystems maximally active without forcing a "shared nothing" architecture onto a system located on a single physical host.

In a world of multicore CPUs, limited L1/L2 cache, and fast kernel schedulers, threaded code can stand toe-to-toe with pure event loops for non-trivial workloads.

>you're just straight-up wrong about file transfers and polling. The select(), epoll(), et. al. syscalls are exactly what you want for large network file transfers.

I think there's some confusion on your end. My assertion was not that you shouldn't use polling for file transfers, but that you also needs to use at least one additional fork\ thread to do the polling in a second event loop to isolate disk-dependent connections, because polling does not work on the disk part of a file transfer, only the network part.

> standard C syscalls can easily indicate when the main thread would block while trying to read or write

Only on the socket end. Not the regular-file end. This has been the case for a long time on linux with O_NONBLOCK and regular files (aka not pipes\sockets\processes). Which OS\kernel version are you using?

Non-blocking sockets and poll are supported in the underlying APR library. There just doesn't seem to be support for the apr_poll stuff in lua-apr yet:

http://apr.apache.org/docs/apr/1.4/group__apr__poll.html

FYI: I'm considering adding a binding for the pollset module.

Awesome. I'm a happy user of lua-apr and would recommend it to anyone.

Hi all, author of the Lua/APR binding here. If you have any questions or comments, feel free to post them here. I haven't gotten a lot of feedback on the binding yet so would love to hear what people think about it!

Nice job. This fills a big gap in Lua.

I think the stewards of Lua are afraid to grow the standard library because most of Lua's early popularity stemmed from how small and embeddable it is.

The correct title for this is "Lua/APR: Apache Portable Runtime binding for Lua". I'm pointing this out because I'm tired of seeing submission titles used to make a point instead of describing what's behind the link. This mightn't be a big deal for other people, but personally, I don't like having opinions[#] forced on me like this.

[#] The opinion here is that Lua needs a standard library. Maybe it does, maybe it doesn't, but I want to come to that conclusion myself, that's all.

This is my fault really, the first paragraph of the homepage contains the following remark:

> APR powers software such as the Apache webserver and Subversion and Lua/APR makes the APR operating system interfaces available to Lua, serving as an extended standard library.

I wrote that description because I like Lua so much that I want to use it not only for embedding but also for writing standalone scripts. However I found that writing standalone Lua scripts that run on both Windows and UNIX is hard because Lua's standard library contains almost no operating system interfaces (e.g. no directory handling and minimal process handling). Also lots of third party libraries either work on Windows or POSIX systems but not both.

After reading your comments and those by compay about Penlight I realized that "standard library" may not be the best way to describe the Lua/APR binding because its focus is really on being an extended operating system interface.

This is very nice. It's good to see examples weaved into the site on how to use it. If anything, main libapr site could use something a bit better than the Doxygen output manual as reading the unit tests from the actual source is more approachable to me than their documentation.

I totally agree on the Doxygen documentation and indeed, while writing the Lua/APR binding the APR unit tests were more useful to me than the public headers and documentation.

I must admit I also looked through the implementation here and there, when the documentation didn't explain the 'grand scheme' and there were no tests to read :-(

That's a very useful library, and also great example of how easy it is to hook into C code from Lua.

Another one which I prefer personally because it's more comprehensive, is Steve Donovan's Penlight. If you have a background in Python you might enjoy using it:

https://github.com/stevedonovan/Penlight

If you are interested in hooking C from Lua or vice versa: I gave a recent talk on the ways of hooking C code from Lua at Open Source Bridge. The slides and code are here: https://github.com/philips/beaming-up-alien

Sometimes really simple examples can be useful.

Don't mean to hijack, but could I be referred to a good introduction to Lua? I keep hearing about it, but it's one of the few languages I know NOTHING about, and I have a free weekend!

The reference manual is very well written http://www.lua.org/manual/

The first edition of "Programming in Lua" is available for free online and might make for more interesting reading than the reference manual: http://www.lua.org/pil/

When I was getting started I bought the "Programming in Lua"[1] ebook, by the author of Lua himself.

There's an older edition available for free online[2].

Then there's of course the lua users wiki and its "Learning Lua" section[3].

I find the wiki a little hard to navigate so I just went ahead and bought the book. IMHO the gems in Chapter 10 (Complete Examples) and the clear explanation of coroutines are alone worth the price. The rest is a very nice bonus too.

[1] http://store.feistyduck.com/products/programming-in-lua [2] http://www.lua.org/pil/ [3] http://lua-users.org/wiki/LearningLua

Thank you (and all other commenters) kindly. I purchased the book.

If you liked Kernighan and Ritchie's "The C Programming Language", then you'll love "Programming In Lua". The first edition is available online here:

http://www.lua.org/pil/

Like K&R, it's style is colloquially terse, yet each sentence is chock full of information. It's a great book for someone who's already coming from another language like Python.

If you're into graphics programming, then http://love2d.org/ would be the perfect vehicle for you to get into Lua while you read PIL.

http://www.lua.org/pil/1.html

This isn't a language introduction, but after you learn it, I've written a bunch of posts at http://playwithlua.com, all about weekend-sized hacks in Lua.

http://love2d.org/

Another interesting combination would be:

https://github.com/justincormack/ljsyscall

this would work with LuaJIT: http://luajit.org/

The FFI in LuaJIT is kind of brilliant. Check yourself.

EDIT: with FFI in LuaJIT you can jump very comfortably between the high-level Lua code and low-level native calls. It's almost addictive. And the code is running X times faster than the Lua baseline (X between single-digit and triple-digit values).

I've made LuaJIT FFI bindings to some "C" precompiled libraries for these platforms: OSX, iOS, Windows, Linux (x86+arm)

There libraries are: GLFW, SDL, ZeroMQ, AntTweakBar, and planning on few more. There is also OpenCL and OpenGL and glu.

Please visit: http://github.com/malkia/ufo

You should not need to recompile, as I've packaged the binaries.

I have to admit, that's kind of brilliant.

Reply on news.ycombinator.com