Lua support in the NetBSD kernel
mail-index.netbsd.org
mail-index.netbsd.org
This is partly driven by "we need device drivers and no one understands C anymore", partly because it's cool, but mostly because Moores law is still alive and well. They are putting a scripting language in a kernel ! Forget write your app in python / perl / Ruby then optimise in C - the raw power argument is going to overwhelm us all.
I heard a stat the other day - that a greetings card, the kind that plays a silly tune, has more processing power than all computers on the planet in 1960. And we should expect a similar growth in the next 50 years. Even if that's out by two orders of magnitude just let it sink in.
I think we are at a Cambrian explosion period - where the goal is to try out every possible new body shape, as fast as possible and see which ones get the Darwin seal of approval.
New organisation forms are possible, some for the first time in human history, new ways of thinking and communicating - it's stuff like this that makes one realise the water around the frog is getting hotter.
(Still it's worth remembering that most train companies in 19C England failed and the average return for stockholders was 10% - just because the world will change beyond recognition does not mean industry stocks is a great return)
This library is not included in NetBSD as far as I can tell: http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/modules/lua/?onl...
But it's an indication that it shouldn't be included in a Kernel.
I'm sure there are sooo many people running Linux on pre-OSX Mac harddrives formatted with HFS that could be hacked with a buffer overflow.
Are you seriously arguing that buffer overflows in the kernel are not a big deal?
Mac just turn off JAVA completely in user space and NetBSD is putting Lua in kernel?
NSA: "I have a dream! Someday, a script language will be in BSD kernel, so I can ping an script into the kernel"
:-)
> This is partly driven by "we need device drivers and no one understands C anymore"
I don't buy this argument. Who are the people
* Using NetBSD
* having cutting-edge hardware for which drivers don't exist
* deciding to write device drivers
* that have device specs/docs needed to write them
* that don't know/want to use C
* that would do it if only they could write it in a scripting language?
Do these people exist? I assert that the far more common case is that the device specs, documentation, BIOS, etc. aren't accessible. If that's true, they couldn't write the driver regardless of the language, C is hardly the limiting factor here
This is spot on. When you hear NetBSD developers arguing about proposals (or commits, ex post facto, like here), there is often mention of "users". But these are mostly fictional.
I think it's safe to say most NetBSD users are comfortable working with C.
I reckon it's not much different with OpenBSD: most changes they make are for the benefit of the developers themselves, to suit personal preferences, than for the benefit of other "users". (And that's perfectly acceptable, I think.) Of course, these changes can benefit users beyond simply those who make them and those who approve them, but those users are not the ones driving the changes.
I think this is true of much open source software. Just for the sake of example, a few years ago the Gnome devs removed some printer config options because they decided the "users" didn't need so many options. It's not that Gnome doesn't have users, but the developers seemed to have some notion of a type of user that didn't really exist.
I think more often than not, fictional users are used as justification for the addition of some nice-to-have feature, but of course I'm just speculating.
If you really want to get into OS programming, I recommend you start with "Computer Architecture" by Hennessy & Patterson along with a course in C programming.
Well, it might not be such a bad thing to write your stuff in Lua and get the lights blinking as a way to work your way up to C and mainline device driver development. If the device drivers interest the novice programmer, let's encourage that. Its not like there aren't a ton of safeguards between the novice and getting their driver in the mainline.
And if it doesn't, then there's certainly a reason to optimize.
I always wondered if a lot of components like this could be either sped up, or moved out of the kernel, using a zero-copy messaging system based on splice() / vmsplice() in the Linux kernel (which works by page table remapping as far as I understand). I guess the prime candidate would be device drivers.
I was looking into embedding lua in a kernel module in Linux, just to hack around with stuff.
How did you strip out the stuff that means it won't compile for kernel space?
btw. there once was a project which added a Scheme interpreter to the Linux kernel. Sadly it's dead now: http://www.abstractnonsense.com/schemix/
I think we will go back to the days of 640k where there is a ceiling and we will just have to refine for "large" projects. Software and not hardware will be Moore's law BUT only for a extreme circumstances. The Golden Age is coming to an end in our life time.
After all, why use C if you have a working alternative? We just need real world experiments to see if Lua is a working alternative for the kernel.
I write linux drivers for a living and I must say it puzzles be a bit. There are times C falls a bit short (if only by its syntax and idiosyncracies). There are times I wish I could use something like a subset of C++ (for templates and destructors mostly) or maybe Rust these days. But a scripting language?
You can already write drivers in userland (using /dev/mem for instance). You can already write device drivers any way you like that way. One of the main limitations is IRQ handling, but do they propose handling IRQs in a scripting language? Maybe I'm too close minded (or just brainwashed by years of C) but it sounds like a recipe for madness.
I actually used lua to write a small userland test driver (it was convenient because the lua interpreter is very small and easy to compile and cross-compile).
It was quite painful really. There's no builtin bitwise operator (although it's now in the standard library), all integers are stored as double floating point values (floating point in the kernel is hardly ever needed and usually very poorly supported). Using integers and bitwise operator is like all you do in a kernel driver (ok, I'm exaggerating, but not by a lot).
I guess they might have changed the syntax a little bit (and use ints instead of doubles for storing integers) but then it means their lua is not compatible with the "real" lua so you can't prototype your code in userland with any stock lua build.
I dug up my userland test driver code, here's what the naughty bits look like:
function avi_unset(addr, val)
local v = assert(devmem.read32(addr))
v = bit32.band(v, bit32.bnot(val))
avi_write(addr, v)
end
For reference here's what it'd look like in C and its bitwise operators: void avi_unset(unsigned addr, unsigned val)
{
unsigned v = __raw_readl(addr);
v &= ~val;
__raw_writel(addr, v);
}Changing the number type in Lua is only a few #DEFINEs away. See http://www.lua.org/source/5.2/luaconf.h.html#LUA_NUMBER and the comment above it.
Regarding the lack of infix bitwise operators: usually, the functions are cached in locals. It is not as readable as C, but less bulky than what you wrote:
local band, bnot = bit32.band, bit32.bnot
function avi_unset(addr, val)
local v = assert(devmem.read32(addr))
v = band(bnot(v), val)
avi_write(addr, v)
end
If it became apparent that infix operators were needed, adding them to the VM is trivial (There's a power patch for Lua 5.1, and a patch for Lua 5.2 was recently posted in the mailing list). If you want to use the stock VM, you can still use a preprocessor (which makes things a bit more cumbersome).As for your improved code I didn't know I could do that, it does look much better indeed.
Again, maybe I'm just nitpicking but my main point was that lua, scripting language or not, just didn't seem better suited than C at writing device drivers.
That being said, having a good in-kernel REPL would revolutionize kernel hacking, but it seems very hard to do right. Lots of kernel code is very timing-sensitive and you can't afford to wait for user input when you're in a critical path. Just look at the difficulty to implement proper kernel debugging to get a taste of it.
The thing is, the NetBSD people know that and more and yet they decided to go with it. I would be very interested to know their objective/motivation. I googled a bit around and found this:
http://netbsd-soc.sourceforge.net/projects/luakern/
The page mentions some use-cases (like power management and packet filtering) but those can already easily be handed to the userland through the sysfs and tun/tap interfaces for instance. I don't get what problem it's trying to solve.
If by stock Lua, you mean lua.c, it is partly intended as a minimal demonstration of how you can embed Lua the library. The authors encourage users to modify the VM to better fit their use case, and changing the number type really is trivial.
> Again, maybe I'm just nitpicking but my main point was that lua, scripting language or not, just didn't seem better suited than C at writing device drivers.
As I said, I'm mostly unqualified to answer that. It's harder to shoot yourself in the foot when using Lua, which may make things easier at the prototyping stage.
Maybe having LPeg[0] could be of some use too.
Speed matters, of course, but I suppose that even script speed is enough for many drivers (and, just in case, we have luaJIT).
Don't sure if lua is a right choice, but it's an interesting experiment - and this is what NetBSD is good for.
You'll have to back up that assertion with some facts here, there's a whole industry and millions of lines of code that contradict you.
Is there a way to say "do this under spinlock"?
Is there a way to handle errors not mixing it with main logic? Not wrapping nearly every line with if/goto?
Is there a way to create a generic collection? No, linux list.h is not that: it requires manual initialization, it does not work with elements, but with their special parts, it's macroses requires logically unneeded information (such as 'member' in list_for_each_entry), and so on. And this is the best you can do in C.
C is too unobvious, too. Google for "wait4 backdoor".
C works, of course. It just requires too much boilerplate code and too much programmer resources.
The industry of million lines of code exists, because UNIX and its system programming language, C, spread into the enterprise when the university students brought it from their labs into the enterprise.
Thankfully the industry is slowly moving to other languages.
The kernel is a mixture of Assembly, C++ and Modula-2, with a JIT.
All user space code compiles to bytecode.
The kernel JIT compiles the applications on first execution. Afterwards the application is validated against generated native code, and re-JITed again if the bytecode has changed between executions.
So Microsoft's attempt was actually nothing new.