Should Apache replace its current httpd.conf with Lua? | Hacker News Reader

Should Apache replace its current httpd.conf with Lua?

thread.gmane.org

58 pointscompay47 comments

While this may sound like it's coming out of nowhere, Lua was designed primarily for use in extension/configuration, and has been used in production this way for about fifteen years. It's a scripting engine, config file format parser, JSON-like data format library, etc., all in a library a bit smaller than SQLite.

The lua interpreter is essentially just a "code = read(); f = compile(code); print f()" loop written in the C API, with better error messages. The C API is primary, not the standalone language, and using it with existing C projects is quite easy. (As a language, it's very much like a twin of Javascript that was able to mature for another decade, rather than suddenly getting standardized. Most of the flaws Crockford describes in "Javascript: The Good Parts" were fixed in long ago in Lua.)

More specifically relevant to apache, running code in a sandbox is trivial. You can use setfenv() to control the environment, global packages, etc. available to any code. If code shouldn't access the compiler, OS or debug packages, etc., just pass it an environment without them.

There's also a chapter in _Lua Programming Gems_ (http://www.lua.org/gems/) (it's "Lua as a Protocol Language" by Patrick Rapin, no sample PDF though) that shows how to set up a simple filter at the bytecode level to block loops, guaranteeing that code will always terminate. With a few minor changes, it's completely safe to use as a network protocol language.

Surprised to see so much discussion on that. Instead, would expect someone to simply implement it, throw it on the table, and provide instructions for building it into your own apache.

If it catches on, then you have a replacement to the current config syntax.

That said, in my experience, admins are slow to change. There are still sites running Apache 1.3 out there.

True, though I think it's probably a very large undertaking for a single person. Bringing it up on the mailing list would be a good way to gain mindshare and contributors.

Open source projects usually fair better with one author until they are somewhat stable, in my experience.

Well, I think that's definitely true in most cases, especially when the direction of a project isn't really nailed down you tend to get a lot of backseat driving and discussions over things that aren't important to the original goals, and people want to make the project about X when it should really be about Y, etc... I guess you need an OSS leader to either lead through strength of character/charisma, or by having already written the bulk of the code, but code trumps everything else so it's better to have written the bulk of the code.

Point being, I think you're right and I think my initial reasoning was wrong. But all the same, a little mailing list discussion generally doesn't hurt.

Admins are not slow to change. Operations and Corporations are slow. Admins generally have more than one version of any software running. More than you can imagen.

Generally they do have development versions daily updated and built and tested under the hood. Generally they do provide support and feedback to these software by bug reports or even with changesets.

But they are not slow.. most of the time they do work out of your eye sight.

I can bet some admins are trying to see if mod_lua can be a good replacement for the current configuration.

For myself I do not like it to happen. Because most of the deployments of apache are huge and this will break too many things.

And yes, if it's working do not change it :)

(edit: gramer and typo corrections)

As someone who has been an admin (both with and without that job title), I'll say that I was slow to change, and often still am. Why? Because every change is an opportunity for something to go wrong, and in my experience, something goes mildly wrong about half the time, and catastrophically wrong way more often than it should. This is after doing all the testing that can usually be done. Either there's no testing server for this client, or you can't replicate the exact conditions of the change because you don't have another Windows 2003 SBS handy to test interaction with, or you run into lots of issues in the test and can't easily roll back to make sure that you can do it flawlessly this time...

Unless you've got an employer or client that can afford you for an additional 20 hours for each major change, everyone involved just crosses their fingers and holds their breath. I've never been an admin in an environment where there was adequate funding, though, so maybe it's just me.

Anyway, I typically put off upgrades and new installs that affected old servers (new apache modules or whatever) as long as possible.

Edit: this is why I just code, now. It's easier to write stuff that works than to find the magic this-THEN-that incantation to make someone else's stuff work. ;)

You are not telling something contrary what I tell. You are not slow to change. Operations are slow. You have some systems and operations on going but a change will drastically impact all the operations. You have changed yourself about the change. You know about it, but you do not know it's effects on the system. So you can't change the system right away.

I bet you tested most of the changes on your own time and on equipment when possible.

For myself I do.

Admin work is weird because when everything's working, no one says a peep. No, "hey, great job with that update last night which we were largely unaware of because it all happened seemlessly!". It's only when something goes wrong that anyone says anything.

The way to encourage adoption is just provide new functionality only possible using the new config syntax. Need dynamic process pool control? Here's a Lua example...

It might be a harsh stereotype, but it has been said that the Apache project is an exercise in bureaucrats doing open source. Alternatively, it's an ideal outlet for bikeshedding.

I wonder what is wrong with the current way of configuring apache. I might be biased, but I have experience with both lighttpd and apache and I always found the apache way of doing things perfectly fine and actually prefer it to the kind-of procedural style of lighty.

Let's just hope they don't switch to an XML based configuratino file. We don't need even more filler-code in our configuration files.

XML just isn't human readable (and writable! and, yes, you DO write configuration files here and then) enough to be convenient. IMHO at least.

As far as I can tell XML config files (at least those intended to be hand-edited) have largely fallen out of favor in non-Java open source projects.

I also don't mind the current config file, but I've been using Apache for more than 10 years so I've probably lost all perspective at this point. Having the config file in Lua would certainly make it eaiser to automate, validate and document though.

well... the XML approach was in fact proposed in the linked thread.

mod_perl already allows for a totally dynamic configuration which should be sufficient in terms of automation.

The documentation is quite perfect already and would IMHO lose quality if the syntax changes this drastically.

The only point I'm willing to hand you is the validation argument, although the current parser seems to be doing a decent enough job already.

No. Discussions like these feel to be motivated by artistic considerations instead of practical ones. Think of all the external documentation such a step would invalidate. Books, Tutorials, Blogs. And think of the external addon modules that would probably break because of this.

Remember the 1.3 => 2.0 switch and how long that took and consider that 2.0 actually had some real advantages compared to 1.3.

...but not every Apache server has mod_perl. Adding mod_perl to a Rails or PHP server would eat more memory than its worth.

"well... the XML approach was in fact proposed in the linked thread."

Yes, but, my point was that I see it as very unlikely to be adopted.

"The documentation is quite perfect already"

Sure it's well-written but "perfect" may be a stretch. Having a more consistent syntax would allow the documentation to be much more terse.

The current documentation reads like a manual, which is great. What's being proposed would mean that the documentation could be presented as a manual, AND as an API.

"The only point I'm willing to hand you is the validation argument"

I think you should hand me "automation" as well. :-) Writing a tool that interprets a Lua config file would be trivial. Writing a tool that interprets the current Apache config is not.

Anyway I'm not really advocating this change. I simply find the proposal interesting because I do a lot of programming with Lua and it's nice to see it get some high-level recognition.

As far as I can tell XML config files (at least those intended to be hand-edited) have largely fallen out of favor in non-Java open source projects.

I think it's too bad that people seem to be diverging towards either a crappy, barely readable data format or full-fledged programming languages. The advantage of XML is that it's a well-specified and reasonably flexible way of representing structured data. You can interpret an XML document as code (e.g., XSLT) but basically XML is just data until you say otherwise. Configuration file formats should be dumb. You shouldn't be able to program your configuration file to play chess. The only significant downside of XML is its verbose and barely readable syntax, and you don't need a configuration file that has its own byte compiler and runtime system to provide readable syntax.

While Lua is a programming language, it was designed to have a subset usable for data. It's very similar to Javascript / JSON, in that regard. (Lua grew along parallel lines to Javascript in many ways.) You can run it in a sandbox, remove recursion, etc., and its compiler has been optimized for reading large dumps of such data.

I've been using Apache for nearly a decade and it still takes me potentially hours of trial and error to figure out how to do something in an Apache config file. I'm pretty sure I'm not a stupid person, but there's something about Apache configuration that just doesn't fit in my brain. The Lua examples in that thread were instantly obvious to me, and I've never written a line of Lua in my life.

Thing being ... mod_perl let you do most of your configuration in code years ago. Yet I've never done that, and nor have the vast majority of mod_perl users I know (and bear in mind these are people self-selected to already like writing perl, so language wars probably don't make a lot of difference to my argument :).

Configuration isn't code, and I don't really think trying to make it such is necessarily a good idea - I suspect it seems superficially attractive to developers but with my sysad hat on it really doesn't seem wise.

I prefer the nginx config style over apache2. Why the hell would anyone need a turing complete configuration file syntax?

Turing completeness is a red herring and not an argument against using a language. Turing completeness is usually practically useful only to prove that certain analyses are impossible.

As to why, sometimes you want the flexibility of inferring settings from third-party data sources, or implicitly from the layout of the directory structure, or naming conventions, or any other thing you choose to mention. IMO the power of a programmable configuration language is that the configuration need not actually be written in the language, but stored somewhere else, and just retrieved using the language.

I like this:

http://thread.gmane.org/gmane.comp.apache.devel/37627/focus=...

> I would like to see the output of configuration be a struct (of structs), and the actual config files are never looked at again (until reload). At that point we don't need to care how it is built, and we can try using lua, vcl, xml, jelly, groovy, windows ini files, or plists :)

I wonder what's going on with this, I just noticed the whole discussion is from early June...

Wait, what?

key=value is really so hard that we need to replace m4 with lua? Honestly? Before you feel the need to flame, imagine if this proposal had been for implementing the config file in COBOL (a language designed to be code readable by non-technical people). Would you still believe it to be a good idea?

What ever happened to wanting to keep things simple?

m4? key=value? Have you ever either read or written an Apache httpd config file?

yep, on a daily basis. I also handle php.ini, java properties files and a slew of others. (Including the ever loving satan spawn of sendmail.cf).

All of which are simple key value pairs. Apache (and a few others) introduce scoping with psuedo XML, but again, it's not complex logic.

I brought up m4 because that was the last time someone tried to solve the configuration problem with a "simple and easy" macro language. Yeah, that worked great.

Layering Lua (or any language) as your configuration option will not simplify things. In fact, I'm willing to say that it will make it much, much harder as folks suddenly become increasingly "clever" about how they construct those config files.

I want my config files stupid. I want them heavily documented so that when I (or someone else) next has to manage them two years from now after working on multiple other projects in various languages and systems, they don't have to spend a week trying to remember what the hell this is doing, deciphering cryptic compile time errors, or wondering what the complex state machine construct someone layered on top of port allocations is supposed to be doing.

clearly you've never ran any big site (yahoo, netflix, unitedheroes, etc) where complex configs are required.

I don't see how that's clear at all. And which of those three do you work at, btw?

heh. I used to work at Yahoo, I currently work at Netflix, and my personal site is unitedheroes.net.

I also know chad and he's being a dork. (hey, i appreciate a well crafted troll.)

Tcl's nice for this kind of thing too, as it's really easy to make your own "DSL"'s (or is that passe`?)

    directory "/usr/local/foo/bar" {
        options bee bop doo wah
        if { $someotheroption } {
            order allow deny
        }
    }

Lua might or might not be better for processing requests: it's sure fast. Config files don't really need to be fast though.

Its a neat idea, but I would echo the sentiment brought up in the thread linked that casual users would have a much harder time with a lua config in comparison to the current one.

Its probably a good idea to move in this direction in the long run, but there is always a risk of displeasing your current users.

Given the complexity to which apache and its ecosystem has grown and the consequences for screwing up this configuration task, this seems quite similar to sendmail in that I think a reasonable argument can be made that casual users should not be configuring this particular system. Period.

You can make that argument, but reality is that casual users do configure apache so it ends up becoming a concern of the developers whether they want it to be or not.

Like I said, in the long run, it might be better to make the switch, but a change can cause significant risk/inconvenience to casual users or users who have such complicated configurations it would require significant time to replicate in another format.

I wonder if the people who made those comments are in denial about the complexity of Apache configuration files. Having never seen one, I Googled "apache sample config file" and I came up on this page: http://www.pantz.org/software/apache/apache13config.html

This is representative of the more complex entries:

  <IfModule mod_alias.c>

    #
    # Note that if you include a trailing / on fakename then the server will
    # require it to be present in the URL.  So "/icons" isn't aliased in this
    # example, only "/icons/".  If the fakename is slash-terminated, then the
    # realname must also be slash terminated, and if the fakename omits the
    # trailing slash, the realname must also omit it.
    #
    Alias /icons/ "/usr/local/apache/icons/"
    Alias /icons/ "/var/www/icons/"

    <Directory "/usr/local/apache/icons">
        Options Indexes MultiViews
        AllowOverride None
        Order allow,deny
        Allow from all
    </Directory>

    #
    # ScriptAlias: This controls which directories contain server scripts.
    # ScriptAliases are essentially the same as Aliases, except that
    # documents in the realname directory are treated as applications and
    # run by the server when requested rather than as documents sent to the client.
    # The same rules about trailing "/" apply to ScriptAlias directives as to
    # Alias.
    #
    # ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin/"
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

    #
    # "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
    # CGI directory exists, if you have that configured.
    #
    <Directory "/var/www/cgi-bin">
        AllowOverride None
        Options ExecCGI
        Order allow,deny
        Allow from all
    </Directory>

    #
    # "/usr/local/apache/cgi-bin" should be changed to whatever your ScriptAliased
    # CGI directory exists, if you have that configured.
    #
    # <Directory "/usr/local/apache/cgi-bin">
    #   AllowOverride None
    #   Options None
    #   Order allow,deny
    #   Allow from all
    # </Directory>

  </IfModule>

That is a language, and it's one that exists solely for Apache config files. I think using an actual programming language would be easier for most people.

http://people.apache.org/~rbowen/presentations/apacheconEU20...

Right, and it's a language with a brittle, inconsistently documented, largely ad hoc syntax. While Lua's syntax has its quirks (the syntax for lambdas is a little cumbersome, arrays are indexed from 1), that's because it is designed to be straightforward for simple use by non-programmers. The advanced features* don't get in the way if you're only using it as a log/config/etc. file format.

* Tail-call optimization, closures, coroutines, etc.

Thank you, I was looking for a link to that presentation this morning and couldn't remember who had given it.

In your example though, the complexity comes from having to know what things like "Order allow,deny" are. If you moved to lua I assume they would still use the same terminology, you would just have to represent it in lua instead of the current format. I think a file like this is easier for someone who has even a little bit of HTML experience to understand versus lua.

To me, the complexity comes from using a pseudo-XML syntax to indicate control flow and function (?) calls.

Programmatic configuration solutions seem easier to me in a Lua world than the current .conf approach, which means it would be easier to develop tools for the casual user.

Small thing, but it's "Lua", not "LUA". It means 'moon' in Portuguese, etymologically related to 'luna'/'lunar' in Romance languages. (Lua is from Brazil.)

<Perl>

#!perl

use DBI;

use Blah;

...

push @{ $VirtualHost{'*'} }, {

        	ServerAdmin     => "webmaster\@$domain",
	        ServerName      => "$domain",
                ServerSignature => "Off",
	        ServerAlias     =>  \@aliases,
	        AddHandler      =>  [qw(cgi-script .cgi)],
	
	        DocumentRoot    =>  "/home/$domain/www",

	        Alias           =>  [
			["/cgi-bin/", "/home/$domain/cgi-bin/"],
			["/stats/", "/home/$domain/stats/"],
		],

}, ...

I think that config files should be simple to write DSLs, and not programming languages. For automation what is really needed is an API exported by Apache via some kind of networking (a TCP socket could do the work) that allows to configure and change the behavior at run-time via a scripting language. This way you have the best of both the words.

Have you looked at Augeas http://augeas.net ?

Augeas is:

- An API provided by a C library

- A command line tool to manipulate configuration from the shell (and shell scripts)

- Language bindings to do the same from your favorite scripting language

- Canonical tree representations of common configuration files

- A domain-specific language to describe configuration file formats

It would be cool if we can use runtime variables in Apache's config

Haha I remember Brian from CNN he was talking about this years ago, I guess it takes a lot of momentum to change something as big as apache.

OK, for anyone who hasn't seen it before, this is what "second system effect" looks like just before it gets really ugly.

Should [X] replace its [Y] configuration with [programming language here]?

Maybe. Depends. Is there a way you can try it out?

Reminds me of Zed Shaw's "The ACL is Dead." http://vimeo.com/2723800

I can also see that turning in to a quagmire.

Reply on news.ycombinator.com