Hacker News Reader

Comment by anowlcalledjosh | Hacker News Reader

Why do web servers always seem to invent their own config file format? Whilst nginx seems to do it slightly more sanely than Apache, it still doesn't use something like YAML or JSON; is there a good/obvious reason for this I'm missing?

View on HN

nginx configs can be difficult enough on their own without constraining them to formats that wouldn't allow directives that take arguments preceding a block, not to mention the extra escaping that would come with quoting already-quoted strings. This is a common formulation:

  location ~* \.(jpe?g|png|gif|ico)$ {
    ...
  }

That would be pretty messy inside of JSON or YAML. You couldn't have the location line be a key for a map/hash, because you can have multiple blocks with the same "key".

Besides, as bad as nginx configs are, just trying to understand which block "traps" a request is where you can spend most of your time; see ifIsEvil [1].

[1] https://www.nginx.com/resources/wiki/start/topics/depth/ifis...

Thanks for that comment. Running a complex nginx config with lot's of special treatment for subfolders on nginx - why and where a requests ends up in such a configuration is really difficult to tell.

However Apache just can't do some things that help you shooting in your foot. Maybe the C++ vs. Java comparison is not too far fetched.

nginx is one of those things that requires its own mentality to really get. Like SQL's set logic or CSS's forward-looking selectors, it's not quite like traditional imperative, control flow-based programming. But once you start thinking the way it wants you to think, there are benefits.

I'm in the middle of a project built entirely in nginx and it's astoundingly performant. The restrictions on what I can do in (mainline) nginx force me to think through how I structure the blocks and directives with better logic representative of a web server and not an application server, which is what I'm used to.

DigitalOcean has a decent tutorial on now nginx decides on server and location block [1]. And, related to ifIsEvil, this blog post [2] goes a little into explaining how nginx "traps" a request. If someone has better resources, I would appreciate them.

[1] https://www.digitalocean.com/community/tutorials/understandi...

[2] http://agentzh.blogspot.com/2011/03/how-nginx-location-if-wo...

I've written a tool to tell us exactly that, called nginspex...hoping to open source it shortly. We use generated configs with tens of thousands of lines, for me this was an exercise in learning what nginx is up to as well as being able to say how a request will be handled.

The tool interprets the nginx conf, rather than compiling any of it (as nginx does with the rewrite rules), makes it easy to log which lines are involved in the processing as it hits them.

> Since its public launch in 2004

YAML's initial release was 2001[0], likely not yet popular or stable enough for consideration.

JSON was likewise "released" in 2002[1] with a similar story.

In all likelihood, we're probably just lucky Igor didn't go with XML.

[0] https://en.wikipedia.org/wiki/YAML

[1] https://en.wikipedia.org/wiki/JSON#History

(s-expression (cough (cough)))

> In all likelihood, we're probably just lucky Igor didn't go with XML.

Okay I'll bite, what's wrong with writing nginx to go with XML?

XML is easily translatable to JSON, so... there's that.

> XML is easily translatable to JSON

Not really. XML is a document markup language, where order is important. JSON is a serialization format, and is a bit more bare-boned. For example, how would you transfer this XML to json, and back to XML?

    <foo bar="baz"><bork>foo1</bork><bork>foo2</bork></foo>

There are many ways to do it, you just need to pick a convention. For example, one convention is that you use "#" as key for element name, "." as key for element children, and attribute name as keys for the attributes. Following those conventions, you can produce:

    {
        "#" = "foo",
        "bar" = "baz",
        "." = [
            { "#" = "bork", "." = "foo1" },
            { "#" = "bork", "." = "foo2" }
        ]
    }

Or, for something more verbose (but maybe more intelligible), you could use "$element" as key for element names, and "$children" as key for child elements / text. (The point of choosing $ as a prefix, is it is not a valid character in attribute names, so cannot conflict with them.)

That is pretty hard for a human to parse and understand, though.

I think it could be reasonably easy to come up with a config file standard in XML or JSON, but that the format will have to rely on the strengths of each. Translating between the two just becomes an unreadable mess. If anything, if I were to write an application that allowed for either format, I would come up with a separate standard for each. More code/upkeep, but when the config files are intended for humans and to be hand-written, the focus should be on the user.

JSON is not all that readable for a human either, if that was the main complaint. I guess let me just ask this: Why does XML suck?

It's not just readability. XML has some serious issues when used as a configuration file format rather than as document markup. It has features oriented towards document markup that get in the way of writing data structures, and it lacks convenient features for writing data structures that yaml-like languages have.

First, a quick disclaimer: Apache conf format is not really XML. It leverages XML-like syntax but it's mostly not XML and avoids most of the serious problems that XML tends to bring, which I'll explain in more detail below.

XML was designed to provide structure to documents, it was not designed as a configuration syntax or a data serialization format. XML is meant for a document that already exists in its own right as a document, where the XML is added on as a layer to aid automated semantic understanding of that document's structure. It is not meant to directly represent programming data structures. As such, XML tags are designed to pop out and be visible from significant amounts of text that is not metadata. When there's more tags than text, as is usually the case when you try to use XML to write programming data structures, XML winds up being hopelessly verbose, and it's hard to avoid errors writing it (like misspelling end tags, forgetting a slash, putting end tags in the wrong order, etc.)

When not used for its intended purpose, XML winds up being hard for humans to write directly and hard(er than yaml and json) to write programs to parse it. In yaml and json, there's a standard, mostly direct mapping to common data structures in most high-level languages. With XML you have to make a lot of trivial decisions to make use of features that weren't designed for what you're trying to do. The most obvious examples are the distinction between attributes and tags: what does each one mean? What do tagnames represent? What do attribute values represent? What do attribute names represent? How do you handle CDATA that has more XML in it? XML is designed to elegantly handle something like this:

    <A>first section <B>marked up section</B> second section</A>

But this kind of structure is horrible for a configuration file, unless the CDATA sections are a parsed language of their own and parsed externally, which is essentially what Apache does. If you're trying to use XML to specify data structures like lists and trees, it's messy. Consider this example:

    <VirtualHost>
       <ServerName>my.server.domain.com</ServerName>
       <DocumentRoot>/var/lib/www/my.server.domain.com</DocumentRoot>
    </VirtualHost>

You might envision "VirtualHost" to be an item in a list, where the value of that item is a dictionary with subkeys specifying "ServerName" and "DocumentRoot." But in fact, there's more to it than that. An XML parser also gives you all the whitespace in between those two tags. You can discard it, you can write checks to ensure that nothing ever ends up in that unused CDATA area by mistake, you can write tools to generate the XML-- but no matter what method you choose it's something you have to think about that just doesn't come up if you are using a language designed for writing programming data structures instead of abusing one designed for marking up text documents.

And that example highlights another problem with XML which is a flat out lack of support for common programming data types such as lists and integers. In the example above, how would you know that "<VirtualHost>" represents an item in a list, but "<ServerName>" should be a key in a dictionary? XML doesn't help you there, every parser decides for itself.

Thank you for the extensive answer!

> There are many ways to do it, you just need to pick a convention.

The fact that you have to pick a convention is the crux of the issue. "Easy" is a subjective term, but the fact is there isn't a direct mapping between XML And JSON.

It means that if you're using XML and converting it into a programming data structure, you have to make a bunch of decisions about how to handle the XML. It means that if you're converting a programming data structure to XML, you have to have a bunch of specific rules for how to generate that XML.

With JSON, you only have to make those decisions if you need to use data structures that aren't supported by JSON.

XML is largely human-unreadable, ludicrously verbose, and has weird character escape requirements. Not a trifecta I want from my config files. I don't think anyone who's ever found themselves tampering with a complex XML file in vim ever wanted to repeat the experience.

I am highly disappointed by Nginx in three major areas:

1. They created a paid version that has some additional basic features such as cache purging, dynamic upstream name resolution, and a few others. Charge for support, charge for some fancy management interface, monitoring, but for basic features (most of them available in Tengine [0]) - thanks, but no thanks. You lost me as an evangelist! In fact, in may aspects, they now are catching up with Tengine!

2. Instead of making LuaJIT integration standard and avoid the need to escape Lua in the configuration files, they invented some subpar JavaScript. People already use Lua widely, it's fast, it's great - don't you have anything better to do than invent yet another language!? I really can't believe pragmatic people would have done this, honestly! Speaks so badly about their thought process! I know can expect anything stupid from them!

3. The configuration language is not very intuitive. If they embedded Lua, the whole configuration could be a Lua script that initializes some internal state. This would have been a dream come true!

[0] http://tengine.taobao.org

The real question is: why didn't they use something like Lua for the config file format?

Web servers can have notoriously complex configs, up to the point where designing a mini-language might a worse idea than stripping down an embeddable language, such as Lua.

Lua, especially when sandboxed, seems like a fitting configuration language: http://stackoverflow.com/questions/1224708/how-can-i-create-...

It's not just that Lua makes a good config file format, it's that serving HTTP requests is a complicated enough business with enough edge cases and corner cases that it merits a turning complete language.

I actually have some special corner case API endpoints that nginx just simply can not handle in the manner I would prefer. Further, the regex based location syntax, and even the prefix based ones, are not really what you want; you want a "Path" object that's aware of what / in a URL means, and does the right thing if you do/don't add it to the URL. (And it's not as easy as "/foo/bar/baz/?")

In Nginx, you can use Lua for complex routing and responses, with the right extensions. [1]

[1] http://openresty.org/

And we do. We don't use openresty (at least at the moment), we use a custom-built nginx that includes — among other things — the lua module. It's a great boon, but something things are still hard or impossible[1], and we don't use it for every request, opting for the location syntax, and all its flaws, for most.

[1]. Conditionally streaming an upload to a backend (i.e., if auth fails, don't stream) is impossible; nginx will buffer the entire request body, either in memory or on disk, and there is no way to change this behavior.

I invented one and wrote an article about why: https://caddyserver.com/blog/deliberate-caddyfile-syntax

(You're definitely not the only one to ask that question.)

This looks really interesting. Can't believe I haven't run across it before.

I've been toying with getting a couple home servers going (replaced home service w/ business service, just installed two router based DMZ, looking at lightweight hardware -- probably will be Fit-PC products). I was going to run a separate reverse proxy and Lighttpd or similar but a quick glance makes it seem like Caddy could be used for both and more easily. Thanks for the link.

(Edit. BTW, you're not here: https://en.wikipedia.org/wiki/Comparison_of_web_server_softw...

JSON doesn't allow comments. So it is an ok-ish config format. Yaml would have been better. But they probably wrote everything from scratch to be optimized and decided if they are writing a parser for a config file might as well invent a config file format as well :-)

Not allowing comments doesn't make JSON an "ok-ish" config format, it makes it completely _unsuitable_ for it.

But it is used though pretty oftne. That's why it is an -ish format ;-)
You can do things like
{ "comment": ["blah blah comment message"], "k1" : "v1", "k2" : "v2", ... }
Or some silliness like that.

Nginx was invented before the days of JSON and YAML...

JSON, sure, but I think YAML pre-dated the launch of nginx (if not the initial development).
https://groups.yahoo.com/neo/groups/sml-dev/conversations/to...

YAML initial release: 2001 Nginx initial release: 2004
Well ok. But I'd still argue that Nginx was invented before the popularization of YAML. I think I didn't see YAML until I got in touch with Rails in 2007, which used the format extensively. And even then, almost everyone I met didn't take it seriously, saying that it is a joke until Microsoft, IBM, etc support it.

{ "//This": "works though!" }

JSON is horrible for a configuration language. Needing to put quotation marks around every single literal is insanely irritating if you ever have to write a lot of configuration. There's also a lot of application-specific syntax sugar that you cannot do if you stick with strict JSON.
Note that it's not really fair to call Apache's configuration language "XML". Apache relies on XML for some structured data in its configuration file, but all the individual directives are parsed separately from the XML.

I agree. When people started to move towards the "better" JSON standard from XML, I was puzzled. The inability to have comments, the need to "escape" every key name, the lack (in the past) of a schema and query language standard, etc.

To be fair, while I agree that JSON is bad for config, I would argue that JSON's objects and arrays eliminate the need for DTDs and standard query languages altogether in many cases. Of course, with sufficiently lqrge and complex data stores, you'll want a documented structure and method for accessing that data no matter what format you use, but unlike XML, JSON provides structure to get started on a small scale very easily. Consider this python:
import json import sys sys.stdout.write("%s\n" % json.dumps(json.load(sys.stdin).get(sys.argv[1], None)))
This will accept JSON on standard input and will return the value of an object with the key name specified by the first command-line argument.
$ echo '{ "value1" : { "sub-value": 5 }, "value2": 99 }' | python json-test.py value1 {"sub-value": 5} $ echo '{ "value1" : { "sub-value": 5 }, "value2": 99 }' | python json-test.py value2 99
Such a program will look similar in any language with a json library that maps objects and arrays to native data structures. Granted, my simple tool will fail if the JSON isn't an object, but it's a very simple matter to extend it to handle lists and literals. For many applications this is a huge advantage over XML, especially if the point of the JSON isn't configuration but rather inter-application communication (aka data serialization).

Have you seen nginScript (1)? It's a syntax for embedding snippets of JavaScript in your NGINX configuration. This allows for scripting application logic into standard NGINX config files via a very common and simple language. There's some more details and simple examples at this blog post.
(1) https://www.nginx.com/blog/launching-nginscript-and-looking-...
(disclaimer: I work at NGINX)

Okay, but you have to escape this in the Nginx config! Same issue with Lua! And this is terrible!

there is these days _by_lua_block.

Thanks! This is really great:
content_by_lua_block { ngx.say("hello, world") }

I know it's probably not "open source-y" enough for you, but IIS just uses plain ol' XML files for configuration.

So I'd say Nginx is actually the least annoying web-server to config of those I've used:
1. Nginx
2. IIS 6 (strange metabase thing)
3. Apache (XML, mostly)
4. IIS 7, 7.5 (XML, but with some of the files scattered through your Windows directory, and also some values aren't valid in some of the files).
5. Tomcat (XML plus madness).
I'd say the thing that makes all web-servers a pain is debugging which rules are passing/failing, and where are they sending their results to.
I think the thing which makes Nginx easier than the others is probably that it doesn't try to support the 'shared hosting' scenario, which adds a lot of mess.

Is that a positive thing? Of course, like in many other cases, IIS configuration is supposed to be generated from tools, but it always needs some tweaking and XML makes it a total pain to edit.

It's not just web servers. I would say that it is because they want it to be human readable/editable, but JSON would probably be just as good in that regard.
I would prefer it if it were more like openssh or supervisor. Though I suspect those styles of configs are would make some of the more advanced configurations a pain.

Reply on news.ycombinator.com