Introducing OpenBSD's new httpd [pdf] | Hacker News Reader

Introducing OpenBSD's new httpd [pdf]

openbsd.org

197 pointsfcambus154 comments

So... Apache was removed from base on Mar 14 2014 in favor of nginx, and nginx on Aug 27 2014 in favor of OpenBSD httpd.

For sysadmins who closely follow the "recommended" way, having to migrate the configurations of the http server twice within half a year must have been a frustrating experience.

Also, I wonder what "removal from base" means exactly - can you still install them (the OpenBSD-patched versions) from the ports collection or something like that?

That's correct. They are still available in the ports. If were operating a complex setup, you probably already used nginx from ports before.

Not being in the base means that it doesn't get the same security attention as its not officially part of OpenBSD anymore.

> If were operating a complex setup, you probably already used nginx from ports before.

Does "before" refer to the period before Apache was dropped from and nginx was imported to base? (Since apparently you can just use nginx from the base when it is there.)

EDIT: It just occurred to me that "complex setup" likely indicates that you would like to change some build flags, so you have to install from the ports. Stupid me.

And the nginx in ports still maintains at least one significant OpenBSD patch: chroot(2) functionality.

Apache was removed from base in 2014, but nginx was added to base in 2011.

Sysadmins had ~2 years to migrate from Apache to nginx (and if they didn't want to migrate, they could continue to use Apache from ports).

For nginx, they only had 6 months to migrate (though again, they can still use nginx from ports).

So no, they didn't have to migrate the http configuration twice a year, more like twice in 4 years, although, considering that OpenBSD only ever included into base 3 web servers, more like twice in 16 years.

Those sysadmins could still use nginx from ports. That's what I did, and will continue to do until httpd gets SNI support. Nothing broke for anyone during this time.

I've been going through the code for the last half hour and I really hope this isn't representative of what the OpenBSD group considers to be defensive C programming.

Stack allocated buffers, questionable logic and a generally terrible style as well as a complete lack of comments.

Don't take my word for it, see for yourself:

https://github.com/reyk/httpd/blob/master/httpd/server.c

The "new" is a bit off too, the copyright runs 2006-2015.

I don't see anything wrong with that code.

There is nothing wrong with using the stack for what it was designed for.

The function names are self-explanatory.

The "style" might not be yours, but it doesn't make it bad.

And as described in the slides, it was derived from relayd.

> I don't see anything wrong with that code.

Look longer.

Let me give you some examples:

Forward declarations for functions that could be avoided by re-arranging the code, using both 0 and -1 to indicate error returns from functions, bits like:

     	s = fd == -1 ? socket(ss->ss_family, SOCK_STREAM, IPPROTO_TCP) : fd;
        if (s == -1)
            goto bad;

Sure, I can read that but it takes more effort than it should and could be re-written much clearer and so on.

> There is nothing wrong with using the stack for what it was designed for.

The stack should not be used to allocate buffers intended to hold data written by other routines or read from untrusted sources for reasons that have become painfully clear over the last couple of years.

> The function names are self-explanatory.

Strong disagree about the function names being self explanatory, plenty of the functions have non-obvious side-effects. The 'style' is asking for trouble and as for it being derived from 'relayd' that pretty much confirms that this isn't something new (which is actually a good thing), but an adaptation of something old to a new role (such adaptations are security wise something to be very wary of, re-purposing old code is a great way to find out what edge cases were missed previously).

Far more symbols are exported than necessary.

> The "style" might not be yours, but it doesn't make it bad.

There's a return at the end of a function returning void for no reason.

#if 0'd old code that should simply be purged.

In the server_log code I think there may be a path to get a double free of 'ptr' where it is used first in the block with the while loop, then not reset to NULL and re-used in the second block and freed if it is not NULL (which it still is from the previous block...).

The style used obscures this possibility.

The assignment in the while boolean condition sets per to NULL at the end of the loop.

Yes, you are right. It should not take such close reading of the code though to make sure that this condition can not occur.

Since you have such valuable insights, perhaps you should discuss them on openbsd-tech.

To be fair, I think that the given block could be written better both in terms of clarity and in terms of knocking out unneeded functions.

    s = fd == -1 ? socket(ss->ss_family, SOCK_STREAM, IPPROTO_TCP) : fd;
    if (s == -1)
        goto bad;

Is the equivalent of

    if (fd == -1)
    {
        s = -1;
        goto bad;
    }
    else
    {
        s = socket(ss->ss_family, SOCK_STREAM, IPPROTO_TCP);
    }

but with one less comparison, assignment of a constant to s in the error condition instead of a variable, which is usually faster, and a more clear, explicit, layout. Now, the compiler may make these optimizations for you, but it still leaves an uglier bit of code. Tristate operators can be useful, but in some cases, like this, they can make the code less efficient.

Your refactor is wrong. You've reversed the ternary and removed the socket() return check. The consolation prize is that you've reinforced the issue of lack of clarity in the code.

You're right. I was thrown off by them putting the exceptional condition as the first part of the ternary operation. Chalk this to overly clever code.

    if (fd != -1)
    {
        s = fd;
    }
    else
    {
        s = socket(ss->ss_family, SOCK_STREAM, IPPROTO_TCP);
        if (s == -1)
        {
            goto bad;
        }
    }

or in rewriting the ternary function for clarity: // If we have a socket, use it, otherwise, try to get one s = (fd != -1) ? fd : socket(ss->ss_family, SOCK_STREAM, IPPROTO_TCP); if ( s == -1 ) // we still don't have a socket, error out goto bad;

Ternary operators should, in my opinion, be put to the 3 am test. If you think you'd get confused at looking at code that uses one at 3 am, then you've not written it properly and should clarify.

>using both 0 and -1 to indicate error returns from functions

I couldn't find any function that returns 0 on failure. Or do you mean a null pointer?

>The stack should not be used to allocate buffers intended to hold data written by other routines for reasons that have become painfully clear over the last couple of years.

You want to see a zero tolerance policy for them? I think this is a bit overprotective. Their maximum size is usually known and all called functions take a size argument (snprintf(), strftime(), strlcpy(), etc.). There's a chance one could make a mistake here, but C is the wrong language to dynamically allocate everything. The probability to forget to free it, including in every error path, is much higher in my eyes. But then again, a memory leak is probably preferable than a buffer overflow (of course, both are bad). There are also already many heap allocated buffers.

>Looking a bit longer at the server_log code I think there may be a path to get a double free of 'ptr' where it is used first in the block with the while loop, then not reset to NULL and re-used in the second block and freed if it is not NULL (which it still is from the previous block...)

I don't think so, but that code really is dodgy. Also, `ptr != NULL` is unnecessary (just mentioning since OpenBSD/LibreSSL developers like to mention that too).

> I couldn't find any function that returns 0 on failure. Or do you mean a null pointer?

server_socket_getport

An unknown address family will return a '0'.

> You want to see a zero tolerance policy for them?

No, but if you use that mechanism then it is preferable to have all the functions hitting that buffer to be visible from the scope of the declaration of the buffer. Passing it on to other libraries can cause problems when/if those libraries' maintainers mess up. So if you're defensive about this then you can't be hurt that way.

> I don't think so,

Agreed, looking still longer it looks like it will always end up with a NULL in it after the while, but that's very ugly to put it mildly.

> Also, `ptr != NULL` is unnecessary (just mentioning since OpenBSD/LibreSSL developers like to mention that too).

Yep. And there are plenty of other nitpicks like that but I don't even care that much about any of those, I mostly care about the way that the code is laid out making it an excellent place to hide some really nasty bugs.

>server_socket_getport

I found this interesting article: http://daniel.haxx.se/blog/2014/10/25/pretending-port-zero-i...

So since 0 can be okay and (in_port_t)-1 (=65535) is okay, it has to be changed to return int and take a in_port_t* to be acceptable for you. The return value may not even matter if only AF_INET/AF_INET6 as family is possible - a comment could be of help here.

> An unknown address family will return a '0'.

In this particular case, returning 0 doesn't necessarily indicate failure. Binding a socket to port 0 means you're asking the operating system to pick an available port for you, which one might argue is a reasonably safe default for unknown address families.

No, there is no 'default' at play here. If you don't know what address family is in use then you should simply abort rather than to let the end user of your product guess that the address family code is the culprit.

Let it crash, as close as possible to the point of origin of a problem is a very good principle.

> C is the wrong language to dynamically allocate everything. The probability to forget to free it, including in every error path, is much higher in my eyes.

There's a trick you can use to make remembering to free it more likely. Some people call it "RAII in C", I like to think of it as "nested allocations and errors": When you allocate something, or do something else which one, can fail, and two, must be undone (freed, closed, etc.), do the allocation and deallocation in properly nested pairs, and use gotos to jump to the code which undoes the last successful allocation if some allocation goes wrong. Basically, you arrange your code in a stack, where the most recent successful allocation is undone first, so you can neatly jump down to the code which undoes all successful allocations and doesn't try to undo the unsuccessful ones.

For example:

    int foo(void)
    {
        int ret = 0; /* Variable we return to indicate
                        success or failure. Defaults to 0,
                        which is success. */

        FILE *inf;
        FILE *outf;

        if ((inf = fopen("foo", "rb")) == NULL) {
            ret = -1;
            goto inf_fail;
        }

        if ((outf = fopen("bar", "wb")) == NULL) {
            ret = -2;
            goto outf_fail;
        }

        /* Do the actual work, now that you know you have
           all the resources you need. */

        fclose(outf);

        outf_fail:
        fclose(inf); /* If we jump here, we know we
                        successfully opened inf,
                        but not outf. */

        inf_fail: /* If we jump here, we didn't actually
                     open squat, so all we can do is return
                     the status variable we set above. */
        return ret;
    }

C++ does basically this automatically, which is called RAII, but in C you have to do it by hand.

Trying to figure out control flow with more than one goto label in a function is going to get complicated. I prefer to use `do { } while 0`:

    int foo(void)
    {
        int ret = -1; /* Variable we return to indicate
                         success or failure. Defaults to -1,
                         which is failure. */

        FILE *inf;
        FILE *outf;

        if (!(inf = fopen("foo", "rb"))) {
            ret = -1;
            return ret;
        }

        do {
            if (!(outf = fopen("bar", "wb"))) {
                ret = -2;
                break;
            }

            do {
                /* Do the actual work, now that you know you have
                   all the resources you need.
                   Use break if something fails.
                 */
                ret = 0;
            } while(0);
            fclose(outf);
        } while(0);
        fclose(inf);
        return ret;
    }

If you really must use gotos then just one suffices if you initialize all your variables to NULL:

    int foo(void)
    {
        int ret = 0; /* Variable we return to indicate
                        success or failure. Defaults to 0,
                        which is success. */

        FILE *inf = NULL;
        FILE *outf = NULL;

        if ((inf = fopen("foo", "rb")) == NULL) {
            ret = -1;
            goto foo_fail;
        }

        if ((outf = fopen("bar", "wb")) == NULL) {
            ret = -2;
            goto foo_fail;
        }

        /* Do the actual work, now that you know you have
           all the resources you need. */

    foo_fail:
        if (outf) {
            fclose(outf);
            outf = NULL;
        }
        if (inf) {
            fclose(inf);
            inf = NULL;
        }
        return ret;
    }

P.S. the return code of fclose() should also be checked, as I/O errors might only be reported on close

What's wrong with stack allocated buffers? They are faster (no malloc/free) and the stack is large enough to hold couple of KiB of data.

Dynamically allocated buffers on the stack are somewhat risky, an overflow of that buffer will allow all kinds of nasty things to happen, some of those things can lead to exploits. So if you're reading data from an untrusted source you normally should avoid reading that data into a stack allocated buffer, especially if the code that does the actual reading is not immediately visible from the scope where the buffer is allocated.

I agree that in case of buggy code stack allocated buffers are much easier to exploit, but in my post I assumed that the code is secure, i.e. the developer doesn't allow buffer overflows.

Yes, everybody always assumes that the code is secure. That's the root cause of the problem.

To assume that the people behind OpenBSD put protections against that into their C programs is one of the less wild assumptions to entertain.

On the contrary, without reviewing the code you call that is the worst possible assumption you could entertain.

But hey, why bother reviewing the code you call, the docs say that it is secure and that makes it so, right?

Calm down Jack. I wrote that it's one of the less wild assumptions: not that it's a reasonable assumption.

To trust the OpenBSD's code with regards to security is more of a safe bet than trusting a lot of other organization's code. But of course, that's a relative statement: maybe it's still a horrible assumption.

I'm pretty calm, it's just that I think that a reputation based approach to security is a bad one, especially because it creates a blind spot.

You know OpenBSD has been built with stack protection since 2003? It was the first OS to integrate what was then the outside GCC patch ProPolice (it has since been interated in mainline gcc and is the standard -fstack-protector).

Given that and the fact httpd is made by people which know and care about the issues you point, there is much street cred to be had exhibiting a buffer overflow on the stack of this program.

And no software has ever been ported over to other operating systems or flavours of similar operating systems...

I don't think they aim for portability, and give valid reasons why they don't. For instance:

http://www.openssh.com/?vm=r

""" OpenSSH is developed by two teams. One team does strictly OpenBSD-based development, aiming to produce code that is as clean, simple, and secure as possible. We believe that simplicity without the portability "goop" allows for better code quality control and easier review. The other team then takes the clean version and makes it portable (adding the "goop") to make it run on many operating systems -- the so-called -p releases, ie "OpenSSH 4.0p1". """

Obviously, I'm quoting from OpenSSH's page here, but I believe it's probably the same philosophy on why they're writing code that relies on the security properties of OpenBSD.

It's funny, sounds like a bad transplant.

I don't think heap overflows are particularly safer than stack overflows. If anything, more dangerous on most systems, since at least stack cookies are fairly common. Heap protection is harder to find.

Agreed, so in this particular case I would have allocated those buffers statically but scoped to the function (so outside the heap). Feel free to call me paranoid ;) I'd have added a sentinel as well.

Locally scoped statics are still the moral equivalent of globals, and a waiting landmine for anyone who might ever run this code in a multithreaded environment.

Yes, that's definitely a problem. But switching from single threading to multi-threading should trigger a complete review of all the underlying code. That's not a trivial change and likely to break very many things if the assumption was that the code would be single threaded. Every data structure would need to be inspected.

Static buffers have even less protections than the stack or the heap. On many Linux and BSD systems, static variables don't even have ASLR applied to them if they're in an executable rather than a library.

Yes, because as a rule it is much harder to get execution of arbitrary code out of an overflow in a buffer allocated like that. The protections are there where they are needed most.

Can you link to specific examples? I'm currently not taking your word for it. The file has comments. Maybe too few and too terse for you, but that isn't what you've said. Yes, some buffers are stack-allocated, but what's wrong with that? No unsafe functions are used to write to them. Discussing style, unless it's really terrible which is also not the case and moreover it is well documented (as OpenBSD KNF), is a waste of time.

See sibling comment.

The code is based on their relayd, so that's why it has the 2006-2015 copyright

Don't forget yet another hand-written HTTP parser, and the complete lack of a test suite.

On a more flamebaity note, I don't know why you'd even want to write something like this in C. Writing a server for one of the most prevalent network protocols on the Internet in C, in 2015, just seems like masochism. This code reinvents so many wheels for the 1000th time in C code history, it's just tiresome to read. C++ would've reduced and simplified the code substantially and there are a growing number of other fine choices these days.

The compilers for any number of other fine choices aren't included in OpenBSD, making the addition of software written using them more complicated.

What other languages does the OpenBSD people use? Shell scripts? Maybe something else?

Jeez, it was actually an honest question.

Perl is in base and used for pkg_add and the other package tools.

There are some larger perl programs (pkg) and some smaller awk/sh/etc. floating around. Not sure about using them for a web server though.

Awk for a web server? It's been done.

http://awk.info/?tools/server

Couldn't they use GCCGO since they already package GCC?

For gcc 4.2?

This is a banal comment.

This certainly is not how I would have written this code, but I will say it is very consistent in its implementation. That alone is extremely important and underrated, so kudos for that.

According to the linked PDF, it follows "OpenBSD's coding practices, and uses style(9) aka. KNF".

> The "new" is a bit off too, the copyright runs 2006-2015.

As the presentation says, it's based on relayd, so the code is not all new, a lot of it is reused.

Is there a technical reason why you would implement HTTPS in a HTTP server? If you ran a separate process on port 443 to terminate SSL connections, and then proxy that request to a HTTP server running locally, there would be better separation of concerns.

For example, this setup would mean that a security flaw in the HTTP server that allowed a user to read memory would not be able to read any private keys used in the HTTPS server.

I guess some downsides would be some extra latency while the request is proxied, and some extra memory overhead for the second process.

I'm interested in anyones thoughts on this.

I'm not into the specifics of https but i've been writing a lot of gateways for other protocols and it's never as easy as "just forward the message". Protocols some times have state that the proxy must be aware of and sometimes the forwarding is conditional which means that the proxy must understand both types of protocol and be able to act based on information in it. Say for example you want to block all requests to a specific resource, if your https server knows about this you might be able to reject the request before you decrypt everything of it.

OpenBSD has added support into libressl for privilege separated processes that hold SSL keys, any operation requiring the use of private keys such as the creation of session keys, and signing things are shuttled off via a small api to a separate process. This is somewhat analogous to what ssh-agent does for openssh clients.

OpenBSD's TLS private key consuming daemons have moved to this model or are in the process of doing so. This helps to mitigate the problem of access to process memory results in disclosed private keys, also the requirement of the daemon's user facing bits to have access to the keyfiles.

http://article.gmane.org/gmane.os.openbsd.cvs/139527/

Proxying typically loses a lot of information, like client address, client certificate info and so on. If you take care to make carry over everything transparently, it would more accurately be called privilege separation (ala sshd).

SNI, for example. If you're running multiple virtual hosts, the proxy would have to be aware of all of them. But yes, SSL termination is not uncommon, especially if you have a frontend/backend architecture.

We run exactly that setup. Apache on the front end proxying and doing SSL for a mish-mash of Java EE, .Net and native apache modules.

There's very little latency added, it allows centralised logging and TBH apache is a ton more reliable than anything else out there. Does about 2-3 million requests a day.

That's how plan9 does it. Ssl is a wrapper around whatever connection.

At work, we run stud in a freebsd jail to handle SSL termination. It uses the haproxy proxy protocol (v1) to send the client IP to the http daemon.

Downsides include: Three sockets per client connection (this gets problematic around 1M client connections). Lack of information about the SSL negotiation in the http context. Stud doesn't have the typical graceful restart options that are typical with web servers.

On the plus side, stud is a lot less code than an http server, so its easier to modify things if you need to. I added sha-1/sha-2 cert switching for example. Would have been doable in an https server too, but a lot more to avoid.

I love these guys:

https://github.com/reyk/httpd/blob/master/httpd/server_http....

Hmm, Comic Sans, weird choice for an error message.

At least they added Comic Neue. I'm a bit disappointed that a project like OpenBSD that is so vocal about free software is promoting a proprietary font like Comic Sans MS.

Not sure if whooshed, but just in case anyone else doesn't get it, you're not going to see Comic Sans unless you load it from a Windows machine or a Unix distribution that's installed the font. IOW, it's just a gentle jab at Microsoft and perhaps those distributions that install the MS fonts. Technically that is promotion in name but not spirit.

Well the OpenBSD devs seem to use Comic Sans in their presentations though

E.g., http://www.openbsd.org/papers/ru13-deraadt/index.html

You're right, they use it as a weapon of snark and for fund raising:

http://www.openbsd.org/papers/bsdcan14-libressl/mgp00025.htm...

You'll see it on the Mac, too, since the MS web fonts are installed by default.

You are correct. I assumed MS Office put it there, not even considering that Comic Sans would be on OSX by default. Hah! It's an even more sublime statement now, as only the untainted free distributions out there don't see it.. and it's also directed at web hipsters, but the result there appears to be unintentionally ironic.

They're not promoting the font at all. Non-windows users will never noticed that the defined font is Comic Sans (unless inspecting the CSS).

That's the entire joke, to jokingly annoy windows users.

I love these tiny easter eggs that the OpenBSD proyect leaves around and only Windows users will come across.

So much passive agressiveness in such a fun way!

Supports TLS using LibreSSL, serves static files and FastCGI.

https://github.com/reyk/httpd/issues?q=label%3Afeaturitis+is...

featuritis tag in die bugtracker for currently denied features. Clearly aiming for as simple as possible while being useful.

More information: http://bsd.plumbing/about.html

What is wrong with compressing static files?

That's a good question. Though a better question might be "why not serve pre-compressed static files alongside regular files?"

Compression will change your server from being network throttled to being CPU throttled. And in this day and age, we can scale CPUs more easily than we can bandwidth.

Pre-compressing is possible, but why shouldn't the http server do it for you? The server can cache the compressed files in memory and/or disk as it sees fit (although gzip was pretty cheap last time I checked). It can also deal with cache invalidation better than humans - I speak from experience :).

Very true. The difference lay in the benefit of not having to implement the compression logic as part of the server, making the server implementation much simpler.

And it's funny, Windows/IIS makes you install dynamic compression separately, in the name of security. (Course, they also make you install the tftp client separately, under the same "reasoning".)

In one of the comments on the "featuritis" tagged issues:

> I add the label "featuritis" to remind us of extra features (eg. ldap) that we reject now but might want to reinspect later.

So it's not that httpd will never be extended beyond the basics, but these issues are simply out of scope right now. I like that approach.

Slides are available here : http://www.openbsd.org/papers/httpd-slides-asiabsdcon2015.pd...

If I may take this opportunity... Does anybody know what I'm supposed to put in /etc/ssl/server.crt for SSL encryption? I have concatenated all six possible permutations of my own certificate ssl.crt, the intermediate certificate sub.class1.server.ca.pem and the root certificate ca.pem, but this gives me the error The certificate is not trusted because no issuer chain was provided. (Error code: sec_error_unknown_issuer) (my Ubuntu Chrome gives me a green lock, though). Feel free to visit my blank site https://ezequiel-garzon.net

Thanks!

Works fine here. Where are you getting an error? Maybe that client doesn't have StartCom's root cert in its trust store.

Thanks for your reply. Ubuntu Firefox, Android Chrome... scary warnings pretty much everywhere. It'd be surprising if all these clients didn't have StartCom's root certificate. If you don't mind, how did you set up your /etc/ssl/server.crt file?

Are you sure they have the same root certificate? About a month ago, I ran in some problems because I got a new certificate from RapidSSL which had been signed by their new-ish SHA256 certificate but older clients (~3 years old browsers) had only RapidSSLs old root certificate installed.

Thanks for answering. I'm not sure, but wouldn't this be a widespread issue, given the popularity (read free as in beer) of StartCom certificates? I'm pretty sure I'm doing something wrong.

If you are using Firefox: click on the padlock icon -> "more information" -> "view certificate" and then on the "details" tab. If you check "certificate signature algorithm" for each of the 3 certificates, you'll see that the root certificate as well as your own one are using SHA256 but that the intermediate certificate is using SHA1. You need to bundle the right intermediate certificate which you can find, together with additional information, on https://shaaaaaaaaaaaaa.com/

Thanks a lot! I will look into tha. As far as I can tell StartCom didn't give me a choice in that regar. Thanks for pointing me to that site. (I'd say it beats Slashdot in difficulty when you have to pass it on verbally!)

In cases like this, turn to Qualys' SSL tester: https://www.ssllabs.com/ssltest/analyze.html?d=ezequiel-garz...

It shows your server as sending only one certificate, the one with "CN=ns.ezequiel-garzon.net". It's missing the next one in the chain, "CN=StartCom Class 1 Primary Intermediate Server CA". I don't know the configuration details for the server you're using, but many servers use a separate "chain" file for the intermediates; if that's the case, you should put the main certificate in one file and the "StartCom Class 1 Primary Intermediate Server CA" in the other file.

And why it works in some browsers? Notice that Qualys listed the intermediate as "Extra download"; some browsers can download the intermediate certificate directly from the CA's web server. Some browsers cache the intermediate certificates they've seen, so if you've visited a properly-configured server with the same intermediate before, the browser will use the copy from its cache. But it's not recommended to depend on this; you should always include all intermediates.

Thanks for pointing to this useful resource. It's clear now that the question is how to make httpd send all three things. I'll bring this up in the mailing lists. Thanks again!

You can probably do it with a bundle, the same way it's done for nginx; append the CA cert to yours:

    cat your_cert.crt CA_cert.crt >> cert_bundle.crt

According to man 5 httpd.conf[1] (via man 8 httpd), it's a PEM-encoded private key. The TLS section has further configuration options.

[1] - http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/...

FastCGI: The protocol provides the single and fast interface to serve dynamic content

That's a bad choice in my opinion. Without reverse proxy functionality httpd can't match the flexibility of nginx.

This is needed at least for PHP. Granted, PHP has a built-in web server, but it's really just for testing and not meant to be a replacement for fpm.

Maybe reverse proxying is coming up next, as well as HTTP2. But once again, the goal is not to have a full-featured server that can replace Nginx. Rather something small, simple and secure.

relayd does reverse proxying in OpenBSD:

https://calomel.org/relayd.html

So there's no need for httpd to do it as well.

please do not link to calomel.org. OpenBSD takes great care to have clear, concise, accurate documentation available both on the system and on the web. link to the man page!

calomel.org is filled with...ignorance, to say the least.

I had a look at the relayd man page, but it's not actually very useful. I wanted a document that would demonstrate the use of relayd as an http proxy server. The man page doesn't even mention http. If you would like to suggest a better reference, i will happily edit my post.

Why is it a bad choice? Why does it need reverse proxy functionality? Why not use some reverse proxy software (e.g. HAProxy) in order to follow the UNIX modular principle? I'm not trying to argue here, but rather trying to understand.

Why bother with FastCGI? Why not just use reverse proxy HTTP instead? Why introduce yet another protocol when HTTP works fine?

I didn't write this, but I agree with it - https://ef.gy/fastcgi-is-pointless

If you want reverse proxy, run relayd. But slowcgi (and therefore bgplg, man.cgi, etc.) already use fastcgi.

Reverse proxying isn't the same. Information that's critical to some apps, like client certs or socket info, isn't available without hacking up custom headers and mixing that trusted data in with the untrusted client message. Due to HTTP's supremely obtuse parsing rules, this can be exploitable even when being defensive. I've personally seen this in SIP proxing (ganked the parsing from HTTP) leading to simply unparseable messages due to irreconcilable differences in deployed software.

Yeah, FastCGI introduces a whole other attack surface, but it's on a trusted boundary, at least. Mixing trust levels within content seems like one of the primary classes of security problems.

Unsurprisingly his "400-ish line C++ header that implements an HTTP server" falls far short of supporting the full HTTP spec.

For instance it looks like multiple occurrences of the same header will be ignored, and it will just close the connection instead of returning status 400 on protocol errors - in the best case.

Sure, you can try to implement just the parts of the HTTP spec that you think you'll need, and hope for the best. But wouldn't you rather use a simpler protocol that you could implement fully without having to worry about subtle errors sneaking in because you parsed the request line slightly wrong, or because your proxy decides to send two X-Forwarded-For headers instead of one.

If you're writing in Java or C++ you'll be able to find a mature HTTP library that handles all of this for you, but if for whatever reason I were writing from scratch, I'd take a limited-purpose protocol like FastCGI any day.

That's because the code was never meant to be used in production. It was meant to be used on a unix socket behind an nginx instance, which pretty reliably fixes up almost all the slight nuances and omissions in the protocol.

That said, closing the connection when you get rubbish input is generally a perfectly reasonable strategy, especially behind a reverse proxy that'll clean up after you. And in the code that is only done at all if asio.hpp claims that the underlying (tcp or unix) socket is in an error state. At which point it is impossible to reply.

As for repeated headers, according to http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 all multi-line headers must be representable as a single header by concatenating the parts with commas. So there was simply no use case for implementing this.

And FastCGI is definitely not simpler or "more limited-purpose" in any meaning of the word. At least not if you insist on implementing the full spec, which includes fun things like the authorisation and filter roles. And I highly doubt you can convince any conformant implementation to parse duplicate headers for you.

So implementing a reasonable subset - e.g. clean queries for HTTP or merely the responder role for FastCGI - is a perfectly reasonable strategy. And given all that, I will take a text-based protocol that in a pinch I can query against directly with a web browser or telnet any day.

I believe relayd is mean to serve that purpose.

Surprised that nobody has mentioned ashd yet: http://www.dolda2000.com/~fredrik/ashd/

Does anyone know if the FastCGI implementation is complete i.e. it supports FastCGI processes in all three roles; Responder, Authorizer and Filter? I've always wanted to use FastCGI more but most implementations (in Apache and Nginx at least) only support some of those roles (or require work arounds using server specific features; i.e. apache filters rather than FastCGI filters)

I think it's "it runs the software we tested with" complete.

I read the config file format and I fell in love.

I really hope this gets the portable treatment.

[update - i read the back story elsewhere and the reason is less boneheaded than i had assumed. still, i think the community needs to focus on higher priority needs and gaps]

this is the sort of thing that makes me happy i'm no longer involved in the OpenBSD world. httpd & previously smtpd are two replacements that (in my opinion) have little additive value beyond existing, community-adopted solutions (e.g. nginx and postfix), diluting effort where it is needed.

does the world need a new httpd? maybe. but the world needs other replacement software to be done first because it'll have a greater impact.

for example, OpenBSD could invest time and effort in maturing static code analyzers to assist in code audits (especially of ports).

i suspect this new httpd was done less because it was needed and more because it could be done. that's the attitude i disagree with.

Not all developers are fungible.

If the world needed OpenBSD programmers to write high-impact replacement software so badly, the world would offer to pay for it. And since evidently no one wants to pay them to do so, it seems pretty reasonable to expect that the unpaid programmers -- volunteering their free time to write programs that they give away to the world for free -- would set their own priorities, address their own needs, and invest their time and effort however they see fit.

OpenSMTPD is much, much simpler than Postfix. I can't stand Postfix's fucking 1000 line multi-file configuration.

Nginx is simple, so, yeah – no need for a replacement there.

i don't know what you are doing there, but Postfix has two crucial files, main.cf and master.cf. your /etc/postfix may have multiple files, and the default main.cf certainly has all the things (commented out), but you absolutely do not need to keep it.

if nginx feels simple it's only because it doesn't copy a good chunk of its documentation into your /etc.

> Nginx is simple, so, yeah – no need for a replacement there.

Nginx is approximately twice the size of the old Apache 1.3 based httpd OpenBSD had in base.

Nginx is an order of magnitude larger than the new httpd.

Perhaps nginx is simple from a user's perspective. But from a code/complexity viewpoint (think of all that must be read and verified to make sure it is correct, clean, simple and secure?) it is not quite so simple.

considering you wrote a book on OpenBSD many moons ago, you'd know that the only things that get attention are the itches /developers/ scratch. since you seem to be happy not being apart of the OpenBSD "world" (aka not developing on or for OpenBSD), who other than yourself really cares what you feel are higher priority needs and gaps?

As an OpenBSD/OpenSMTPD developer, I feel I should ask:

What makes you think that if I wasn't spending _MY_ spare time working on projects I like, I'd spend _MY_ spare time working on projects _YOU_ prefer ?

I work on projects because I need them and want to work on them, not because someone else feels I should do it. You say that you suspect a developer wrote code because it could be done and you disagree with that attitude, but I'd argue that there's much more to be said about the attitude of people thinking they are entitled to decide what _VOLUNTEER_ developers should do with _THEIR_ free time...

OpenBSD seems to have caught a bad case of the 'not invented here' sickness. If they didn't like where nginx was going, why not just fork it and have a working web server with a known codebase? The forks would diverge but they could still grab fixes from nginx whenever they wanted to.

See how well forking has worked in the past? Especially with large components like compilers or web servers (see old apache, old gcc). Such forks end up antiquated and more or less stagnant.

Besides, grabbing fixes just like that doesn't work so well once you've diverged enough and the upstream is constantly growing and changing. RTFA and you'll see this point discussed.

This wasn't a question about having a functional webserver or not. It was a conscious decision against running code without reading it first. The "run but don't read" strategy belongs into ports, not base. Hence nginx moved to ports.

Forking nginx was actually discussed when a proposed nginx update diff was too large for proper review. Tons of complex regex parsing code was added with nobody willing to go through it all in detail.

The forking option was quickly dismissed. The tipping point happened when someone (reyk?) pointed out that relayd had most of the guts of a complete web server anyway, including OpenBSD-style privsep which has to be bolted on to virtually all software imported from elsewhere, including nginx. It took about a week from that discussion happening to having a relayd-based functional and peer-reviewed web server in base (+ a couple of months of shaking out a few bugs and adding minor features).

> he "run but don't read" strategy belongs into ports, not base. Hence nginx moved to ports.

Why is there a webserver in base?

For the same reason the other 20+ net-daemons are: http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/etc/...

With OpenBSD, you can actually avoid the ports tree all together and still play a critical role in your infrastructure.

If you're going to distribute a system that doesn't do much of anything on its own, and delegate the responsibility of security (and choice!) to "the ports tree" you're not doing any better than Linux or FreeBSD.

The majority of users are not serving static pages, so unless they bundle their own JDK, Python, PHP, Ruby installs you have to go to the ports tree anyway?

The FAQ has the canonical answer:

Why is a web server included? It isn't needed by many people!

Because the developers want it. [1]

[1] http://www.openbsd.org/faq/faq1.html#HowAbout

What would be the elegant way to implement websockets on the new openbsd arrangement? Would it be to use relayd instead of httpd? Or is websocketd suitable for the openbsd base?

Probably to install nginx from the ports and use that. I don't think supporting web sockets would fit the scope of the base system.

Why do they even have an httpd in base? They like to say they're smaller and simpler than FreeBSD, but FreeBSD doesn't include a web server in base!

It's very handy to wrap services in HTTP to expose them outside the machine, limiting outside access to a single port with clear FastCGI delegations to specific functions. If you don't build this in, then services shouldn't assume it to be operational and can't auto-configure themselves.

One reason is that network installs require http. The base system would be unable to self-replicate across network installs without httpd.

Why not FTP? Or even TFTP?

you can do that as well.

FTP is dead.

How I wish this were true.

Yeah, don't you just love how "Enterprise" vendors make it more secure with going with ftps instead of picking a different protocol?

Why is a web server included? It isn't needed by many people!

Because the developers want it. [1]

[1] http://www.openbsd.org/faq/faq1.html#HowAbout

Did OpenBSD just standardize on an HTTP server they wrote in 2 weeks, has no tests, doesn't fully implement the spec…and then brag about it?

No.

any performance benchmarks in the wild?

Https authentication support for Subversion could be the killer feature.

Why use a name that's already in use as a general descriptor? At least the other httpds have names that can be used to differentiate them: http://en.wikipedia.org/wiki/Httpd

Yeah, it needs a real name.

In practice, everybody will just say "OpenBSD httpd" and there will be little to no name confusion.

Ugh, there's 10 syllables in that. Maybe they should call it o10d?

I'm amazed we still use configuration files like that. Why not have some capable programming language (python, javascript) handle the configuration? This would let one more naturally describe things, and will eventually reduce the possibility of obscure configuration commands that do one very specific thing, and hence lead to frustration when the user needs something slightly different. Using a programming language, one could even configure using callbacks, etc. and is much more flexible.

I'm not a lisper, but this makes me pine for some universal, maybe lisp-like language. It's a total waste of time to learn a new config syntax for each application.

This is getting pretty offtopic, but this is what GNU has tried to do with guile https://en.wikipedia.org/wiki/GNU_Guile

the ironic thing here is that Lispers fucking love sticking new DSLs wherever they possibly can. And I imagine most developers of these pet config files would argue the same thing. Unfortunately.

It was Microsoft of the 1990s that surprisingly brought some sanity to the table, with INI files. I'm glad tools like Git use this format.

Then they threw sanity out the window with the registry.

Lua is often used for that purpose. It is not a Lisp but it fulfills this role well enough.

The choice of configuration file is described in the PDF:

"The httpd.conf configuration file is using OpenBSD's modern style of a sane cofiguration language, that attempts to be exible and humand-readable. It does not use a markup with semicola or tags, just english keywords and blocks identified by curly braces (\ fg"). This is commonly called the parse.y-based configuration within OpenBSD, because it originates from the grammar and parser that was written for pf."

But still, the developer of the configuration file format has to think of all possible use-cases of httpd. It seems to me that life for the developer, as well as the user, could be much simpler with a programming language instead.

Can you give an example of a config file being used for what you're describing? Because I've always written software to treat config files like #defines that don't need a recompile, and maybe setup a variable number of strictly defined structs.

If your application is written in a language that could be used as a configuration language, then this is quite simple. I've come across it in Ruby and Python - for example, Ruby's rackup config file is Ruby code:

https://github.com/rack/rack/wiki/%28tutorial%29-rackup-howt...

However, if your application's language is not suitable for use as a config language, then this becomes significantly more painful. You have to link in an interpreter for the configuration language, and then expose your application's internals to the interpreter. The easiest this could be is probably using JavaScript as a configuration language in Java, because the Java runtime already includes a JavaScript interpreter which can reach into the Java object model. The hardest it could be is using pretty much anything with C, where you'd have to add a separate interpreter, and then expose all the relevant internals by hand.

Reply on news.ycombinator.com