Show HN: My embeddable C/C++ webserver | Hacker News Reader

Show HN: My embeddable C/C++ webserver

github.com

84 pointsudp44 comments

This webserver is the most complete part of a high-level, cross-platform networking library I've been working on. The rest of the examples: https://github.com/udp/lacewing/tree/master/examples

ajax.cc is probably the most interesting - it's an example of long-poll AJAX with jQuery and Lacewing::Webserver as a backend. Lacewing uses epoll/kqueue/IOCP rather than a traditional select/poll approach, so it should scale quite well.

I've been developing Lacewing pretty much in private for quite a while, now - it's quite daunting when I start thinking about having to write all the documentation and such, but this morning I just thought "screw it, it's going on github, undocumented or not". It might as well be public while I'm getting it all together...

Interesting. Is there a specific purpose to this webserver?

I suppose it's just meant to be easily embeddable into just about anything. It could be used to add webserver functionality into an existing application (think web control interface for a daemon), or it could even be used to write a complete C/C++ web-application or backend.

I don't want to make any claims about the performance of Lacewing (aside from anything else, it's too early for that) but one would assume a backend written in C/C++ would be a little faster than one written in a scripting language. Although probably quite a pain to write..!

From a technical point of view, I found that most webservers were tuned for high performance on Unix, but on Windows they wouldn't run very well at all. The news earlier about Node.js having Windows support on the roadmap (which is mainly what prompted me to release Lacewing today) suggests that it's worth supporting, and Lacewing has used IOCP and other tricks on Windows from day one.

what about embedding in an iOS app? Any thoughts on this and if Apple will let is pass?

It would certainly be interesting to try - do you know if kqueue is available on iOS? If not, I guess the only thing to do would be writing a select() or poll() fallback for the EventPump class. I might have a look at it sometime.

Having said that, I'm not sure how much use the server stuff would be on iOS, and I haven't yet written the Unix implementation of the Client class (not that that will be very much work).

Yes, iOS has working kqueue.

I have some reason to believe you can get libevent working on iOS with minimal drama; I think there's at least one app that embeds it.

Which, by the way, is what I would suggest you do with your code; if you're so dependent on kqueue that it's a portability issue, you have little to lose and lots to gain by ditching the custom event loop.

I support epoll and IOCP, too, and I don't consider event handling something complicated enough to justify bringing in an external library.

I quite value the freedom of dependencies Lacewing currently has (except the obvious OpenSSL, etc - although it doesn't use OpenSSL on Windows, it uses native Windows SSL).

libevent is small enough to simply bake in and embed, which is what I usually do, but if you like your own event library that's fine.

Good Reader seems to use a web server to allow you to upload documents, so I don't think this is an issue.

I downloaded an app that seems to embed an FTP server in itself (comicbook lover), if that passes muster then HTTP might be OK?

Thanks for sharing! I was actually looking for a lightweight embeddable C/C++ web server not long ago, perhaps this will fit my needs. Only thing I'm wondering about is does it (or will it in the future) have windows and mac support? Ideally I am looking for something cross platform for what I have in mind.

(My current solution is JVM based (over JNI) and uses Jetty, but I really want to eliminate the JVM dependency - I use it for more than just the web server, but the other stuff is easy enough to replace with C++ solutions)

Yes, I've used it on both Windows and OS X, as well as Linux and FreeBSD servers.

It uses IOCP on Windows and kqueue on OS X. There's a project file in the 'msvc' folder which should build out of the box on Windows - on OS X you probably need macports with autoconf/automake/libtool to build it.

Awesome! Thanks! I will definitely dig into this more over the coming days.

You can also check out libebb - http://tinyclouds.org/libebb/

Thanks for the suggestion, I'll check it out.

Wow. Your code is a joy to read. Really clean and well-laid-out.

Thanks, that's really nice to hear! :-)

Some parts need improving, mainly because of how long I've been working on it - there were many gaps in my knowledge when I started. I don't think I've learnt so much from any project!

Thanks for sharing. How does the basic functionality compare with, say, http://www.gnu.org/software/libmicrohttpd/ or http://cpp-netlib.github.com/ ? (the latter looks like it might eventually be incorporated in boost).

+12? +8?

  char * NameDecoded = (char *) malloc(strlen(Name) + 12);
  char * ValueDecoded = (char *) malloc(strlen(Value) + 12);

  if(!URLDecode(Name, NameDecoded, strlen(Name) + 8) ||   
     !URLDecode(Value, ValueDecoded, strlen(Value) + 8))

I'm pretty sure you also don't want to accept and honor any arbitrary 64 bit value for Content-length.

And I'm also pretty sure Content-length can't be negative and doesn't belong in a signed integer.

Ouch, I honestly can't remember why I did that - I'll change it.

It's harder to spot stupid mistakes when you're working on your own - thanks!

Small potatoes but this:

        for(char * i = this->URL; *i; ++ i)
        {
            if(i[0] == '.' && i[1] == '.' && (i[2] == '/' || i[2] == '\\'))

misses the case where the double dot is not followed by a slash (eg http://example.com/foo/..). Currently you're almost certainly okay (read(2) will _generally_ (OpenGroup says "implementation specific") return EISDIR) but beware if you should implement a mod_autoindex-alike.

ps. Somebody else said it already but: nice read :)

Thanks, I changed it to block ".." in any form.

Doesn't this also read right off the end of the string?

No, because if i[1] is 0 (the null terminator), i[1] == '.' will fail.

Duh. Sorry.

No, it will hit the NULL terminator and short circuit.

Bigger potatoes: output sanitization?

Eg only whitespace is trimmed from cookies, then in webserver.Outgoing.cc:

    Socket << "\r\nSet-Cookie: " << Current->Key << "=" << Current->Value;

Lacewing splits input lines on "\r\n" but browsers often happily accept "\n" as a line delimiter (see 19.3 of RFC 2616 for Postel-esque tolerance discussion). So the client can send a request header like:

  Cookie: foo=bar\n<naughty stuff>\r\n

to manipulate the server response, cf "response splitting attack".

This is not discussed in the 37 pages(!) of the new cookie RFC 6265 :-/

It doesn't automatically send back any cookies the client sends, only the ones that have been modified by the application code (eg. via Request.Cookie("name", "new value")).

Making the parser split input lines on both "\r\n" and "\n" is a good idea. Thanks!

Done that, it's on github.

Cool.

> only the ones that have been modified by the application code

Sounds good (after a quick skim again I still can't see that but I'll take your word for it!).

I've had two attempts to build and test but unfortunately failed: currently can't build (src/unix/Client.cc:242: error: cannot convert ‘sockaddr_in’ to ‘const sockaddr’) and hello_world.c dumped core on me this time yesterday (Ubuntu 10.10).

"Work in progress" as you say... but it may just scratch an itch. I will come back again in a couple of months :) Thanks.

Someone actually just submitted a pull request to fix this on Github, which is now merged :-)

I just cloned the source and compiled it successfully on Debian, and it all seems to work alright (I can't get hello_world.cc to crash.) If it happens again, could you possibly get me a stack trace?

By the way, if you look at the cookie output code:

    for(Map::Item * Current = Client.Cookies.First; Current; Current = Current->Next)
        if(strcmp(Current->Value, Client.Input.Cookies.Get(Current->Key)))
            Socket << "\r\nSet-Cookie: " << Current->Key << "=" << Current->Value;

The strcmp() line checks the value of the cookie when received from the browser (Client.Input.Cookies) against the value of the cookie with any modifications (Client.Cookies).

In src/webserver/Webserver.Incoming.cc you have an ad-hoc HTTP parser that internally buffers partial request data and does some questionable things like recursing on each request header line.

Consider using a finite state machine parser for HTTP like this one [1] or this one [2], which will reduce your overhead, and help you push the buffering and eof problems back up to the main event loop code.

[1] https://github.com/ry/http-parser

[2] https://github.com/mongrel/mongrel/tree/master/ext/http11

Yes - the actual HTTP parsing code was written a long time ago, and I'd been meaning to improve it. Thanks for the links.

Awesome way of doing assertions! https://github.com/udp/lacewing/blob/f1c6386bf33f470ff61b275...

I remember I used to do something like that in C++ because throwing exceptions on error situations gave me zero debug information about what happened. But forcing a segfault instead you can get the stacktrace.

As someone else commented: kudos for you code! It's really clean and well laid out.

Encouraging to see more development in this space. Thanks. I'm curious, though. How do you think this compares with the mongoose (http://code.google.com/p/mongoose/) project. Other than the fact that it's C++-based what would you say are the relative advantages/disadvantages?

Hmm, looks nice :-)

At a glance, I'd say it looks both more lightweight and more high-level than Lacewing (Lacewing won't host anything at all unless you add in a handler which calls SendFile or whatever, whereas Mongoose seems to have a concept of a document root and things like that).

Mongoose seems to use select() and nothing else (I only had a quick read through the source, mind), whereas Lacewing opts for things like epoll/kqueue/IOCP, which should mean that Lacewing might scale better to many clients.

Mongoose also appears to use multithreading, which Lacewing currently avoids in favour of the reactor-esque Lacewing::EventPump. Lacewing has, however, been designed with multithreading in mind, and it's something I'll be adding as an option for the Server and Webserver classes in the near future.

By the way - Lacewing does provide a C API too, despite being written in C++.

I honestly believe that you should avoid multithread support and instead focus on the current single threaded reactor pattern. Any developer found easily just attach something like a zmq ipc queue in order to support multithreaded handling, given that the request object is thread safe. This approach would just leave you with work which ensures that the protocol parser doesn't trump any I/O overhead.

This looks very cool, thank you for sharing.

Mongoose is only selecting for the listening socket, so the comparison there isn't particularly valid. Mongoose scales as well as threads on its platform scale.

Mongoose also does a fair bit more than your software does, including SSL, CGI, HTTP Authorization, and file uploads.

Neither of them, frankly, are particularly trustworthy in their present state (Mongoose has apparently actually been audited by someone, but it too stores lengths in signed u64s). But Mongoose, for all its extra functionality, is actually somewhat smaller than yours.

I'm glad you're playing with this and wish you the best of luck, and would only urge caution about advocating that people adopt this code (or Mongoose). Most developers would almost always be better off with a better tested server, or one implemented entirely in a high-level language.

Hey, mine does SSL and file uploads. ;-)

The advice about developers being better off with a better tested server is very true, though, of course. I hope Lacewing might fit into that category one day, too.

Sorry for my ignorance, but could somebody point me to a more mature well-tested C++ webserver, as mentioned?

Have you considered using Scons as your build system, instead of Automake?

It's possible - I can't say I gave it much consideration, I just used automake because I knew about it and it's widely available.

I shouldn't need to include autogen.sh or require people to have automake/autoconf installed to build it, but I had problems using a configure script/Makefile generated on one system on another. I guess I'll sort it out when I come to understand the tools a bit better.

I'm putting together a pull request with some cleanup for Automake files. I'll take a look at porting the Automake configs to Scons next.

I think a question people should often ask themselves is: Have you considered if you need a build system at all? Is this something I can properly build with a plain old Makefile?

Reply on news.ycombinator.com