Show HN: A little web server in C
github.com
A little web server written in C for Linux.
Supports: CGI, Reverse Proxy.
Single threaded using I/O multiplexing (select).
github.com
Supports: CGI, Reverse Proxy.
Single threaded using I/O multiplexing (select).
And therefore, an HTTP/2 server calling itself compatible with HTTP/3 clients is OK
You need to implement a variety of ciphers, manage certificates with expiration dates, etc... And if you wanted to implement all that yourself, people will yell at you for doing your own crypto. So yeah, you need a library. But not just that. You need a way to update your certificates, so you can't have a package (or even a single executable) that you can just run and have a server that serves static pages. You could make a self-signed certificate that lasts a thousand years, but good luck getting it accepted.
In classic HTTP TLS was layered beneath it, providing important degrees of freedom, including the freedom to not use TLS, which can be especially important for experimentation and development.
Prediction: If HTTP/3 manages to substantially replace classic HTTP+TLS, QUIC is destined to become a kernel-provided service like TCP, shunting all that complexity behind an OS abstraction and freeing user space. The fact QUIC uses UDP is an important aspect here because a performant userspace QUIC stack conflicts with classic, high-value abstractions like file descriptors and processes; abstractions which make it viable (i.e. cheap) to have a rich, diverse ecosystem of languages and execution environments in userspace. More importantly, HTTP will have come full circle.
Do you mind expanding on that with a sentence or two?
"File" abstraction layer has predefined start, end, length, order, and so on. You need some buffering magic layer on top of UDP that provides facilities mandated by file abstraction layer.
IOW, with QUIC a file descriptor no longer represents a particular server/client connection, let alone a particular logical stream. From a server perspective, a user space QUIC stack is effectively equivalent to a user space TCP/IP stack. (Relatedly, user space QUIC server stacks are less CPU performant than HTTP stacks using TCP sockets, unless they use something like DPDK to skip the kernel IP stack altogether, avoiding the duplicative processing.)
The BSD Sockets API for SCTP (the original, non-UDP encapsulated version) permits 1-to-1 socket descriptors for associations--logical streams within a connection--in addition to an API for 1-to-many--1 socket descriptor representing a connection, over which you can use recvmsg/sendmsg to multiplex associations (logical streams). So you have 3 options for using SCTP from user space, from a dead simple, backwards compatible sockets API that matches how TCP/IP sockets work, to the low-level QUIC model where user space handles all routing and reassembly. I would expect kernel-based QUIC to work similarly, except combined with kernel-based TLS extensions, which is already a thing for TCP sockets.
What do you mean exactly? UDP is not a stream protocol, it is an unreliable datagram one. A single read will return exactly one UDP datagram.
It is TCP where a read will return incremental data which is not necessarily aligned to what the other end wrote.
Of course QUIC builds a reliable stream abstraction on top of it, but that's not different from TCP.
> What do you mean exactly? UDP is not a stream protocol, it is an unreliable datagram one. A single read will return exactly one UDP datagram.
This is exactly what makes UDP a stream of datagrams. Data is defined over one single UDP datagram - UDP gives exactly zero guarantees about data across datagram boundaries. There is no first, there is no last, there is no next/previous datagram, there no guarantees and expectations whether more datagrams will follow.
> Of course QUIC builds a reliable stream abstraction on top of it, but that's not different from TCP.
Key word "reliable", which is subset of streams. In goes stream of datagrams, out goes ordered (and otherwise identified) stream of bytes. Reading your comment it seems that you equate "stream" with "asynchronously fetched buffer" meaning from C++ and others, which is a stricter definition of streams.
I can take a bunch of XMLs, "stream" them into a TCP socket on one end, feed that socket to "XML stream processing library" on another end and... Watch it blow up, because apparently the only streamy thing about the library is that it can handle asynchronously filled buffer, but will nevertheless blow up if fed anything but a single well-formed xml ¯\_(ツ)_/¯
SOCK_STREAM
Provides sequenced, reliable, two-way, connection-based
byte streams. An out-of-band data transmission mechanism
may be supported.
> Not sure what your reference to buffering or async buffers and XML has to do with the topic. An illustrative point for what happens when assumptions from a contrived case are applied on a more relaxed case.
I think this might happen in the opposite direction to how you think: the prevalence of containers is likely to thin down the OS layer to just arbitrating the PCIe bus, and you'll get monolithic applications that handle all the layers inside themself. So the QUIC layer and the web server and the app-routing and the app itself will all happen inside the same process.
Reinvented a bit later as libOS (library operating systems) and unikernels.
Even Linux got in on that a bit:
https://lwn.net/Articles/637658/
It isn't a stupid idea, but the only way to update anything is to rebuild the world. Luckily it's a small world after all.
Only because the implementations of fds in popular kernels are absolute rubbish at allowing userspace to extend them, except for perhaps ptys. A userspace implementation of a protocol can make a domain socket or just pass the client one half of a socketpair, but it cannot change the way accept() behaves, let alone add its own sockopts.
Even if you extend a kernel to do this, I suspect there are going to be interesting security implications when processes can suddenly receive fds that behave in funny unexpected ways. Much like the current mess with Linux namespaces: not because the idea is inherently bad, simply because we waited this long to try it.
When one uses TLS today, chances are very good that it's using something from djb, someone who "made his own crypto". Maybe the assumptions, stated as "rules", do not apply equally to everyone. For example,
"And if you wanted to implement all that yourself, people will yell at you for doing your own crypto."
In fact, before HTTP/2 existed, djb did exactly that, as a demonstration that it could be done.^1 It succeeded IMHO because it worked. People can "yell" all they want, but as above, these same people if they use TLS are probably using cryptography developed by the person at which they are "yelling". Someone who broke the "rules". Perhaps there is evidence that HTTP/2 would exist even were it not for the prior CurveCP experiment. But I have yet to find it.
The word used in the parent comment was "implement" and the suggestion is that attempts to "implement" would not succeed. Perhaps the reason they might "fail" is not a technical one. Perhaps "success" in this instance really refers to acceptance by certain companies that are making "rules" (standards) for the internet to benefit their own commercial interests. It may be possible to implement a system that works even if these companies do not "accept" it. If so, then the problem here is the companies, their fanboys/fangirls (watch for them in the comment replies), and the undue influence they can exert, not the difficulty of implementing something that works.
IMHO, getting something "accepted" by some third party or group of third parties is a different type of "success" that getting something to work (i.e., "implementing"). It's the later I find more interesting.
But yes, humans are humans and everything that is "simple" will get built upon and then become the backbone of something complex that will engulf and smother it as it evolves.
google isn't really concerned about creating elegant abstractions so much as they are about improving the performance of their browser talking to their server, though sometimes these do coincide
Or it can be neither. But it can't be both.
If you want fast web at all cost (and you need encryption), you get HTTP3/QUIC.
The IP model won over the OSI model due to lower number of larger layers.
I think it's a very reasonable question, but also the HTTP3/QUIC advantages do seem to offer advantages.
We built pianojacq.com without all that, just plain vanilla JS and a minimum of dependencies, see: https://gitlab.com/jmattheij/pianojacq
Certainly some applications need not have access to the entire HTTP suite. If the goal is not to offer a "full-featured" client then an HTTP client may not be terribly difficult, it can just fail gracefully if the other party tries to do something it doesn't support?
But loosing one of the main benefits of scripting languages for some convenience? You could argue that you use webpack anyway. Same question applies here...
If you have to use JS anyway, be glad that you can have an extremely shallow toolchain for deployment for smaller projects. I am not a JS dev, I just use it for stabbing at bits occasionally.
https://github.com/robdelacruz/lkwebserver/blob/main/lkstrin...
It's fun to write this (and read others' versions) the first 3 or 4 times, but it gets old quickly.
and never mind that Go has had generics for over a year now right? sometimes having a small, stripped down language is better than having a huge bloated monster. I would point to examples, but you know what they are.
While Go has had generics for a while now, the vast swath of existing Go code doesn't. The situation is definitely improving but there is still a lot of non-generic legacy cruft.
you know that Go has interfaces right? and I dont just mean the "any" interface. you can write your own interface similar to io.Reader and others, then add types that implement that interface, then no type matching is needed.
> non-generic legacy cruft
AKA normal, fine code. generic code is not some magic pixie dust that makes bad code into good code. plenty of awful bloated slow generic code around as well.
With regards to your second point, there are definitely situations where generics are vastly preferred: type-safe collections being a big one. For instance, the standard library containers (https://pkg.go.dev/container) are still non-generic with no generic versions in the standard library yet. This is the kind of cruft I mean: generic collections can be turned into concrete collections with type safety, but not the other way around. I make no claims about use of generics making your code being absolutely good or bad, but I do make claims that use of generics can make your code less error-prone and more safe.
Right. this is the issue right here. people only see that Go didn't have generics, and they never stop for a second to think WHY Go didn't have them. Generics have an implementation cost for the Go project, and maintenance cost for the Go project, and end user negative impacts in regards to time and memory. but advocates often dont know or care about these drawback, and only howl that GENERICS ARE MISSING until they are added, consequences be damned.
I'm just saying that while generics are useful in some cases, they are not always the right answer, and it shouldn't just be assumed that every language needs them, nor that any non-generic language "sucks".
It's funny to me, because this comment chain exists because I said I got tired of rewriting the same functions with minute changes, the exact problem generics is intended to solve. I didn't say anything sucked, or even that I didn't like Go, just that this one aspect got tiring.
And yet, this small comment lead into a person feeling their language was being attacked.
I have thought about this. I concluded that Go doesn't have generics because it's a poorly designed language whose designers elected to ignore decades of progress in programming language design for... reasons. Hence why you have to do the moral equivalent of passing around void pointers and checking return codes in $current_year
I'm saying that I wanted them...? I didn't say need. I was able to operate without them. I simply _wanted_ them.
Also maybe consider the way you're communicating? I didn't say Go was wrong. I said I didn't enjoy it. I'm allowed to not like things, and I'm allowed to post about them.
_Also_ this concept of "need" is amusing to me. You don't _need_ a garbage collector, or a statically linked binary, or static typing, or IDE support, or a debugger, or, or ,or.
And yet, people _want_ them.
> and never mind that Go has had generics for over a year now right?
Their generics implementation is pretty bad IME. Interfaces lacking type parameters seems pretty untenable to me. I'd rather just not use them.
> sometimes having a small, stripped down language
Of all the things Go is, I wouldn't say its either of these.
But also, if you respond, please try to not be so defensive. You can like Go, that is a valid thing to do.
1. Generics are bloated and don't allow us to have a single-pass compiler, which we need.
2. You don't need generics because go generate covers all the cases that generics cover (please ignore that this is a second pass).
3. Go has generics!
When new C++ standard libraries are written, there is now active, public effort to review what exists in other languages. Finally.
1. The current proposals for generics are bloated and don't fit properly with our vision for Go.
2. We will do heaps and heaps of work over many years until we get something we like.
3. Go has generics!
Also, AFAIK Go is not a single-pass compiler, at least not in the way I learned about compilers.
> 2. We will do heaps and heaps of work over many years until we get something we like.
I suppose repeatedly suffering the pain of not having generics until you finally get enough of your community to admit generics aren't just bloat can be described as "heaps and heaps of work". ;)
I'd love to hear what meaningful differences you think exist between early proposals for Go generics, and later proposals for Go generics. Or Go generics and generics which had existed in other languages for decades, for that matter.
The result of this silly delay will be felt basically forever, because millions of lines of go were written with error codes, which could have been much-less error-prone option types. I'm not sure it would even be a good idea to start using option types at this point because it would create such inconsistencies in codebases.
> Also, AFAIK Go is not a single-pass compiler, at least not in the way I learned about compilers.
Correct! But it was single pass and that was very important because multiple passes were bloat. But multiple passes aren't bloat any more, because now Go has multiple passes and Go doesn't have the bloat of other languages. See how that works?
"The past was alterable. The past never had been altered. Oceania was at war with Eurasia. Oceania had always been at war with Eurasia."
Go couldn't have Option before, and it cant have option now. Option is not only generic, but its also an Enum, which Go still doesn't have:
https://doc.rust-lang.org/std/option/enum.Option.html
so its not really clear what you mean by this.
It’s not like we’ve seen some massive regression in compiler performance. Circumstances changed, and the Go developers changed their minds. What would you do?
1. Yes, we didn't see a massive regression in compiler performance. Which begs the question why they didn't benchmark the options to see that might be the case.
2. Circumstances didn't change. The options for how to implement generics are roughly now what they were in 2009. What we need from languages in Go's sphere, is roughly the same as well.
3. What would I have done? a) Benchmark before making claims about speed. b) Assess features from other languages to see what works and what hasn't (literally nothing about the problems with lacking generics was unforseeable--I was talking about them early on and I'm not some sort of prophet). c) Admit rejecting generics was a mistake instead of pretending they were part of the plan all along.
In terms of changed circumstances, obviously something caused them to spend the time and effort to get it done. Maybe they thought modules were more important? Features don’t just magically appear, they need resourcing. The rest of your points are asinine. You mention benchmarking twice which begs the question, how did they maintain good compiler performance for all these years? Was it just luck?
This feels a lot like those conversations where the winners get what they ostensibly wanted, but then they want a pound of flesh to go with it. If you don’t like Go then use something else. The world is full of wonderful programming languages.
Yes, exactly--wonderful programming languages whose mind share and funding is stolen by Go, a language with a few famous programmers and Googles' names attached, and almost no actual innovation. In fact, in reinventing the wheel Go seems insistent on re-making all the same mistakes other programming languages already made, and take us backwards. Part of the problem is exemplified in this conversation: the Go community's refusal to even acknowledge mistakes were made.
And sure, I can go use another language, if I can find a job that uses it. Luckily I'm fairly proficient in a few languages which are in no short demand, but none of them are new or interesting.
I would love if there was significant job stability in Elixir, for example. But there isn't. Meanwhile Go has the 11th spot on the IEEE list[1], through no deserving qualities of its own.
I'm not a winner getting what I wanted, and then wanting a pound of flesh in addition. I'm more of a frustrated loser here--sure, I was right about Go needing generics, but it's a Pyrrhic victory, because Go still has a bunch of market share while better languages languish in obscurity. And it's not just me losing: we all lose if we have to use poorly-designed languages and the lower-quality programs which result from them.
[1] https://spectrum.ieee.org/top-programming-languages-2022
Seriously tho, take a break. Go has strengths and weaknesses like any language; generics is a weirdly specific thing to get hung up on, especially since they admitted it was a weakness and they fixed it.
I mean, Java didn’t have generics for, what, 10 years? And Java generics are - in my opinion - a shitshow. And Java is #5 on your list.
I mean - just look at the top five languages. C doesn’t even have generics, and it’s number 2. C++… not gonna go there. I have never used c# in anger so can’t comment. And I’ve tried to use Python and I hate it for all the reasons I enjoy Go - and it’s number 1.
So what? Each of these languages fills a niche. It’s not worth getting upset about it.
I’ve been developing software for a very long time, and I’ve forgotten more languages than most people ever learn. I like Go and I think they have made perfectly reasonable trade offs.
You’re welcome to your opinion, but complaining about a moderately popular language stealing mind share is kinda weird.
It's too late, some problems caused by not having generics are unfixable at this point. As I said before, we're stuck with error codes for the forseeable future, because they chose to go with 1980s solutions to problems and not prioritize generics when everybody with a brain said they were needed.
> I mean - just look at the top five languages. C doesn’t even have generics, and it’s number 2. C++… not gonna go there. I have never used c# in anger so can’t comment. And I’ve tried to use Python and I hate it for all the reasons I enjoy Go - and it’s number 1.
Yes, those languages have problems. My point isn't to give you the opportunity to point the finger at other people, it's that we need better languages. This is why it's so fucking irritating that people latched onto Go, instead of one of the many amazing languages which isn't total shite. We need languages that have fixed these problems, but what we got was another shite language which learned almost nothing from the past.
At least C and Python have the excuse that they are products of their time. At least they learned from and fixed the mistakes of their predecessors. Go brings 0 innovation to the table: its best features, like coroutines and message queues, have existed in other languages for decades, and meanwhile they've made obvious mistakes that other languages already made and learned from. If I'm going to have to write code in a shite language, I might as well use one of the ones that knows and admits its problems, and has well-known solutions to them.
Yes, many of those languages made mistakes--a minimum of 10 years before Go existed. So why didn't Go learn from those mistakes?
> I mean, Java didn’t have generics for, what, 10 years? And Java generics are - in my opinion - a shitshow. And Java is #5 on your list.
My list? Bruh, this is a list of the most popular languages list. I didn't pick these, and I wouldn't pick those if I was making a list of the best languages. If you think that's what I'm saying, you're completely missing the point.
What you seem to be missing about Java is that their generics were a big step up from C++ templates--they were an improvement when they came out. They made mistakes, and C# learned from them, and brought in a better version of generics. You know, progress! Unlike Go which went with error codes and went with C-style casting and/or code generation which were already obviously problematic in the 80s, in 2009. Great language you got there, 25 years behind when it started.
> You’re welcome to your opinion, but complaining about a moderately popular language stealing mind share is kinda weird.
And you're welcome to yours, but defending an overhyped, innovation-less language that doesn't deserve its popularity is kinda weird too. We're both just wasting time.
What circumstances changed, specifically?
https://github.com/robdelacruz/lkwebserver/blob/main/lkstrin...
This is distinct from checking the parameters; if lks is null then the user of the API has made an error. Some libraries may sanitise user parameters, others don’t. At any rate, an assert would be the wrong choice to check user parameters since this would result in a (recoverable) user error leading to an abort unless the assert is disabled at compile time (-DNDEBUG), returning an error would be a better choice.
Personally, I wish HTTP messages were closer to something like ASN.1 DER; there's little in the way of string manipulation necessary for those, and all the lengths are prefixes instead of "try to find the terminator" (and don't forget to not run past the end of the buffer...)
This has also had serious security bugs because it's so hard to understand.
I mean, you can always just use an existing string library or reuse your own. There's no reason to rewrite string operations for every project.
Curious to know how it compares to micro_httpd [1] which is about 200 lines of C. Or others like thttpd and tiny_httpd.
WARNING: select() can monitor only file descriptors numbers that
are less than FD_SETSIZE (1024)—an unreasonably low limit for
many modern applications—and this limitation will not change.
All modern applications should instead use poll(2) or epoll(7),
which do not suffer this limitation.22 years ago, I worked for Zeus Web Server, which was built entirely around one-process-per-core webserving off select(), and it was so much faster than Apache for serving static content that the developers had built a business out of it. At the time it could saturate a gigabit ethernet link off the largest HP-UX server we could find.
Sure, you should use the modern interfaces, but 1024 connections per process can get you surprisingly far.
http://canonical.org/~kragen/sw/dev3/httpdito-readme
http://canonical.org/~kragen/sw/dev3/server.s
it's a 2-kilobyte executable written in i386 assembly that can handle 20000 requests per second on my laptop, but only serves up files from the filesystem; no cgi or reverse proxy
instead of being single-threaded or preforking it just forks a child per request
Love it!
what are the best options out there for hosting websites built as HTTP-serving executables (either Windows or Linux)? is it possible to do this relatively cheaply?
I ask because I've been working on a framework[0] for building websites in a compiled language recently, and while it's been a ton of fun to build and test locally as a hobby project, I have absolutely no idea if it's even remotely financially viable to host a (small- to medium-sized) website made this way, compared to all of the managed hosting solutions out there for PHP/Node/etc.
I don't want/need to pay for a whole dedicated server—I just want to serve HTTP (eventually HTTPS) from a single executable, using one or more SQLite database files. ideally, it would cost as close to your typical shared PHP host as possible.
I have almost zero experience with "cloud" hosting—I made a small game with Node on Azure years ago, and accidentally racked up charges just playing around with it in development—so I don't know if this, or AWS, or whatever else is a viable solution for this. I've seen that it is indeed possible to host a single executable on Azure, but I haven't actually tried it myself, or determined what the pricing for this would end up being.
Cheapest AWS solution would be just EC2 instances (=basic virtual machines). t3a.nano instances cost just $3.5/mo and do not require additional load-balancers.
The modern cloudy approach would be to look into stuff like CloudFlare Workers, iirc they can run WASM, so if you manage to compile your code to that then it might work.
Otherwise, why not just get a cheap VPS and host the binary there? I’ve used Vultr and it’s $3.50/month all-in, at the low end. There are even cheaper providers, although I don’t know about their quality. I bet this option would be the cheapest.
This is the beauty of a single binary—it’s trivial to deploy!
this is the question driving the framework I'm building—it even has support for simple HTML templates, but they're interpreted, type-checked against the structs that get passed into them, and baked into the executable, all at compile time. this is all coming off of building a website for a client using PHP for the first time in over a decade—on one hand, I appreciate the relative simplicity and ease of deployment compared to modern backend stacks, but on the other hand, it's still an interpreted language, with all the baggage associated with that. I believe it is possible to take the ease of use and speed of iteration of interpreted languages, and the benefits of strongly-typed compiled languages, and get the best of both worlds—at least, for the scale and complexity of website that I want to build and maintain.
Multiple websites served by golang written server, some static sites, also gitea and Jenkins.
The web app is fronted cloudflare (free tier). On the box itself I have caddy set up as a reverse proxy with the cloudflare cert, then uvicorn serving my python asgi (starlette) app. The app uses a local SQLite db.
I’m still working out some of the operational stuff like backups and monitoring but so far I am very pleased with the setup. I’m learning a lot and for the first time I do not feel like there is some monstrous pile of complexity behind a curtain.
Setup takes some time but I have detailed notes and it gets easier every time I run through. Feel free to get in touch if you’d like to hear more details.
This is exactly the type of web server I'm looking for my project.
We have tokio to handle all the IO stuff, we have hyper to handle HTTP parsing, and we even have tungstenite to handle websocket out of the box. While I appreciate your work but it will not be practical to write C anymore in the modern age. Well, unless you need to target something LLVM isn't there yet and maybe you need some weird GCC toolchain (cough cough AVR)
been hearing that for 20 years
Both approaches are valid and serve different purposes, you seem to have misunderstood the purpose here.
Religious wars will persist