Show HN: Hosting my website using my C web server
github.com
github.com
This is one that has always baffled me. If there's no specific reason that a reverse proxy is helpful, I will often hang an app with an embedded Jetty out on the internet without one. This has never lead to any problems.
Infra or security people will see this and ask why I don't have an nginx instance in front of it. When I ask why I need one, the answers are all hand-wavy security or performance, lacking any specifics. The most specific answer I received once was slow loris, which hasn't been an issue for years.
Is reverse proxying something we've collectively decided to cargo cult, or is there some reason why it's a good idea that applies in the general case that I'm missing?
Proxying is always less-performing than serving directly since you add another layer in between, right? Or am I missing something?
Maybe in Java-land it’s overused, but everywhere else it makes sense.
I'd also say that there is no such thing as a "general case"; I've launched and/or supported countless (must be hundreds?) of web projects and even the "simple" ones were each a bit of a snowflake.
But at the cost of having a separate build step that deploys your static assets somewhere. Jetty is actually pretty fast - I've built some fairly high-volume internal apps this way.
> Also allows you to run two binaries at once for a rolling update.
You don't necessarily need an extra reverse proxy layer for this, though I will concede in some environments it's probably the easiest way to achieve it.
Also, most other rolling update solutions will end up being more complex than having a reverse proxy. What do you have in mind that would be simpler? NixOS?
Each is running on a different port, but I want them all accessible publicly from different URLs and I only want to expose port 443 to the internet.
I also want to have TLS autorefresh for each domain.
I need a reverse proxy for the former and caddy does both.
If you’re running a single server and that server does TLS termination then you don’t really need a reverse proxy.
If going IPv6-only, the need for a reverse proxy is seriously lowered. You could spin multiple servers up (even on different machines), listening to 443. Have each service handle its certificate renewal, etc.
That "original need" is exactly what GP is talking about.
It removes a lot of friction from "I wonder if making this service is a good idea?" and because I am self hosting I am not tying myself down to any of the "all in one" hosting platforms.
For the minimal case you don’t need it, but in production (with a single host) it allows for rolling releases, compression, TLS, fast static file serving, potentially A/B testing capabilities.
The layer of indirection between the request and your server can be very useful.
I mean for me this is pretty much already enough of a reason to always put an rp ahead of my apps. It's requires minimal setup, most of the tools are fire and forget so I see no real downsides. But having the ability to just point it somewhere else, or to split traffic across app replicas, is more than enough.
For security, you want a strong OS with this little code as possible in your overall system. Proxy-style apps can be very simple compared to web, application servers. They can filter incoming traffic, validate the input, or even change it to something safer (or faster) to parse. They can also run on OS’s that are harder to attack: OpenBSD; GenodeOS; INTEGRITY-178B. On availability, putting load-balancing, monitoring, and recovery in these systems is often safer since app servers are more likely to crash.
On performance, the first benefit is that the simple, focused app can have a highly-optimized implementation. From there, one can use hardware accelerators (CPU or PCI) to speed up compression or encryption. Also called offloading. The most, cost-effective setup has many commodity servers benefiting from a few, high-cost servers capable of offloading. Some have load-balancing to route incoming traffic to servers able to handle it best to minimize use of costly resources.
So, there’s a few ways that proxy-type servers can help in security and performance.
Overall, the benefit is that you can keep your origin server protected and only serve relevant traffic. Also, lets say you offer custom domain to your own customers and in that case, you could always swap out the origin server (if needed) without worrying about DNS changes for your customers as they are pointing to the reverse proxy and not your origin server directly.
Similarly if you start load balancing, you can put some server in the middle yes. But the ideal solution is at the DNS level I think, unless there's some serious compute going on (which a website loading a page from disk is not).
URL rewrites should not be a thing unless you have a clusterfuck, and Security is best accomplished in my experience by removing, rather than by adding.
So Ingress -TLS-> Container (pod).
We implemented LinkerD for this, which runs as a sidecar in the pod. Since the sidecar and the main container communicate on the same machine, this is OK.
I automated all that with a few scripts that included sanity checks with `nginx -t`. After the update looked good I would shut down the old app server without any time crunch. Only the Nginx config was time-sensitive.
I'm not sure if you can do that without some kind of reverse proxy as an abstraction layer. At least a TCP-level proxy.
And as everyone said, virtual hosting.
The url-specific web servers you're proxying tend to need a whole lot more, at least filesystem access to serve html content, at most program execution like CGIs and interpreters.
Separating these concerns makes a lot of sense, and brings little to no overhead by modern standards.
So you are using a proxy server, just an embedded one. Most prefer simply prefer not to bundle their application with one.
1) you can share multiple apps or sites with one server listening on port 443/80. 2) You can redirect to another backend on your infrastrcture 3) You can enforce certain login/sso/restrictions 4) You can configure all these things in one place.
Of course, if you don't need all that, then it's somewhat moot.
Really? I am curious.
You are not talking of monkeys?
It's a matter of risk management. On the one hand is your service that speaks http. Maybe it uses a good library for it, maybe not - but even if the library is good are we sure you used it correctly? Even if you used it correctly, has it been as thoroughly tested and proven as nginx?
On the other hand you have nginx - a deeply understood technology that has served trillions and trillions of web requests, has proven itself resillient against attacks again and again, and has been reviewed with a fine-toothed comb by security engineers deeply for years.
So just from the starting point, your software is riskier. Even if you're the best software engineer who's ever lived, it's a higher risk profile to deploy new unproven software than the one that's been battle tested for decades.
It's also a matter of mitigation - if your software does have a vuln, are you going to notice it? Even if you do notice it, how long til you understand the problem and fix it? What to do in the time between discovery and deploying the fix? On the other hand if there's an nginx vuln, there are almost certainly juicier targets than your software to exploit first, and the bug and the fix are far more likely to be found and deployed long before someone even tries it for your site.
Yeah, that’s ridiculous. “Cargo culting” is when people imitate processes without understanding the underlying purpose, but reverse proxying is widely used for valid reasons—like security, load balancing, caching, SSL termination, etc. It’s not just mindless mimicry. Dismissing a best practice as “cargo culting” because they don’t understand it is lazy. Just because it’s common doesn’t mean it’s done without purpose. Worst case? You get people following a pretty good practice.
He mentions bots make great fuzzers, but I think he should also do a bit of actual fuzzing.
http://git.annexia.org/?p=rws.git;a=tree Requires: http://git.annexia.org/?p=c2lib.git;a=tree http://git.annexia.org/?p=pthrlib.git;a=tree
the really surprising thing about that was that when your memory map only has five 4k pages in it, linux gets really fast at forking
comet was a pretty compelling reason to write your own web server 20 years ago
The chat code [I really should upload the code as the company has been dead for at least 10-15 years] worked by browsers holding an infinitely loading frame, so each client held open a connection for several hours. IIRC there was some Javascript that reloaded the connection after a few hours.
To handle 1000s of HTTP connections we had to implement our own fairly lightweight threads. It also had a cool inversion of control where you could write straight through code and it was turned into event-driven callbacks automatically. The webserver couldn't make use of multiple cores, which was lucky because the server had only 1 CPU!
Also used a pool allocator, which is very well suited to server applications.
nowadays people use websockets for comet
yeah, protothreads type stuff and pool allocators are great fits for that kind of work
P.S. Love the indentation
I love the connection-specific functions and related structs and arrays for your connection bookkeeping, as well as the poll fd arrays. It's very reminiscent of how it's done in lots of other open source packages known for high throughput numbers, like nginx, redis, memcached.
Great work!
It made me think that anyone should know and try every possible language (programming or otherwise) - "thinking" in a language is such a unique experience. The different contexts make everything feel different, even though it's more of the same. The perspective change, and changes the subjective experience.
For example - to really understand the nature of linux or git, you have to speak its language and understand the nuances that are usually lost in translation. Tangibly, to understand the true subjective meaning of the word "forest" in russian one has to speak and understand russian.
The context changes the perspective, so sometimes it changes everything.
I remember learning about the effects of language on cognition in a psychology course I took a while ago, it’s interesting to think about how that could apply more broadly.
It occupies an ever-shrinking ecological niche on the Pareto frontier.
The only major challengers to C in the last 50 years are C++ and Rust. I think that’s a testament to the quality of the language.
Totally agree. And actually using them is even more satisfying. I'm starting to get curious about email protocols..
> I'm kind of surprised that the vanilla poll() can put up numbers like you were seeing
Me too. I assumed I was going to go with epoll at some point, but poll() is working great.
> As of 2024, the althttpd instance for sqlite.org answers more than 500,000 HTTP requests per day (about 5 or 6 per second) delivering about 200GB of content per day (about 18 megabits/second) on a $40/month Linode. The load average on this machine normally stays around 0.5. About 19% of the HTTP requests are CGI to various Fossil source-code repositories.
Essentially build and run modules, and they can be combined (including mixing Lua/Python + C.)
Don’t get me wrong this is an awesome project but if you really care about this kind of thing in a production scenario and you’re serving mostly static content… just use a CDN. It’ll pretty much always outperform just about anything you write. It’s just boring.
(though I confess it's been some years since I've worked in this area)
You are vastly overestimating HN front page traffic. Any reasonable system on any reasonable machine with any reasonable link can do this. And I really do mean reasonable: I've served front-page traffic from a dedicated server in a DC, and from a small NUC in a closet at home, and both handled it completely fine.
Honestly, HN front page traffic isn’t much. For most, it probably peaks at about one page load¹ per second², and if your web server software can’t cope with that, it’s bad.
Even if your site uses PHP and MySQL and queries the database to handle every request, hopefully static resources bypass all that and are served straight from disk. CPU and memory usage will be negligible, and a 100Mbps uplink will handle it all easily. So then, hopefully you’re only left with one request that’s actually doing database work, and if it can’t answer in one whole, entire second, it’s bad.
(I’m talking about general web pages here, not web apps, which have a somewhat different balance; but still for most things HN traffic shouldn’t cause a sweat, even if you’ve completely ignored caching.)
Seriously, a not-too-awful WordPress installation on a Raspberry Pi could probably cope with HN traffic.
—⁂—
¹ Note this metric: page loads, not requests. Requests per second will scale with first-party requests per page.
² From a quick search, two sources from this year: https://marcotm.com/articles/stats-of-being-on-the-hacker-ne..., https://harrisonbroadbent.com/blog/hacker-news-traffic-spike.... Both use JS tracking, but even doubling the number to generously account for we sensible people who use content blockers has the hourly average under one load per second.
Well then sites on average are sadly "bad" by your standards. Lots of sites that get on the front page of HN go down.
I love it
It’s fun and not that bad really.
Hell if you’re really lazy you can forgo responding with the http headers and just socket.write(“hello world”) as the response and all the major browsers will render “hello world” to the user. Properly formatted http headers are just a text string extra and the html is just text. There’s not much to it.
Everybody speaks HTTP/1.0 and it is even simpler.
$ wc -l *
31 Makefile
910 config.c
314 control.c
34 css.h.in
257 http.h
100 httpd.8
1262 httpd.c
882 httpd.conf.5
843 httpd.h
19 js.h.in
218 log.c
319 logger.c
2563 parse.y
309 patterns.7
713 patterns.c
46 patterns.h
829 proc.c
1484 server.c
849 server_fcgi.c
826 server_file.c
1997 server_http.c
10 toheader.sed
14815 total
[1]: https://man.openbsd.org/httpd.8There's lots of bloat in practically every software not sure how much it affects performance but it's nice to build something from scratch.
Congrats to developer
That would make it not only faster but also safer.
Although if you start caching contents in memory (which is faster) you would have to at least kill the server and restart it. Or signal a reload.
It's very inspirational to see that you've just said screw it, I'm going to host my own HTTPS server, and also hey reddit, do your worst, try to break it. Now I want to work on my similar project again.
For anyone similarly inspired, but who doesn't know where to begin making an HTTP server, check out this excellent tutorial that walks you through everything you need to make an HTTP/1.0 server, and then grow it to handle HTTP/1.1: https://www2.cs.uh.edu/~gnawali/courses/cosc6377-f12/p1/http...
You have the opportunity to really make something custom and of high quality, hard code the paths of your files and avoid a whole class of vulnerabilities for example.
Configuration files? That makes sense when programmer and sysadmin are distinct, you can just modify variables and recompile.
I’m waiting for someone to chime in and explain why that would be a bad idea cause I can’t think of it from a security perspective.
vs just putting things in a subfolder of your repo or whatever and having the default handling not accept `..` path components
For example, you implement a super secure solution and no one hacks your website, but you end up being very unproductive and can't find a job. You lost food security.
In covid, bank systems in my country were so hard to use, there were like 6 passwords to login. Not only was usability compromised in the sense of security, but people, especially old people, started lining up in banks, compromising health security.
To say nothing of the scenarios were users just bypass obnoxious exaggerated security systems, like leaving a post-it note with a password on their screens.
One question: you say that "Transfer-Encoding: Chunked responds with 411 Length Required, prompting the client to resend with Content-Length". Is there a reason for doing this (security perhaps), or is it just a choice?
I think that was ewd?
- how much code actually gets read outside of top 2-3 projects?
- how many of those readers can detect security problems?
- why are others inherently better at detecting problems than the author?
Wouldn’t 1000 lines read by 2 people be better than a million read by 10?
And possibly serve attacker-controller content to other users.
I've always wanted to undertake a project similar to this but chunked encoding has always been the thing that put me off the idea... I never even though about just not supporting that :)
I've written many http/1.1 servers in the past but only for internal stuff that I also controlled the clients. Guess perfection was the enemy of good for me.
It does mean you have to read the client's headers to see if it was requested, though.
#define LIT(S) ((string) {.data=(S), .size=sizeof(S)-1})
#define STR(S) ((string) {.data=(S), .size=strlen(S)})"But if you actually do this, WAT" – https://www.destroyallsoftware.com/talks/wat
As with much of HN, this is fun, a good thing to learn while making and reading about... but it likely needs the caveat that doing this is production isn't a good idea (although in this case the author does not appear to encourage production usage).
Edit: And sure, if the author is lucky, then maybe a handful of people will gather around the code and try to make it "production ready". But since the README doesn't say anything about the topic at all, just let people have fun and learn things along the way?
I used smallhttpserv - a program that weighs a couple of dozen kilobytes and works even in early Windows
surprisingly, it still works
So that accounts for about the top 5 ...
It’s fast!
I have always wanted to try out something like this.
Good job!
$ curl http://playin.coz.is/index.html
curl: (7) Failed to connect to playin.coz.is port 80 after 166 ms: Couldn't connect to server
$ python3 -m http.server
$ ruby -run -e httpd .