I abandoned OpenLiteSpeed and went back to good ol' Nginx | Hacker News Reader

I abandoned OpenLiteSpeed and went back to good ol' Nginx

arstechnica.com

83 pointsmikece44 comments

This makes sense. Nginx is reliable and well-tested, and OLS is neither.

See my recent Shmoocon talk (yes yes, self promotion) for an example of a vulnerability that would be caught immediately in Nginx, but persisted in the OLS codebase for >=10 years: https://yewtu.be/watch?v=aKPAX00ft5s&t=2h29m28s

Really nice work.

> ... I was profoundly disappointed that the OLS access log didn’t give me useful information and left me flying blind—the LSPHP processes shouldn’t be pegged to the wall while the access and error logs both show absolutely no requests being serviced of any kind.

I was profoundly disappointed that the article didn't go on to describe how strace works, in the service of delivering a satisfying answer as to why the CPU was pegged.

Sadly, I'm too dumb for that.

> I didn’t know too much about OpenLiteSpeed (“OLS” to its friends) other than that it's mentioned a bunch in discussions about WordPress hosting—and since SCW runs WordPress, I started to get interested

This was a trend for a bit but I don’t know of any serious hosts off the top of my head who run LiteSpeed these days. Most are running either nginx or good ol Apache (sometimes both!).

(We run enterprise scale WP on nginx, and our competitors at this scale basically all do too.)

What about Caddy, me and my friends have moved onto Caddy over a year ago and haven't experienced any issues.

I’m not aware of anyone running Caddy at any sort of scale for customers. From what I’ve seen, it tends to be only self-hosted people running it.

The overall trend in WP hosting has been away from shared and VPS hosting towards managed hosts, who provide a WordPress-specific environment and do things like upgrade WP and plugins for you. As a result, I can’t imagine there’s a huge percentage of sites using Caddy.

(That said, even a small percentage will be millions of sites, since the WP install base is massive.)

> I’m not aware of anyone running Caddy at any sort of scale for customers.

Well, to name a few...

- Stripe (https://twitter.com/caddyserver/status/1559591673511813120)

- Mercedes-Benz (https://github.com/caddyserver/caddy/pull/5275#issuecomment-...)

- Approximated.app (https://dev.to/carterbryden/how-to-allow-end-user-custom-dom...)

- FusionAuth (https://fusionauth.io/blog/unlimited-domains-fusionauth)

The problem is those using Caddy are shy, not that it's not used at all. I know this because I see users removing the `server` header on the Caddy forum all the time, and many of the large users are just shy of their technology stack when it comes to Caddy.

Disclaimer: Member of the Caddy team

I mean with WordPress hosting - I don't doubt Caddy is used in production by companies elsewhere. :)

Caddy is actually ideal for scaling WordPress: https://frankenphp.dev

Not sure, who handles .htaccess then? Some plug-ins heavily rely on it.

Most every cPanel based hosting company, including the large ones, will offer a product with LiteSpeed. It's sometimes re-branded, but it's there. The main benefit is that it's a dropin replacement for Apache on cPanel servers and performs much better when it's tuned for that purpose than Apache.

Good to know. That said though, my understanding is that the vast majority (let’s say >85%) of sites these days are on managed hosting rather than shared hosting (or even VPSes) with cPanel.

WordPress famously used LiteSpeed (and still does) and the only benefit I know of from the time is that LS has a much better PHP SAPI than fcgi with php-fpm. But now we have nginx unit with native php support (no fcgi), which halves the php-fpm base latency and can be run standalone or behind nginx (via an http proxy over tcp or preferably a Unix socket) so the advantages of an LS (or OLS) configuration are probably non-existent.

"halves the php-fpm base latency"

Have you a source for that? I've been having this discussion (that php-fpm is slower than needed) but I haven't found any hard data.

There are benchmarks readily available if you search “nginx unit php benchmark” (some that show an order of magnitude reduction in latency) without the quotes, but I hesitate to link to any of them as configuring nginx is highly complicated and has so many knobs, particularly with backend upstreams that can be configured in a myriad of ways.

My advice would be to try and see.

> WordPress famously used LiteSpeed (and still does)

I presume you mean WordPress.com here (WordPress = open source project, WordPress.com = product by co-founder of WordPress; it's confusing). They did run LiteSpeed a long time ago, but haven't in at least a decade; they moved to Nginx.

(There's a comment quoted at https://news.ycombinator.com/item?id=6341947 from Barry Abrahamson, who's the .com systems guy - sadly, Wired appears to have removed the original. I also checked with colleagues who were there at the time.)

Edit: Also, this HN thread might be useful for further info: https://news.ycombinator.com/item?id=4578258

Either time flies or the information I feel I recently read that WP.com was still using LS (which surprised me) was incorrect. I stand corrected.

Anyone else confuse LiteSpeed for Lighttpd ("lighty") for a moment?

https://en.wikipedia.org/wiki/Lighttpd

Ummm no, but just to give credit, I'm using lighttpd for production for years now and happy with latest patches and stuff (supports HTTP/2 and several improvements over time), give it a try

Really interesting write up. As someone that has spent a lot of time debugging these kinds of issues I feel the pain. I still have a lot of Apache instances lying around just on the basis that the stability, simplicity and consistency on offer outweighs the performance benefits of switching even to nginx.

That said I am slowly migrating them to nginx as I get more comfort with it. Always interesting to hear about experiments with other stacks!

I've tried OLS before, but it's always slower than my nginx setups for general page load times.

nginx + cache is a nice simple solution that just works well.

I have to say this is very well written. So much long form stuff is just fluff, but this had some nice narative, a beginning a middle AND an end.

It's too bad they didn't delve into the technical details of why the processes were pegging the CPU. Would have loved to see that level of Linux debugging narrated.

I would love to be able to narrate that level of linux debugging!!

I can recommend Ellen Ulman, The Bug. She's a treat.

nginx can be a cache and an ssl termination point so I’m not sure why the author didn’t do this in the first place versus wrapping it in haproxy/varnish.

maybe as a layer of indirection so nginx and varnish could be independent backends? I wasn’t entire sure of this either

Author here. I settled on the haproxy/varnish/nginx sandwich through an accretive process that started back in 2012 with wanting to speed up nginx by taking advantage of scads of extra RAM on the box I was hosting with at the time (and also being entranced by all the cool header games one can play with Varnish). Fast forward to 2015 as LetsEncrypt started to be a thing, and HTTPS started the mainstream push from a nice-to-have to an ought-to-have feature. I was comfy with Varnish + Nginx and I wanted to keep my cache layer, but Varnish has no SSL termination capabilities. I looked at a few options, including Hitch, but decided on HAProxy because it was more widely used and therefore easier to google answers to questions. Everything worked, so I never re-evaluated until 2022 when I got curious about OLS.

As to why Varnish + Nginx versus just Nginx + FastCGI or proxy cache, I was chasing the performance dragon. AIUI, Nginx's fastcgi and proxy caches just store the static-ified versions of dynamic content in flat files, and those files are accessed via the file system, which means they get cached into RAM whenever the file system decides they get cached into RAM. I had the free RAM, and Varnish's RAM-first approach felt like it would be faster. Plus, this was all for personal hosting, and playing with Varnish was fun. (Or at least "fun.")

Have you considered caddy?

No, I'm happy to retreat to the familiar, and nginx is pretty familiar. After having my tail feathers burned with OLS, I doubt I'll switch to anything else for the next several years without an extremely compelling reason.

Sure I'm not suggesting it now as a follow up, I just wondered if it was something you were aware of and dismissed for some reason over the last several years before opting to try OLS (which I hadn't heard of) instead; especially since you mentioned Let's Encrypt.

Thanks for the walkthrough.

One particular comment:

> "Nginx's fastcgi and proxy caches just store the static-ified versions of dynamic content in flat files, and those files are accessed via the file system, which means they get cached into RAM whenever the file system decides they get cached into RAM. I had the free RAM, and Varnish's RAM-first approach felt like it would be faster."

The pagecache should do a pretty good job holding file contents in memory for you! https://tldp.org/LDP/sag/html/buffer-cache.html

I haven't had to tune for this scenario, but I think this might be the one place where you want a high swappiness (and some swap spaces). This will let Linux move unused bits of memory into swap & let even more ram be used for pagecache.

On the WordPress side, consider using Batcache to generate appropriate Cache-Control headers. While it’ll also cache into your object cache (memcache/Redis), you can disable or ignore that functionality, and then you get a quite good integrated system. We use it with CloudFront’s page caching.

Oh man, Batcache. Now there's a name I haven't heard in a long time. I'll take a look!

Feel free to send me an email if you run into any issues! :)

Makes sense, particularly with the “fun” bit.

In the future, a ramdisk is a great way to make files from disk fast in a manner where the caller is naive to this. For instance you could create a ramdisk and then give that path to nginx for the caching config.

If I were going to do that, I'd just use tmpfs — i'm lazy. But I'm under the impression that given how aggressive and intelligent file system caching is these days, ramdisks are almost never the correct solution because the kernel is waaaaaaaaaay better at figuring out whether or not to cache files than I am.

Same same. s/ramdisk/tmpfs

I'm curious, how many requests-per-second have you managed to get with Varnish and how many with your NGINX FastCGI / Proxy caching settings?

I vaguely recall Nginx being taken over (maybe just founders arrested?) by Russian officials at some point? Is Nginx open source enough where the community would notice any shenanigans Russia might do to compromise the integrity of Nginx? No important parts are black box binaries or anything?

If you feel insecure about NGINX, you can use OpenResty that bundles LuaJIT and help you push its limits even further.

NGINX the company is owned by F5, Inc, a Seattle, Washington company. All of the open source infrastructure for NGINX is maintained by F5 as a result, and work on NGINX is managed by Maxim Konovalov, an F5 employee. F5 moved all the developers they could from Russian offices to offices in the US, Dubai and Poland a while back. Edit: Also, at the time of acquisition, NGINX, Inc was headquartered in San Jose, California.

Neat, thanks for the info!

Reply on news.ycombinator.com