Hacker News Reader

Comment by jiggawatts | Hacker News Reader

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so.

I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app!

People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages.

Trivial stuff like static content being treated as static content (files on the disk!) not as a distributed cache in front of a database can do wonders.

Am I just old and jaded?

View on HN

Excellent post, good technical content, amazing feat.

That said, I echo that the amazing feat is that they can fit modern inefficient tool choices with poor mechanical sympathy into that budget. The last decade of web-dev tooling has been pushing the TCO of systems through the roof and this post is all about how to struggle against that whilst using those tools.

If they went old-school they'd get another order of magnitude savings. Many veterans know of systems doing 10x that in 10x less cost. Remember C10K was in 1999.

> If they went old-school they'd get another order of magnitude savings. Many veterans know of systems doing 10x that in 10x less cost. Remember C10K was in 1999.

How to learn more about the old-school way without getting a job related to it? Like, topic or book recommendations.

I'm also curious about this, since in many areas of computing (not only webdev) the old-school guys take some stuff as so obvious that they don't even bother writing about it or explaining it beyond "This is obviously possible, dude". They know how to achieve this level of performance, but for everyone else, we have to cobble together fragmented insights. So if anyone out there reads this and thinks like the GP, please do write about it ;)

What topics specifically are you interested in? And where do all the people like you hang out?

I'm not an old-school guy by any means .. but I might have something to contribute.

> What topics specifically are you interested in?

Well, for example, what's the old-school alternative to mangadex's solution?

> And where do all the people like you hang out?

We are here on HN.

No. I think you have a healthy perspective and we should all be questioning if current trends are beneficial/sustainable.

I haven’t read the article, but the headline alone to me seems alarming, $1,500 a month is a lot of money for only 2k rps.

Maybe its a lot of money just for the web servers, but for the entire infrastructure stack its pretty reasonable IMHO.

There is more to it that http request response. Mangadex also need to store a lot of images and distribute them.

CDNs have already solved this problem and are much cheaper than $1500/month.

I've ran far more complex sites with much higher traffic for less.

Mangadex can't use cloudflare because of privacy reasons. They may be facing similar issues with other popular CDNs.

I am sure they must be using some kind of CDN for sure, however, those options are unlikely to be free

Privacy reasons? It's all static content that is publicly accessible. I don't understand what the privacy reasons could be under this context.

Are they worried about CDNs logging the images their visitors access? Seems like an absurd edge case to worry about in my opinion.

> however, those options are unlikely to be free

I wasn't even talking about free CDNs :)

They’re basically hosting illegal content, or at least a good chunk of it is copyright-infringing so they cannot use cloudflare or any of the other off the shelf offerings

Not disputing your statement, just made me laugh a bit because almost every single site I visit recently that offers links to copyrighted content [stored in file lockers] sits behind Cloudflare

I wouldn’t say illegal content, since most of that gets removed relatively quickly. But definitely a lot of content in the grey zones of copyright law.

Hm? There are lots of copyright-infringing sites using Cloudflare, and Cloudflare seems pretty content to generally ignore infringement notices.

I see. That does complicate things somewhat.

I wonder if there's merit in them approaching studios with a proper business plan?

Now with the way the manga, Manghwa, and WEBTOON industry is tied up. But I believe that is their end goal eventually.

I think privacy as in the mangadex team don't want to get sued. So they avoid popular services who are more willingly share their identity.

Then you should read the article ;)

I think even a distributed cache in front of a database shouldn't have any trouble handling 2000 requests per second.

The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading images from a filesystem directory.

Valgrind claims httpdito runs a few thousand instructions per request, but that's not really accurate; what happens is that the kernel is doing all the work. httpdito on Linux can handle about 4000 requests per second per core, nearly a million clock cycles per request, almost all of which is in the kernel. Of course it doesn't ship its logs off to Grafana. In fact, it doesn't have logs at all. But it would work fine for reading manga.

> The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading images from a filesystem directory.

I assume they are talking about their more dynamic content serving in this post (for things like search, tracking which chapters are read, new chapter listing based on what user follows etc.).

They have a custom CDN that is hosted by volunteers to serve the images for the manga pages. They provide some metrics for that at https://mangadex.network, there are also some older screenshots where they hit 3.2GB/s.

Interesting! Thanks! It still doesn't sound like the kind of thing that would require load balancing, but maybe it was easier to write it in Python or PHP or something, and that made it so heavy that it did.

> distributed cache in front of a database

Already an over kill.

Think smaller. Think simpler.

A single machine serving files directly from the file system (yes, from the SSD attached to the machine) will be able to handle a LOT more.

Well, that's what httpdito does: it serves files directly from the filesystem. That's why I mentioned it. But, for some applications, such as the website you're using right now, it's useful to display pages that haven't been previously stored in the filesystem.

Also that 2000 request per second has to happen 24/7 not only quick demo session.

httpdito is nothing if not consistent in its performance. It doesn't have enough state to have many ways to perform well at first and then more poorly later, or for that matter vice versa. (Not saying it couldn't happen, but it's not that likely.) Linux is pretty good about consistent performance, too, though it has more state.

In 2011 a company i contracted for was testing some new dell 1U servers with around 1-2TB of ram. There was a postgres database with 4000qps that could fit into tmpfs, and so i restricted postgres to 640Kb of memory and we got replication working, it took about 6 hours of babysitting.

We threw the switch and watched as postgres, with 640Kb of ram and a tmpfs backed store proceeded to handle all of the query traffic. There were some stored procedures or something that were long-querying or whatever - i'm not a DB person at all, so we switched back to the regular production server about 8 minutes later.

Yes, we did it in production.

Postgres handles low memory situations well. It'll kill memory intensive queries before it crashes. I wonder if your application was getting a lot of errors back instead of successful queries :)

the application performed fine, even though we made the switch around 15:00 PST. The DBA was concerned because of the few long queries.

Obviously the tmpfs was doing the heavy lifting, there - and if i had to do a postmortem, i'd wager that filling the OS caches was the main reason the long queries took so long. We didn't do any sort of performance tracing.

The main purpose was to show that these $35k servers could essentially replace the older machines if need be, even though the old ones had FusionIO. I just removed the middleman of the PCIe bus between the application and the memory. It was a near constant argument on the floor about whether or not we could feasibly switch to SSDs in some configuration over spinning rust or even FusionIO, i wanted a third option.

Basically, serve out of registered, ECC memory in front, replicate to the fusionIO and let those handle the spindled backups, which iirc was a pain point.

For real, 640 kilobits?

K isn't the abbreviation for kilo, so if you're going to rag on the fellow for the 'b', then you should at least be asking what a Kelvin*bit is.

640KiB is very little and I'm wondering if it's a typo, given that the servers had 1-2TiB available. Postgres 9.0 released in 2010 already had 32MiB as the default for shared_buffers (with a minimum of 128KiB): https://www.postgresql.org/docs/9.0/runtime-config-resource.... and 8.1 released in 2005 used 8MB (1000*8KiB): https://www.postgresql.org/docs/8.1/runtime-config-resource....

i interpreted it as "we wanted to turn the shared buffers ~off, but in a hilarious way that would suggest to someone reading the configuration file that something was going on" (bill gates, mumble mumble)

but, wtf do i know, i'm the crazy guy who tries to interpret comments generously.

Yes, it was a direct reference to Bill Gates "640 kilobytes is enough for anyone" and i typed the comment right before i fell asleep.

“The binary meaning of the kilobyte for 1024 bytes typically uses the symbol KB, with an uppercase letter K.” [0]

0. https://en.m.wikipedia.org/wiki/Kilobyte

The question was more about the kilo part, even though I didn't clarify. Seems orders of magnitude too small?

2k sockets on a test bed vs 2k real user request in production is very different. I doubt you ran a top 1000 Alexa site on your laptop. Today we need to deal with SSL which eats from the performance budget.

> SSL which eats from the performance budget.

That was a short-lived thing, and has now become a myth perpetuated by companies like Citrix and F5 that sell "SSL offload" appliances for $$$.

Have you benchmarked the overhead of TLS?

In my experience, a single CPU core can easily put out multiple gigabytes of AES-256 (tens of gigabits). This benchmark shows 3 GB/s (24 Gbps) for recent AMD CPUs, and nearly 40 Gbps per core for an Intel CPU: https://calomel.org/aesni_ssl_performance.html

A multi-core server is very unlikely to have more than a 1-5% overheard due to TLS. Even connection set up is a minor overhead with elliptic curve certificates.

This is thanks to the AES offload instructions, which are present in all server CPUs made any time in the last 5-7 years or so. As long as the modern Galois Counter Mode (GCM) is used with AES, performance should be great.

Meanwhile, Citrix ADC v13 with a hardware "SSL offload card" actually slows down connections! I had a very hard time getting more than 600 Mbps through one. It seems to be the way the ASIC offload chip is architected: it seems to use a large number of slow cores, a bit like a GPU. This means that any one TLS stream will have its bandwidth capped!

The problem with these benchmarks is they measure the bandwidth you can push through an established tls connection. Try to build 2000 new tls connections a seconds (yes many are still active and dont need to be restarted) that is what is the really slow part. Not sending the data over already established channels.

This exactly. Most CPU time in our Haproxy is spent on crypto/new TLS sessions handling thousands of new connections per second.

To add to that, in 2005, a Cloudflare engineer showed that you can receive 1 million packages per second (https://blog.cloudflare.com/how-to-receive-a-million-packets...). Without processing though.

You're not alone in making comments like this, and as I read them I'm not sure if people are missing the point or being disingenuous.

Much of the work I've done professionally has happened in the webhosting space, and I'm not a new hand at it - the first "professional" website I ever ran was hosted off of a machine running on a Quantum R5000. I have served (and still serve) plenty of static content as files off the disk. My own first impulse when building anything is to use as few moving parts and as simple of a setup as possible.

Requests per second are not a good metric for the amount of work being done by a system, because not all requests are made equal. You say you got 3k requests out of your laptop and that it wasn't a trivial app, but you have only provided a trivial amount of information. Taking a quick look at the features provided by the site, they offer quite a bit of flexibility on both what is and how it's displayed. Filtering options based on original and translated language, adult content filtering, multiple types of search, request routing to low quality images to save on data, category and tag filtering, tracking of what you have read and what your progress is on it, follows and notifications, permissions systems for uploading and updating content, etc. This is all on top of the basic "Display images and metadata about the images" functionality.

They have concerns around privacy and being able to withstand active attackers due to the content they host. They have their own requirements about logging, analytics, etc. Their own concerns about data availability.

You could not meet the design requirements of this website and serve 2k requests per second on a 2 core VM in 2007. I'm not saying there aren't inefficiencies in their architecture and further places where they could save money/increase performance/etc., but acting like two cores of lower-clockrate lower-ipc compute from a decade and a half ago could do all of the work needed to support the features and design requirements they have for this website is pretty disparaging towards the people who built this infrastructure.

Not just you, but if it works for them, that's completely fine.

But there are many ways to achieve 20K RPS without this type architecture and especially without k8s, for less than $1,500.

>20k RPS.

If this metric is what you are chasing, there are ways to reliably break 1 million RPS using a single box if you don't play the shiny BS tech game. The moment you involve multiple computers and containers, you are typically removed from this level of performance. Going from 2,000 to 2,000,000 RPS (serialized throughput) requires many ideological sacrifices.

Mechanical sympathy (ring buffers, batching, minimizing latency) can save you unbelievable amounts of margin and time when properly utilized.

I frankly don't see where containers could lower the performance.

Basically a container is a glorified chroot. It has the same networking unless you asked for isolation, then packets have to follow a local (inside the host) route. It has exactly no CPU or kernel interface penalty.

Maybe you wanted to say about container orchestration like k8s, with its custom network fabric, etc.

> I frankly don't see where containers could lower the performance.

Have you seen most k8s deployments? It's not the containers, it's the thoughtspace that comes with them. Even just using bare containers invites a level of abstraction and generally comes with a type of developer that just isn't desirable.

So it's not due to container, but nowadays "container" in prod tend to mean k8s or similar clustering.

Even loopback is significantly slower than a direct method invocation.

Reply on news.ycombinator.com