Hacker News Reader

Comment by PragmaticPulp | Hacker News Reader

> The bulk of the traffic is serving static files.

The bulk of the data transferring is static files because they're hosting large assets, but the rest of the article is about their API, database, etc.

A single server in a single datacenter isn't comparable to a global CDN. They made it clear in the article that they value global latency, and they're willing to pay more for it.

> Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59

That has mechanical spinning hard disks and a CPU that was released literally a decade ago.

You're not going to replace Cloudflare, Firebase, their API server, and a global CDN with a single 10-year old machine serving files from a mechanical spinning hard disk, hosted by a company that almost nobody has heard of.

View on HN

Sure you are for something like serving large image files (like, say 3D asset files?)... I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month doing pretty much that, multi-terabyte image datasets and gigabyte-sized DL models. Plus a few websites, which you've probably seen. Zero optimization, just rsync and nginx with defaults, works fine. (It'd be way cheaper except I wanted another 20TB or so of rust to get room to add more datasets to serve - after all, the server is barely breaking a sweat.)

> I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month

You could probably do it with Cloudflare for a similar price, but that's not what this article is about. The article says that their Cloudflare (not Argo) bill is only $40/month, and that's only because they have two domains. It's $20/month per domain.

The $400/month figure isn't just for static file hosting. It includes their API, database, and global CDN. They make a point to say that low-latency, global CDN is a priority for them, which is why they're paying extra for Cloudflare Argo.

I'm not sure why everyone is comparing what they're doing to a single fileserver hosted somewhere, serving up static files without regard to global latency. It's apples and oranges.

> They make a point to say that low-latency, global CDN is a priority for them

Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe.

It is like saying you are trying to optimize an ocean cargo ship for acceleration time.

> Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe.

Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte.

They're also not a static file serving website.

If they were just serving static files and they didn't care about latency, they could pay $20/month for Cloudflare Pro and be done with it.

I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.

> Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte.

I've built three CDNs in my career, one at the Tbps scale. Latency only has two factors in throughput: time to scale window size, and retransmits. Modern TCP stacks handle the prior just fine, and the latter is only an issue with packet loss. You can also turn on HTTP/3 and remove your reliance on TCP entirely.

> I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.

Because I actually looked at the site and read the article. The vast majority of the site is large static files. The frontend is a static JavaScript app that calls an API server. That API server is running on a $5/mo VM.

My offer still stands if you'd like a tour of a datacenter and see the sausage being made at scale. I'll throw in lunch and answer all your scalability questions if you like. But this thread is growing quite long and veering off topic.

> My offer still stands if you'd like a tour of a datacenter and see the sausage being made at scale.

I might take you up on that offer, as I might need that for some new project I'm preparing.

Where are you located?

Email is in my profile.

You can use different TCP congestion control algorithms, not just cubic. BBR2 has no issues with high RTT even with packet loss and is fair to other TCP streams.

It serves 25Mbps stream in real time to users in Japan and the US from a data-centre in Europe with no issues.

Yahoo, Google, and eBay were built with gasp spinning hard disks.

But fine, if you don't believe it I will happily extend an invitation to come see in person my racks of spinning rust and 10 year old servers in Los Angeles or Fremont that are running services that are used by millions of end users. My email is in my profile.

> Yahoo, Google, and eBay were built with gasp spinning hard disks.

Which they ditched as soon as physically possible. The internet has changed a lot since then. That much should be obvious.

You can't reduce this article to "millions of end users" and then equate it with every other service. There's more to this article/service than you're suggesting, and there's more to a website than just users/month.

What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM... that completely negates any SSD advantage. SSDs have little to no benefit in serving static files.

If you're really strict on this it's trivial to preload these files in RAM after compiling your assets so you don't have to wait for a request.

> What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM

They're running a dynamic website with an API and other dynamic functions.

It's not a static fileserver.

It might make more sense if you visit the website in question: https://polyhaven.com/

Higher up someone calculated that they have 2TB of data, considering their $11/mo Backblaze bill. I calculate $11/$0.004/GB and get 2750 GB.

If you have 2.75TB of files to serve and have a bunch of simultaneous requests for different files:

- everything doesn't fit into the RAM page cache, unless you have 3TB of RAM in your server

- for files that aren't cached, you have to read from the drive

- when you have a bunch of different requests for different files, you're going to have a lot of disk seeks. Seeks are expensive on rotating media.

Serving a large collection of static files at scale can be quite challenging. At my previous e-commerce company, back in the early 2000's, we did our own image serving with RAID1 multiply-mirrored drives (before SSDs!) so we could multiply the seek capacity by the number of drives.

24 GB of RAM means a lot of ability to cache data in RAM, which the OS does already by default. It of course depends on the type of content you have but generally traffic follows a power law, so most content can be served from RAM directly. The write side I'm a bit more worried about, as e.g. SQLite issues a sync request on each transaction (not sure about the other SQL implementations, should be similar there tho), and those might require seeking.

The article explains that they have database and API usage. You can't run a high-throughput database on spinning disks unless you disregard data integrity.

It's not just a static fileserver, despite some of the comparisons in this thread.

Like SSDs bring any new angle to this.

> You can't run a high-throughput database on spinning disks unless you disregard data integrity

Of course you can. That's the way it was and still is done. I'd say one can't run a database with any data integrity unless there are disks spinning somewhere very near.

I'd take another box for like $40 there just for the replica, which would sit mostly idle anyways.

But piling on CDNs, clouds and variuos other opex and indeterminate risks instead of some thinking out of this little cloud shoebox is just baffling.

Spinning disks lose data integrity? You will be surprised where the cloud servers store your backups then. They are doing 80TB/month, you don't need 1GB/s nvme for that(assuming everything is uncached).

Why not? We did that for years if not decades.

Reply on news.ycombinator.com