Hacker News Reader

> A capable dedicated server with 2 TB of disk and 1 Gbps unlimited bandwidth will run about $30 in a major European metro, maybe double that for the US.

Not even close to comparable. I think you're ignoring all of the functions they're paying for. They're not just hosting a few files. Also, your "worst case" is literally the most optimistic best case for a standalone single server, which completely disregards any redundancy and assumes that the $30/month server is truly unlimited/unmetered in every way. That's not a good assumption.

Cloudflare is a global CDN that will be fast for everyone regardless of location and will soak up bursts of traffic with ease. It's also fast in a location-independent way, which won't be true for a single server in a single datacenter somewhere.

Finally, the amount of effort it takes to do this with Cloudflare is trivial. The amount of effort it would take to maintain and optimize a standalone solution is not negligible.

Cloudflare seems like a very good deal to me, unless you value your time at $0 and you have access to these truly unlimited, high-performance $30/month servers.

But they could deliver potentially ~300TB with a 1gbps unmetered link. Not that hard to come by, could even get to 3PB for $300/mo.

Back in the day we used Texan Colo data centers with DrFTPD to do just this at massive scale.

Bytes in, bytes out. Not a popular opinion here, but I firmly believe it's not necessary to play the CentralizedFlare game to get a winning outcome.

> But they could deliver potentially ~300TB with a 1gbps unmetered link. Not that hard to come by, could even get to 3PB for $300/mo.

Cloudflare Pro is $20/month per domain.

Trying to run everything through a single server in a single datacenter just doesn't compare for globally-accessed websites like this, even with a 1Gbps unmetered link.

I must be missing something, are you able to elaborate on how CentralizedPro Flare for $20 solves the OPs problems?

I've accessed plenty of sites physically located in the USA from various EU countries, not to mention AUS and I didn't notice much difference as long as they had a good interconnect.

Maybe not the best for YouTube, but there's only a few of those.

Don't get me wrong, I see the appeal and fell for CF early on but not a fan anymore since they've grown into such a behemoth.

Absolutely power corrupts absolutely.

Are you referring to the linked article? It literally explains what Cloudflare does for them.

Everything might make more sense if you visit their website: https://polyhaven.com/ It's not just a dumb, static file server.

EDIT: The questions below are literally explained in the article, so I'm going to give up on this thread.

Yes, and if they could pay 1/20th of their current bill to solve it, I'm sure they would.

I feel like we're heading into the weeds. Do you get the broader point I'm trying to make?

Note: my comment is no longer relevant and looks insane because Badwolff made edits to completely change their message.

> Everything might make more sense if you visit their website: https://polyhaven.com/ It's not just a dumb, static file server.

At a glance, it still looks like mostly a highly cached static website + search. Its not exactly a high complexity backend.

Things are always more complex at scale, so there's probably more to it then what appears at first glance, but its not obvious what that is.

> Don't get me wrong, I see the appeal and fell for CF early on but not a fan anymore since they've grown into such a behemoth.

> Absolutely power corrupts absolutely.

This seems to capture the gist of the argument clearly. I hadn't realized cloudflare had grown large enough to fall into the "too big to not be evil" bucket already.

brace yourself before you read up on cf's origin

Good network connectivity makes a huge difference.

Back in the day when Joyent ran their public cloud you could beat Cloudflares free plan from Joyents Japan datacenter by a tiny amount in most cases.

Anyway, those days are gone and Cloudflare provides great value and probably the best quality network available. So if you have the problem they solve then you would be a fool to not put some serious thought into considering them.

For the average person, why make it unnecessarily hard on yourself?

If it’s not your core competency and unless you possess the vast breadth and depth of skills to dig yourself out if something goes wrong, it seems like a distraction at best and a fatal mistake at worst, with the end result of saving 1-2 engineer hours worth of money per month.

For some self-hosting is a no-brainer, but I suspect that community is much smaller than many here would expect.

It seems disingenuous to count engineer hours at $200USD/hour for charitable donation-run projects. Is it charitable to funnel your donor money to CF? I'm sure CF is grateful for the donation..

Fact: Things break most often because of humans doing things - changes, like deployments and config modifications.

Set things up right to begin with, and in my experience you can leave them running for a long time without intervention.

Downvote all you like, but it's the truth.

I guess the question is what do you consider a long time?

RedHat usually break setups every 3-4 years which I personally consider way to frequent.

If I configure a server with automatic updates then I would like it to run with very little maintenance for at least a decade and a long time would be two plus decades.

> Is it charitable to funnel your donor money to CF? I'm sure CF is grateful for the donation..

Is the job getting done for the donors? I bet they don’t care about $300 if it means the website is highly available and the maintainer isn’t burned out from giving away $200/hr of opportunity cost all the time.

> Fact: Things break most often because of humans doing things - changes, like deployments and config modifications.

Totally agree, and sometimes success in your project forces your hand as you are pressed to add functionality or need to scale. If you can guarantee that you’re doing this work once, ignoring hardware failure or scaling, the scales may very well tip toward self-managed.

For a vast majority of projects, I think it just plainly makes sense to go this route, hence the success of these centralized hosts. It is the pragmatic option.

PS. For what it’s worth, I’m sorry you’re being downvoted simply for having an viewpoint that others disagree with.

While you are right CF provides a lot more, it really must be noted that Hetzner SB servers start at 28 EUR and they _do_ come with unlimited bandwidth and 2 x 3TB disks. For 30 EUR you can get a E3-1245 with 2x4TB disk. Just noting.

Yeh, but that Hetzner server is only in one location so doesn’t reduce the RTT between your visitor’s and your server (particularly for visitors who are far away)

And for 99% of cases that doesn’t matter at all. People don’t like hearing it but it’s true.

That would be a single node though, in contrast to cloudflare's edge, and would probably slow down download speeds for everyone outside europe

> Wholesale IP transit prices are anywhere from 10 cents to a dollar per Mbps depending on volume.

10 cents is far from the lower end for high quality IP transit.

Cloudflare doesn't pay anywhere near $14.70/mo for 80 TB/mo. Even I spend much less than that on 80 TB in a month. On top of that, peering makes it even cheaper for Cloudflare, as you said. Cloudflare is very likely using peering for much more than 40 % of the traffic now, so it's even cheaper for them.

The worst peering region for Cloudflare was North America in 2016 with 40 % peering according to Cloudflare blog posts.

I was conservative in my estimates for IP transit because I don't know what CF pays, but the numbers only get better for them.

Cloudflare might not pay that on a per TB basis, but where are you getting 10 gige transit for under $1k??

Market rate for a 10gbps port is now less than $1k/mo

Where? I'm talking about 10gbps transit.

HE.net is advertising "Get BGP+IPv6+IPv4 for $0.09/Mbps!" in public

Wow, but I suspect that's going to be for 100g.

Linode offers a 50 dCPU with 10Gbps out for $960. Or if you just need the bandwidth and don't need the CPU power, you can combine three $30 servers with 4Gbps each, and put them behind a $10 load balancer, for a total of 12Gbps out at $100/mo.

https://www.linode.com/pricing/

That 10gbps comes with a 10TB traffic limit. The other poster was talking about a dedicated port with no limits.

There are no limits on traffic. It costs $0.01/GB across the board.

This is not what 10gbps transit means.

By all means, enlighten us.

Plenty of providers, depending on how many 10G/40G links you're willing to buy, check out https://blog.telegeography.com/global-ip-transit-prices-decl... to see how much prices have gone down.

An actually capable dedicated bare metal server with a good amount of RAM, CPU and disk, and 1Gbps will be more like $250-300/mo USD.

This would be approximately equivalent in spec to what you could build for $2500 purchase cost if buying a 1U machine and colocating itself.

For 30 bucks a month you'll get something very old and weak.

The bulk of the traffic is serving static files. We figured out how to do this with "old and weak" hardware in the 90s.

Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59

> The bulk of the traffic is serving static files.

The bulk of the data transferring is static files because they're hosting large assets, but the rest of the article is about their API, database, etc.

A single server in a single datacenter isn't comparable to a global CDN. They made it clear in the article that they value global latency, and they're willing to pay more for it.

> Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59

That has mechanical spinning hard disks and a CPU that was released literally a decade ago.

You're not going to replace Cloudflare, Firebase, their API server, and a global CDN with a single 10-year old machine serving files from a mechanical spinning hard disk, hosted by a company that almost nobody has heard of.

Yahoo, Google, and eBay were built with gasp spinning hard disks.

But fine, if you don't believe it I will happily extend an invitation to come see in person my racks of spinning rust and 10 year old servers in Los Angeles or Fremont that are running services that are used by millions of end users. My email is in my profile.

> Yahoo, Google, and eBay were built with gasp spinning hard disks.

Which they ditched as soon as physically possible. The internet has changed a lot since then. That much should be obvious.

You can't reduce this article to "millions of end users" and then equate it with every other service. There's more to this article/service than you're suggesting, and there's more to a website than just users/month.

Why not? We did that for years if not decades.

What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM... that completely negates any SSD advantage. SSDs have little to no benefit in serving static files.

If you're really strict on this it's trivial to preload these files in RAM after compiling your assets so you don't have to wait for a request.

> What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM

They're running a dynamic website with an API and other dynamic functions.

It's not a static fileserver.

It might make more sense if you visit the website in question: https://polyhaven.com/

Higher up someone calculated that they have 2TB of data, considering their $11/mo Backblaze bill. I calculate $11/$0.004/GB and get 2750 GB.

If you have 2.75TB of files to serve and have a bunch of simultaneous requests for different files:

- everything doesn't fit into the RAM page cache, unless you have 3TB of RAM in your server

- for files that aren't cached, you have to read from the drive

- when you have a bunch of different requests for different files, you're going to have a lot of disk seeks. Seeks are expensive on rotating media.

Serving a large collection of static files at scale can be quite challenging. At my previous e-commerce company, back in the early 2000's, we did our own image serving with RAID1 multiply-mirrored drives (before SSDs!) so we could multiply the seek capacity by the number of drives.

Sure you are for something like serving large image files (like, say 3D asset files?)... I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month doing pretty much that, multi-terabyte image datasets and gigabyte-sized DL models. Plus a few websites, which you've probably seen. Zero optimization, just rsync and nginx with defaults, works fine. (It'd be way cheaper except I wanted another 20TB or so of rust to get room to add more datasets to serve - after all, the server is barely breaking a sweat.)

> I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month

You could probably do it with Cloudflare for a similar price, but that's not what this article is about. The article says that their Cloudflare (not Argo) bill is only $40/month, and that's only because they have two domains. It's $20/month per domain.

The $400/month figure isn't just for static file hosting. It includes their API, database, and global CDN. They make a point to say that low-latency, global CDN is a priority for them, which is why they're paying extra for Cloudflare Argo.

I'm not sure why everyone is comparing what they're doing to a single fileserver hosted somewhere, serving up static files without regard to global latency. It's apples and oranges.

> They make a point to say that low-latency, global CDN is a priority for them

Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe.

It is like saying you are trying to optimize an ocean cargo ship for acceleration time.

> Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe.

Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte.

They're also not a static file serving website.

If they were just serving static files and they didn't care about latency, they could pay $20/month for Cloudflare Pro and be done with it.

I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.

> Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte.

I've built three CDNs in my career, one at the Tbps scale. Latency only has two factors in throughput: time to scale window size, and retransmits. Modern TCP stacks handle the prior just fine, and the latter is only an issue with packet loss. You can also turn on HTTP/3 and remove your reliance on TCP entirely.

> I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.

Because I actually looked at the site and read the article. The vast majority of the site is large static files. The frontend is a static JavaScript app that calls an API server. That API server is running on a $5/mo VM.

My offer still stands if you'd like a tour of a datacenter and see the sausage being made at scale. I'll throw in lunch and answer all your scalability questions if you like. But this thread is growing quite long and veering off topic.

> My offer still stands if you'd like a tour of a datacenter and see the sausage being made at scale.

I might take you up on that offer, as I might need that for some new project I'm preparing.

Where are you located?

Email is in my profile.

You can use different TCP congestion control algorithms, not just cubic. BBR2 has no issues with high RTT even with packet loss and is fair to other TCP streams.

It serves 25Mbps stream in real time to users in Japan and the US from a data-centre in Europe with no issues.

24 GB of RAM means a lot of ability to cache data in RAM, which the OS does already by default. It of course depends on the type of content you have but generally traffic follows a power law, so most content can be served from RAM directly. The write side I'm a bit more worried about, as e.g. SQLite issues a sync request on each transaction (not sure about the other SQL implementations, should be similar there tho), and those might require seeking.

The article explains that they have database and API usage. You can't run a high-throughput database on spinning disks unless you disregard data integrity.

It's not just a static fileserver, despite some of the comparisons in this thread.

Like SSDs bring any new angle to this.

> You can't run a high-throughput database on spinning disks unless you disregard data integrity

Of course you can. That's the way it was and still is done. I'd say one can't run a database with any data integrity unless there are disks spinning somewhere very near.

I'd take another box for like $40 there just for the replica, which would sit mostly idle anyways.

But piling on CDNs, clouds and variuos other opex and indeterminate risks instead of some thinking out of this little cloud shoebox is just baffling.

Spinning disks lose data integrity? You will be surprised where the cloud servers store your backups then. They are doing 80TB/month, you don't need 1GB/s nvme for that(assuming everything is uncached).

These are pretty capable in my experience: https://www.netcup.eu/vserver/

I’m a happy customer of theirs. The UI for managing things is a disaster, but once you get ssh access, it doesn’t really matter.

How would you define actually capable? I can find servers with recent Xeon processors, good amounts of RAM and NVMe SSD with a gig uplink for around the 100 dollar mark.

Do tell where you found your 1Gbps unlimited bandwidth for $30. Sure it wasn't $300?