We handle 80TB and 5M page views a month for under $400
blog.polyhaven.com
blog.polyhaven.com
It cost less than $2K/Month.
The cloud is crazy expensive. Private servers are beasts, and they are cheap.
Of course, for this price, you don't have redundancy and horizontal scaling.
You also don't have to maintain and debug a system with redundancy and horizontal scaling.
-But google/facebook/amazon...
-But uptime needs to be 99.999
-But everyone uses cloud
Most businesses are not a trading-market, have less then 100 peoples (aka you are probably not another amazon), and no bonus using a cloud/kubernetes etc.
But it's the same old story, in the 00's i used the ~same arguments against buying OracleDB ;)
I worked at a company once that, from higher up, said that they had to have five nines of uptime. We had some really good cloud engineers there (one guy set up a server / internet container for the military in Afghanistan; in hindsight he said they should've just sent a container of porn dvd's), and they really went to town. For that five nines uptime, you're already pretty much required to set up your infrastructure to use multiple availability zones, everything redundant multiple times, etc.
Of course, the actual software we wrote was just a bunch of CRUD services written in nodejs (later scala because IDK), on top of a pile of shit java that abstracted away decades of legacy mainframes.
Some stuff makes sense to put on iaas, dns often does for example.
If it’s 2k/month for a non resilient solution, it’s in the order of 4k for a resilient solution (you need to relocate assets both ways but it’s in that order of magnitude)
All means, no end.
If they didn't but cared about their business (they know I hope) and make hard requirements to IT, that would help. IT should then come up with solutions to those real-world problems. We're not talking just about hobbyists, do we?
That's what Eric Evans talks about in DDD as I got it.
Oh absolutely true, you don't want to look completely clueless in front of people who take your money to setup an infrastructure ;)
>and make hard requirements to IT, that would help
That would make stuff so much more easy. Often it even lacks even a inventory of used applications/hardware....network.
Just one example:
Made a plan for new hardware and network (new cabling etc), then i walked around the workshop and there was that dusty machine running...i asked what it is...a dos-machine with...wait...TOKENRING. That machine was an integral part of the whole workshop ;) However we made a virtual FreeDOS machine and buy'd a software converter for the machine protocol, the 25yo cnc-machine needed a new card (ethernet instead of token-ring, very lucky we found that thing) so there was that one little DOS-machine no one though about who could stop the whole "modernization".
That's why I give them money, so they bare with my cluelessness on their field. Would do it myself otherwise.
Or the way round: "I hire smart people for a lot of money, why would I tell them what to do" (Steve Jobs)
> that one little DOS-machine
The donkey does the work and the horse gets the fame.
But no matter how logically convincing your arguments were, most of the time upper manglement just went on buying Oracle, right...?
Isn't AWS down like every two months for a few hours? That's far off the 99.999% mark. No one can guarantee 100% uptime and sometimes it's even better to have that under your control (eg. have a dedicated server and a backup one from different providers).
My point is that, if you want the highest possible uptime, you shouldn't rely on a single (cloud) provider.
And if that one sys-admin wants to go on vacation? Or travel away from a computer? Or takes another job?
You can never have “just one” admin handling a server and being on-call 24/7.
Would you really want a job where you could never, ever be away from a computer because you’re the only on-call person? This doesn’t work.
Works fine for me. $20K a month for two people doing f*k all is insane.
You don’t even need a single employee to manage a single server…
With Cloud and SaaS. They are so abstracted from Hardware that their knowledge on basic hardware and server, everything from CPU, Storage I/O and Network are close to zero.
Side Note, aren't they 2U?
The production boxes are 2U, but it can be done in a 1U box.
Where? Costs vary hugely across the world
What? You set up the deployment once, and then you only need to touch it when things go horribly wrong, which is every couple of months, or to make minor quick tweaks and run some updates. Let's be generous, and say you need 10 h/month, which is about 1/16 of a person-month. And if things go horribly wrong, everybody drops what they are doing to fix things, anyway, no matter if you're on AWS, dedicated/colo or run your own data center.
When you significantly change your architecture/deployment, then you need to put in more time again, but if you build your code with need to scale and such things in mind from the get-go, then that won't come up much or at all.
Right, which is exactly why people pay extra for cloud managed services.
If things are going “horribly wrong” every couple of months then you must necessarily be on call 24/7 and never take vacation or time away. In practice, you need at least two people to manage on-call coverage so you’re not completely uncovered if someone gets sick, decides to take vacation, wants to travel away from a computer and so on.
In the real world, once most hosting platforms are up and running, the maintenance overhead is pretty low.
> It cost less than $2K/Month.
The solution in this article is serving on the order of 100TB/month for $400/month including a high speed global CDN, their API and database servers being hosted reliably, and redundancy and backup being handled by someone else.
Your solution is hosting on the order of 1000s of TBs of month (ignoring the database and other aspects of this website), but the price is an order of magnitude higher. You’ve also given up all of the automatic redundancy and hands-off management, and you don’t have the benefit of a high speed global CDN.
But more importantly, you have significantly higher engineering and on-call overhead, which you’re valuing at $0.
If anything, that only makes Polyhaven’s solution sound more impressive.
> Of course, for this price, you don't have redundancy and horizontal scaling.
Which is a huge caveat. The global CDN makes a big difference in how the site loads in global locations. Maybe not a big concern if you’re serving of static files with a lot of buffering, but they have a dynamic website and a global audience and they said fast load times are important.
> You also don't have to maintain and debug a system with redundancy and horizontal scaling.
But you have to do literally everything else manually and maintain it yourself, which is far from free.
All of these alternative proposals that value engineering time at $0/hour and assume your engineers are happy to be on-call 24/7 to maintain these servers are missing the point. You pay for turnkey solutions so you don’t have to deal with as much. Engineers don’t actually love to respond to on call events. If you can offload many of your problems to a cloud provider and free up your engineers for a nominal cost, do it.
Labour isnt expensive if you're operating towards a minimum needed to function, and your systems are sufficiently operationally stable.
Engineering labor is definitely expensive.
The entire $400/month bill for this linked website will only get you 2-3 hours of consulting time. They’re getting an enormous value by just offloading the work to someone else and not having to worry about it.
I'd argue it actually need a more expensive team.
Barring in-depth research (which I'd love to read if someone has any links), it's not clear on a 1:1 basis what's cheaper. Paying for someone's time to research hardware and talk to vendors, run POs for them, figure out where/how to install them (Equinix is expensive), and RMA hard drives as that comes up; versus not paying for that and instead paying a cloud vendor for that privilege. Throw on top a changing hiring landscape (how much 'sysadmins' cost vs 'devops') and it really depends on the size of this hypothetical fleet that we're trying to manage, and how complicated the backend of the site is. If there's no real backend to speak of, Cloudflare's CDN for static assets is going to be way cheaper, and available now, vs anything you could possibly build from scratch, that would maybe be ready in a couple months.
The entire team is composed of half of one dev.
I'm 100% sure it's way cheaper than anybody that has AWS on resume.
Of course, some things have to give, like the global CND, and some data guarantees.
Everything is a compromise. It's all depend of what is important for your project.
EDIT: also, my comment was not meant to oppose the article, but rather confirm the view that you should calibrate your setup to your project. Doing so will lead to great savings in hosting, and project complexity. A lot of projects don't need the cloud.
Who is on-call 24/7/365, never takes vacation ever, and is always available to fix the website?
It’s weird how much HN hates jobs with on-call requirements, but every time cloud services come up the solutions always involve forcing someone to be permanently on call to save a few hundred dollars per month in hosting costs.
If the system has a problem during the night, the users will wait until the morning.
The world doesn't stop because a streaming service is down for a few hours. It's not a medical service, or something thousands of businesses rely on.
It just looses a bit of money and users are grumpy for a day because they had to wait until they could access new content.
It's ok.
This one dev is literally on call 365 days a year and can never be away from a computer on vacation. If he leaves, the project has no one.
How is that not a problem? Surely you can see that this isn’t reasonable for anyone who wants to run a business, or any employee who doesn’t want the website to be their life.
> If the system has a problem during the night, the users will wait until the morning.
> It just looses a bit of money and users are grumpy for a day
If you’re running a website where extended outages are no big deal and you don’t care about lost revenue, then it’s not really a valid comparison to the typical business website.
Your situation is unique, not a model by which other companies should follow.
Whether the cost of trying to add another 9 to your uptime is worth the marginal benefit is for each company to decide. Each 9 gets exponentially more expensive. A lot of companies who think otherwise actually can afford (and will, sooner or later, be forced) to be down once in a while.
If you look at how frequently sites like Reddit used to have downtime, it doesn't seem to matter too much for consumer products. Having half a day downtime once a year might be completely acceptable.
I think you're severely underestimating how many businesses make a significant amount of money from their website, but doesn't actually have full-time developer available. An extended outage would cause significant revenue loss, but it's typically not a problem because outages are surprisingly rare when you (1) have a very stable traffic pattern and (2) you don't spend a lot of time adding features and refactoring. Pretty much every cloud outage we've seen was caused by a human configuration error, not fault machines.
No, I’m well aware. But there’s a simple solution to this problem: Don’t try to run and maintain your own servers. Pay a little extra to use cloud hosting and let it be someone else’s problem.
I take issue with these calls to setup and maintain your own custom solutions and servers, while also suggesting that the cost of engineering and maintaining such a custom setup should be ignored.
Running your own servers and not having developers is a recipe for an endless stream of contracting invoices that are going to cost far, far more than just using a hosted cloud solution.
The whole idea is that it's not a little extra, but 2 orders of magnitude.
No, apparently not.
> But there’s a simple solution to this problem: Don’t try to run and maintain your own servers. Pay a little extra to use cloud hosting and let it be someone else’s problem.
But that's only if it IS a "problem" in the first place. You have defined it as such, although Bitecode themselves said that for them, it simply isn't. (To paraphrase: "If the site is down, then it's down; so what? We'll fix it when we're in the office again.")
Just plain ignoring whether something is "a problem" or not is hardly being "well aware".
This just seems like a bad decision from a business perspective. You are willing to endure a significant outage that will cost a lot of money but not pay to prevent it? Machines can and will fail.
It seems it's criminal to run a service on the cheap, there must be a terrible human being behing it.
Well no, the dev is not attached 365 days on its computer.
A freelancer is hired part time for the duration of the vaccations. It cost a full dev salary for one month, taking in consideration training time, that's all.
> If you’re running a website where extended outages are no big deal and you don’t care about lost revenue, then it’s not really a valid comparison to the typical business website.
Most services can actually go down once a month, and be a viable business. You are not google or facebook.
In fact, most human service goes down for days: bakeries, lawyers, teachers, plumbers.
The fact your think internet services should be up all time is only in your head. Humans adapt perfectly.
It's not that a big deal. Most of our softwares are not as important as we want to think.
If you really want a 99.99999% up time, you going to increase your service quality by 10%, and your service cost by 1000000%.
The funny thing is, the downtime of ourservice has not being more than github's downtime in the last few years. So honestly the freelacer is mostly hired to have drinks on the house. Because monolythes are very reliable in the first place.
> Your situation is unique, not a model by which other companies should follow.
Every situation is unique, I never, ever stated it was " a model by which other companies should follow". You did.
There is no such thing as "a model by which other companies should follow". You must adapt to the situation and goals. Engineering is about compromises.
My post is simply stating the reality that you can get very far with good old tech.
And a lot of projects don't need the cloud, or high availability. Yet they pay premium for it.
You pay a full dev salary for one month every time someone wants to take a break?
It’s baffling that anyone can read an article about someone spending $400/month on cloud services and then start proposing things like this as an alternative.
Engineering labor is expensive. Cloud is surprisingly cheap once you factor in engineering costs.
> Cloud is surprisingly cheap once you factor in engineering costs.
No, it's not, since you need somebody qualified to operate it. And such qualificatied employees is very expensive. And you will need them on call anyway, since it will break, just in different way than a bunch of private servers.
I'd argue the cloud would be more expensive, even if hosting were not, because you need a more expensive team to run it.
How does the cloud solve this?
Like.. what are you expecting to happen? You'll just gaslight them into thinking they dont know how their business works?
The company I work for has a lot of stuff in the cloud. We seem to have quite a few position's worth of people permanently on call.
The percentage of "data center"-type on-call situations has perhaps gone down somewhat since moving into the cloud, but it has not gone to zero, and it was never the majority of problems anyhow.
It seems like you're sneaking the idea in that if they would just pay a lot more money, the person on call wouldn't have to be on call. I'd like to know what cloud you're using, because it doesn't seem to be any of the ones I know of. If your service is critical to your business or project, you've got someone on some sort of call (maybe not overnight call, which is the really rough bit), period, or you've got a business that can disappear at any second.
I have some clients who use AWS and others who prefer colo and/or dedicated servers from traditional datacenters. The latter group can afford to over-provision everything by 3-4x, even across different DC's if necessary. DC's aren't yesterday's dinosaurs anymore. The large ones have a bunch of hardware on standby that you can order at 3 a.m. and start running deployment scripts in minutes.
I get the impression that a lot of the critics in this thread don't really understand Cloudflare, how cheap it is, or even the concept of CDNs in general.
$20/month for Cloudflare Pro is a steal for what you get. Spinning up a dedicated server in a single datacenter somewhere isn't going to give the same results, especially if your users are geographically distributed like in this case.
If not free then very cheap, as I understood TFA: Wasn't that why they have two separate domains and serve static assets from one of them, to be able to use the cheapest Cloudflare tier for that domain = those assets?
You’re talking past the point here. It doesn’t matter how cheap if you’re fundamentally opposed to enabling cloud flare to reach its meat hooks further into the Internet.
This is no different from arguments about embedding google analytics or “just paying for windows” instead of using Linux.
You should take issue with all the other companies that have failed to deliver something as compelling.
What point? Nobody said anything about "cloud flare to reach its meat hooks" in the article or the above thread except you?
The OP mention "cloud flare to reach its meat hooks" in the thread attacking those who haven't jumped into Cloud flare's bandwagon by putting up a strawman on how that's only due to ignorance.
OP clarified that misrepresentation by pointing out the risk of allowing a single company to control the CDN market specifically and serving web content in general.
What? No. Cloudflare reported a revenue of half a billion dollars, and already controls about half the CDN market.
Let's put things in perspective: in comparison with Cloudflare's business, AWS is a minor player and an underdog with less than half of Cloudflare's market share.
Cloudflare is by no means a small company or an upstart or a David among Goliaths. Cloudflare is in fact and by far the Goliath of the CDN world.
Maybe not, but is the target audience that shills out $20/month really the type of people who have optimized their site to such an extent that shaving 50ms off the request latency by having your edge cache geolocated is really the type of thing that makes the difference? most of that group could probably do a lot of other optimizations that probably count for more.
[1]: https://stackoverflow.com/questions/536300/what-is-the-short...
I just think if your site takes 700ms, is there really a difference between that and 650ms?
3.44 seconds to do a search for "donkey"
The common mistake is to pick a server geographically close to yourself, only access it from low-latency connections, and then assume that everyone in the world is seeing the same thing.
Or to only visit your own site with everything already in the browser cache. If you're not seeing cold start loads, you're not seeing what every new visitor to your website is seeing.
Consider the Photopea.com website. The author explained in a comment below that he spends $60/month to host the site without a CDN. Several of us loaded the site and it took 2.5 - 5.0 seconds to load. He could sign up for a cheap Cloudflare account, reduce the size of his server (due to caching), and the load times for everyone would drop by a significant amount.
If you're hosting simple, static content like a blog for an audience that doesn't care about load times, then of course nothing matters. But for modern, content-rich websites (photos especially) it can actually be a substantial improvement to add a CDN even if you have a single fast server. You may not see it, but visitors from distant locations definitely will see a difference.
This is a conclusion i am extremely doubtful of.
Ping time new york <-> tokoyo is about 180ms. So lets say as a worse case the ping time to the single server is 180ms (its probably not that bad), and lets say the latency to cloudflare edge server is 20ms.
So using cloudflare on a cache hit (best case), you save something like 160ms per roundtrip.
Which don't get me wrong is a huge savings and worth it (although this scenario is hugely exagerated).
However say you want to load the page in under 1 second instead of 5 seconds. In this scenario you would basically have to have 25 round trips to bring the site from 5 seconds to 1 second just on rtt savings of having a geo located edge server. If your site needs 25 round trips to load, something else is clearly wrong. (And this is an exagerated case, the real world the benefit would probably be much less)
To be clear i'm not saying that geo located edge caches are bad or useless. They are clearly very beneficial thing. Its just not the be all and end all of web performance, and most people in the demographic we are talking about probably have much more important things to optimize (otoh using cloudflare is cheap and doesnt require a lot of skill, so it is a very low hanging fruit)
Per packet. If you're doing a cold start, you'll pay that latency cost several times over: first the TCP handshake (3 roundtrips), and then the TLS handshake (2 more roundtrips). That's 800ms of extra latency before you even get to sending the first HTTPx request.
Cold start latency matters a lot.
You’re forgetting that the TCP protocol itself is bidirectional. High latency connections will have lower throughout, especially at the beginning of transmission, because the data isn’t literally just streaming in one direction.
A CDN is more times than not the wrong answer to a real problem. Shave off your website and consider content-addressed protocols for big static asset download (like the textures from the article). If you run your website as a lightweight glorified Bittorrent index you'll notice your costs are suddenly a lot less, and you can still have a smaller "Download over the web" button as fallback.
I tend not to realise when my site goes viral, as I'm based in Australia whereas my largest audience is in b the US (and I'm a bit of a Luddite!)
Looks like with some dead basic optimisations (free versions of WP Fastest Cache and Autoptimise), my Wordpress site can handle around 1500 requests per second on a $5 DigitalOcean VPS before it starts to slow down.
On the old site, running on a shared host with less optimisations it would crap out at less than 10!
Seems like I don't need to worry about this after all.
For the rest of us who would rather not let our site fail, a quick one time $20 tarp over dumpster to handle the traffic in the meantime is good.
It's your choice if you don't want redundancy in place if any incidents happened.
Same thing applies to millions of 10kb files. Whether or not files are large is irrelevant to whether a CDN is a good idea.
If every file only ever gets requested once it may be pointless.
To pay USD $20 and unload a problem to some 3rd party service takes under 30min, while running a scalable, high performance web-server on low budget is hard and time consuming - even impossible for devs with no sufficient devops/admin skills, which is sadly a majority.
Back in the day, as a student with no money and a time to spare I used to do it all myself too, guerrilla style. Nowadays tinkering with my private servers would mean taking time from my real job, and that just doesn't make sense financially, my time is way more valuable and scarce now.
That's why we have the economy of specialists in the first place. One can do everything in DIY fashion, but in our civilization it's usually cheaper to hire a plumber's or carpenter's services than to invest in learning the skills, buying the tools and then doing it, if it's not your primary source of income. It's no different with CDN services.
If you are only talking about food, it may stretch a bit further, but you’re still far away from the majority of countries.
Once cloudflare captures the web market we'll all pay back with interest. They are not a charity.
hm
This occurs on many company blogs as well operating under a subdomain like blog.whatever.com
To be clear, this is a very tangential and irrelevant nitpick and I understand it does not contribute to the content of the website itself.
Or so goes the theory anyway.
Even if 5 of them would go down at the same time, the site would still work as intended (thought probably couldn't handle peak load with less than 3 or 4). If one of or two are down, nothing happens.
Also completely reinstalling a server takes around an hour.
Also, comment I was replying to mentioned a 48 euro budget, it is a price of a single server.
I'd be really interested to know, since to the best of my knowledge, they don't have PTP solutions in their datacenter.
In the finland datacenter, it appears there is a PTP running, though the offset is ~2.33 seconds off from NTP. Chrony says its a false-ticker, but I haven't really figured out how to get it configured correctly nor have I asked Hetzner for help. I've mostly just played around with it.
I did manage to also accidentally discover a local ISP's stratum 1's server is extremely close to me, as in a few microseconds away (I accidentally put hetzner's servers as a pool instead of a server and my NTP 'discovered' the stratum 1) ... I'm not using it, but I've thought about reaching out to them to ask if I can use it if I'm very nice.
Is the 2.33s stable? If so, after deducing the stable offset, it could still be valuable.
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
==============================================================================
PPS 64 40 67m -6.827 33.059 -72ms 99ms
PTP 6 3 20 -9.585 0.005 -119ms 10ns
192.168.100.1 6 3 88h -0.159 0.000 -12ms 3815ns
192.168.100.2 6 3 29h -0.176 0.299 -11ms 2817us
ntp1.hetzner.de 14 10 154m -0.003 0.006 -47us 12us
ntp2.hetzner.de 11 6 103m -0.001 0.005 -21us 5711ns
ntp3.hetzner.de 9 6 137m -0.006 0.007 -72us 8949ns
once you place value on determinism (in regards to time spent on a task) you want a tightly specced distribution mechanism and/or a feedback loop to communicate busystate back to the LB.
oh the irony
> Google Firebase is nice and convenient, but it is quite expensive. We could investigate some other managed database options in future.
I've only seen people get annoyed with Firestore over time, and migrating out of it. People do end up worrying. At first, They seem to choose Firestore because it's strongly marketed and seems suitable for a new project. And then data modeling, high prices or scalability becomes a problem.
So choosing something which makes it "one-click" to set up but total madness to manage is a really short-term optimization, only worth it for a pure prototype which you will throw away no matter how successful.
If you know you need those things to reach success, then it is better to make the up-front investment to get good tools for those.
If you still want to go with a cloud provider, AWS Amplify has some interesting tooling. I've build products both against Amplify and Firestore (and Firebase). Yes, Firebase is a few days to a week faster to set up (integrated user management, as you say), but AWS gives more sophisticated control and is built around scripted deployments.
You pay for it, of course, and I'm not arguing AWS vs running your own server. I am saying if the choice is AWS or Firebase, that a few days researching the choice would give you knowledge you could use for launching the next 10 prototypes you have in mind.
I think quite a lot of other people have mentioned in the thread that they are getting a lot of other "benefits" from using multiple services, but I don't see how these help solve the problem of data delivery besides taking advantage of the Cloudflare + Backblaze alliance which is $31 if their main website is a static one.
Let's say a month has 30 days -- 5 million views a months is 166,666k per day, 6944 per hour, 116 per minute, and ~1.92 per second.
1.92 qps.
Of course it's not expensive, it's a tiny amount of traffic!
A raspberry PI could handle the compute.
I don't think they are overpaying for what they are getting. $400 is not a lot for a proper site that serves a lot of bytes. I just don't think it's that impressive either -- it's just yet another website serving static assets through a CDN, with low QPS too, so I don't see why it's noteworthy.
Also note that they say "page views" which can translate in many more requests per each page opened.
So they're saturating 25% of a gigabit uplink every second of the month.
Have you considered using Firestore in Datastore mode[0]? It might make all of your reads free[1], though migrating could be a project.
[0] https://cloud.google.com/datastore/docs/firestore-or-datasto...
I'd also encourage you load your fonts late via JS. Your main JS package competes right now with WOFF files from Google Fonts for priority and there's no need for that.
[0] https://www.webpagetest.org/result/220106_BiDc42_428a3caec56...
can't tell if joke, so:
there are already enough sites that display content for a split second and then some script runs (or fails?) and there is either nothing on screen or an error message.
this is ridiculous - please stop!
What you’re describing has been thankfully avoidable for many years.
be sure i blame ad-tech and analytics
Now, I upgraded to $60 a month. I never used any CDN.
[0] https://www.udemy.com/course/vector-drawing-on-the-ipad-with...
> Now, I upgraded to $60 a month. I never used any CDN.
Why not go back to the $40/month plan and spend $20/month on Cloudflare Pro?
I was able to load photopea.com in about 5000ms, uncached. That's not terrible, but it was slow enough that I wondered for a few seconds if the site was broken. A CDN would cut that load time massively, and it wouldn't even be a net cost increase because you could downsize your server.
I mean really cold cache from the above user is at 5000ms that's pretty horrible. When I tried I got 2800ms. Take a look around it seems to be mostly from fetching static content which is a classic problem CDNs solve.
People have been citing $30/month. Even if you value your time at min wage, its probably still cheaper than setting up your own varnish (and that's assuming you don't care about improving the last little bit of latency with geo located edge servers)
One doesn't even need pro. In our use, Cloudflare's pages.dev and workers.dev serves single-digit TB traffic with triple-digit million hits at $30/mo. We pay $0/mo for origin servers since there aren't any.
To answer the actual question - its the obvious answer. They make a product that works well and is relatively cheap. Is it without drawbacks? obviously not, nothing is. However, for a significant segment of the market the value proposition makes sense.
So what? Unless your website is offering some superficial junk that can easily be found elsewhere, you’re not going to lose a user.
For something like photopea, there aren’t sub-second alternatives out there.
> For something like photopea, there aren’t sub-second alternatives out there.
if there is't one now, there will be. Besides, with a little CDN and tweaking you improve your users enjoyment. Why not?
FYI I tried to bookmark it with cmd-D, but the app has hijacked that combo!
Regards oftentimes forgotten Linux user
If I update a single file, how long it takes until nobody in the world can access the old version anymore? Does it take seconds / minutes / hours? Also, some files should not be cached at all (like PHP requests).
I am afraid it would take me days or weeks to learn everything and to cofigure the CDN properly, and I am risking being offline for a part of the world during that time. Also, if there is a problem at the CDN, my website would be broken, too. If someone could help me, you can write me at support@photopea.com
I use it myself, but just as a normal static file CDN, but they have dedicated tabs in their UI for video stuff. Their bandwidth pricing is also very reasonable ($5/TB with their "bulk" option and $10/TB for standard)
Two thoughts:
1) CloudFlare offers incredible service. What would need the team of netadmins/sysadmins can be handled via their UI easily. I use Argo in one project and yes, it is a very efficient tool. CloudFlare is becoming like AWS, knowing its tools is a skill. Maybe one day well see an official CloudFlare Solution Architect certification?.. :))
2) For such traffic heavy websites I don’t think there’s a better combination than Static Websites hosted over CDN, which is cached aggressively and for the dynamic part serverless as a backend. If you configure it properly (cache optimization, rate limiting to not get a huge paycheck if DDoSed, etc…) this setup is like set-up-and-forget-it, without the need to invest loads of resources into it.
Not sure how most CDNs work, but for Cloudflare, content is not actively pushed to edge nodes, it’s only pulled to an edge when needed and then cached there usually for an hour or two.
The caching is still distributed, and all of the tcp+https roundtrips are done to a local data center which makes things faster.
> what the source article describes is essentially just that it is cheaper to run a static website with a CDN than without - right?
Right. The cost savings comes from caching done by the CDN, and public static content is super easy to cache.
The distributed part makes things faster, but doesn’t necessarily save money.
I believe it's also against Cloudflare's TOS to use it as an asset-hosting platform.
I'm currently clicking through various European hosting services and they seem to offer great dedicated servers at good prices. I cannot wrap my head around these ridiculous costs I keep hearing about. A guy I know was telling me how they were spending tens of thousands of dollars per month on AWS at his company.
Is it because everyone is writing their stuff on node.js, putting 500kb of JS in every web page they serve, putting Docker everywhere when a chroot would suffice, microservices, Kubernetes, or hell even writing SAPs when we don't need them?
I am genuinely confused
Also, you know what you're getting security-wise with AWS, and no one will blame you when your website / service goes down because AWS is down.
My response may come off as rude, but that's not the intention. From the above quote you've vastly over simplified all but the most simplistic environments. Someone could run all of the above for well under 10k month - those are not the reasons why costs are high
> putting 500kb of JS in every web page they serve
that's not much and CDNs serve that without issue or significant cost
> putting Docker everywhere when a chroot would suffice
Docker does not have much overhead over chroot with a large drop in security/isolation
> microservices
there are good and bad ways to do this - does not need to cost much. saying "microservices" is so broad to have no meaning in this context
> Kubernetes
think about why someone might run kube. Again, not that expensive unless you're really small where having master nodes would have a big impact on costs
> or hell even writing SAPs when we don't need them?
Do you mean SAP? SAP is the largest non-American software company by revenue. I don't think anyone likes working with SAP. You think people use it for without reason? There are reasons. Think about what those might be
But those are just details - when building anything of a decent size, nothing is a simple CRUD. There are always exceptions, limitation, biz logic, migration issues, schema issues. I don't care if you use Postgres, Mongo, Kafka or $SOMETHING_COOL
Then there's data retention. It's very very easy to keep PB of data in s3 or other data stores for biz or compliance needs
Then AWS gives IAM, which is very helpful in teams > 5
Docker adds plenty of attack surface that could easily be avoided by using any sandbox or a systemd unit file.
Because most web apps aren't simply CRUD apps. It's a meme said by armchair engineers who believe they could build twitter in weekend.
Only people here on hn seem to think that everyone is slinging k8s and 1000 layers of microservices to build the most complex things on earth; the rest is earning their paycheck by building some forms in php.
Sadly many here are overengineering crud apps and then somehow making them out to be something more than crud apps because somewhere in the future they might be (probably not); as in, you can build 'a twitter' in 1 weekend and it will even probably handle good volume of users ( more than Twitter did at startup; it was often down or very slow) etc if you just take postgres with php on a few $/mo server. It is a crud app at that time (the flow for these user authored posts is built into every trivial and complex framework these days). Also it will most likely be bankrupt in a few months as that is what happens to startups; I rather spend money on building the business first and then scaling the tech; which is exactly what Twitter did. On hn it seems a lot of people work backward in that regard but that probably has also something to do with being a techie and VC interest in scalable tech.
If you're building a basic CRUD app and you have low traffic numbers, you don't need to spend tens of thousands of dollars on AWS.
But it's not as simple as picking a dedicated server, setting it up, and hoping for the best. At minimum you need periodic backups, testing and staging environments, solutions for rate-limiting your API, and so on.
And the "everything is just a CRUD app" meme is just that: a meme. Usually engineers who repeat this have only ever worked on simple CRUD apps, so they don't understand what it's like to work on anything different.
> I'm currently clicking through various European hosting services and they seem to offer great dedicated servers at good prices. I cannot wrap my head around these ridiculous costs I keep hearing about.
If you're working on the types of problems that are a good fit for setting up a single dedicated server on a random hosting provider, you're not working on the same types of problems that necessitate $10K AWS bills.
> I am genuinely confused
I've worked on projects with $10K+ monthly AWS bills. It wasn't node.js or Docker or Kubernetes. It was the sheer volume of connections we had to maintain and data we had to process.
But even if we could reduce our $10K/month AWS bill to $1000/month with a lot of engineering effort and manual management of our own servers, what would we gain? If I had to hire a single additional devops person or engineer to help manage this custom solution, the entire savings would be wiped out. And then some!
Wasting AWS resources isn't smart, but trying to DIY your solution to everything rarely makes financial sense when you look at how much engineering effort it takes and how much engineers cost. If I can spend $10K/month to avoid hiring 1 additional engineer, it's a financial win. I do not care if someone thinks they can do the same thing in $1K/month with endless amounts of custom setup and maintenance. I don't want it.
It also reduces the number of moving pieces that we have to manage manually, which reduces the on-call burden, which keeps people happier.
I really don't understand the anti-cloud hate on HN. It doesn't mirror the real engineering world at all.
Complexity inflates HN readers CVs and keeps them employed.
I've had this argument too many times to count. Every page view does not need to be dynamic per user request. Creating a sane cache policy reduces origin resources, servers, cost, etc... and surprise gives a better user experience at the same time.
A few years ago when I hosted what felt to me like a popular site, I was serving serving > 150K page views a month from a 1.5Mbps adsl uplink. With that said, back then you could gzip everything and there was no letsencrypt or CloudFlare walls across the board (didn't exist yet).
Good thing bandwidth is easier to come by these days.
HDRIs, high res textures & 3D models
'Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."'
> Running a massively popular website and asset resource while being funded primarily by donations has always been a core challenge of Poly Haven.
My brain went: poly.. fill.io or something
Thank you, sorry, am embarrassed.
No, that's low by an order of magnitude. It is 16MB (16,000KB) per page view.
They are pushing about 245 Mbps out of Cloudflare (averaged over the month). Wholesale IP transit prices are anywhere from 10 cents to a dollar per Mbps depending on volume. Cloudflare dumps 40% of their traffic over peering, putting their price at $14.70/mo. Ignoring fixed capex of servers, Cloudflare is making about $25/mo on this customer.
Given the $11 Backblaze bill, I estimate about 2 TB of data.
A capable dedicated server with 2 TB of disk and 1 Gbps unlimited bandwidth will run about $30 in a major European metro, maybe double that for the US.
With a grand total of $370 vs. $60 worst case, they are spending 516% more to be "serverless."
Edit: Yes Cloudflare has more than one server. Double the price and put one in the US and you still come out ahead.
Edit 2: I'm not saying one way or the other is better. Just that the title is very clickbaity for a "put a credit card into a website" payoff.
This would be approximately equivalent in spec to what you could build for $2500 purchase cost if buying a 1U machine and colocating itself.
For 30 bucks a month you'll get something very old and weak.
Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59
The bulk of the data transferring is static files because they're hosting large assets, but the rest of the article is about their API, database, etc.
A single server in a single datacenter isn't comparable to a global CDN. They made it clear in the article that they value global latency, and they're willing to pay more for it.
> Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59
That has mechanical spinning hard disks and a CPU that was released literally a decade ago.
You're not going to replace Cloudflare, Firebase, their API server, and a global CDN with a single 10-year old machine serving files from a mechanical spinning hard disk, hosted by a company that almost nobody has heard of.
But fine, if you don't believe it I will happily extend an invitation to come see in person my racks of spinning rust and 10 year old servers in Los Angeles or Fremont that are running services that are used by millions of end users. My email is in my profile.
Which they ditched as soon as physically possible. The internet has changed a lot since then. That much should be obvious.
You can't reduce this article to "millions of end users" and then equate it with every other service. There's more to this article/service than you're suggesting, and there's more to a website than just users/month.
If you're really strict on this it's trivial to preload these files in RAM after compiling your assets so you don't have to wait for a request.
They're running a dynamic website with an API and other dynamic functions.
It's not a static fileserver.
It might make more sense if you visit the website in question: https://polyhaven.com/
If you have 2.75TB of files to serve and have a bunch of simultaneous requests for different files:
- everything doesn't fit into the RAM page cache, unless you have 3TB of RAM in your server
- for files that aren't cached, you have to read from the drive
- when you have a bunch of different requests for different files, you're going to have a lot of disk seeks. Seeks are expensive on rotating media.
Serving a large collection of static files at scale can be quite challenging. At my previous e-commerce company, back in the early 2000's, we did our own image serving with RAID1 multiply-mirrored drives (before SSDs!) so we could multiply the seek capacity by the number of drives.
You could probably do it with Cloudflare for a similar price, but that's not what this article is about. The article says that their Cloudflare (not Argo) bill is only $40/month, and that's only because they have two domains. It's $20/month per domain.
The $400/month figure isn't just for static file hosting. It includes their API, database, and global CDN. They make a point to say that low-latency, global CDN is a priority for them, which is why they're paying extra for Cloudflare Argo.
I'm not sure why everyone is comparing what they're doing to a single fileserver hosted somewhere, serving up static files without regard to global latency. It's apples and oranges.
Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe.
It is like saying you are trying to optimize an ocean cargo ship for acceleration time.
Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte.
They're also not a static file serving website.
If they were just serving static files and they didn't care about latency, they could pay $20/month for Cloudflare Pro and be done with it.
I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.
I've built three CDNs in my career, one at the Tbps scale. Latency only has two factors in throughput: time to scale window size, and retransmits. Modern TCP stacks handle the prior just fine, and the latter is only an issue with packet loss. You can also turn on HTTP/3 and remove your reliance on TCP entirely.
> I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.
Because I actually looked at the site and read the article. The vast majority of the site is large static files. The frontend is a static JavaScript app that calls an API server. That API server is running on a $5/mo VM.
My offer still stands if you'd like a tour of a datacenter and see the sausage being made at scale. I'll throw in lunch and answer all your scalability questions if you like. But this thread is growing quite long and veering off topic.
I might take you up on that offer, as I might need that for some new project I'm preparing.
Where are you located?
It serves 25Mbps stream in real time to users in Japan and the US from a data-centre in Europe with no issues.
It's not just a static fileserver, despite some of the comparisons in this thread.
> You can't run a high-throughput database on spinning disks unless you disregard data integrity
Of course you can. That's the way it was and still is done. I'd say one can't run a database with any data integrity unless there are disks spinning somewhere very near.
I'd take another box for like $40 there just for the replica, which would sit mostly idle anyways.
But piling on CDNs, clouds and variuos other opex and indeterminate risks instead of some thinking out of this little cloud shoebox is just baffling.
I’m a happy customer of theirs. The UI for managing things is a disaster, but once you get ssh access, it doesn’t really matter.
Not even close to comparable. I think you're ignoring all of the functions they're paying for. They're not just hosting a few files. Also, your "worst case" is literally the most optimistic best case for a standalone single server, which completely disregards any redundancy and assumes that the $30/month server is truly unlimited/unmetered in every way. That's not a good assumption.
Cloudflare is a global CDN that will be fast for everyone regardless of location and will soak up bursts of traffic with ease. It's also fast in a location-independent way, which won't be true for a single server in a single datacenter somewhere.
Finally, the amount of effort it takes to do this with Cloudflare is trivial. The amount of effort it would take to maintain and optimize a standalone solution is not negligible.
Cloudflare seems like a very good deal to me, unless you value your time at $0 and you have access to these truly unlimited, high-performance $30/month servers.
Back in the day we used Texan Colo data centers with DrFTPD to do just this at massive scale.
Bytes in, bytes out. Not a popular opinion here, but I firmly believe it's not necessary to play the CentralizedFlare game to get a winning outcome.
Cloudflare Pro is $20/month per domain.
Trying to run everything through a single server in a single datacenter just doesn't compare for globally-accessed websites like this, even with a 1Gbps unmetered link.
I've accessed plenty of sites physically located in the USA from various EU countries, not to mention AUS and I didn't notice much difference as long as they had a good interconnect.
Maybe not the best for YouTube, but there's only a few of those.
Don't get me wrong, I see the appeal and fell for CF early on but not a fan anymore since they've grown into such a behemoth.
Absolutely power corrupts absolutely.
Everything might make more sense if you visit their website: https://polyhaven.com/ It's not just a dumb, static file server.
EDIT: The questions below are literally explained in the article, so I'm going to give up on this thread.
I feel like we're heading into the weeds. Do you get the broader point I'm trying to make?
At a glance, it still looks like mostly a highly cached static website + search. Its not exactly a high complexity backend.
Things are always more complex at scale, so there's probably more to it then what appears at first glance, but its not obvious what that is.
> Absolutely power corrupts absolutely.
This seems to capture the gist of the argument clearly. I hadn't realized cloudflare had grown large enough to fall into the "too big to not be evil" bucket already.
Back in the day when Joyent ran their public cloud you could beat Cloudflares free plan from Joyents Japan datacenter by a tiny amount in most cases.
Anyway, those days are gone and Cloudflare provides great value and probably the best quality network available. So if you have the problem they solve then you would be a fool to not put some serious thought into considering them.
If it’s not your core competency and unless you possess the vast breadth and depth of skills to dig yourself out if something goes wrong, it seems like a distraction at best and a fatal mistake at worst, with the end result of saving 1-2 engineer hours worth of money per month.
For some self-hosting is a no-brainer, but I suspect that community is much smaller than many here would expect.
Fact: Things break most often because of humans doing things - changes, like deployments and config modifications.
Set things up right to begin with, and in my experience you can leave them running for a long time without intervention.
Downvote all you like, but it's the truth.
RedHat usually break setups every 3-4 years which I personally consider way to frequent.
If I configure a server with automatic updates then I would like it to run with very little maintenance for at least a decade and a long time would be two plus decades.
Is the job getting done for the donors? I bet they don’t care about $300 if it means the website is highly available and the maintainer isn’t burned out from giving away $200/hr of opportunity cost all the time.
> Fact: Things break most often because of humans doing things - changes, like deployments and config modifications.
Totally agree, and sometimes success in your project forces your hand as you are pressed to add functionality or need to scale. If you can guarantee that you’re doing this work once, ignoring hardware failure or scaling, the scales may very well tip toward self-managed.
For a vast majority of projects, I think it just plainly makes sense to go this route, hence the success of these centralized hosts. It is the pragmatic option.
PS. For what it’s worth, I’m sorry you’re being downvoted simply for having an viewpoint that others disagree with.
10 cents is far from the lower end for high quality IP transit.
Cloudflare doesn't pay anywhere near $14.70/mo for 80 TB/mo. Even I spend much less than that on 80 TB in a month. On top of that, peering makes it even cheaper for Cloudflare, as you said. Cloudflare is very likely using peering for much more than 40 % of the traffic now, so it's even cheaper for them.
The worst peering region for Cloudflare was North America in 2016 with 40 % peering according to Cloudflare blog posts.
This reads like a paid marketing post for Cloudflare.
It's astonishing how many times the author conflates browser cache-able assets with cached assets on a CDN. When a browser downloads a static asset if the web server is configured properly those files will be cached and there won't be a need to re-download them for a very long time.
There is of course the issue with modern web frameworks like React generating a single massive js/css file that bundles everything all over again in a unique file busting all previous cached versions across all users' browsers just because a comma was added to a sentence.
Keep your js/css files small, serve them from your own web server and set a reasonable expire header, no need to pay Cloudflare $40/mo to continue gatekeeping the internet.
If anything, I’m fine with a new competitor to BigCloud that’s been unchecked and increasingly hostile (cost-wise).
Asset storage: Backblaze B2 – $11 (replace with Cloudflare R2)
Web hosting: Vercel – $20 (replace with Cloudflare Pages, $0 cost)
Database: Firestore – $100 (replace with Cloudflare)
API: Vultr – $5
Image hosting & optimization: Bunny.net – $27 (replace with Cloudflare)
Domains: Cloudflare – $4
Email fees: MXroute – $3
R2 is 0.015$ per GB while backblaze B2 is just $0.005.
Does Cloudflare have a database offering? Do you just mean the Worker KVs, or is there a full relational database?
If you were OK with your content only updating periodically you might be able to do away with Firestore, Argo, and Vultr.
You definitely do a great job of mitigating db hits. Consider publishing all the data to a CDN using something like Gatsby.
In this scenario your CI/CD would build a static website from all the assets and publish it to a static server.
You mentioned that only your view counts are the only really dynamic part. You could just estimate and emulate those or design them a different way.
Martin Fowler has an article on this: https://martinfowler.com/bliki/EditingPublishingSeparation.h...
Then at the end he says he uses Bunny as the CDN for images. Why use Bunny instead of Cloudflare? One more moving part.
I presume it is cost related?
I speculate it's because Cloudflare's TOS prohibits you from using their CDN to serve "video or a disproportionate percentage of pictures, audio files, or other non-HTML content" on typical plans. See section 2.8 on https://www.cloudflare.com/terms/ . Since PolyHaven seems to be a purveyor purveyor of 3D assets and they use Bunny to host "all of our images shown on the website (thumbnails, renders, previews, etc.)", I'm guessing a lot of their assets exceed Cloudflare's TOS.
PolyHaven mentions using Bunny's image optimization service, so that'd factor into the decision too.
- Storj has the lowest cost per TB, but charges for egress - Wasabi doesn't charge for egress, but has a fair use policy where if you egress more than you store they can kick you off their platform[2] - Wasabi is also a better fit only if you plan to keep your files around for 90+ days (they have a minimum object retention period) - The bandwidth alliance is only available for HTML related content unless you're on a specific paid plan[3]
[0] https://www.storj.io/ [1] https://wasabi.com/ [2] https://wasabi.com/paygo-pricing-faq/#free-egress-policy [3] https://www.cloudflare.com/terms/#28-limitation-on-serving-n...
*opinion is mine, not my employer's
You can sign in with
Username: hn@hn.com
Password: hackernews
-
The site is currently in demo mode, and the db will be wiped before launch - feel free to sign in and poke around. Also it's hosted in Australia currently, so site may be a little slow for those in the US.
Source is here: https://github.com/jjcm/soci
I want to make a live video streaming website a la Twitch.tv. How much would CloudFlare charge me to stream 8 Mbps to ~80,000 viewers for 4 hours?
I thought about doing this a while ago, after I left my job with an ISP. I would just have bought transit from them (and maybe made them regret their 10Gbps for $1000/month plan ;)
19,200,000 minutes at $1 per 1,000 minutes. https://support.cloudflare.com/hc/en-us/articles/36001645087...
Though, mux.com is decent too, from what I've heard; for certain workloads, LiveStream and Vimeo might be cheaper.
There's Peer5, PeerTube, and BitTorrent in the P2P space (among many such solutions).
First time hearing of them (due to my own ignorance probably). Any details on them?
I am asking this because YouTube seems to have a massive monopoly both on technology as well as content/audience/network-effects. Even if you can get all the people of Youtube to shift over, you still need to solve the problem of bandwidth and the costs associated with that.
That's massively overpriced, with current prices you should fit into a 5 dollar budget.
They have sweet deals with the cloud providers?
Also see their post about it https://blog.cloudflare.com/aws-egregious-egress/
The more relevant number for this comparison would be our gross margin — much we have to spend on things like bandwidth and servers to service our customers divided by the revenue those customers generate — which in Q3 2021 (our last reported quarter) was 78%. Which is… pretty good for a services business like ours.
I don't know the specifics of this customer, but I don't see anything that leads me to believe our margins would be out of the ordinary with them. There are a lot of scale economics in our business. In other words, we can definitely do things for less money because we service a lot of customers than any one customer could hope to do it on their own.
My two cents from a regular guy to a billionaire, :).
Noting they are operating at a slight loss, during a period where they want to drive growth, isn't "naysaying". The question is whether the current pricing is tied at all to the "operating at a slight loss". He replied to say it mostly wasn't.
Remember, for example, when Uber rides were really cheap?
Foul play?