Image hoster pushes 1.8 PB per month through Cloudflare CDN | Hacker News Reader

Image hoster pushes 1.8 PB per month through Cloudflare CDN

postimage.org

71 pointssheraz45 comments

Yeah unlimited does have it's limita....OMG you're funneling 1.8 petabytes?!?!! That's like 1,800,000 gigabytes or 1,800 terabytes every month. EVERY MONTH. Unlimited does not mean you can serve the bandwidth of a small country through 1 ISP.

"Please contact us if you have a CDN that is capable and willing of serving 1.8 Petabytes of outgoing traffic per month free of charge," -PostImage

Is the company run by spoiled entitled millennials? It takes a lot of equipment and staff to create a network that can serve that and they just want it all for under $1k? You know data centers have costs too. Other startups spend years of their lives putting together a team that can assemble infrastructure to handle that and they want it for free? For their barely-cash-flow-positive project that doesn't earn them a decent living...

I can't believe they just asked that. Sorry but PostImage sounds like the biggest moocher in the web right now. If your ISP provider says "unlimited data" does that mean you can funnel the traffic of Russia through your internet connection? You want for "free" what others are paying thousands and thousands of dollars for?

What is wrong with you? Seriously? It's like you're trying to run a top-tier web company on pocket change.

> If your ISP provider says "unlimited data" does that mean you can funnel the traffic of Russia through your internet connection?

I mean, yeah. Unmetered+uncapped bandwidth is a dumb idea that creates bad incentives, and anyone offering it deserve to be punished by abusers until they rescind their offer.

If people can't proxy all of Russia through your LTE data plan—or serve a billion image hotlinks per day through your caching reverse-proxy—then you should put that in the contracts you make them sign. That's what contracts are for. Say precisely where you're drawing the line, or you don't get to complain when people act as if you haven't drawn one.

Probably don't need years to make a team capable of serving 10-20Gbps of static content.

I do wonder what they thought would happen. CF's ToS are very clear. Why did they think they have a project, let alone a business?

Because some developers have an unreasonable expectation of what services should cost, and why those costs exist for real underlying resources.

Recall the uproar of that open source project that was breaking a piece of Github's infrastructure, and even when Github explained politely how to properly architect the fix, people were upset the project was going to need to pay for a CDN for their binaries.

Because early in the negotiation phase CF sales rep said that a $1k/mo plan would be a good enough upgrade, and this was an unpleasant but a sustainable figure.

I agree, but to be fair, I think "unlimited anything" is almost always misleading, Why not advertise with a generous free tier with clear limits?

Services offer this because if you don't have any users, it's easier than building in and enforcing limits. If your service gets traction then it makes sense to spend the time and money on development, not before. I personally wasted a lot of time on building limits for a service that has yet to gain traction.

People say "unlimited rates" but what they really mean is "indefinite rates".

Because if it was static human readable text content, they'd let you push that much though just fine. It's unlikely that you ever would get to that level, but they don't mind if you do.

I'm glad we figured out how to pin this on millennials.

Exactly, the real culprit here are either the baby boomers for spoiling us or gen z kids who grew up on freemium tier cloud services.

While I agree with the main thrust of your post, a couple of things jumped out. First, Cloudflare is a CDN, not an ISP. Second, 1.8 Petabytes per month is 5.48 Gigabytes per month. That is alot, but it isn't THAT much. For comparison, CDN's occasionally handle several orders of magnitued more traffic (although admittedly not for cheap).

Equivalent pricing:

Amazon Cloudfront: $59770 per month

Google Cloud CDN: $50000 per month

Fastly: $144200 per month

etc etc. If they can't get together $12000 per month for Cloudflare, they sure as hell don't have a viable business model.

Or like $1000-$1500 per month for a dedicated server with 10G unlimited bandwidth. This is as low as it gets.

Could you run the service off a few OVH dedicated boxes with 3Gbps? Sure, you could, but it's going to be painful, and at that point imgur is the better choice anyways.

No, it's not painful at all. I think multiple boxes on different ISPs is the way they should go, but it's going to be more expensive, than a single cheapest 10G box.

One server would not be nearly enough. PI is running on 14 dedicated machines, and even this is not enough to handle the load without a caching CDN. Source: I'm with the team.

If you have the engineering capacity, and your business model can at least sustain some amount of overhead, you can build your own CDN.

Look into PhoenixNAP and other similar providers that are located at non-profit peering sites and have reduced bandwidth costs.

PhoenixNAP (as the example) will lease you a mid range bare metal server with 16GB/RAM and 100TB of data/mo in the $150.00/mo - $300.00/mo range (depending on your processor needs; well optimized nginx doing just proxying should scream).

You can't exceed some unwritten bandwidth cap because you're explicitly paying extra for the 100TB of BW. For 20 caching servers firing 2PB/mo you're looking at around $3000 - $6000/mo...

Seems like you could optimize this down with fewer high powered caching servers and paying for the extra bandwidth... is the engineering time and $2500/mo really out of reach?

Best of luck.

Would one server be enough for their load though?

I can't believe they never researched CDN prices beforehand.

It probably started as a hobby project to fill a need and grew way more than they expected it to.

The project is now certainly larger than it was expected to become, although this doesn't absolve the team of not taking preemptive action sooner. Disclaimer: I'm with the team.

Sure as hell we don't. Setting up AdSense and content recommendations widgets did the trick for a long time and allowed the project to grow; the team has not been active enough in exploring better monetization channels. Also, before PostImage came to use CF, it used a number of cool tricks to conserve bandwidth that had since then been cut out as they were incompatible with CDN caching. Source: I'm with the team.

Yeah I priced Azure as well, and it clocked in at just over $60k USD

The solution is really simple, don't host images through a CDN. Though it might not be as fast, it will be way cheaper.

I'm really suprised they didn't look at all the stories of people before trying to start image or video hosts on Cloudflare and getting booted, as if all the other image hosts were too stupid to use Cloudflare for $200/month instead of paying CDNs tens of thousands a month.

Of course we were puzzled by CloudFlare's pricing structure. However, all of their advertising material kept reiterating that they were making money off low-tier customers by collecting lots of data required to properly serve their business and enterprise customers. So our sense of alarm and disbelief was somewhat suspended until CF contacted us about our bandwidth consumption, but even then we didn't realize how much trouble we were facing, as they first told us that a $1k subscription would do the trick.

Watching competing projects also added some degree of assurance. Without giving out any specific names, we are aware of a number of competing projects that even now keep heavily relying on CloudFlare (although I am obviously unaware of how much they pay), so we didn't consider it an abusive behavior. In contrast, we were also aware of at least one competitor that kept pushing a considerable portion of their uploads to imgur and basically parasitizing on them for a long time until imgur's abuse team finally took notice and brought down the hammer.

However, all of their advertising material kept reiterating that they were making money off low-tier customers by collecting lots of data required to properly serve their business and enterprise customers.

This makes it sound like we are doing some monetization of low-tier customer data. We are not doing that with any data (low-tier or anything else). That would be hella creepy.

We do track abuse (DDoS attacks, etc.) whoever they hit and use that information to protect other customers. So, in that sense low-tier customers help our overall business, but it's a common lie from our competitors that we are somehow monetizing traffic. Cloudflare's business is pretty simple: work out how to operate our services as cheaply as possible, charge web site/application owners more than that amount.

Apologies, that was a poor choice of wording on my part. While I do entertain the theoretical possibility that CF silently uses low-tier customer data in interesting ways (such as data mining and tracking user behavior), my actual expectations of the way CF actually uses this data exactly matches your description (DDoS mitigation & things like that).

Cloudflare claims unlimited bandwidth then says you are using too much bandwidth? Isn't this one of those blatant false advertising claims?

Also it's super unfortunate if they go down breaking so many images hosted so many places.

Cloudflare's terms of service specifically deal with this: "Using an account primarily as an online storage space, including the storage or caching of a disproportionate percentage of pictures, movies, audio files, or other non-HTML content, is prohibited."

If they put their website behind Cloudflare, but not their image serving domain, presumably that would be fine. Cloudflare is made for a site that serves up web pages with some content like your logo, a few images, etc. Those images can be cached and it's a small amount of data since something like the YC "Y" image is the same served for everyone on HN. A site like Postimage serves image content that's hard to cache since it's mostly different for each visitor.

Cloudflare is pretty explicit that it's not meant for a site that will be mostly serving up loads of pictures.

> Cloudflare is made for a site that serves up web pages with some content like your logo, a few images, etc.

For an interesting counter-example: 4chan (including its image-CDN subdomain!) is served through CloudFlare. It seems that CloudFlare really is willing to host and distribute multiple petabytes of image content per month, as long as it's for the sake of something else (e.g. pages of threaded conversation) that's not an "online storage space."

Though, I think in 4chan's case it might also help that pages and their image resources both get expired off of the site quite quickly. You can create a page that hotlinks image URLs from 4chan's CDN subdomain, but the correspondent resources at those URLs won't be there five hours later, so there's little point to doing so. Unlike most image-hosts one could name.

Postimage is an image hoster. 20 million of their 28 million unique monthy visitors are viewing images on pages they do not control.

This means the majority of their bandwidth is going to people whom:

a) Cannot be monetized via ads b) Are not on Postimage.com thus the site is breaking Section 10 of Cloudflare's services.

1.8PB/month is an average of 5.48Gb/s. Assuming you can make do with peak capacity of ~4x average, you can buy 20Gb/s of transit for ~$2k (these guys for example https://www.fdcservers.net/ip-transit.php). You can probably get a decent deal from somewhere more well known if you ask for a quote, with that kind of traffic, especially if it get a little publicity.

You'd get something, but it won't be a CloudFlare level CDN.

At best, you can get AdvanceHosters. They're Russian based but have 7 CDN pops across the US and Western Europe, they'll let you push 1.2PB for 7500 $/month

You won't get asia/oceana/australia traffic for those prices, and you certainly won't get S. American of S. African traffic for those prices.

Building your own CDN is not an option? Something small only to reign in the costs

Building your own CDN these days requires you to own a large ip block and ASN, and colo at several locations so the DC will actually bother routing to you, in order to deploy anycast.

Certainly too expensive for a small deployment which is why people buy from CDNs instead of setting up their own.

If it was as easy as just doing dns geolocation (which is awful due to geolocation failing and ISP caching), few people would bother buying from CDNs.

> If it was as easy as just doing dns geolocation

It is that easy. The importance of anycast is just PR, nothing more. Some very large CDNs use DNS just fine.

You shouldn't be anycasting TCP, only UDP. DNS is sufficient.

If anycast TCP didn't work, then cloudflare wouldn't exist. DNS is not sufficient. It takes forever to handle downtime because ISPs don't respect TTL.

You are incorrect, I'll try to explain.

Anycast is essentially an SPOF. Well, not only anycast and not anycast per se, but a single AS it is under. It breaks from time to time because of various mistakes, bugs, etc. and brings down every server as a consequence. This occurs roughly every couple of years and takes hours to resolve.

So, with anycast, if you have 10 servers in different places, you get hours of downtime for 100% of users from time to time.

With DNS, on the other hand, if one server goes down, it affects only 1% or so of users of a particular server, that have incorrect TTL in the resolvers they use, others see change in DNS right away and use working server. But, those 1% of users don't all go to that server at the same time, only small percentage of them does and also sees the old record. Leaving us with let's say 10% of that 1% on 1 out of 10 servers, or 0.01% of all users unable to see the new DNS record for an hour or so. If a typical server on some random AS goes down five times a year, you get 0.01% * 5 * 10 or 0.5% of users affected for an hour per year. Now if you use round robin and let users see multiple records nothing is even going to stop working in the browser for them, just going to make them wait longer until they see a set of working records.

To summarize, anycast is 100% of users not able to reach any server for hours every couple of years, while DNS is 0.5% of users experiencing slowness for an hour per year. In other words: anycast alone cannot be reliable enough for a CDN.

I never said you can't use dns on top of anycast.

You have to understand that anycast comes from ISP/networking people, who are biased towards network level solutions for anything and don't care about hard numbers, no matter how badly anycast looks there.

EDIT: Anyway, resilience, just like security, needs a threat model to avoid wasting resources on things that don't actually work, otherwise it's all just hype.

As someone who has looked into deploying a DNS based CDN, I have frequently found ISPs ignoring TTL.

Routers handle bgp updates much faster. Anycast routing is much more superior to geolocation which gives a completely incorrect location as high as 10% of the time. Anycast ips are also the best way to handle DDoS attacks. Attackers can easily shut down ips listed in DNS while people's ISP and browser repeatedly try to access the same dead IP for hours.

If DNS was an acceptable option, I would have gone with that instead of paying a CDN for an anycast solution.

These are all from my personal observations, they are not something I just heard from "ISP/networking people".

Never heard of them. Does imgur say how much traffic they have per month?

Never heard of them before but apparently they have been around since 2004 and they support 450k websites and their only revenue model is advertising. I doubt they had a solid technical and business plan behind this service.

Reply on news.ycombinator.com