Peering agreements between Backblaze and Cloudflare | Hacker News Reader

Peering agreements between Backblaze and Cloudflare

reddit.com

243 pointsdecrypt61 comments

Public internet exchanges are huge in Europe, you can basically reach every network in Europe by connecting to a few large exchanges for a fraction of what you'd pay for tier 1 transit (granted, you still probably need it to reach some eyeballs). I wish we'd see the same concept pop up in the US, as these exchanges are definitely not as popular there.

https://www.de-cix.net/en/locations/germany/frankfurt/statis...

https://www.ams-ix.net/ams/documentation/total-stats

https://portal.linx.net/

A big part of it is that simply connecting your own network to these exchanges is a lot more expensive in the US due to increased land size and corrupt legislation.

Due to the intense regulations (which are bought and paid for at the legislative level, and mostly exist to protect the large incumbents like Verizon, AT&T, Comcast, CenturyLink, et al from any new competitors) you end up having to buy your "last mile" loop (which, due to the US's size and general car-centric design, is much more than just a mile in most cases) that connects your network (in your office, or data center, or whatever, over to the exchange point) from a small number of monopolists.

They're thousands, or tens of thousands of dollars per month.

"We don't care. We don't have to. We're the phone company."

Internet Exchanges doesn't mean that a peering agreement is also in place automatically. For example: https://www.ams-ix.net/ams/documentation/general-terms-and-c... 3.3 Peering arrangements are not covered by these AMS-IX General Terms and Conditions or the AMS-IX Connection Agreement. The Customer is responsible for the negotiation, conclusion and implementation of peering arrangements with other users of the AMS-IX Infrastructure.

While you are correct, many networks connected have in fact an open peering policy: https://www.ams-ix.net/ams/connected-networks

sure and FRA IX also has a non-need for individual agreements policy

https://www.de-cix.net/en/locations/germany/frankfurt

but since I've seen the issues in my country in Europe where peering is not taken for granted, I wanted to make it visible that IXs are great yet open peering doesn't always come with them

These used to be more common in the US and some still exist but the big telcos have all stopped upgrading their equipment which deliberately chokes the available bandwidth at those locations. They also deliberately don't on-board any new ones. This is the same strategy they used against Netflix if you recall - deliberately don't upgrade the linecards at choke points to ensure Netflix packets get dropped. Netflix eventually caved (and raised prices) so we're all paying for it on both ends: as ISP customers and Netflix subscribers. Who doesn't like a bit of double-dipping?

They really really want to go back to the long-distance telephone days where people paid according to some kind of metering plan. That's the whole push behind getting rid of Net Neutrality (which is how the internet always worked up to that point - so much so we had to invent a new term to describe what was taken away). Why invest in new plant, fiber, or equipment? Just stop upgrading or upgrade very slowly and let increases in data usage dump money directly into your pocket. A great gig if you can get it.

The reasons residential ISPs don't peer is two-fold.

a) They're nearly all Tier-1 networks, and Tier-1 networks have very hard to meet peering requirements; AT&T, Verizon, and Sprint have been Tier-1 since forever, and now they're the dominant wireless carriers. AT&T and Verizon are also large residential ISPs, and CenturyLink is another, and they're a Tier-1 too. Comcast is effectively Tier-1 in the US, but they don't have an international network (yet?).

b) residential connections are setup to be unbalanced flows, and peering policies at networks with restrictive policies generally require roughly balanced flows. Very few networks that do business with consumers are going to get balanced flows. Maybe a consumer oriented backup service can, if they mostly receive backups and don't send a lot of restores; 1-1 messaging without persistent server storage should be pretty much balanced, but nearly everything else is pretty much never going to be balanced. Of course, tier-1 networks also won't peer with you if you're a customer or have been a customer recently; so if you do end up with appropriate flows to your transit provider, you'd need to switch to another provider for some amount of time before coming back to ask for peering, and they may increase requirements in the meantime.

Things were different when we had local/regional ISPs (dial-up and DSL line sharing). As those got big enough to justify multiple transit providers, they would gravitate towards peering places, because you can pickup multiple transit providers easily, and also peer to reduce costs for both networks when connecting to other networks at the peering place. A local ISP doesn't really care about traffic balance, they just want traffic off their transit link; and content networks don't care about traffic balance, they just want traffic off their transit link.

If I had my way, we'd get mandatory line sharing for the last mile (and something for wireless?), so that we could have meaningful competition in IP networking at the local level again. Something where all the last mile infrastructure providers are prohibited from offering retail service would be best; they could keep their tier-1 ISP business, and their last mile business, but someone else would have to at least glue the two parts together.

> you can basically reach every network in Europe by connecting to a few large exchanges for a fraction of what you'd pay for tier 1 transit

Erm.

No. No and, erm, NO !

Members at peering exchanges can be split into two categories "old boys club" and "everyone else".

The former category are mean as mouse shit.

They won't peer with "everyone else", in many cases they won't even peer with you even if you have a decent amount of traffic related to their AS.

They certainly won't publish ANY routes on the exchange route servers.

For example, BT (incumbent UK telco for those who don't know) are famously restrictive about who they will peer with at LINX. There is an old joke between LINX members, "Q. How do you know you've made it as an internet network ? A. BT agreed to peer with you over LINX". Basically the template answer when you ask BT for peering is "NO. Go away now you silly little man, you should be ashamed for asking."

That's not to say European exchanges are worthless. They are great for interconnecting with other mid/small networks. They are great for connecting to CDNs, the hyperscalers, and the Facebooks, Googles etc. of this world.

in all fairness, the original op did say "basically"... I am peering in a few different IXes in Europe for personal stuff, mostly learning, and i get Apple, Google, Cloudflare, Facebook (even though i dont use them), Netflix and a lot of other places included... I do also agree about the whole "old boys club". My main ISP is Liberty Global (Virgin Media Ireland) and anyone i have talked to about getting a link with tells me its paid peering... would be handy, but too expensive...

Also, they dont peer with Cloudflare direct either... Ahh well...

Huh, that sounds interesting. How does this peering work?

Is it mostly just a bunch of intangible configuration and routing, or do you actually have equipment racked in a colo that offers peering arrangements?

I'm also curious how the pricing side of things works out. Sounds like it's not that expensive...

So... It's mostly though vms from companies who run smaller educational IXes and the likes of Vultr who have bgp offerings... I have some details on https://as204994.net about it. Costs are currently in around $150 per month including v4 and v6 leasing... Details of the IXes I'm in are up there too...

That certainly takes me back. In the early days of my career, I tinkered with BGP and VPNs amongst friends and cities, over radio and internet. The legacy lives on.

https://lodge.glasgownet.com/2015/10/26/the-old-backnet/ https://web.archive.org/web/20050421022318/http://wiki.backn...

We even had BGP over AX.25 for a very short while if memory serves.

Sounds very like what dn42 does now. That's how I started with bgp... Then got my own v6 and then v4 space and ended up on the real internet. Dn42 defiantly helped in learning. This is cool that it's been around that long!

It does indeed! Had never heard of dn42, although I've been out of the loop with regards to this stuff for years now. Will certainly be having a look at dn42 :-)

Several of the big incumbent ISPs in most European countries are not as eager to peer because they also have a B2B side of their business that sells you the bandwidth you would otherwise need.

But for most smaller end-user networks it's great to be able to peer with Netflix, GCP/Youtube, AWS, Azure easily in the exchanges, because that's where the majority of their user traffic comes from.

It's a bit more complicated in reality. Most of the ISP consumers, even if present on the exchanges, aren't openly peering on it. It's great to interconnect with smaller networks and providers, but consumers ISP ? You have to either have a lot of traffic _they_ send to you, otherwise, peering isn't possible, or complicated, or you have to pay.

My brain is a bit fuzzy today but this comparison seems... off? Yes an interconnect inside a colo is going to be dirt cheap. But you can’t compare it to a huge backbone spanning continents that gives you reach across the world. Everyone should peer with everyone they can locally. Use the tier 1s for reach beyond what you can peer with. (This is not a defense of tier 1s and their peering policies or pricing.) Also this is why it’s always amused me when DC guys quote their raw switchport capacity with pride. Gotta multiply that by distance traveled to get a real comparable metric.

Yeah, this is a bad take from Backblaze. If ISPs had 99% profit margin you'd see it in their quarterly reports. Spoiler warning: they don't.

Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it.

Frankly, I’ve never seen a good take from Backblaze once I got past their marketing blog posts.

Zero-knowledge encryption? You don’t need that[0][1].

Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3].

A pattern of violating of basic and well-known software security principles[4][5][6]? Bodge in fixes. Don’t tell customers about it. Cancel the public bug bounty programme, claim it’ll be back in an indeterminate period of time[7] and replace it with a private one instead[8].

[0] https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_...

[1] https://help.backblaze.com/hc/en-us/articles/217664798-Secur...

[2] https://old.reddit.com/r/backblaze/comments/igaqse/please_al...

[3] https://old.reddit.com/r/backblaze/comments/k764xq/backblaze...

[4] https://twitter.com/zetafleet/status/1304664097989054464, more discussion at https://news.ycombinator.com/item?id=24842871

[5] Hard-coded credentials in deployed apps: https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backb...

[6] Relying on client-side validation only: https://www.youtube.com/watch?v=1LC0aEZFVz8

[7] https://news.ycombinator.com/item?id=24849334

[8] https://www.backblaze.com/security.html

> Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3].

There is basically never going to be a meaningful reduction in SSD life from that amount of wear.

Every file is written once (when you change it) and then a second time (when the client writes the same file into a backup chunk). Halving the SSD lifetime is not something I’d personally call meaningless, especially for QLC drives which have two orders of magnitude fewer P/E cycles than SLC drives[0], especially since there’s no good reason for writing these chunks to disk in the first place.

[0] https://www.techradar.com/news/nand-and-cells-slc-qlc-tlc-an...

> Halving the SSD lifetime is not something I’d personally call meaningless ...

No, it doesn't halve the SSD lifetime. It is a little odd of you to make that claim.

First of all, you are assuming that 100% of SSDs die from being written, and die so quickly and so often from being written to that this is an enormous concern everybody using an SSD in a laptop should be worried about. In reality, a modern SSD you buy today might last you 10 years of normal use (including running Backblaze), but the rest of your computer simply won't last that long. Personally, I've never had one of my SSDs die. For any reason, including too many writes. Often I throw them in the trash perfectly working because I have upgraded their size for less money. Other times I get a whole new computer which comes with a whole new SSD. Backblaze runs SSDs in all of our "monitoring" computers that write the ever loving bejeebus out of their SSD drives 24 hours a day, 7 days a week FOR YEARS. I think we've had 1 SSD go bad after YEARS of writing at a rate 10x up to 100x higher than any home laptop user could achieve. And we're not sure it died due to too many writes.

So even if Backblaze doubled the writes (it doesn't, see below) it wouldn't have any measurable effect on your SSD's life. This is just provably false by watching servers that aren't running Backblaze write 10 or even 100 times as many times for 5 years as any consumer could ever achieve, and they still don't kill the SSD drives.

So, is Backblaze doubling the writes on your computer? Heck no. Backblaze only backs up valuable data files, not your operating system and certainly not your log files. The vast majority of writes to an SSD are not writing your one photo that you took today to disk - they are database transactions and log writes and system log write and system registry writes, etc. Backblaze puts in a lot of effort to exclude "chatty" log files (log files that are written all the time, all day long) because it saves our customer's network bandwidth and saves Backblaze space in our datacenter. So IN REALITY Backblaze's behavior is adding maybe a small single digit percentage of additional writes. The exact profile will matter what you use your computer for, but take email -> if you get 200 emails a day saved into an Outlook PST Inbox you might have 600 write transactions per day. But Backblaze won't back that up after every email, it would waste too much customer bandwidth. Backblaze might only back that up once per day. And only the part of the PST file that changed! So in reality, Backblaze only added 3 or 4 writes out of 600.

> especially since there’s no good reason for writing these chunks to disk

Different customers have different needs and different profiles, and Backblaze has to accommodate them. One reason a full snapshot is taken of the large files is that Backblaze wants a consistent "snapshot" of a file. On slow network connections it takes some customers 4 days to upload their Outlook PST file ONCE. Meanwhile they keep getting mail. If Backblaze didn't take a clean snapshot at one moment of time of this file, you'd get this corrupted file where the first 10 MBytes were from a file on Monday, and the last 10 MBytes were copied on Thursday from the same filename but totally different contents. Surely you see how that might be an issue?

And customers may or may not have the RAM required to hold a 10 GByte file in RAM, and none of them have enough RAM required to hold a 500 GByte file in RAM. Backblaze needs to store these "snapshots" someplace, so it stores them on disk.

Saying this is "for no reason" is disingenuous.

> And customers may or may not have the RAM required to hold a 10 GByte file in RAM, and none of them have enough RAM required to hold a 500 GByte file in RAM. Backblaze needs to store these "snapshots" someplace, so it stores them on disk.

But at the same time, customers are very likely to have enough RAM to hold a 100 MB file, and definitely have enough for a 10 MB file in RAM. It might be better (faster, easier, less abusive to the drive) to store small files in RAM (for some well-chosen value of “small”, of course. Likely one that depends on the system RAM size and bandwidth.)

> to store small files in RAM (for some well-chosen value of “small”, of course...

The Backblaze client code currently puts that dividing line at 100 MBytes. Any file less than 100 MBytes we call a "small file" and those don't get subdivided into 10 MByte chunks on disk, they get slurped up into RAM and just held there for the entire duration of being sent. For that code path, Backblaze can be configured to send up to 30 of these 100 MByte files in 30 separate threads which means (if the customer configures it this way) it can eat up 3 GBytes of RAM at peak. If you have that much RAM, it can really rip. If you don't, we recommend using fewer threads. :-)

The sub-division into 10 MByte chunks taking a one time "Snapshot" of the large file is for all files larger than 100 MBytes.

The security issues are disconcerting. And their attitude is not encouraging at all.

What are some decent alternatives to Backblaze?

"What are some decent alternatives to Backblaze?"

If only there were a cloud storage provider that gave you an empty UNIX filesystem to do anything you want with, using any tool that worked over stock-standard SSH.

If only ...

To be apples-to-apples, you (2-2.5c/GB) do charge 5x what B2 charges (0.5c/GB) (for just the storage part) for small customers.

For a lot of people, a 5x price difference disqualifies the term "alternative", although you are in line with industry/S3 pricing. It's just that Backblaze has specifically differentiated by being super duper cheap.

It's why I use them for my personal projects.

Agree. It’s relatively straightforward to underprice ‘brand name’ cloud object storage, S3 and the like.

rsync is likely making a margin choice here rather than marketshare choice.

For people who, like you, may not be aware - there is a "HN Readers" discount as well as other (.edu, FOSS author, etc.) ways to chop the headline price down substantially ...

Your snark aside, I pay $5 per month for Backblaze to backup my entire PC. I would pay many multiples more for rsync.

Just an aside, but I found that I was overpaying for backblaze backup. I switched to duplicati backed by backblaze b2 and went from $5/mo. to more like $1.70/mo. for the backup. As always though, YMMV.

You shouldn't really worry about object storage security, and your approach to using it should be as if everything you store in/on it will be published in the newspaper tomorrow.

Wtf am i reading here:

"HTTPS doesn't hurt anything and actually has some nice side effects (it separates backups onto a separate port than the majority of your web traffic, allows for traffic shaping if you want to do it)."

Disclaimer: I work at Backblaze so you should check up to see if what I say is true and keep me honest.

> Zero-knowledge encryption? You don’t need that[0][1].

This is not our position, and I feel it is disingenuous of you to say that. If you read YOUR TOP LINK from TWO YEARS AGO I explain that Backblaze specifically offers 4 levels of security, one of which is Zero Knowledge, and we think that is a perfectly valid decision for some customers. If you want zero knowledge, choose it at Backblaze! But some customers have other requirements, and you are insisting that we remove part of our product line up that is very useful to other people.

Here are the four levels of security Backblaze offers, most of this is from this post 18 days ago I wrote: https://www.reddit.com/r/backblaze/comments/kroqhn/private_e...

1) Security Level 1 - no security. Backblaze B2 can serve public websites, on purpose, the way stuff that you want to go viral and share with everybody. https://www.ski-epic.com is supposed to be readable, not locked. These are totally open files for anybody to download.

Security Level 2 - username/password/2-factor. This is a good choice for the customers who would rather error on the side of recovering their passwords than losing all their backups. In this level of security, your Online Backup is secured by your username and password, and every file is "encrypted at rest" (all the files are always encrypted when stored on disk). In this mode, all it takes to decrypt your backup is to sign into the Backblaze website with your username and password, and 2-factor verification, and you can prepare a ZIP file restore to download. You can ALSO prepare an encrypted USB restore hard drive to be sent to your home. This particular level of security has the advantage (or disadvantage to the security sensitive) that if you forget your password, you can "recover" it through your email account. If you use 2-factor (like we recommend), a hacker with your username and password will STILL not be able to gain access to your files. This is a good choice for a customer who is not super overly concerned about hackers possibly getting their data, and just wants to backup a public website like https://www.ski-epic.com (which anybody could get from the website anyway), or some photos of their wedding. It errors on the side of being able to recover the data no matter what. Some things you want BACK more than you want to destroy the files in the event of a hacker breach, or if you forget your password.

Security Level 3 - Backblaze Personal Backup with a "custom" unrecoverable private encryption key. In this mode, your account is protected with your username, password, 2-factor like the above "security level 2", but also an ADDITIONAL "unrecoverable" passphrase that Backblaze does not know in any way, shape, or form for years. Without the passphrase your files cannot be decrypted. You only provide the "passphrase' in the event of preparing a restore, and then your passphrase is never stored on disk anywhere at Backblaze, it is held in RAM. For years your files are encrypted at rest where even if Backblaze is ordered by government subpoena to hand over your files Backblaze cannot comply even if we wanted to, we have no way to decrypt your files. If you choose this level of security, DO NOT FORGET that passphrase because there is no possible way to "recover" it, and without it your files are GONE. You cannot recover them, Backblaze cannot recover them, the CIA or FBI cannot recover them - they are GONE. Now, as long as you don't forget that passphrase, then years later when you actually need to prepare a restore, there is a security "window of exposure" for as little as 20 minutes ONLY IF (and when) you go to restore. If you are under arrest -> just don't prepare a restore, and the FBI simply cannot get the contents of your files. An alternative strategy is if you have some particularly sensitive files, like incriminating evidence of your crimes or your tax returns or a file with all of your passwords to your bank accounts, put these few files in a small encrypted file on your laptop, and EVEN IF you prepare a restore the FBI (or Backblaze, or hackers) cannot get the contents of those files. Now, the reason we allow the customer to provide this passphrase is it is STILL relatively friendly, and we can prepare 8 TByte USB restore drive (that is encrypted) and sent to the customer's home. While there is that tiny exposure if the restore servers were ACTIVELY hacked during the 20 minutes while the restore is being prepared, some customers (especially if they are just storing wedding photos and cat pictures and public websites) prefer this option. Many of our customers are naive (not computer expert) customers, and many, many, many customers find this particular security level, convenience, and tradeoff useful.

Security Level 4 - "Zero Knowledge". Customers can use Backblaze B2 with a zero knowledge product such as some of the products listed on this web page: https://www.backblaze.com/b2/integrations.html Some of those 3rd party tools are even open source if a customer doesn't trust commercial products and wants to read the source code. This is a very useful security level for customers that would rather LOSE THE DATA than ever have it intercepted by law enforcement or a hacker. Backblaze offers this level if you want it! However, there are some very real world drawbacks of this level of security. First of all, Backblaze cannot prepare an 8 TByte hard drive with all your files organized correctly as they were backed up and send it to you fully organized, because "zero knowledge" demands Backblaze never, under any circumstances, know any of your file names. This is a more secure system, but it is less convenient to restore. Also, some of our customers don't have the bandwidth to download 8 TBytes conveniently, so you may have to pay more money for a faster internet connection to make this type of backup work for you. The other thing that is "dangerous" or less convenient and might lead to data loss in this scenario is that if a customer stores the "Private Key" on the laptop that is being backed up, and the laptop SSD dies, they actually lose the backup also, because you need the keys to decrypt the backup. So any customer who chooses this security level needs to make several copies of their "Private Key" they never give Backblaze, probably on multiple different external hard drives in their home (in case one of those copies of the key "goes bad" you need multiple copies). Beware of a house fire that destroys the laptop, and all the extra copies of the security keys, because this will result in loss of the backup also! So one idea is to store the keys in a DIFFERENT online service (or two or three online services) elsewhere on the internet (not at Backblaze, because that is what is demanded by "zero knowledge"). Again, this is a great choice for customers that PREFER DATA LOSS and extra time and thought and effort and cost over allowing the FBI or a hacker to gain access to their files. This is a perfectly valid choice, and Backblaze offers it.

Now sometimes people accuse Backblaze of not being a "zero knowledge" backup. What they are saying is that only #4 is "valid" and they really want Backblaze to stop offering #1, #2, or #3. I reject their insistence that we not offer easy backups for cat pictures and that customers must all share their technical ability (and bandwidth, and time) to download the encrypted data instead of getting a USB restore hard drive prepared with all their files in a friendly fashion sent to them. Some customers have different use cases, and Backblaze strives to support all four of these use cases! Pick which is right for you!

> This is not our position, and I feel it is disingenuous of you to say that.

Every time I’ve seen someone bring up ZKE, you demur or say that most of your customers don’t need it because they are non-technical and their data is not sensitive enough to warrant that level of protection. Every time, you strongly imply that the only people who would need zero-knowledge encryption are criminals. (You’ve just done all of this again here.)

If your position is not what I claimed, and you do think ZKE is something users need, I’d recommend that you spend your engineering budget on implementing it, instead of using that time to do things like rename your Git branches[0].

> Now sometimes people accuse Backblaze of not being a "zero knowledge" backup. What they are saying is that only #4 is "valid" and they really want Backblaze to stop offering #1, #2, or #3.

Could you please give a link to an example of someone actually saying this?

> Security Level 3 - Backblaze Personal Backup with a "custom" unrecoverable private encryption key.

The private key for decryption is automatically generated and sent to Backblaze without any passphrase at install time. It doesn’t get replaced when the passphrase changes, so the original unprotected private key could just be stored and reused later. Setting a passphrase sends it to Backblaze (necessarily, since the private key is only stored on Backblaze servers), so that too could easily be stored (accidentally or intentionally). Restoration requires decryption on Backblaze’s servers, which both requires the passphrase to be sent and also puts all the user’s decrypted data onto persistent storage.

All these design flaws make “Level 3” not meaningfully more secure than “Level 2”. Because of this, it seems to me to exist primarily as a marketing tool, rather than to meaningfully protect users from threat actors. I think we will probably have to agree to disagree on this point.

> Security Level 4 - "Zero Knowledge".

This is not a thing that Backblaze offer. Saying “hey, we have storage, use some third party software to get actually encrypted backups” is no different than me selling some server space and advertising it as “zero knowledge”.

> This is a more secure system, but it is less convenient to restore.

Outside of the issue of forgetting your backup passphrase, there’s nothing more inconvenient about a properly designed zero-knowledge backup. In fact, I would say that the way Backblaze works right now—requiring users to go to a web site to download a zip file, or have a disk sent in the mail—is way more inconvenient than most other backup software which restores directly from the client, including ones with ZKE like Arq and SpiderOak.

[0] https://www.backblaze.com/blog/code-and-culture-what-happens...

What seems to happen when there's insufficient competition for ISPs is that they under-invest in network upgrades. So you wouldn't expect to see crazy profits, it's more a matter of high prices and inefficient operation.

"Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it."

As evidence for your point I would point to Hurricane Electric (he.net) who I have found to be very progressive, cost-competitive and easy to work with.

For historical reasons we[1] are connected with init7 throughout Europe but in the United States and Asia we work with he.net anywhere that we can.

[1] rsync.net has a long-standing (15+ year) relationship with init7 in Zurich.

FWIW, HE has also been pushing for IPv6 to be awesome waaaaay longer than IPv6 has been very popular.

I don't know them, but I am supremely glad for their 15+ year efforts along these lines.

Cloudflare is providing the huge worldwide backbone and saying it's cheaper for them to peer where you are and transport it on their backbone as needed rather than pay Tier 1's to do it.

The difference in cost largely comes from being peering not transit though. I.e. they aren't handling traffic for 3rd parties on their backbone (well, in this case, they do offer that though), just traffic between the mutual peering entities. As a result you're never sending them something they didn't already have to pay to get.

Also it's not as simple as distance. A lot of data between 2 world class datacenters is pretty easy and cheap, the fiber is already there. A lot of data to the rest of the world is a PITA and a physical/logistical nightmare.

Edit: ok you expanded your initial comment. So yes they aren’t providing transit and your own comment explains why this isn’t a good comparison, it appears cheaper than it really is because Cloudlfare aren’t paying transit to get your data.

Disclaimer: My mum works for AT&T on internet related stuff.

This post has large elements of truth to it, especially for dumb hops over tier 1 which should be much cheaper than they're currently billed, but there's a lot more going on at AT&T than most people realize. Especially across the oceans. It's not quite as simplistic as this take makes it out to be, since the 5%ile case for networking a packet over tier 1 really is expensive to handle, maintain, bill, and provision.

That said, there should be better regulations on tier 1 inclusion. On the other hand, they did make deep investments into infrastructure that took decades to become profitable. Often times with government assurances. It's one of the reasons telephony had such high penetration in Canada (75%) at a time when it was much lower in France (~25%), Bell was given assurances that we're now paying for in more expensive internet. Not saying it's right, but it's like this for understandable reasons, and it's not strictly raw corruption.

> Ok, so here is my proof that the "tier 1" networks are over charging: if Backblaze routes a packet over the "tier 1" network is comes down to about 2/10ths of 1 penny per GByte. That's $0.002/GByte.

That roughly matches the information I've been able to find when looking into transit pricing.

And then AWS/Azure/GCP gouge you to an absolutely ridiculous extent even beyond that.

You'll likely see prices in the same range with most providers if you have some volume and are willing to commit to it.

By "most" do you mean excluding those three? I know it's not too hard to get a port at a good rate if I have my own server, and it's easy to rent a server with plenty of bandwidth. But if I want to use something like glacier, I'm screwed. Amazon's "contact us" threshold is $28k per month, and anyone under that is paying 30x-40x as much as if it was $.002/GB.

Cant speak for others, but there are some specialized SKUs in Azure you can opt, for non-metered egress data transfer.

https://docs.microsoft.com/en-us/azure/expressroute/expressr...

It starts about USD 1200 per month, for a 1 Gbps line.

That's still price gouging imho. For the same amount of money you can get 5-10G of transit bandwidth from large carriers.

How can it be cheaper for the big carriers to transport my packets halfway around the planet than for Microsoft to transport them inside the local metro-region to their own datacenter?

Data transfer within a region is already free, and that service won't send the data any further.

So I'm paying $1200 a month just to plug in a single low-speed cable. It's an awful price.

Why doesn't Cloudflare buy GTT? They can easily afford to.

I always thought peer agreements are under NDA. Surprise to see they talk openly.

Disclaimer: I work at Backblaze.

> I always thought peer agreements are under NDA. Surprise to see they talk openly.

Both Backblaze and CloudFlare are very open about this. It is called the "Bandwidth Alliance", you can read about it at these two links:

https://www.backblaze.com/blog/backblaze-and-cloudflare-part...

https://www.cloudflare.com/bandwidth-alliance/backblaze/

For Backblaze, it's part of our business strategy to be open and honest with customers. That feels funny to type, but it's true. We have historically disclosed things that other companies have traditionally kept secret, like our internal drive failure rates. I don't know if this is surprising or not, but it has worked well financially for us. Meaning we explain what is going on, more potential customers hear about us, more customers trust us, our sales increase.

I'm actually curious why other businesses keep their operations so secret?

so why do vps providers provide a "1-3"TB a month transfer when it costs nothing? talking about cheap plans here

It most definitely does not cost nothing. The whole point here is that the Tier 1s price-gouge everyone else, and from there everything is far more expensive than it needs to be.

Prices only go up from there.

Cheaper plans sold by smaller outfits are likely being sold to end users through 2-3 layers of resellers. This creates a tug-of-war between multiple resale overheads and customer interest in something at a low price point, so the baseline of what is being offered (CPU, RAM, speed, monthly transfer) can get fairly wobbly.

For larger companies, to take Contabo as an example, they basically get to dictate whatever performance metrics they want, because they own the entire stack. So I might have more-or-less-unlimited 100Mbps and 6GB RAM, but CPU performance is... not great, I'm pretty sure my KVM instance winds up almost entirely swapped onto an HDD occasionally, and there are some really harsh QEMU disk I/O quotas.

funny that you bring up contabo. i have been thinking about setting up a selfhosted email instance and that could lead to more stuff later on but bandwidth is like night and day. on contabo cheapest vps is like €4.99 / month, 4 vCPU Cores,8 GB RAM, 200 GB SSD, 200 Mbit/s Port, 1 Snapshot and "32 TB of transfer is already included in each VPS package."

while on vultr.com, the ipv4 cheapest is like 25 GB SSD, 1 CPU, 1024 MB RAM, 1 TB transfers for $5/mo.

that is such a huge difference that i dont know what is real.

i want like 10TB storage vps from contabo but apparently they dont have it. do you know anyone like them? i have looked at scalaway and others but they dont say pricing of bandwidth past the allowance. like only for their s3 they say pricing which is exorbitant like the ones mentioned in the article

edit: vultr object storage says "$0.01 per additional GB transferred " https://www.vultr.com/products/object-storage/

and scaleway says Inter-regional* and external outgoing data transfer to other products from a different region and the Internet: 75 GB free every month, then €0.01/GB

same pricing for backblaze b2 for downloads

Oooh, so they actually throw a number down (32TB), nice to know. (Others just say "unlimited" everywhere and let you figure it out, yay.)

In amusing timing, I was literally just looking at the homepage and specs sections myself - my 2CPU/6GB/500GB/€6.99 plan is kind of getting a bit old, although I do admittedly have 100GB more storage, hmm.

My thinking is that Contabo is located in areas where the peering and power are reasonably inexpensive. Also, based on the tour photos and webcam views they publish, they're using tons and tons of desktop-form-factor systems they presumably assemble and maintain in-house, as opposed to ordering more traditional rack systems (with $$$$ server boards/proprietary SKUs inside). For a small-to-mid-scale operation, it would appear this offers better value for money (ironically), including compute/heat density considerations etc.

As for the price disparities... the sad thing I think is happening is that, by pricing themselves into the corner of the market (which they're basically dominating), customer perception may be "oh they're the cheap/low end", which... they are and they aren't. They offer dedicated (whole machine) for €150+/mo and semi-dedicated (CPU+RAM reservation, not sure about storage IOPS reservation though!) for €35+/mo, but these are fairly new offerings and I wonder if Contabo is trying to balance out customer perception about where they sit in the market. (As an aside, I kind of take those prices as vague hints about how much oversubscribing is (cough) technically going on with the low end offerings... ahem.)

An environment like Vultr seems to hold a position similar to eg DigitalOcean et al, where customers have learned to expect "it costs this much" and just accept the pricing. Sigh.

I guess the only way to really know is to actually order a VM, generate an eye-watering amount of traffic (which might be a project in and of itself), and see what happens.

As for 10TB of storage, that's a tricky one.

Naturally the first thing that comes to mind is Amazon :) which reveals that this size hovers around the US$260-$300 mark. Get two, mirror them with ZFS, add a small UPS, and you're done for a steep but once-off total cost of around US$1000.

Second to that is the Hetzner Server Auction, which allows users to bid for rental of currently-unused custom configurations. Filtering for >9TB, https://www.hetzner.com/sb?hdd_from=9000, currently returns a box with a Xeon-E3, 32GB ECC :) RAM, and 4x10TB (yes) HDD, for €82/mo. (With "unlimited" traffic ("yay"), apparently at 1Gbit.)

Second option: much more accessible, both in terms of price and "other people using it" :)

First option: would be paid-out after one year of Hetzner rental (1000/82=~12).

the usecase i have been looking at is hosting a peertube instance and video generates a lot of traffic and storage. it is becoming clear to me either have own min.io physical servers at a one off capex cost and then pay for bandwidth...

this one would need to do arrangements with ISPs because they dont want a brand new DC they cant handle.

Hmm. Wow, looking at this, I'm honestly trying to figure out how instances generate revenue. It looks entirely like a sunk cost situation?!

Video is definitely a complex usecase to solve for - and YouTube sets the bar almost impossibly highly, storing multiple copies of videos to raise the chances that whatever you want to load is pulled from a cache in a PoP nearby.

It looks like PeerTube's approach is to distribute costs amongst whoever has the money to throw at hosting stuff they're interested in. Certainly one way to sidestep the need for advertising, but... ouch, it certainly makes the barrier to entry a bit high, and IMO lends quite a bias to the results as well.

yes. you are right about that. the benefit of using peertube is at scale, it uses webtorrent tech and that results in massive distribution of bandwidth on the server. like i have seen live streaming screenshots where data from server was 1 GB and 12 GB from peers. Still you need a server that can handle users, p2p saving is secondary. i also found out about backblaze b2 and cloudflare bandwidth alliance where you pay for only b2 storage and cloudflare takes egress of data. if that works for peertube, that would be awesome. i will definitely set something up as a test in the coming days

Ah, I see. Interesting!

I'm very curious to hear about the results of the tests you speak of as well.

From what I have read, these networking centric companies have a reputation for noncompetitive pay. Is this true? They otherwise sound like great places to work.

so i want to start a peertube server, what storage is best, size and ample bandwidth. $0.01/Gb is too much for transfers....

oh, wasabi is not for video, i tried

Reply on news.ycombinator.com