Cloudflare Calls
blog.cloudflare.com
blog.cloudflare.com
All it will take is one major outage for everyone to see this is a bad idea.
Why trust a cloud provider who could go down and take half the Internet with it? Why centralize it that much where that is even possible?
Their position in the stack has a great deal of market relevance, which translates to risk if they realize that potential, because they could well end up being a critical part of more of the Internet than AWS is.
They are not mainly a CDN and aren't even particularly interested in competing with other companies that are mainly CDNs, which becomes crystal clear if you ever negotiate enterprise pricing with them. The CDN's just a means to an end.
Nb. despite all that their public-pricing plans are such excellent values (though, beware, last I checked the $200/m one was the only one with any kind of SLA whatsoever, and not an impressive one) that if I were creating a start-up CloudFlare might well be the very first service I signed up for. If you're a small fish it's damn hard to justify not using them. And the coils squeeze a bit tighter....
For many (most) use cases, CF will operate at a resilience and stability and professionality level far above what they can achieve themselves.
So the question seems to be does the internet going down at the same time outweigh the internet being down for larger periods in aggregate? I don't know, honestly - seems like a tossup.
Is there a better angle to view this from?
edit: My issues with centralization are more about privacy, incentives, points of authority/leaks/autonomy, etc. Downtime seems the least concerning to me.
Yes.
> If that's true for everyone, then the internet will, in aggregate, be down less with CF than if we distributed better.
That depends on what we define as “the internet”. If we use any single service as a point of measure, then “the internet” will have more downtime. But my desire to use the internet is very seldom to use one specific service. Instead, I want to accomplish a specific task, and if my usual service goes down, with any luck they will have a competitor which is still up. This is why I think this alternative is better; it will encourage competitors to exist, which will provide a level of redundancy above the simple network layer.
When something big like AWS goes down, it’s just understood by users that stuff is all broken everywhere. It’s not really an opportunity to get more users just because your thing is still up during this huge outage.
On top of that, if the alternative is less reliable than CF, any marginal gain in users during that outage (users that were only interested in your service because it was still up) will again be lost during subsequent outages for the exact same reason.
Obviously if you need uptime better than AWS, don't use AWS, or use AWS and someone else. The reason people are fine accepting this is because the impact of "50% of the internet goes down" is hilariously unimpactful - 99% of the internet is just not anything to care about.
It’s like with stocks. A single stock I own might go bust, but with a diversified portfolio, I won’t really care. But if ⅓ of all stocks go bust at the same time, that’s a market crash.
With a healthy dash of "What are people actually trying to accomplish?"
The weakness at hyperscale is that all products feel like some mistranslation of the generalized form of an HR request: almost for everyone, but perfect for no one. Probably because nothing less than a TAM of "everyone" moves their revenue needle.
It's actually kinda nice to have half the internet go down at once. People can just stop work, wait a few minutes, and it magically comes back up. Making downtime somebody else's problem is a huge advantage...
However, people continue to use cloudflare because it is easy, solves problems people don't like dealing with, and does the job. I don't know what the alternative pitch is to businesses so that cloudflare isn't so central to the internet.
The problem is that governments worldwide have done little to curb abusive behavior that makes this all but necessary to survive on the Internet:
- India (for US/UK based callcenter scams) and Turkey (for German based) don't do shit against scam callcenters. There have been multiple high-profile Youtubers making videos exposing these scammers and police there hasn't done anything, some have even boasted about having connections to bribed police officers protecting them.
- Russia, China, North Korea and Iran haven't been kicked off of the Internet despite both nations actively running hacking campaigns and sheltering hackers and "bullet proof" hosters.
- Western governments still don't mandate open source or at least audits for Internet-connected appliance software, which means that there are tons of devices (smart cameras, other smart home systems, routers, ...) out there that end up compromised, and on top of that residential Internet connection speeds routinely cross 100 MBit/s these days giving compromised appliances an awful lot of leverage for DDoS attacks (which is the chief use case for employing Cloudflare, AWS Cloudfront+WAF and others).
There simply is too much abuse in the system
In the example, now instead of sharing my IP with a therapist, (who I presumably trust enough to... not ddos me?), I'm sharing the fact that I was talking to a therapist with a company I possibly didn't even know existed.
Better yet, I suppose I can now be barred from accessing webrtc services if said company decides I'm a "threat" based on all the metadata they've been collecting through their other services.
This allows CF to construct a person graph, which is the only power Facebook have in the advertising business. ;)
This will also allow CF to police WebRTC and block people out, like they already do for the rest of the internet. Get ready to answer webRTC captcha(TM) on every call if you use linux or such.
That said, I’m not sure that leaking an IP address is a big deal for most people. (It might be important in Ukraine, though.)
And when the inevitable curation / editorial / policing challenge of running half the internet does knock on their doorstep, they go "well we're not the ones who are supposed to be policing it, but what are you gonna do?!"
Can someone clarify this? WebRTC is encrypted generally even if you leak metadata like IP address. Is Cloudflare stating they will be the middleman and therefore have access to the decrypted video stream?
edit, yes it's encrypted:
> Finally, all video and audio traffic that passes through Cloudflare Calls is encrypted by default. Calls leverages existing Cloudflare products including Argo to route the video and audio content in a secure and efficient manner.
Yes, WebRTC does end-to-end encryption by default. The IP is "leaked" because the peers directly connect to one another, so they will naturally require each others' IP address (which is required to talk to one another).
There are both upsides and downsides to direct P2P connections.
1. Pro: The minimal number of parties can analyze the call.
2. Pro: The call depends on a minimal number of parties.
3. Pro: The call is generally more performant, limited only by the connection between both peers.
4. Pro: No need for third-party services other than a network connection.
5. Con: The peer learns your IP which may be used to help identify you or DoS your internet connection.
6. Con: Intermediates anywhere on the network can see which two peers are talking. (With a SFU only the SFU knows the ends of the connection for sure)
> Is Cloudflare stating they will be the middleman and therefore have access to the decrypted video stream?
I see nothing in this article that suggests that they will have access to the decrypted video. However I wouldn't be surprised if that is added in the future.
The reason is that in order to to big calls you need to support multi-quality streams. This can in theory be done on decrypted connections but not all browsers support this right now (notably Firefox). So if you want the widest support you need to do video transcoding at the SFU.
There are also other features such as recording and live-streaming that (generally) require access to the raw video. (Of course this can be done as adding the recorder/streamer as a "peer" to the E2EE call when needed, but that is still giving the keys to the company at this point).
It definitely used to be true that most p2p routes were lower latency than bouncing through a server at, say, an AWS data center. In 2019 we looked closely at this and it was fairly rare to see cases where latency was improved by switching over from a p2p connection to an SFU (media server) connection. Now, the reverse is true. It's usually the case that routing through a media server at AWS (or any other major provider) is as good or better than a p2p route between any two end users.
Early in the pandemic, we assumed this was a temporary thing. ISPs had not built out their networks expecting much upstream traffic. But they'd adjust.
well, ISPs have evolved. Now we see much better performance in general than we did early in the pandemic. But we still see better performance to "the backbone" than we do between ISPs.
Another step in the Internet become less of a decentralized network, perhaps.
That being said for big calls you start wanting to do selective forwarding and you probably need to drop down to a lower layer in the WebRTC stack to manage this and allowing the Selective Forwarding Unit (SFU) to be allowed to drop chunks without messing up the connection. However it is definitely possible to do all of this over WebRTC with full E2E encryption (see Jitsi Meet).
Chrome implements experimental user-space media stream processing APIs that allows you to build "end-to-end encryption" at the javascript level. But, to me at least, it's a bit hand-wavy to call that "end-to-end encryption" because the keys are created, managed, and accessible from user-space. And neither Safari nor Chrome yet support these APIs.
There's ongoing work on this: https://datatracker.ietf.org/wg/perc/documents/
1. IPFS PubSub can be used for sharing this info (although you do still need to bootstrap the IPFS DHT).
2. You can share blobs over text chat. (Including services like Jami which are distributed)
A traditional form of griefing is "get your victim's IP address from a direct-connecting service like Skype and DDOS them while they're doing something latency-sensitive"
For health data that's what you want. So I'd like to see how it would go with the GDPR agencies if used in a medical app.
[1] https://blog.cloudflare.com/announcing-our-real-time-communi...
https://www.vonage.com/communications-apis/video/ https://aws.amazon.com/chime/chime-sdk/ https://www.daily.co/
If Google had just opened their APIs, they could have provided this to everyone...
What do you suggest? Cloudflare should stop releasing products? Regulation that you are only allowed to handle x% of the total internet traffic?
So we're supposed to go use one of thousands of other tiny cloud platform providers?
Okay let's entertain that idea. The first bare minimum items on your vendor approval checklist are things like "can I trust this business to exist in 10 years" , "do they have enough resources to support me when shit hits the fan", and "are they mature enough to deliver on the shiny bullet points on their homepage and in their sales pitch".
Isn't this process going to naturally select a small handful of providers? What am I missing here?
No. It's possible to not do a lot and still last a very long time. Consider zippers, YKK has existed for almost a century and they only manufacture zippers.
followed by
"On September 19, 2007, YKK was fined €150.3 million by the European Commission for running worldwide price-fixing cartels and sharing markets with zipper-makers Prym and Coats."
Perhaps not the best example? (source: https://en.wikipedia.org/wiki/YKK )
Capitalism requires competition. If it’s only natural that one company grows larger and better than all others, then this is bad for consumers, and in this case bad for all of us, since it limits who can even be on the internet in any meaningful way.
I do not at all understand how anyone is walled off from being on the internet, if anything I feel like it's massively insanely easier to do that today than it was twenty years ago.
It's not like capitalism doesn't have its faults, but using competition to forge winners is literally what it's meant to do. At least in my very basic layman opinion.
Or when there's no real difference in product so there can't really be a winner (sugar water/Pepsi/Coke).
Keeping markets competitive often does require regulation. (For example, common carrier regulations.) Thriving markets don’t happen by accident; they are often tough to get going and don’t necessarily happen without a stable government that allows trade to happen.
I do think we should point the finger at companies like Amazon and Microsoft before Cloudflare though.
We would also likely not have those companies without capitalism to begin with. Or computers. Which actually sounds pretty nice... haha
It hasn’t happened because revolving doors, fascism, etc. The state, tasked with doing these things, is not doing them.
> I do think we should point the finger at companies like Amazon and Microsoft before Cloudflare though.
Luckily, I have more than one finger to point. There is certainly enough blame to go around. But if you try to argue against criticizing Cloudflare because others also deserve blame, then you’ve lost me.
It seems to me that competition does not imply winners. At least not permanent winners. The difference between capitalism and a traditional competition being that a traditional competition has an end point (at which point a winner can be declared), whereas capitalism has no ending point and thus can only have a winner for a time.
Imagine a sporting competition that started with 20 teams in a league and every year the bottom team was eliminated until after 19 years there was only 1 team left. We would not want to leave the competition in that state, we would want to introduce more teams to sustain a level of competition. And if the introduced teams consistently had no chance of winning due to the dominance of the top team, then we'd likely change the rules to level the playing field.
So yeah, not being able to handle more than x% of the internet traffic (unless they're running a real dumb pipe with only IP routing logic) sounds great. I'd welcome anther Bell systems breakup.
Regulation sounds about right. Monopolies are regulated in the real world, so why don't we do the same in the virtual one?
Accusations of hypocrisy is not an argument. Instead of accusing me (and all other detractors) of not criticizing others enough, please elaborate why this isn’t what I described.
Cloudflare (and others) keep releasing products which makes their central role more central and less vulnerable to competition. They ought not to do that, and I would argue for laws which prevent them from doing that if necessary.
Regarding the problem, this kind of problem should not be solved by one central actor. Instead, these problems should be solved by new network and protocol designs.
Also, the comparison is flawed, since neither Linux nor Nginx are network services.
Approximately 1/3 of the most popular websites use Cloudflare[1]. That's not really an argument against the fact that Cloudflare might want to be 'the central server of the internet', but it's a suggestion that they have some way to go yet. I'd bet that Google Tag Manager and some AWS services are integrated into more than 1/3.
Or “Cloudflare cannot possibly be taking advantage of their market share, since they have competition!”?
AWS and Cloudflare, on the contrary… (and also Google products like fonts.googleapi.com, or probably anything under googleapi.com)
Which take is that? An opinion or outlook that differs from your own?
>"You never really see that if AWS adds a product, or GCP adds a product or any other products from bigger CDNs."
Sure, you do. When AWS released it's DocumentDB(MongoDB competitor) and "Open Distro for Elasticsearch" there was plenty of uproar, both form the companies behind these products as well as the community. Those concerns were also registered on HN.
I'm really getting tired of this kind of hand-wavey response.
I keep hearing this term 'fireside chat' used like this, and ever time there's no actual fire and it's not intimate (10k viewers?). What is it supposed to mean?
>"With a traditional WebRTC implementation, both the patient and therapist’s devices would talk directly with each other, leading to exposure of potentially sensitive data such as the IP address... When using Calls, you are still using WebRTC, but the individual participants are connecting to the Cloudflare network
> Calls uses anycast for every connection, so every packet is always routed to the closest Cloudflare location.
Is this true for the UDP media (and data channels) traffic, or just for the initial signaling and connection setup?
If the UDP traffic is all anycast, that's truly impressive engineering work. Bravo!
It is true, both media and signaling is over anycast and advertised from every Cloudflare location. We manage things like ICE and DTLS state in a distributed way.
Super happy to be part of the super talented team that made this happen!
I don't think they've talked much about what happens if the connections gets routed to a different PoP mid-stream.
Should we be reading deeper into this?
Longer answer...
WebRTC was designed as a fundamentally peer-to-peer protocol. The spec defines (and basically mandates) the use of end-to-end encryption. Which is great! Among other things, this means that browsers can implement e2e in a standardized and provably secure way.
On top of WebRTC's fundamental peer-to-peer-ishness, you can build an architecture to forward or process media and data streams through media servers. This is what Cloudflare has done, and what every major WebRTC platform/project does in order to scale up participant counts, improve performance by moving routing closer to the network edge, and implement things like recording. But there's no support (yet) in the WebRTC spec for encrypting media streams so that they can be handled and routed by a media server without decrypting them.
There's ongoing work on this. Here's a nice blog post covering how the early working group effort was being organized: callstats.io/blog/2018/06/01/examining-srtp-double-encryption-procedures-for-selective-forwarding-perc
Relatedly, I didn't know that Stream had launched SFU cascading. That's awesome! I'd love to compare notes sometime if you're up for it. (I'm a co-founder of Daily.)