elithrar

We looked at a number of existing options — trying to strike a balance between languages we run at scale (tooling, ops, security), memory safety, and long-term maintainability as a growing company.

We ended up writing an MQTT broker from scratch in Rust, with our own Durable Objects[1] for consistency and co-ordination across our hundreds of edge locations (and tens of thousands of machines). The MQTT spec itself[2] is very approachable, which made this undertaking easier still.

One of the increasingly cool parts of Cloudflare is that we use more and more of what our customers use: our Broker service sits behind Spectrum[3], and we use Workers + DO for all of the metadata/session handling. The next set of new products will invariably start to build on Pub/Sub itself, too :-)

[1]: https://developers.cloudflare.com/workers/learning/using-dur... [2]: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html [3]: https://developers.cloudflare.com/spectrum/

Put really concisely: you want to add serverless functions to your own SaaS platform. This is how you do it.

Expanded: Imagine (e.g.) a first-party Slack or Discord "Functions" offering, or a CRM tool (e.g. "Notion Functions") that allows you to consume + handle events directly as a user. Don't need to ask your users to sign up for, configure, maintain, (etc) $SERVERLESS_PLATFORM on their own, outside of your product. Give them an editor, do your own pre/post hooks, and plumb through relevant metadata that they can consume in their functions out-of-the-box.

(Disclaimer: work at Cloudflare, not on Workers, feedback on blog content is helpful)

We’re iterating on a standard for this - our current thinking is to use well-known URIs to publish both a public key (is the hash legit?) and hash endpoint URIs for clients to pull.

The extension requirement is an “unfortunate” friction point due to needing a separate security context for hash verification. Doing this part natively will be the biggest adoption win.

This—improving trust in the system and making it provable—is something we're looking to do. Having the publisher sign the hashes and publish the public key is one approach we've discussed. You'd be able to validate this out-of-band. Most users, however, likely won't go to this extent - a lot of the motivation behind this work was around automating the stuff that folks just don't do (like comparing hashes).

Longer term we'd like to get more of these capabilities into the browser itself - comparing the hash in a separate context, validating signatures, etc - so that this is scalable.

Ultimately these systems are also still built on trust: at some level there are humans in the loop, an assumption that the user's machine itself isn't compromised, and/or that the code itself is actually "correct" (for some definition of correct).

A “man on the street” isn’t browsing via IP address, and in the modern days of TLS SNI, you practically can’t just type in IP addresses to visit websites anyway. Names matter.

One of the replies to Ben’s tweet includes a diagram and references to the NSA’s RJ-45+Ethernet capturing device within the RJ-45 jack itself.

Out of curiosity: by whom? I don't think this is likely at this point, short of some truly benevolent benefactor pushing tens of millions of dollars per-year into a competing foundation.

None of the "alternative" browser companies today built upon Gecko or forked Firefox itself, because _it's really hard to maintain a browser engine_. Mozilla itself barely manages to keep up, and Apple sort-of-does (and can riff off work on Blink due to WebKit's shared history). This is also why Chrome & Chromium tend to drive de-facto standards (some good, some not so good) with an implementation-first approach.

Building browsers is hard. Really hard, especially as we consume more media and build richer applications on top of the web. Dedicated teams for video & audio, WebRTC, Wasm, CSS, JS itself (huge, let alone the security challenges), etc.

From TFA.

Some folks are insistent that their PR and their specific problem be addressed by your project, and the effort they put into writing code before discussing it just increases their attachment.

Understanding the # of unique names, peak QPS, % of response types, etc will help.

I consider it a “bug” if you can take the same profile of DNS queries to another resolver and see no issues ;-)

The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe.

If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps.

(Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)

We also have WebSockets via Cloudflare Workers for layering more complex logic on top: https://developers.cloudflare.com/workers/learning/using-web...

TURN (as a protocol) doesn’t change this on its own, but our network backbone is going to help minimize jitter and keep latency as close to optimal as possible.

I would agree (if this is your implication) that this is very different from what we consider “real time” when it comes to control systems and safety systems.

(I’m a co-author of the blog post)

Not libraries - we’re building APIs that allow you to create rooms/channels, connect participants, manage authn/authz, ingest via non-WebRTC protocols (and still broadcast to participants), and trigger serverless code based on events in those channels.

Those clients will still connect over WebRTC using any browser, native client, or client abstractions you build - one of the best things about WebRTC are its browser APIs and existing set of clients.

In the meantime, we’re launching a “managed TURN” service for teams who already have WebRTC infrastructure but who need help scaling and securing.

(I work at Cloudflare and co-authored the blog)

Couldn't agree more.

We see the opposite when it comes to broadband monopolies: "barely good enough" DSL infrastructure, congested HFC, and adversarial relationships w.r.t. subscriber privacy and experience.

When it became worthwhile to invest in not just peering but also last-mile because poor Netflix/YouTube/Disney+/etc performance was a reason for users to churn away, they invested.

This isn't to say that this is all "perfect" for consumers either, but this tension has only been good for consumers vs. what we had in the 90's and early-mid 00's.

They're likely paying even less. $0.20/TB ($0.0002/GB) is aggressive but at their scale, connectivity and per-machine throughput, it's lower still.

A few points to take home:

- They [Netflix, YT, etc] model cost by Mbps - that is, the cost to deliver traffic at a given peak. You have to provision for peaks, or take a reliability/quality of experience hit, and for on-demand video your peaks at usually 2x your average.

- This can effectively be "converted" into a $/TB rate but that's an abstraction, and not a productive way to model. Serving (e.g.) 1000PB (1EB) into a geography at a peak of 3Tbps per day is much cheaper than serving it at a peak of 15Tbps.

- Netflix, more so than most others, benefits from having a "fixed" corpus at any given moment. Their library is huge, but (unlike YouTube) users aren't uploading content, they aren't doing live streaming or sports events, etc - and thus can intelligently place content to reduce the need to cache fill their appliances. Cheaper to cache fill if you can trickle most of it during the troughs as you don't need a big a backbone, peering links, etc. to do so.

- This means that Netflix (rightfully!) puts a lot of effort into per-machine throughput, because they want to get as much user-facing throughput as possible from the given (space, power, cost) of a single box. That density is also attractive to ISPs, as it means that every "1RU of space" they give Netflix has a better ROI in terms of network cost reductions vs. others, esp. when combined with the fact that "Netflix works great" is an attractive selling point for users.

You sure? ISPs can, and do, this fairly readily. They wiggle on what “your data is” (“CPNI”) but browsing history via DNS queries is not uncommon (and cheap, to boot).

Go read Spectrum’s policy and how they wiggle around it as just one example: https://www.spectrum.com/policies/your-privacy-rights-opt-ou...

Beyond that:

• You can turn down the sensitivity of the challenge pages: https://support.cloudflare.com/hc/en-us/articles/200170056-U... • Since Cloudflare Pages is fully hosted on us, we can soak the DDoS and don’t have to worry about letting things through to origin infrastructure. Benefit of serving directly from the edge :-)

(I work at Cloudflare, if that’s not clear)

Please send me details (silverlock at cloudflare) here - AWS has our geofeed.

If you can include resolution details - e.g. dig @1.1.1.1 <cloudfront-host> +nsid - with the incorrect CF results, we can provide them to AWS.

Folks did geo-routing with DNS long before ECS was included, and there’s a privacy trade-off to be had. We’re exploring ways to make this better but there is no free lunch.

They (AWS) should not accept traffic on that hostname until ownership - tightly bound - is proven.

This also helps avoid related cases, such as deleting a bucket (which now frees it up) - other services like Heroku have similarly been vulnerable to this takeover hack because of a lack of strict verification: https://0xpatrik.com/subdomain-takeover-providers/

Every new bucket, service or account handling traffic for a domain should require re-verification. That verification should never persist beyond the lifetime of that resource.

The association between IP and user/endpoint is changing, especially with the advent of Apple’s Private Relay, other privacy-protecting proxies, and increased CGNAT.

Website & hosting providers will have to adapt, but right now we’re certainly in a transition state.

But this is much, much harder to do once you are already streaming the response - if the time to first byte (TTFB) is quick, but the connection is low-throughout, you can’t do much at this point. But nearly all modern implementations stream the bytes to all clients immediately; they don’t try to fill the cache first (they do it simultaneously).

Some implementations might avoid fanning in too much - maintaining a smaller pool of connections rather than trying get to ”1”, but that’s ultimately a trade-off at each layer of the onion, as they can still add up.

(I worked at both Cloudflare and Google, and it was a common topic: request coalescing is a big deal for large customers)

They have the right to it, but organizations are not air-tight, especially when it comes to policies that f-ck over employees.

From what I observed here, it was more internal DNS related: Newegg was serving an opaque “DNS failure” error page from Akamai’s front-end which is likely because their infra was failing to resolve names internally.

Agree - and many people are surprised to learn both: a) how fast mutex ops are in practice, and b) how channels use mutexes under the covers.

https://caniuse.com/#search=vp9

[1]: https://developer.nvidia.com/video-encode-and-decode-gpu-sup...