eastdakota

Baby steps toward truly free egress.

The name has nothing to do with where we believe the attacker came from. We borrowed it from Google. At Google they have a procedure where, in an emergency, they can declare a Code Yellow or Code Red — depending on the severity. When it happens, it becomes the top engineering priority and whoever is leading it can pull any engineer off to work on the emergency. Those may not be the exact details of Google's system but it's the gist that we ran with. We'd had an outage of some of our services earlier in the Fall that prompted us to first borrow Google's idea. Since our logo is orange, we created "Code Orange" to mitigate the mistakes we'd made that led to that outage. Then this happened and we realized we needed something that was a higher level of emergency than Code Orange, so we created Code Red. At some point we'll write up how we thought of the rules and exit criteria around these, but I think they'll become a part of how we deal with emergencies that come up going forward.

This came up before and it was super confusing to me because I had no idea what it was referring to but I also believe Tavis isn’t one to make something up. So I took some time to investigate.

Turned out, no one on our management, legal, communications, or public policy team had any idea what he was talking about. Eventually I figured out that a non-executive former member of our engineering team was dating someone who worked as a fairly junior staffer at the FTC. On the employee’s personal time they mentioned being frustrated by how the disclosure took place to the person they were dating. I believe the employee’s frustration was because we and Project Zero had agreed on a disclosure timeline and then they unilaterally shortened it because an embargo with a reporter got messed up.

There was never anything that Cloudflare or any executive raised with the FTC. And the FTC never took or even considered taking any action. The junior FTC staffer may have said something to Tavis or our employee may have said something about telling the staffer they were dating, but that was the extent of it.

I understand Tavis’s perspective, and agree it was inappropriate of the former Cloudflare employee, but this was two people not in any position of leadership at either Cloudflare or the FTC talking very much out of school.

I was the one who made the call to bring in CrowdStrike. It had zero to do with PCI/DSS or any other compliance obligation. It was to 1) bring in a team with deep experience with a broad set of breaches; and 2) to make sure our team didn’t miss anything. The CrowdStrike team were first class and it was good to confirm they didn’t find anything significant our team hadn’t already. And, for the sake of clarity, no system breached touched customer credit card or traffic information.

We have very good relations with our network vendors (in this case, Cisco, Juniper, and Arista). The CEOs of all of them 1) immediately got on a call with me late on a weekend; 2) happily RMAed the boxes at no cost; and 3) lent us their most senior forensics engineers to help with our investigation. Hat tip to all of them for first class customer service.

I think that’s unlikely. São Paulo was a new, not fully provisioned facility that didn’t have all our security hardening in place. That’s why the threat actor likely targeted it. That it was in Brazil is most likely incidental.

Had customer data been impacted we would have disclosed it immediately.

100%

This is so awful. Carta is the Facebook of B2B software (i.e., hoover up user data then find lots of ways your users don’t expect to monetize it). Their business model has always been premised on a bait-and-switch. It’s how they’ve justified their crazy high valuations (e.g., “Imagine it we could create the largest market for private securities!! We’ll be rich!!!”). Otherwise, the market for cap table management just isn’t that big.

As a founder you should maintain tight control of who is on your cap table. Carta disagrees. That may be good for them but is terrible for startups. If you’re using Carta, call your support rep and insist they never market, offer, or induce any buyers or sellers of your stock without your explicit permission. If they refuse, change vendors.

If you're building your own facility, there's a lot of things you can do with design. In our case, these servers need to go into other people's facilities (e.g., ISPs around the world) so we need to live within their constraints. Things like liquid cooling either wouldn't be possible or would add significant complexity to install and maintain.

We use both. This post was about the x86 version of our edge servers.

I was surprised we didn't get a single question about it from an analyst or investor, either formally on the Q3 call or on any callbacks we did after. One weird phenomenon we've seen — though not so much in this case because the impact wasn't as publicly exposed — is that investors after we've had a really bad outage say: "Oh, wow, I didn't fully appreciate how important you were until you took down most of the Internet." So… ¯\_(ツ)_/¯

Of the incident? Someone on my team called me about 30 minutes after it started. It was challenging for me to stay on top of because it was also the same day as our Q3 earnings call. But team kept me informed throughout the day. I helped where I could. And they handled a very difficult situation very well. That said, lots we can learn from and improve.

Thank you for providing product roadmap. Label all the above: coming soon.

Yup. Bandwidth at scale is effectively free.

The greatest trick AWS ever pulled was convincing the world you needed to pay for bandwidth.

Not abuse. Thanks for being a customer. Bandwidth at scale is effectively free.

Not impacted.

From some first-hand experience over the last few months… these suggestions and patch will help prevent a single client from overwhelming an NGINX server, but it will do little to stop even a modest botnet from generating enough requests to be a problem. Keeping some state on IPs and downgrading those that exceed limits to HTTP/1.1 I believe is the only effective defense. Tuning those thresholds to get them right is… challenging.

It's effectively a unit of time benchmarked to what we can accomplish in that time as of Sept 27, 2023 (launch). The challenge here is that because we're abstracting away the underlying hardware it's not the same as renting a VM for a period of time. We also don't want to create perverse incentives that keep us from making the underlying system faster. It's similar to how AWS standardized EC2 to a standard compute unit. Over time, as we continue to add faster and faster hardware and better optimize models we expect the cost of a neuron will trend down but the amount of AI inference work that you can do with a neuron will remain relatively constant.

That was our early alpha cooperation with NVIDIA. We've learned a lot since then. Not to mention, the AI ecosystem has grown up a bunch. But you are correct: this is something we've been planning for for a loooooooong time.

I think the confusion is what is meant by "in the Worker." From a hardware perspective, the GPU may be in the same machine as the CPU that's powering the Worker. Or they may be across different machines in our network. We are not routing requests to some third party. And we will try to run the inference task as close as possible to who/whatever requested it. The whole idea of "serverless" is you shouldn't have to worry about what machine where runs whatever unless you're on the team building the scheduling and routing logic at Cloudflare.

But that’s easy to solve for to get to the crux of the issue: what if you realized it and then, on behalf of and with the blessing of your employer traded on the information?

Yup.

¯\_(ツ)_/¯

Think the super unsatisfying answer is:

¯\_(ツ)_/¯

I think a lawyer would advise that that trade would come with a ton of risk. But the law isn’t clear. Generally, the SEC’s goal is to make sure markets are “fair.” What makes a market fair is hard to define.

If you do a ton of work to launch satellites to fly over Walmart parking lots and then model the correlation of how full they are to what the company’s next earnings will be: that seems like you worked hard and earned an edge you can trade on without getting in trouble. Feels like anyone has a theoretically equal opportunity to do the same work you did and get the same trading edge. That feels fair.

Your hypothetical feels less fair. Is it unfair? Maybe? So unfair that it’d be prosecuted? Probably depends on a number of things, including how much you made on the trade. At a minimum it’s an area of unsettled law. And you would almost certainly be in for serious scrutiny and a legal fight.

Supposedly one idea for Google’s business model early on was that they should use search query data to trade equities. After they researched it they concluded it would be considered insider trading. Though it’s hard to distinguish from overhearing something on the train, which (not legal advice) generally has not been. Think the difference at some level is scale and intention. And, I’d guess, if you made it your business to ride the Acela every day between Greenwich and NYC, bought special hearing aides that let you better eavesdrop on conversations, and made significant profits trading on the information then you’d be more likely to be successfully prosecuted.

But… how is that different from flying satellites over Walmart parking lots?

¯\_(ツ)_/¯

Sometimes the law is intentionally a bit unclear. Usually in areas like this where you care about a general concept of fairness and want some caution and buffer at the margins.

Yes. The one bit you’re missing is that a call option is the right to buy a stock at a certain price typically on or before a certain date.

To make the numbers simple, imagine a stock trades at $10/share. If someone came to you and said: how much would you be willing to pay to have an option to buy the stock for $100/share? The correct answer is: it depends. If it’s the right to buy the stock for $100/share at any point over the next 10 years then that’s worth more than to buy the stock at $100/share in the next day. A stock trading at $10 is unlikely to jump to $100 in a day so the option to by it for $100 is not worth much. It could happen, so it’s worth something. But it’s unlikely. So, again to make the numbers simple, let’s say it’s worth $0.01/option to buy a stock at $100 in the next day when it’s trading at $10 today.

Now imagine it’s the next day and the company with the $10 stock discovers the cure for cancer or invents time travel or perfects cold fusion. News breaks and now it’s trading at $1,000 per share. Now how much is the right to buy the stock at $100 per share worth? The answer is going to be something really close to, but maybe a small discount from, $1,000 (current value of the stock) - $100 (how much you pay based on the option) = $900. So what was worth $0.01 yesterday is worth $900 today.

Let’s say you have $10,000 to invest. If you know in advance the news is going to break you can do two things to (probably illegally) try and profit from it.

1. Buy 1,000 shares of the stock for $10/share. 2. Buy 1,000,000 options to buy the stock for $100/share tomorrow with each option costing $0.01.

With strategy 1 you spend $10,000 to buy something that, after the news breaks, is worth $1,000,000. Not bad. But with strategy 2 you spend $10,000 to buy something that’s worth $900,000,000 after the news breaks.

In both cases you’re likely to at least be investigated. And strategy 2 seems especially suspicious because the risk is so high and the non-illegal reasons for doing it are so few and far between. Very few reasons you’d buy a bunch of call options that only pay off if something causes a stock to move dramatically in 24 hours.

Finally, while short-dated, out-of-the-money call options are not something many if anyone should be playing with, they’re just a different flavor of something very familiar. To put it in context a lot of HN readers will understand more intuitively: a call option is what you often receive when you get equity in a startup. It’s the right to purchase shares at a price (strike price) before a certain amount of time (typically 10 years).

Do you have any idea how many customers we have?

The association of IPs to individuals and particular services is the original sin of the Internet. At Cloudflare, we’re working to fix that sin. That doesn’t mean governments don’t have the right to regulate content on networks in their boarders: they absolutely do so long as they follow principles of Rule of Law. But blocking on an IP can never be transparent — and therefore violates the Rule of Law — and will always cause collateral damage. Terrific that the Austrian authorities recognized that.

I think the use of a chopped up SVB credit card is clearly the cherry on top of this story.

I’m smiling thinking of the reaction I’d get on our marketing team if I suggested: “Hey, I have an idea, let’s post a negative story on Hacker News so then people in the comments will say things and I can reply.”

John (our CTO) and I care about HN because we’re engineers at heart. Unfortunately, most the people we sell to today have never even heard of let alone participated in this community. But, over time, I hope as many of the people in this community get C’s in front of their titles and manage hundred million dollar IT budgets, I hope you’ll stay engaged here. More engineers in the C-suite would be great.

Though what you’ll find, sadly, is there’s still lots and lots of chaos.