lclarkmichalek

There are an incredible number of websites that have incredible read:write ratios. I would suggest doing everything you can to avoid writing to the database on every request.

I mean, I do.

I don't know any major service at FB that works this way. There could be risks from increased uptime, but I don't know any service that has OOMing as a regular, expected behaviour.

Much cheaper than CPU cgroups if you want some corse grained isolation when stacking workloads

That’s called a replay attack and is absolutely something e2e encryption protects against

Doesn’t sound too dissimilar to schools before the pandemic. They weren’t exactly libertarian paradises

These follow ups aren't super compelling IMO.

> To do this, the first and smallest step will be to phase out the BinaryPack library and make sure we run a more extensive testing on any third-party libraries we work with in the future.

Sure. Not exactly a structural fix. But maybe worth doing. Another view would be that you've just "paid" a ton to find issues in the BinaryPack library, and maybe should continue to invest in it.

Also, "do more tests" isn't a follow up. What's your process for testing these external libs, if you're making this a core part of your reliability effort?

> We are currently planning a complete migration of our internal APIs to a third-party independent service. This means if their system goes down, we lose the ability to do updates, but if our system goes down, we will have the ability to react quickly and reliably without being caught in a loop of collapsing infrastructure.

Ok, now tell me how you're going to test it. Changing architectures is fine, but until you're running drills of core services going down, you don't actually know you've mitigated the "loop of collapsing infrastructure" issue.

> Finally, we are making the DNS system itself run a local copy of all backup data with automatic failure detection. This way we can add yet another layer of redundancy and make sure that no matter what happens, systems within bunny.net remain as independent from each other as possible and prevent a ripple effect when something goes wrong.

Additional redundancy isn't a great way of mitigating issues caused by a change being deployed. Being 10x redundant usually adds quite a lot of complexity, provides less safety than it seems (again, do you have a plan to regularly test that this failover mode is working?) and can be less effective than preventing issues getting to prod.

What would be nice to see if a full review of the detection, escalation, remediation and prevention for this incident.

More specifically, the triggering event here, the release of a new version of software, isn't super novel. More discussion of follow ups that are systematic improvements to the release process would be useful. Some options:

- Replay tests to detect issues before landing changes

- Canaries to detect issues before pushing to prod

- Gradual deployments to detect issues before they hit 100%

- Even better, isolated gradual deployments (i.e. deploy region by region, zone by zone) to mitigate the risk of issues spreading between regions.

Beyond that, start thinking about all the changing components of your product, and their lifecycle. It sounds like here some data file got screwed up as it was changed. Do you stage those changes to your data files? Can you isolate regional deployments entirely, and control the rollout of new versions of this data file on a regional basis? Can you do the same for all other changes in your system?

> they care about the extremes too of course, since they wouldn't want a bunch of payouts at the same time

Isn’t that pretty much handled by reinsurance?

Pretty easy to find another role internally though. Few hiring managers want to take a risk on a random external candidate over an internal transfer that knows how things work

What you saw likely wasn't a UPS, but an RPP

> A data center typically spans four rooms, called suites, where racks of servers are arranged in rows. Up to four MSBs provide power to each suite. In turn, each MSB supplies up to four 1.25 MW Switch Boards (SBs). From each SB, power is fed to the 190 KW Reactive Power Panels (RPPs) stationed at the end of each row of racks.

https://research.fb.com/wp-content/uploads/2016/11/dynamo_fa...

The UPS role is taken more by BBUs (Battery Backup Units). https://www.youtube.com/watch?v=KNsposM0sJE has lots of info about them.

(I work for FB, on entirely unrelated things)

Yeah probably. Its support for easily configuring daemons to run under dynamic users, with private tmp, lesser capabilities, read only root, etc etc makes it quite attractive.

Even more attractive if you’re writing a blog post about hardening Linux against ill specified ill specified threats. Writing a bunch of config files to “lock down” random daemons seems like it’d be right up this guy’s alley

The classic would be a geographically distributed datastore with intra-datacenter link being fine, but inter-datacenter being down/degraded. Clients within the datacenter would still be able to talk to the server fine, but server to server inter-datacenter is not going to be OK.

Another part of it is that all hell breaking loose can sometimes look very similar to clean partitions, depending on the network. When network QoS classes are available, what appears to be increased latency or packet loss for one class may be 100% packet loss for the other. Though, you're totally right that all hell breaking loose can be very different - the set of systems that handle flapping & other ugly connection failures well is smaller than the set of systems that handle clean disconnections well :)

Well, that’s a novel take on the customer is always right

Make sure you file a bug - there are a myriad of sources internally, but we can often hunt it down easily enough (assuming it gets triaged to eng). Important info is the host of the urls being crawled, and the User Agent attached to the requests (headers are also good too). A timeseries graph of the hits (with date & timezone specified) can also help.

I work at FB, and yeah, I'm not too fussed by this. Lying about where you are working from goes under the "don't be a fucking idiot" category, and I don't have much sympathy for idiots making life harder for their colleagues. Re "threats", location tracking of company equipment is pretty much a necessity to handle regulatory issues w/ geographies under sanctions (want to be an even bigger idiot? lie about where you're working from, and move to crimea!). Beyond that, internal systems also react to your location to do things like change the timezone displayed in UIs, adjust notification preferences, etc. HR having access to that data would be entirely unsurprising - if you're trying to defraud your employer, this is already in your threat model.

Both the replies here miss the fact that the relationship between Netflix and ISPs is often pretty cooperative. Specifically, Netflix will provide ISPs with devices to do caching. The ISPs need to install and run these devices, configure the routes to send traffic to them, etc etc. Details: https://openconnect.netflix.com/en/. Because of that (and various other things), it's very easy for ISPs to attribute traffic to Netflix.

ET has the somewhat undesirable characteristic that if you cat /dev/random or some other large file, you're going to have to kill the ET process if you want your terminal back. Mosh does not suffer from that

I mean, that heavily depends on the kind of tree. London’s plane trees should have a very long lifespan, but at the moment the oldest will be around 300 years old, and most nowhere near that. For things like silver birch, the life expectancy is firmly under 100 years.

I mean, Camden. Go two or three stops up the northern line, and you can find something much more sensible. The rental market might be insane in certain areas, but this isn’t representative of what 2k dollars a month will get you

Max is pretty horrible, as any comparison requires identical cohort sizes. P99.99 is much better if you want to go wild.

Given that the service probably did not have a 100% success rate, and almost certainly had timeouts, the "max" would also likely be at the timeout.

It's a JIT compiler nowadays: https://github.com/facebook/hhvm/

The good news is that barely anyone at FAANG scale is using Hadoop any more

The article attributes some of the shift to remote work, which would seem to be related to tech?

Ok so how would key distribution work here? Tofu?

> Group admins will have the option to have public or private groups and also be able to toggle whether the group is visible in search results.

You could def provide an upper bound, though I imagine that'd just cause more people to be annoyed on this thread, complaining that linux doesn't give them memory when there's tons available. And to be honest, if you want this behavior you can basically get it already; check your mem.pressure and react to that. Linux isn't going to give you a null pointer, but you can def react to memory pressure.

> It is perfectly possible for a system kernel to execute in bounded memory

Is it possible while maintaining anything like the feature set that linux provides currently?

> Why can't the process itself decide how to handle what should be a null pointer return from malloc or mmap et al?

At the point where the kernel knows that it should be returning a null pointer, you're probably already operating in a severely degraded state. You will be scanning the page table constantly, looking for pages to reclaim, wasting tons of CPU. You will have reclaimed all of the reclaimable pages you can reclaim, so io utilisation will be through the roof - calling a random function will cause a disk read to page the executable into memory. Your system will not be functional. And that's ignoring the reality that modern MMUs don't actually let the kernel know when all the memory is in use.

If you want to handle memory pressure better for a given application, shove the application in a cgroup and use a user space oom killer. It's not possible for a program to react gracefully to a system OOM.

Your system would grind to a halt anyway; swap is simply giving the kernel more options to deal with the issue. Without swap the kernel has no options when it cmes to unreclaimable pages.

Obligatory link: https://chrisdown.name/2018/01/02/in-defence-of-swap.html

Which we now have on Linux :D https://facebookmicrosites.github.io/cgroup2/docs/pressure-m...