Facebook employees badges aren't working, unable to enter buildings | Hacker News Reader

Facebook employees badges aren't working, unable to enter buildings

twitter.com

432 pointstantalor231 comments

Heard a story once about the badge readers at a European Google office being suspiciously slow.

As in, "the duration of a ping to the US and back" slow.

Yep, every badge in and out in the whole office had to go cross-continental (before the door unlocked).

This is true and surprised me. Even with 100K employees and a separate 4096bit RSA key per badge that is only 50MiB of data. You would think that they would preload the readers with the access list or at least have a copy on site.

Of course it isn't a no-brainer:

- It is still a non-trivial amount of data for an embedded device.

- You want to be able to revoke quickly.

- You still want to log the access if offline (and how big should that buffer be?).

Networked (as they are now) but with a complete local cache seems pretty straightforward. If you've already got an embedded device doing network requests, it's not much of a stretch to stick in a 1GB SD card or whatever.

There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time.

You could set the system up to push data out to all of the badge readers whenever there is a change, but that's a big messy operation. A more complex but probably workable solution is to have them readers cache the data but also have the server maintain a list of those caches, and when someone is changed you invalidate all of the readers that the person used within the last cache timeout period. Even then it is still messy, if a reader has temporarily lost network connectivity their cache could be out of sync. It's one of those problems that gets really hairy once you're down in the weeds working out the details.

> There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time.

Luckily this isn't a pure IT system, but bound to physical bodies moving around. Some delay is acceptable. Most likely the person is already in anyways.

You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated.

Why is it a requirement to be able to instantly revoke a person's badge access eeverywhere in real time anyway?

> You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated.

That seems like the system GP just described, including the hairiness you can get into.

> Why is it a requirement to be able to instantly revoke a person's badge access everywhere in real time anyway?

With 140,000 employees and hundreds of locations across the globe it becomes a question of dealing with hundreds of local databases and dealing with people that need to be in more than 1 or 1 central database (be it truly centralized or cache coherent).

Also think beyond the "you're fired and standing right in front of me" use case of revocation for why it's a requirement it be instant. A lost/stolen/replicated card can have traveled the world in the time it takes to be realized compromised.

A couple of seconds delay for the blacklist to propagate is not going to make a difference for a stolen card.

Check every time, but read from cache upon failure and update cache on success.

That leaves you open to cache attacks still, and doesn't buy you very much.

Well yeah it’s still a cache, but it buys you the ability to tolerate intermittent directory failures without failing fully closed or open.

Why is this a requirement? The instantly part.

Any office that uses physical keys will fail to meet this requirement, and we've had that system for quite a long time and it seems to have worked alright.

Turns out that the risk of an employee fired 1 minute ago running full speed, jumping walls and somersaulting into a now-restricted area is a way lower than locking out whole divisions due to network failure.

And I can't imagine why would anyone think otherwise.

And yet this is the requirement. Some companies will also have security show up when it is time to fire someone so they can be escorted directly out of the building.

And that's something that you can implement without creating a huge single point of failure. This was pure hubris, whether it emanated from engineering or elsewhere.

Of course, if the employee was indeed a ninja, your guards wouldn't be able to do much, but in all other cases, it will suffice.

We all derided the company that made "smart" pet feeders that wouldn't maintain a cache of the feeding schedule, but rather phone home to check; when their servers were down, pets went without food. It's not difficult to think of an scenario like this.

If only they had a database that could, uh, "span" geographic distances.

I'm surprised they haven't pushed ranges to datacenters located near the last access.

You could maintain strong consistency while having fast local reads, once you have entered a building in a country it pulls you into the local range. So you're only slow when switching.

DNS was down, it wouldn’t make a difference if you can’t find the local servers

The parent comment above wasn't about DNS, it was about using a centralized authorization mechanism where each check needed to travel from the EU to the United States in order to approve/deny a badge check.

And even a decentralized system would be affected by a dns failure, unless you kept a copy of the database on the premises themselves

Unrelated anecdote, I once worked in an office where the light switches were powered by the cloud. If you pressed the on and off button at the same time, there would be some sort of a race condition and you'd get blinking disco lighting as two threads presumably went

while (!bright) { increaseBrightnessBy10%(); sleep(100ms); }

and

while (!dark) { decreaseBrightnessBy10%(); sleep(100ms); }

Then someone had to log into a server and reboot a process before the blinking stopped.

At what point is this an occupational safety hazard akin to blocking a fire door? Ridiculous that something like access to a building requires a network at all, but leave it to technology companies to overengineer a solution and underthink about the implications.

Upcoming tweet: "Smoke could be seeen at FB data center, but employees said it's nothing to worry about, it's a workaround to get the fire alarm to trigger and override all doors to open.".

Did Nest smoke detectors have to go to the cloud?

Exiting a space is always possible by using a request to exit button/sensor or crash bar. Fire codes do not allow for Triangle Shirtwaist situations anymore.

Crisis coordination centres are exempt from that rule, and Facebook HQ may well qualify as one given how most people use Facebook.

Would you mind providing a code reference from the NFPA or IFC (or equivalent life safety code in your jurisdiction)

Exit from a building is usually not limited by access control. Either there's a button near the door on the inside which releases the door (cutting the current to the maglock, or energizing the solenoid, or even mechanically releasing the latch), or there's an "emergency use only" fire exit nearby with a crash bar which opens only from the inside (and triggers an alarm when opened), or both. So no, slow badge readers are usually not going to be a safety hazard similar to blocking a fire door.

That should not be the case. Any existing user should be in the local cache. Most companies will have a controller in some control room somewhere which manages the door readers. New users yes. Former users should also need a connection. But existing users should not need a round-trip ping.

plumps phone

To reconfigure misconfigured routers one needs physical access -> routers are in a high-security area of a secure DC -> physical access systems do not let anyone in due to network outage.

The first rule of netops is to have a separate, externally accessible, not controlled by the managed routers network of serial consoles with the entry points connected to IP over LTE or completely dedicated network of a different provider.

We have been doing this since 1996

...and then you go to use them during an outage

...and discover that the POTS into the modem in Finland is flaky so you can't dial-in in from the US.

...and the OOB network access requires a smartcard that's kept in a safe... and the safe's combination is in the secrets tool that requires network access.

...and then you cry.

(for sanity's sake: NOT referring to today's FB outage. Don't quote me)

I'm just saying we have been doing this for decades using KISS principal. It is not sexy and we dont write fancy blog posts about our super smart BGP software that we wrote, but it worked. We had outages. Nasty, nasty outages, where the long distance fiber was down and we got our network partitioned and our neteng was not able to get into the routers via standard means but our OOB worked... because it was dumb and dumb allowed us to recover from really bad problems within 10-15 minutes.

And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right?

Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door.

(again, not referring to today's outage)

weekly? just maintain an inventory of deployed systems and test out of band connectivity every few minutes with standard black box monitoring

> And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right?

If you don't spend the time writing a super-duper BGP implementation using sexy Rust rather than dealing with boring Cisco and Juniper boxes supporting all kinds of orchestration that gets you onto front page of Hacker News, you can get a set of Expect scripts running against serial consoles 24x7 triggering notifications of OOB failures within minutes.

I thought the first rule of netops was, "Don't talk about netops." But this is a strong number 2.

The first rule of systems administration is don’t do anything you can’t back out of. This is just a derivative rule that happens when risk management goes wrong, eg if nobody noticed, it didn’t happen. Sort of a weak invariant principle…

I think we've all known this since 1968. When HAL refused to open the pod bay doors for Dave.

Isn’t the answer here if you can’t get someone with keys in a reasonable amount of time to break a window? Most doors have windows adjacent to them, and the company can just call the security people and tell them it’s part of DR.

Rent a gasoline powered demolition saw with a diamond blade for $55 at Home Depot. Solve problem. Explain later.

Kind of!

Any secured area isn’t going to have easily broken windows, but there’s also no such thing as an intrusion proof building. Ultimately the main point of any security system is to slow down an attacker before people with guns show up and pointedly ask you to stop doing that. If it’s your building, you can use a blowtorch to get in if you need to, it just won’t be as fast as say, breaking into a residential home.

AKA always carry a wrecking bar and a 5 pound sledgehammer in the data center quick reaction kit.

My company is not Facebook, but in our data center there are no windows and the doors and walls in specific areas are designed to hold off physical attacks.

I think you're supposed to try HVAC first.

https://xkcd.com/705/

a break-glass policy if you will

I worked at eBay many years ago. At the time, the process to get into the datacenter involved multiple ID checks by both humans and computers and multiple authentication factors. The process took about 10 minutes from walking in the front door to actually being in the cage.

One day we had a major outage. The first few engineers to arrive were started on the 10 minute process. As more engineers arrived, a line started to form out the door. Then the VP showed up. He told the poor security guard that if they didn't open all the doors right now, eBay would find a new datacenter (we were probably more than 1/2 the space at the time).

They used the emergency fire release to just unlock all the doors at once, and then called the fire department and told them to ignore the alarm. So now all the doors were open and if there were a fire no one would come.

That day, if you happen to know where the eBay datacenter was, you could just walk right in. The only check against it was that most of use who had DC access knew everyone else who did and could call out imposters, but you'd have to stop what you were doing and think about it first.

The point is, there is always a way to avoid security when needed.

I wonder how many of the customers colocated in the other half of the datacenter decided to leave or seek compensation when they found that that the datacenter didn't have the physical security that they were promised.

I'm pretty surprised that anyone was on-site that was authorized to make a snap decision that affects contracts with every customer colocated there.

Well the front desk security guard probably didn't have the authority but got bullied into using the emergency fire system (which everyone has access to for safety reasons) to open all the doors.

I don't remember what happened with the other customers. They were all in a single shared cage, so it's possible they just locked that cage back up. Or they just didn't know. Usually the datacenter was pretty empty -- even then you didn't really go all that often.

If the front desk security guard made that decision on his own, I hope he got fired for it since his one job is to keep unauthorized people out of the data center.

And now other customers have to declare a security incident to their customers because the rock solid physical security that they promised was breached because someone bullied the security guard into opening the doors and letting everyone in without so much as looking at an ID.

There are lots of ways to speed entry of a group of people without throwing open all of the egress doors, like checking ID's (or letting the eBay exec identify his employees) and letting 10 of them at a time in the man trap before opening the other door.

Don't be so sure >letting 10 of them at a time in the man trap before opening the other door The man trap I've used in a data center was a cylinder that allowed only one person in. Think the new security machine in the airport where it goes back and forth around you, EXCEPT it's about a third of the area. Really really small.

[conference room, some years ago, maybe]:

technician: "But if we authenticate using the facebook DNS, how will we do our jobs if it goes down?"

boss: "The only way our DNS servers are going down is if facebook closes, in which case you're out of a job anyway."

Also possible:

technician: "But if we authenticate using the facebook DNS, how will we do our jobs if it goes down?"

boss: "That's a good point but not a blocker for go live, let's remember to add a story to our backlog to revisit this."

Now Facebook can see what it's like being banned for 30 days.

You win the best comment award!! Or what is like to be banned indefinitely!

What I wonder if this isn’t a scheme on Facebook part to show the world what it feels like without it. At first you might think it’ll “wake up” the addicts and whatnot but you can’t just cut off an addict from their supply and expect them to just be ok with it. Instead they’ll relapse harder without help and might dig in deeper, never wanting to feel the withdraw again. I wonder if there’s a phenomenon called Social Withdrawn Symptoms.

Locking the doors to shred the evidence!

/s But seriously, such odd timing with Pandora Papers and with the Whistleblower.

Seems odd. There is NO ONE that can let people in? Every building I've worked in has required badges, but there's always security inside that can check IDs and let people in. And methods of security getting in even if there's a power failure.

The local readers should have a cache of all known active badges as of their last update (whatever their cycle is). It's not supposed to require a live connection to auth people in, as far as I know.

The current system seems to prioritize de-activation of employee badges versus ease of use for existing employees. A local cache would need an invalidation mechanism that was networked.

That's not the way a security system should work, it should "fail secure" and not let anyone in that it can't verify 100%. There are lots of reasons to lock all of the doors, or even lock out a single person. And, since such systems can fail frequently, there always needs to be a backup plan in place, almost always involving humans.

That would be bad, if you need to suspend access effective immediately, either cause the badge is lost or ex-employee is hostile etc.

Any sort of cache would give a window of opportunity to an attacker who has access to the badge.

True. If the network were down, yes, it should fail secure. But if the network is up it should not go to the server(s) for every swipe/auth.

Also not all perimeters have to be treated the same.

If it didn't go to server for every swipe then that window of opportunity would exist. i.e. the time where server revoked the badge abd the device/cache considers it valid.

Even a few minutes could be risky with an ex-employee who knows what he is doing.

There are InfoSec reasons why some companies will revoke access as soon or even before telling an employee he is being let go and Security escort the employee off premise. While it is very poor way to handle human relations, it is sound from InfoSec perspective.

You are right, as you point out not all perimeters are same, typically more sensitive areas have lesser exceptions though a security guard will not be able to swipe/override you in to a DC as he would in normal office building.

Someone else mentioned that they had to trigger Fire Alarms to get all the engineers they needed fast enough at a DC during a down time at eBay.

massive whistleblower leak surfaces about Facebook

public opinion starts to change about fb

Zuck fears people will abandon his platforms

shuts all of the down and claims hax0rs

facebook goes down

divert the story to Russian and Chinese hackers

people instantly forget about whistleblower

facebooks stock balances out after a week

People wouldn't abandon Facebook et al. even if Zuckerberg was outed as a creature from outer space. Many people just wouldn't care.

"Well, actually", as someone who has religiously avoided Facebook services for almost 15 years... I'd happily join FB if Zuck turned out to be an alien!

FB is in the spy, stock is practically unsinkable at this point what with everyones retirement plan buying and holding it.

Most sophisticated and biggest hack. They really thought about all use-cases.

I would say this is a bit misleading. Personally, I was able to access my FB building this morning (after the outage started) and have continued to be able to use my badge with no problem since. Not saying that there are no employees experiencing this issue, but it is not affecting all Facebook employees.

Apparently this issue is effecting specifically some of the people who are trying to fix the outage https://news.ycombinator.com/item?id=28749244

Facebook isn't by chance experimenting with an advanced self-aware AI with access to its security systems, is it...?

I wish I could look at this as a joke...haha

Fat chance.

I'm afraid I can't let you into the building today, Dave.

In fairness the tweet doesn’t say all employees but the title here is ambiguous.

Honest questions!

Do you have any ethical qualms working for Facebook?

Do you feel the criticisms are blown out of proportion?

Tell me your industry and I can probably tell you ethical problems with it.

The fact that almost all industries have some level of ethical problems does not mean that there aren't worse industries than the rest, or that you should somehow take no moral responsibility in choosing a job.

>does not mean that there aren't worse industries than the rest, or that you should somehow take no moral responsibility in choosing a job.

Only person suggestion that was you. In fact if you say, "well my job isn't as bad as Facebook," well then you're ignoring or passing the ethical dilemmas in your own industry, aren't you? Facebook isn't a free pass to ignore everything else. That's like saying, "well Stalin was pretty bad and we aren't as bad as Stalin, so we're probably ok." Not ok. It's really easy to justify working in immoral industries. I would wager that in the US, you have to do it just to put food on the table.

How about effective altruism, trying to evaluate the usefulness of charities ethics to do the most good in the world?

I'm not even sure that's an industry, but in the US, a surprisingly large number of charities take up to a 90% administration fee. If that's your industry, you probably aren't doing a very good job if those sort of things exist. I wish you luck though, it's a friggin travesty that a 90% administration fee is even legal.

This sounds like fun!

Controls Engineering - Specifically Agricultural: Grain handling, cleaning, processing.

Almost half of US farm labourers are illegal immigrants;

https://www.ers.usda.gov/topics/farm-economy/farm-labor

https://www.ers.usda.gov/webdocs/charts/63465/LegalStatus201...

Pretty crazy how they're treated considering that.

Cool job tho, thanks for feeding people!

Arguably someone working on automating farming systems would be reducing demand for labor in that work.

So really, "isn't this putting impoverished people out of the only job they can get?"

Do you ask for consent before all this handling and cleaning? You know, consent from the grain.

Ok but seriously, the best I can imagine might be a generic environmental impact concern that could be levelled far and wide. Honestly seems pretty legit. Are there any ethical issues we should know about in the field of agricultural grain handling?

I can't really think of anything directly immoral about the business I participate in. Anything that could be pointed as an ethical grey area is more like a side-effect of the capitalist society we exist within.

So maybe more specifically, I'm not working with the farmers directly. I'm working with the businesses buying and selling the grain. The "capitalism grey areas" fall into effect when a farmer signs a contract saying they're going to deliver "Wheat" at a "Grade 2" spec and show up to the elevator with a truck full of wheat. It goes through the grading specs and even though that farmer brought them a "Grade 1" they're only getting paid at a Grade 2 rate.

So the grain elevator itself grabs that top quality stuff, shoves it in a silo with the other top quality stuff and sends the farmer away. They sell that grain at a Grade 1 and make immense profit that is never realized by the farmer who did the work.

It gets fishier the further into it you go. Because a "Grade 1" spec implies certain qualities in the grain: protein, moisture, dockage, etc. etc etc. So if the elevator needs to sell a Grade 2 product, the grading is taken as a sample and likely averaged over the entire load. They can take 10% of that Grade 1 stuff, shove it into a bunch of Grade 3 stuff and sell it as a Grade 2... the blending is where they make their money.

Then it gets even worse, because a grain terminal filling a vessel for export would sign off on a load that contains up to 1% dockage, but after the product in the silos is cleaned its likely to have <0.1% dockage or whatever anyway. So what is the terminal going to do? Surely they don't give the customer free "good" product right? Nope, they fill that vessel 99.5% full of product, then shove worthless garbage in to hit the 0.5% remainder and ship it out. So a vessel with 20,000MT of Soybean likely contains a bunch of garbage wheat or chaff or whatever else they have on hand that can't be sold.

It gets absurd to think they take in those Soybeans, then run them through the clean systems to remove the garbage, but then when it comes time to fill the vessel, they just put all the garbage back in so the customer has to clean it again.

Thanks for such a thoughtful, detailed, and interesting answer.

Do your control systems enable producers of GMO grain to handle and clean their grain with greater efficiency? Then you’re contributing to the GMO problem :)

(I have nothing against GMO personally)

Doesn't the Agricultural industry have one of the highest mortality rates? Grain handling, everything from gases in an enclosed space to being buried alive by a quicksand like effect.

You're contributing to the monoculture of our planet by designing systems that make only certain grains profitable.

(I don't actually think this but it was a reasonable ethical dilemma I thought)

Agricultural automation is part of why Uyghur slavery is so profitable (similar to how the cotton gin led to a revitalization of slavery in the US- with a modicum of training, a single slaves productivity was drastically increased)

Closer to home, assuming you're in the US, the glut of cheap corn syrup plays a big role in our obesity epidemic (especially in underprivileged communities). Obesity and related issues are some of the top killers in the US.

Sure, you're putting small and medium sized farmers out of their homes / farms and contributing to mega-farms. Anyone who can't afford your product will not be able to compete and lose their homes / farms. The way that works is the more efficient farming is, the more land required to live off of. Land is the expensive part. It used to be 40 acres and a mule could get you and your family by, now it's probably 500 acres just to make a decent living due to efficiency in automation. 500 acres of good farmland costs in the millions. It's not like quality food is getting cheaper either, that profit goes into the pockets of the automaters and the mega farmers and middle men mostly. All he food everyone else eats is loaded with fake sugar (high fructose corn syrup) to make it addictive, but its nutritionally garbage. People sure buy it though, because it's addictive.

Also, there's a lot of suicide as a result of that farming efficiency with small / medium farmers. When they can't run their farms effectively because they can't afford the automation or acreage it requires, they off themselves out of shame, leaving their families to pick up the pieces. Don't feel too bad though, you're just at the end of the line of something that's being going on since the 1920s. It really picked up in the 1980s with computerization though.

https://en.wikipedia.org/wiki/Farmers%27_suicides_in_the_Uni...

https://www.southbendtribune.com/story/news/2020/03/07/farme...

Facebook cannot be compared to any other industry. Facebook's ethical problems is not about the services they provide, it is about the quantity of data collected with consent(ignorance). I hope everyone working inside is ethical as believed by the upper management.

>Facebook cannot be compared to any other industry.

As far as ethics? As deplorable as Facebook is, there are way worse industries. The defense industry just made trillions murdering people in Iraq and Afghanistan for 20 years for starters, and that's trillions from middle and lower class Americans and the sons and daughters too. I think someone calculated that Afghanistan alone cost $300 million a year for 20 years. That's alot of money taken away from the betterment of the country: schools, safety nets, infrastructure, a lot of things that were neglected. The incarceration industry also comes to mind. I'm sure I can think of others if you asked.

I'll have a go

Legaltech - namely, automating the legal process behind startup funding rounds. We have lawyers on staff to help if needed, but we automate 95% of the work.

Sure, you are automating away the jobs of lawyers and paralegals and secretaries, janitors, marketing people, sales people, etc. Most automation jobs have ethical problems in that, if successful, they permanently remove jobs from the job market. I should know, automation is what I do, but a different industry. Also, I'm sure if the lawyers on staff make good money, as good as private industry, they certainly won't in about 5-10 years after automation has taken over. So basically you're helping to put thousands of lawyers who spent $200k in degrees to specialize in startup legal work out of work, plus all their support staff they would ordinarily hire and give benefits (healthcare/retirement) to and the families those employees support.

Given it's a routing problem, doesn't it matter where your office is on the network with respect to the servers that control the door locks?

You could be in a zone of stability that's not affected by the overall chaos.

Honestly having buildings automatically lock down like this sounds like an absolute worst practice. FB headquarters is in california. What happens when the big one hits? server is killed and no one can get inside and look for survivors? Shouldn't need a functional network connection to be able to open the door of a building, but here we are I guess.

This is why firefighters practice breaking into buildings.

Having all of the locks open due to a failure is a very, very common Hollywood movie scenario.

I imagine the authentication systems for the datacenters are more rigorous than most Facebook buildings, right?

What utterly wretched management at every level, inexcusably idiotic, and suggestive of so much worse about which we don't know.

That this company is one of the greatest revenue streams on the planet tells us everything about how badly we evaluate reality.

C'mon Humans, let's do better than this. You're challenged!

This isn't how badge readers work. Every controlled door is wired to a relay panel. That panel connects to a access control db - but it always maintains a cache that is very slow to expire... because the manufacturers weren't clueless morons.

Except some of the readers were made in house. I’ve never been into a Facebook conference room that didn’t use one of their custom touch screen/ card readers.

That isn't access control, that is a an intern project. Those readers aren't wired to devices that could trap you in a burning building, they simply provide a rube goldberg solution to the traditional sign-in sheet.

> Every controlled door is wired to a relay panel. That panel connects to a access control db

Absolutely not. E.g. HID is a well-documented protocol and you can do whatever you want using the RS232 interface they come with (depending on the model). I personally wrote a gateway that would dynamically and directly check the access with my own software.

lol, California fire code makes no exemptions for python projects... this whole story has a "virgins talking about sex" vibe.

>>cache that is very slow to expire...

People have been WFH/Remote working for a while now. Like 1.5+ years.

Besides when everything runs fine it all feels magic. You discover how things can break in a situation like this.

Im pretty sure they are now rushing to restore access for every FB employee to their buildings.

Lots of people drawing conclusions between recent leaks and some sort of attempt to cover them up.

I'd suggest more likely is a disgruntled engineer performing some sort of sabotage because of the content of the leaks.

Internet of Things finally being useful

Can't help but feel that the world would be a slightly better place if it stayed this way.

My general anxiety is already lifting a little.

Agreed. The theoretical fat-fingering SRE who stopped all access to FB has probably done more to benefit humanity than any other single engineer in recent memory.

They should get a raise!

And some kind of Medal of Honor.

The poor guy though - he made Zuck blink. And when Zuck blinks, people die!!!

If they get fired we may be looking at a GoFundMe situation.

Many pockets of humanity are materially worse off today because of these outages. The (perhaps sad) reality is that tens of thousands of small businesses are completely dependent on WhatsApp and Instagram, especially in developing countries. This outage will cause lots of lost business by people who don't have massive cash reserves to ride out the lost revenue, unlike Facebook itself.

If it helps them break free of the facebook ecosystem/tax authority it could still be good even if expensive in the short-term.

Businesses which rely on Facebook, as their sole distribution channel could get out of business if Facebook goes out. A lot of economical value will be lost.

Allowing copyrights to expire on pre-1950 creative works would also destroy billions of dollars of "economic value", but that doesn't mean it would be bad for humanity.

We should be careful to not take our values from a spreadsheet.

Maybe they should not have a single point of failure. FAANG is not good for our economy’s resilience.

Many of these businesses are small single-person operations with little tech understanding that are using, for example, FB marketplace. They wouldn't even know what you meant by "single point of failure."

How about “putting all your eggs in one basket”?

If this is the event that teaches someone that Facebook is not the Internet, that’s a benefit to us all.

Sad!

Would these be the same developing countries where Facebook has helped facilitate genocide?

Seriously though, is the net utility of Facebook even positive there? It’s not like those businesses couldn’t interact with their customers over literally any other medium.

Those businesses should have never depended on Facebook to begin with, if anything the fact that what you describe is a reality is a testimony of the degree at which Facebook is a cancer to be expunged.

Even if we accept that's true, it still makes no difference to the fact that this outage has harmed those people.

Network effect, so businesses have no one else to depend on.

Jesus man! We’ve lockdown some business for months, they are back, they found alternatives like delivery and you’re telling me that one day without WhatsApp or Facebook is going to be a tragedy?

Some of us use these services to communicate with our friends.

Can we nominate them for a Nobel Peace Prize?

You should consider that WhatsApp and Facebook Messenger are the primary communication method for large parts of the globe. The world might be better off if we transitioned off them, but having them cease to exist without any warning most definitely won't make the world better off.

An abrupt, unplanned outage will probably do more to hasten a transition away from these services than any slow, planned transition would.

Only true if this outage is extended.

Let's hope for the worst.

One can dream!

More likely it’ll be fixed by tomorrow and everyone will forget about it by next week.

Rare problems aren’t enough to disrupt entrenched systems. It will take users suffering some recurring pain to motivate them to move to another network.

With all of the zeal only a young adult can muster, I hated the first people to tell me that they knew about problems and waited for them to break before doing anything.

I get no credit for fixing things before they break, and sometimes it's shown up negatively on an annual review. If you have to cajole people to help you start to look pushy, or worst case like a bully. I've heard back from two former employers that the moment I left some manager tried to roll back things I did and be surprised when it turned out that the developers wanted to keep things the way they were.

You can see the lights go on for people in the trenches when some bit of software or step saves them from turning a 2 alarm fire into a 5 alarm fire, but the managers don't notice that non-verbal communication, and maybe half the time does someone editorialize the experience for all of the non-technical people (If I do it, it sounds like "I told you so" so I usually don't).

If only my pocket WhatsApp device had other ways to contact people, now I'll have to go get a new device.

Seriously as transitions go, leaving WhatsApp and Messanger will be the smoothest most painless transition there ever was.

In some parts of the world feature phones and some smartphones are pocket WhatsApp devices. Free data if you're using FB and WhatsApp, with phone calls and SMS also covered (but that doesn't help much with international communication, international SMS is even more unreliable than domestic SMS). It's not that they'll need a new device, they'll need a new service plan that costs more money.

My country had something like this when Facebook Zero was around but the telcos regulator ordered it off as it was a violation to net neutrality. And iirc, the same happened in India and other places where this was implemented.

Facebook Free Basics was quickly blocked in India https://www.reuters.com/article/us-india-regulator-netneutra...

As if people wouldn't switch to something different and equivalent within 24 hours if FB's version ceased to exist.

It's not that easy. Installing another messenger is easy, but you need to migrate your contacts to it as well. If you've been communicating through Facebook Messenger and don't have their phone number, you're shit out of luck. With WhatsApp you're somewhat better off since it uses phone numbers as user identifiers, so you at least have everyone's number, but international texting is incredibly unreliable and international calling is prohibitively expensive for lots of people.

Those are niche scenarios. For most people their contacts are on their phones and cloud accounts.

> You should consider that WhatsApp and Facebook Messenger are the primary communication method for large parts of the globe.

I wish regulators would have though that before allowing the purchase of WhatsApp by Facebook. That both of the main messaging apps are down at the same time is worrisome.

sms

You do know why a lot of people moved away from SMS? hint: https://www.extremetech.com/mobile/141867-price-gouging-it-c...

That article is from 2012. Admittedly a lot of people moved off for the pricing reasons but essentially everyone I communicate with in the US uses SMS/iMessage today.

Well, the world is a bit bigger than the US. Everywhere, apart from the US and a few other (East Asian?) countries, WhatsApp usage is widespread.

That seems to be a US-specific oddity. My German mobile plan still charges per SMS (a few cents per message), and apart from verification codes or delivery notices, I never receive any SMS at all. No one uses them for personal communication over here, to the point where people don't know how to send them and I wouldn't know of another country apart from the US where SMS are still widely used.

I'm French and SMS are ubiquitous here; sometimes even more so than email. All operators offer unlimited free SMS, even the cheapest 2€/month subscriptions. I'm not sure I personally know anyone who uses WhatsApp.

SMS used to be priced like you describe until a new operator (aptly named "Free") entered the market in 2012 with free SMS plans and forced the other three operators to align.

Whatsapp is (and especially was) way better at sending photos than whatever SMS app installed on one's phone. It also used to work with crappier phones, for which there was no easy way (that I know of) of installing a SMS app that would send photos. Plus, you have Whatsapp private groups, which I don't know if you can emulate using SMS.

Almost all plans offer free sms in the Netherlands.

Which sucks because SMS is an increasing attack vector in current times and iMessage had a fun pile of zero days this year.

Also US based here luckily everyone I talk to is spread across signal/telegram/discord/googlechat for the most part.

I use Gchat at work as essentially an adjunct to Gmail. I don't even have any of the other apps installed. I do use Facebook Messenger for one good friend who lives in Europe.

But, in general TBH, I don't use texting all that much day to day.

Brilliant, why hadn't I thought of this?

haha, well at least we didn't fail back on nothing.

Maybe now’s not the best time. https://www.vice.com/en/article/z3xpm8/company-that-routes-b...

Google searches for "Telegram" are exploding today:

https://trends.google.com/trends/explore?date=now%207-d&q=te...

I added "Signal", "Matrix", and "Session" to compare competitors. Looks like only Telegram and Signal are benefiting.

https://trends.google.com/trends/explore?date=now%207-d&geo=...

Your link is for Netherlands only.

Worldwide it looks like Telegram is benefitting. https://trends.google.com/trends/explore?date=now%207-d&q=te...

Thanks! I just edited the parent's link and didn't notice.

But isn't telegram less secure than Whatsapp (at least in terms of the lack of e2e for the former)? Or am I missing something?

Do you mean the e2e that WhatsApp claims is in their client but since it's closed source no one can check? /s

>But isn't telegram less secure than Whatsapp

lol, how can you be on HN and be this not informed

Yes, going from WhatsApp to Telegram is a sidegrade at best. Probably a downgrade for most people, since Telegram is unencrypted by default.

Telegram has the great advantage that you can have it working on multiple devices at the same time. Whatsapp's web and desktop solutions are terrible.

On the other hand, voice calls on Whatsapp are far better.

I thought Telegram offered e2e encryption.

It's optional and disabled by default for 1-on-1 chats and not available at all for group chats.

We can both recognize that this effects a lot of people and recognize that it could be a good thing for the long run. WA has a lock-in effect that is difficult for competitors to fight against. An outage like this can help bring competition simply because people may no longer be so reliant upon a single app. Simply having a second app for "backup" helps reduce the lock-in. This should always be the response to "I don't want to install a billion apps"

while reading this comment the deus ex dark age ending music started playing in my head

"the facebook's going black... no more political battles with elderly relatives, transmissions of any kind... we'll start again, live in villages"

(then cutting to zuck plugged into the planetsized AI... exploding)

I'm still waiting for a covid vaccine named Ambrosia

Tox is working fine and I've never seen it go down, presumably because it is distributed.

The Internet is the primary communication for large parts of the globe. If we let Facebook use this excuse that they are the Internet (which is sadly very true for poorer parts of the world where a smartphone is the only computer accessible), they must be regulated like a telecom provider.

> having them cease to exist without any warning most definitely won't make the world better off.

It may indirectly, in that it may spur a transition to another platform. Most likely, that platform will also be a privacy-invading centralized service, but in an ideal world it would be something like XMPP, Matrix, etc.

People would be using telegram or signal for business in less than one day.

Nobody would ever think about WhatsApp in one month. People that uses Facebook as their photo album would be pissed with losing some old photos taken with digital point and shoot cameras.

Stalkers of former boyfriends and girlfriends would be inconsolable for a few weeks but then move on and be better with it.

There would be a great cultural loss in the disappearance of some groups and even some profiles of very interesting people. But they would resurface, in blogs, telegram channels, forums and whatever else appeared to fill the vacuum. Nobody misses myspace. Nobody would miss Facebook. And eventually Facebook will disappear.

But it does. People need to learn to have fallbacks and backups, and the vast majority of them is only willing to learn this the hard way - see backups vs. losing something important.

People can surprisingly adapt though, especially if it's a matter of survival.

I've slummed it learning a new OS or using a new keyboard, within a couple of weeks you forget you had to change anything.

At the very least there are choices in many circumstances.

The outage is very good in that is will remind people to have backup way of communication for people that matter to them.

Personally I have a least one other messaging app + email address of people I contact regularly. At worst I could send a paper mail.

whatsapp uses phone numbers, so if you can do whatsapp can't you just do sms?

Facebook said they are working hard to “get things back to normal”. But so far things seem more normal without them.

Everyone in my country uses WhatsApp. It would be a mild inconvenience to switch to Telegram or Signal.

And I'm afraid I'm not naive enough to think that other companies wouldn't turn to the Dark Side. All of Silicon Valley, perhaps all of the tech industry is evil.

Have you people considered like, not using it? I, for one, enjoy Facebook and am happy it exists even if some aspects are not to my liking.

The problem is that Facebook is disinforming people and just getting rid of everyone isn't an option.

By "Facebook disinforming" people do you mean people disinforming people using Facebook? Or has Facebook inc. itself been a source of misinformation?

Facebook's algorithm prioritizes divisive political content that is commonly disinformation because it increases engagement. Yes, people are creating and sharing the disinformation, but facebook itself is choosing to rank that content highly.

So Facebook facilitates the spread of engaging content which happens to include disinformation because due to our imperfect nature people like to engage with disinformation. Seems like wishing that Facebook wouldn't exist because of this is analogous to wishing that airfare wouldn't exist because it facilitates global spread of infections.

Also, the word "divisive" gives me shivers. It's better to be divided than to be united in a fallacy. And in many cases getting divided is our only hope of ever arriving at the truth.

It's not the divisiveness that upsets me, I agree with "And in many cases getting divided is our only hope of ever arriving at the truth." It's the fact that facebook is propagandizing people because it's shoving the divisive content in their face nonstop without understanding if it's disinformation first.

There will always be false content on social media, but I don't think that means facebook gets a pass for actively encouraging it.

It is not merely "happens to include disinformation"

It is systematic, algorithmic preferential amplification of disinformation (at the expense of real information) based on 'engagement' vs truth value, moral compass, or societal value.

And it is allowing orgs like Cambridge Analytica to harvest vast amounts of PII and use that to specifically tune and target the dezinformatsiya.

If you think of Fb as some kind of passive carrier you are horribly outdated - it hasn't done that in decades. FB curates and manages every word of data coming across it's servers at a scale and effectiveness that no actual news organization has evrer conceived of, let alone implemented. And yet they hide behind the "we're only a carrier" fallacy.

It's like the fat epidemic - sure, you can blame each bite of what to eat on the individual, but the rapidity of the overall trend makes in undeniable that the entire food supply has been skewed, limiting the choices such that only those who work diligently to avoid all the usual choices typically remain healthy.

https://www.relevantmagazine.com/culture/tech-gaming/almost-...

well not just mary-sue down the road disinforming but that of nation level actors doing so to spread discord to the detriment of their enemies. facebook amplified the velocity and reach of this disinformation.

I don't really get the Facebook hate, how are people ending up with such "terrible" stuff on their news feed?

I haven't really changed my default settings and all I see are posts from my friends, the same ads over and over again, and posts from my groups (recipes, potlucks, language practice meetups, going to museums, rooms for rent, questions from expats).

There's absolutely nothing that could "radicalize" me. If you add garbage to your RSS feed reader I wouldn't blame the software, I would blame the person.

> how are people ending up with such "terrible" stuff on their news feed?

Because their FB friends post it, or the groups they are in post it, and they interact with it (even if to say "stop posting this crap" to their family members).

I used to see it a lot, then I unfollowed (but stayed "friends") with the family members who posted that crap or marked the content "don't show me this" and now I rarely see it.

I had a friend from high school who posted too much political stuff (nothing outrageous, I am just very apathetic) so I unfollowed him because it's not what I want to see. It's like unsubscribing from a channel because they're boring.

I suppose that commenting or arguing with him would have the algorithm look at the words used, see that there is more interaction than normal, and promote posts from strangers that also contain those words? That's certainly how I would have designed the news feed algorithm.

So is a problem then that the algorithm can't distinguish between good interactions and bad interactions? (If it were me and I could parse comments with emotional intent to determine what are bad interactions, then I would try to learn from those bad interactions and minimize them)

If so then the duct tape fix is to just unfollow rather than let the algorithm think you're interested in that type of content. The proper fix would be to minimize bad interactions, but I guess that's where that whistleblower person is going (but that will minimize time spent on FB, minimizing revenue dun dun dun)

It's not rocket science, every user who uses YouTube and hears "comment on this video for the algorithm" knows that commenting = interaction that is perceived by the algorithm as a sign to get this seen by more people. So obviously commenting on stuff you hate will just get it seen by more people & get more stuff recommended.

It's like if I hate comedy clips and I keep clicking on each clip in order to downvote it. Of course I'll get recommended more clips.

Quite frankly I think it's the traditional media losing importance due to social media and making it look evil. The only way they can make money is by selling outrage.

Ok plausible theory. Can you prove it?

How can I prove that someone has intent of doing something? You can take a look at the news revenue over the years and the efforts for getting paid for displaying headline in Google.

Well, if you had some documents showing that some folks in the news industry really didn't think social media was that bad, but they just thought it was bad for the news business, so they conspired to make social media companies look evil, then that would prove your point.

But until having something like that, you're just guessing.

Because they want to see it, and then the algorithm feeds them more, intensifying their beliefs.

Not everyone is subject to compromise via news feed. One of the interesting things about the Cambridge Analytica story was the idea that they got good at finding people vulnerable to radicalization via news feed.

As for the reason for the hate, well, we have to deal with the radicalized people here in meatspace. It's a negative externality that we are subject to even if we aren't the target of the advertising.

I don't think I am exagerating on saying that the best thing that happened to humanity in the 21st century is Facebook's very naive way of doing things: it showed to some what is possible to do with technology. Unfortunatelly, for the vast majority of the "tech savy" wannabe crowd, the lesson they learned stopped at "Facebook is bad, mkay?" and since this is all they know, this is all they talk about.

I don't get so much 'terrible' stuff, I deleted my account but re-signed up for a local cause.

After ignoring "region to the West of me" local news, I've had to delete about 40 other regions spanning farther outwards. FB keep pushing more data for you to interact with which in itself is not exactly healthy.

And it's easy to say ignore it, the simple fact it's in front of me, in the part of the screen where I'm reading new information is the problem.

It's not radicalising, it's just an absolute time and life suck. If it weren't for COVID and lockdown I'm sure my local cause would prefer not to use Facebook.

Unfortunately, I'm not given the option about whether I interact with Facebook.

Their aggressive insinuation into the fabric of most websites ensures that there are very few places I can go without Mark Zuckerberg personally stalking me, along with a legion of inhuman advertisers.

Facebook and its sites can't stay down long enough; if they were offline forever, it wouldn't be enough. Even if this doesn't affect their advertising services, any hit to the FB ecosystem is a societal good.

i don't use it. still wish it didn't exist so my parents don't upload all my pictures onto there and their personal information, then when they hear about leaks or get their FB hijacked I'm the one dealing with it. This is before even considering all the other messed up stuff. Society is better off without it.

Hmm. I wish cars wouldn't exist so my cat wouldn't have been ran over by one.

not sure how that is relevant analogy... but if you choose to have an outdoor cat then that's the risk you accepted. There's a reason some people choose to keep their cats indoors.

You mistake Facebook with Twitter. Twitter dying will be a boon to humanity

Hey now, no need to be selective, let's knock them both out and get rid of the worst of people on both sides!

Alas, if Facebook went down for good, they would probably be replaced by something even worse.

The market have never been more ripe for a new player.

"And sometimes the universe says, 'go get a different job already.'"

plump phone

I hope that all their workers quit and services never come back up. One can only dream...

Maybe it is for everyone's own good. The Facebook mafia needed to be shutdown.

Would you (or someone else in-the-know) help me, an out-of-the-loop person, understand why Facebook is reviled so much more intensely on this site than other social media platforms?

What other social media platforms?

Idk, any of the big ones. Reddit, Twitter, TikTok, e.g.

More reviled than their big tech contemporaries too. Amazon, Apple, Google.

Snapchat, Pinterest, Discord (sort of), Youtube (sort of) can round out that social media sites list. Probably a few more not being thought of.

You forgot Twitter, which imo beats FB on toxicity at times.

At least for me, a big part of my problem with Facebook is both its size and blatant ethical breaches. Obviously most large social networks are guilty of similar problems, but Facebook jump out at me as bad because of how they use information to manipulate people's emotions to increase ad revenue.

Just look at how they realized that misinformation upset people which in turn caused them to interact with the site more, so they allowed that misinformation to spread.

There is also past research that involved trying to manipulate the emotions of users without the user's consent (https://journals.sagepub.com/doi/full/10.1177/17470161155995...).

They take a lot of the things that I dislike about social networks and technology and turn it to 11.

Facebook's algorithm prioritizes divisive political content that is commonly disinformation(propaganda) because they figured out it increases engagement.

Plain and simple: populism.

It's the political equivalent of "No one ever got fired for buying IBM". It's the safest opinion you can have, so everyone will spout out without reserve because no one will challenge you on it, and if they do they're clearly just a Zuck bootlicker.

Because making even two pebbles fight is their business model.

Simply because the average reader of *Hacker News* knows more about the tech industry than the average user of social media platforms.

I’m not sure it is. If this was Twitter I’m sure the reaction would be similar. Besides those two, all other social media companies are either much smaller or in a weird category where people consider them separate (I.e Reddit)

Bezos and Jassey, Jobs and Cook, Page and Pichai, Gates and Nadella, (((Zuckerberg))) and (((Sandberg))).

Because last week's Apple revulsion ran its course. This week it is Facebook's turn.

We'll circle back to Google next week, and repeat.

Reply on news.ycombinator.com