Meta outage
metastatus.com
metastatus.com
It's like a power plant grid failure, except for attention instead of energy.
When meta is down, a hoard of internet users desperately seek somewhere else to place their attention... But the system is designed with the expectation that meta will take all that traffic... And boom! everything starts falling over. Wild.
If your service is down, please say "my service is down".
This should be a fun postmortem.
HN never fails us when a big website is down.
I was also almost ready to reset my password.
I fear my initial attempts to 'reset my password' and getting that same security(at)facebookmail.com email with the SAME reset code, have not helped me - if only I'd been asleep, I probably could have slept through it.
At least Insty came back up - though to be honest I cant remember that password either and now terrified to try and recover or change that now too!
This bug has been there since the first time I installed Spotify on a smartphone and it's still there a almost 15 years later.
I remember one time a company I worked at received a support message where the title was (paraphrased) "DONT HURT MY CHILDREN" from some person who saw an error message saying something about "couldn't dispose of child" or something similar, when the frontend broke. "Child/children" being kind of common in programming, I'm sure others faced similar scenarios.
I'm not sure if they were genuinely scared or just decided to have some fun with us while reporting the issue, but after that we made the error messages even more generic, so nothing could be misunderstood.
I guess it is this quest to not alienate the lowest common denominator that is the reason behind the stupidification of error messages. On one hand, people won't get scared, on the other hand, people get less context about why the error happened in the first place.
Wonder how many of the average users actually care?
>The Bomb icon is a symbol designed by Susan Kare that was displayed inside the System Error alert box when the "classic" Macintosh operating system (pre-Mac OS X) had a crash which the system decided was unrecoverable. It was similar to a dialog box in Windows 9x that said "This program has performed an illegal operation and will be shut down." Since the classic Mac OS offered little memory protection, an application crash would often take down the entire system.
Unfortunately, the Mac's bomb dialog could cause naive users to jump up out of their seat and run away from the computer in terror, because they though it was going to explode!
https://www.youtube.com/watch?v=zQGX3J6DAGw
And Window's "This program has performed an illegal operation and will be shut down" error message was just as bad: it could cause naive users to fear they might get arrested for accidentally doing something illegal!
Got a weird email with the same code every time from facebookmail.com
It's particularly egregious with critical stuff like banking.
I have gotten 4-6 "here's your facebook reset" type emails over the last month. All unrequested. I've always assumed they're casual attacks (mostly interested in seeing if my password can be stuffed into another, more valuable account)
Inexcusable to catch people in a reset loop during a login outage, and not confirm which password is the current one.
I just wanted to use their oauth login to buy a jonsbo n3 case from AliExpress. Sadge
A user called in early Saturday evening, saying that the system was down. After a bit of debugging, wondering where our alerting had failed, I concluded that the system was perfectly fine, but the authentication service had been returning 500 errors for around an hour. When I called the user back he made a comment that rather changed how I think about monitoring and systems. He says: Well, from my point of view it really doesn't matter which part isn't working, it's just down.
I was logged in when the outage happened on machine A and phone B - immediately tried to reset password, which took me into the hellish abyss many of us are experiencing now...
This evening - about 18hrs about the event, I fire up machine C - it logs STRAIGHT INTO Messenger. OK... This is something, too scared to open a browser in case that triggers the session disconnect...
So I fire up laptop D - STRAIGHT INTO MESSENGER... OK, open browser, STRAIGHT INTO FB. Log into Google password manager, VISUALLY CHECK password, and it is my last known good password (in use at time of outage).
Fire up iPad E - Straight into Messenger!!! All these machines are on the same network!
Back to Machine A - clear cookies and try to log in, no joy; different browser and try to log in, no joy; try to reset password on this different browser (I might add, I did get a new Change password Token number off this attempt), but no joy; clear cookies and restart, then attempt to log in, no joy!
WHAT THE F?!?!?!?
I initially thought it may have been a 24hr block due to password change attempts? But now not so sure... I've also tried logging in on via Machine A in a VM from a different O/S to see if it may have something to do with it - but again no joy - this environment had NOT been logged in to FB before...
Thoughts????
But on the flip side, I was able to create a back up profile in a different VM on this machine.....
Then i remembered i have a very long and secure password, then immediately panicked about someone having access to my Gmail.
The sense of security is more brittle than i thought.
Facebook data is more sensitive. Not so much the data people go there to see, cool memes that their friends liked, but the list of friends and interests.
Other places I worked had the ability for Ops to push out a change saying the site was down for maintenance. After a while we stopped using it and just took the hit of a bunch of 5xx errors. Basically when the planned down times became shorter than the time to propagate the down setting.
Didn't help that I had just posted a lukewarm spicy take on how linguistic prescriptivism is BS.
All the while the website felt like it was unstable, hard to describe, but it felt like it was bouncing around between URLs too much and reloading a lot.
Definitely feels like a botched update on their end.
E: Instagram is misbehaving as well, banner loads but big "Something is wrong" error on the feed.
E: now youtube has "Something went wrong" - WTF. I can't believe I'm saying this, but thank goodness for reddit and X[itter]???
E: interesting, seeing a big spike across multiple platforms on downdetector, including AWS: https://downdetector.com/status/aws-amazon-web-services/ I'm not able to log in right now, but that could be PEBCAK, I have too many saved IDs and I don't want to fail2ban myself
This would prevent people from panicking they've been hacked and/or unnecessarily resetting their password.
That said, getting there strikes me as pretty challenging. Automatically detecting a down state is difficult and any detection is inevitably both error-prone and only works for things people have thought of to check for. The more complex the systems in question, the greater the odds of things going haywire. At Meta's scale, that is likely to be nearly a daily event.
The obvious way to avoid those issues is a manual process. Problem there tends to be that the same service disruptions also tend to disrupt manual processes.
So you're right, but also I strongly suspect it's a much more difficult problem than it sounds like on the surface.
For example, if the service that authenticates a user stops working but the service that shows the login form works, then you get a complex interaction. The resulting messaging - and thus user experience - depend entirely on how the login page service was coded to handle whatever failure the authentication service offered up. If that happens to be indistinguishable from a failure to authenticate due to incorrect credentials from the perspective of the login form service, well, here we are.
At Meta's scale, there's likely quite a few underlying services. Which means we could be getting something a dozen or more complex interactions away from wherever the failures are happening.
If you can't distinguish those, then that is bad software design.
https://read.engineerscodex.com/p/how-facebook-scaled-memcac...
Yes, with the additions of sheer scale, a vast number of services, multiple layers, and the difficulty of defining "down" added in. I think the difficulty of reporting useful error messages is proportional to the number of places an error can reasonably happen and the number of connections it can happen over, and by any metric Meta's got a lot of those.
No, in that detecting when you should be reporting a useful error message is itself a complex problem. If a service you call gives you a nonsense response, what do you surface to the user? If a service times out, what do you report? How do you do all this without confusing, intimidating, and terrifying users to whom the phrase "service timeout" is technobabble?
If this occurred during the authentication process, I think I would tell the user "Sorry, the authentication process isn't working. Try again later." rather than "Invalid credentials". And you could include a "[technical details]" button that the user could click if they were curious or were in the process of troubleshooting.
Same auth system system used to validate logins to the bastions that have access to DNS. Voilá.
Those people would be wrong. You can take all unexpected errors and stick them behind a generic error message like "something went wrong" but you should not lie to your users with your error message.
If you have different messages for invalid username vs invalid password, you can exploit that to determine if a user has an account at a particular service.
"Invalid credentials" for either case solves this problem.
But sure, let's report infra failures different as "unexpected error"
Now, what happens if the unexpected error is only when checking passwords, but not usernames?
Do you report "invalid credentials" when given an invalid username, but "unexpected error" when given a valid name but invalid password?
If so, you're leaking information again and I can determine valid usernames.
So, safe approach is to report "invalid credentials" for either invalid data or partial unexpected errors.
Only time you could safely report "unexpected error" is if both username check and password check are failing, which is so rare that it's almost not worth handling. Esp. at the risk of doing wrong and leaking info again.
In this case, Facebook does not treat hiding username usage as a requirement. Their password reset mechanism not only exposes username / phonenumber usage, but ties it to a name and picture. So yes, Facebook returning an error that says credentials are incorrect when it has infrastructure problems is absolutely a defect.
Well, in principle, the frontend just has to distinguish between HTTP status 500 (something broken in the backend, not the fault of the user) and some HTTP status code 4xx (the user did something wrong).
And this caused a lot of extra trouble on top of the incident.
This is a pretty weird outlook to have - looking at any group awash with resources, whether it be governments or other companies, and you can clearly see that even with those resources, failures still happen.
You can jump up and down and pretend that this is solvable, or you can look at reality, look at all the evidence of this happening over and over to almost everyone, and conclude with some humility that these things just happen to everyone.
(Looking this reality in the face is one of the things motivating my beliefs around e.g. AI safety, climate change, etc.)
I hate it because it's bad UX, but that's the thinking behind it.
The argument here is the kind of nonsense cargo cult security that pervades the industry.
- in general, if the system is broken enough to be giving false-negatives on valid credentials, it's broken enough that there isn't much planning to be done here because the system's not supposed to break. So if they give me "Sorry, backend offline" instead of "invalid credential," they've now turned their system into an oracle for scanning it for queries-of-death. That's useful for an attacker.
- in the specifics of this situation, (a) credential reset was offline too so nobody could immediately rotate them anyway and (b) as a cohort, Facebook users could stand to rotate their credentials more often than the "never" that they tend to rotate them, so if this outage shook their faith enough that they changed their passwords after system health was restored... Good? I think "accidentally making everyone wonder if their Facebook password is secure enough" was a net-positive side-effect of this outage.
This is security by obscurity of the worst kind, the kind that actively harms users and makes software worse.
That information is accessible to two cohorts:
- authenticated users (sometimes; not even authenticated users get access to errors as low-level as "The app's BigTable quota was exceeded because the developers fucked up" if it's closed source cloud software)
- admins, who have an audit log somewhere of actual system errors, monitoring on system health, etc.
Unfortunately, I can't tell if the third cohort (unauthenticated users) is my customers or actively-hostile parties trying to make the operation of my system worse for my customers, so my best course of action is to refrain from providing them information they can use to hurt my customers. That means, among other things, I 403 their requests to missing resources instead of 404ing them, I intentionally obfuscate the amount of time it takes to process their credentials so they can't use timing attacks to guess whether they're on the right track, I never tell them if I couldn't auth them because I don't recognize their email address (because now I've given them an oracle to find the email addresses of customers), and if my auth engine flounders I give them the same answer as if their credentials were bad (and I fix it fast, because that's impacting my real users too).
To be clear: I say all this as a UX guy who hates all this. UX on auth systems is the worst and a constant foil to system usability. But I understand why.
Anyway, when FB thought my password was wrong I calmed way down. I thought maybe FB corrupted their password DB or something, so I just tried to reset my password, got into an odd workflow loop, and then quacked "downdetector facebook".
We have the same approach to password management!
Have a look at https://www.passwordstore.org/ and also https://github.com/kmag/store_password_gpg
As an individual, this is pretty confusing. I don't have much to lose. I am glad I spent 10-15 bucks on favorite 2-3 posts only. I can imagine many others to be more affected. What is normally to be expected for SME users? Does Meta resume the ads automatically? Do they make good for the lost time since the clock seems to be ticking -- although no one saw any impression.
Edit: I submitted a ticket to Instagram Help and they responded by asking for a screencast video. The first time I sent, the video bounced. I have re-sent this by trimming the video.
Out of curiosity I want to know firsthand how Meta handles the small customers.
What is the impact of it being Super Tuesday? Are people worried they can't vote without social media?
If the main social media used by liberals goes down, while the one used by Trump (forgot what it's called) stays up, surely that is an advantage for him?
For example, you can then pose the conspiracy that the fact it was Russian interference conspiracy was a conspiracy to justify more social media policing during the real election.
This one is already decided though. Biden on the Democrat side, Trump on the Republican. If anything it might hurt the Trump side a bit, as people may not realize that the Supreme Court only yesterday ruled states can not block him.
Can't you register the same day as the voting happens? Seems utterly stupid that you have to register to vote to begin with, but if it's a requirement, you should at least be able to register the day of the voting.
And states like Texas often have nearly one-party rule, so the primary pretty much is the election.
Because of the party that currently owns the non-urban parts of Texas, I usually vote in that primary, despite not voting for its candidates in the general elections.
I'm not very conspiracy-minded but this does smell a little weird.
At the very least, "there's a big event in the country of one of our biggest userbases, maybe hold off on risky deploys until tomorrow"
Shutting down social media has gone to the top of the list for regimes either "attempting to fabricate positive election results" or "attempting to combat the spread of misinformation about elections".
More sympathetically, for better or for worse (definitely the latter) there will be people trying to look up election information ("what are my local polling hours", "who is on my ballot") on social media websites, who will now not be able to be guided to the correct information.
We have a small livestock operation, and won an online auction late last night for a pig about four hours away. Facebook was the only listed means of contacting the person, and we were planning on driving to pick it up this morning.
Now I get to re-arrange my day today to deal with that, and will probably have to take a PTO day from work to drive there later in the week.
Real businesses are in fact impacted by Facebook being down - including those not based around Facebook and that you might never expect.
I really enjoyed reading a few of your comments, and would like to get in touch.
If you're open, drop a mode of contact in your bio/here?
Either way, global GDP should be up :)
I really would like a social network just for friends without all the garbage bloat.
I dream of a "facebook-like" app where you can only add someone as friend via a bluetooth protocol, forcing you to only add people that you've met in real life. Then text only, or with very limited image options.
Or is the report of Google auth being down actually tied only to meta logins?
My gmail seems to be working fine, both personal and work.
That rollout is staggered over time, so not all users receive it at the same time. It's unlikely to be related.
Which, I guess, is the best possible redesign: one that freshens up without rocking the boat.
Facebook + Insta makes up a huge share of the social media market, and when they go offline, it'd be natural for their competitors to receive large sudden upticks in trafic they're not immediately prepared for on a Tuesday morning.
But thats naive because ad serving isn’t totally sold out so they can make up for it by increasing the density of ads in the next time window. If the outage is short, then the impact is small.
But some markets are totally sold out and there’s no making up for lost impressions.
Do they share data about this?
Or raise the prices of ads.
When you've more-or-less monopolized a lot of the web's content sharing you get to tell your clients to pound sand. Where else they gonna go? Twitter? The incel white supremacist dollar is not what advertisers call "the good dollar".
no need to do that, for google search, people will come back later to make the search
Not only that but the bigger spenders will have more budget so the bidding after a large outage should return higher bids on average leading to increased profit per ad slot.
Meta 2023 ad revenue was $131 billion. To make it easy, let's assume an even spread for # of users and ad revenue generation per hour/minute of the day and day of the year (which I'm sure is not the case).
This would be:
$358 million per day
$15 million per hour
$249k per minute
This also assume a minute down won't be somewhat or totally offset by a spike in users when it comes back online.
Overall it was a good learning experience. I didn’t get reprimanded; several months later I got a promotion.
IG is not working as well. Feeds, profiles, messages are all blank.
Bad UX, Meta. Bad!
I would bet those who are not working in full time jobs are a lot more active though.
Whew.
My Messenger and FB on my phone magically logged back in a day or so ago, but on my PC, no luck - and that PC doesn't seem to recognise the password I thought it was before the outage.
A lot of people would probably also don't bother log back in, if there will be a password prompt?
[0] https://metastatus.com/whatsapp-business-api at "5:32 PM GMT+2"
https://metastatus.com/facebook-login
Platform Status : No known issues Mar 5 2024 4:38 PM GMT+1
The service is up and running with no known issues.
Joking aside, I wouldn't be surprised if this was the case, because during their last major outage (something related to DNS iirc) they were having issues pushing fixes because they couldn't login because their DNS was down.
EDIT: seems like the status page was recently updated.
I started to go through the password reset process and that failed as well. Then I got here.
- YouTube
- Google Play
- T-Mobile
- X (Twitter)
- Discord
- TikTok
- Pokemon Go
- Snapchat
It looks like they all have the same failure point.
Edit: actually a more attractive theory, given the very short timelines and near simultaneity of all those failures, is that downdetector itself had a failure, possibly a Meta-dependence, that they noticed and corrected quickly.
It's as good as asking a neighbor what happened with a loud noise down the street. Sometimes you'll get something good, sometimes it'll be completely wrong.
> It's as good as asking a neighbor what happened with a loud noise down the street. Sometimes you'll get something good, sometimes it'll be completely wrong.
Asking my neighbors if they know what some loud noise was or about some local disturbance has been extremely reliable in my experience. The one time someone gave me an explanation about something which wasn't mostly right they qualified it with something like "So-and-so said it might be such-and-such but I don't know if it's true".
Car exhaust :: gunshot Appliance delivery truck liftgate :: gunshot Transformer explosion :: gunshot Garbage truck :: gunshot 787 at 25000ft :: complete ruining of peace and quiet Any police activity :: probably someone robbed a bank
For the record, my city has (statistically indistinguishable from 0) homicides and bank robberies and, by American standards (I know, I know) no particular issues with gun crime.
One time I heard a loud boom. A few hours later I saw a neighbor outside and asked if he'd heard it and if he knew what it was. He told me a house a few neighborhoods over had exploded. I was a bit skeptical of it but he turned out to be right.
Sources:
https://www.statista.com/statistics/942043/laboratory-incide... - meth lab incidents are down to about 900/yr and have been far higher in the past (presumably because the labs have moved to things besides meth)
https://rpgaspiping.com/blog/critical-safety-tips/gas-safety... (286 natural gas incidents per year) - I've tried to find a more credible source for this number but keep seeing it cited in various places and have no better source, higher or lower.
I can’t tell if you’re trying to demonstrate the problem you have with your neighbors ;)
It was a gas leak, not a drug lab. The utility failed to fix things in a neighborhood where there’d been reports of leaks for years: https://www.wbur.org/news/2023/08/17/eversource-fine-gas-exp...
I'm confused. Isn't listening for spikes in complaints about outages a great way to detect them? I know for a fact some service companies monitor social media channels for this purpose (among others). I'd be surprised if that wasn't more or less standard practice.
I've checked Down Detector for ISP outages in my area many times now. It's always confirmed them before my ISP did.
Maybe they should have a backup password (if they site allows it, f-ing Spotify doesn’t), but it’s still effectively down for them!
When there's a major ISP outage, people report problems with all the major sites. When Facebook's down, people report problems with any site that has "Login with Facebook" as an option.
It's almost never actually an outage impacting all of FAANG at once.
If users log into your site with Facebook, then the login functionality of your site effectively is down when "Login with Facebook" is down.
From the user's perspective, your subcontractors, including authentication subcontractors, are a problem for you to deal with and never show them. From your perspective, you could have architected your site in a way that logging in doesn't "go down" when Facebook login is down.
If the user chooses "Login with Facebook" over other authentication options available, and they don't want to use other options, educating them with a good error message might help. Or you could remove the Facebook login option, if you (totally reasonably) don't want Facebook's failures to reflect poorly on you.
There are plenty of sites where "Login with Facebook" is a convenience but hardly the only way to log in. Reddit, for example, has "Login with Google" and "Login with Apple"; it would be highly misleading to claim "Reddit is down" if Google's OAuth flow was having an outage.
> educating them with a good error message might help
Nothing in the API or OAuth flow would make that doable in an automatic fashion with this outage. It'd have to be something you put up manually as a banner after hearing of the outage.
> Or you could remove the Facebook login option, if you (totally reasonably) don't want Facebook's failures to reflect poorly on you.
I don't particualrly care; we're talking about why DownDetector isn't necessarily ideal for assessing. It can be a useful signal, in some scenarios, but I've seen plenty of spurious signals come from it.
That is fair: if I choose to architect my site such that a user-critical feature goes down when a 3rd party service goes down, it behooves me to monitor the 3rd party service and do whatever necessary to properly inform users what's going on.
I edited my post unfortunately after you replied, but another option is removing the parts of your site that rely on 3rd parties, if you don't want the failures of those 3rd parties to reflect poorly on you (which they reasonably would).
>we're talking about why DownDetector isn't necessarily ideal for assessing. It can be a useful signal, in some scenarios, but I've seen plenty of spurious signals come from it.
Indeed, and if a bunch of users say that a feature of your site is down, even if it's a result of a 3rd party failure: chances are, that part of your site is down, and it's partially your fault for relying on a 3rd party for that feature. The users correctly don't care what the root cause is, they expect you to either mitigate it or don't have a feature they rely upon be unreliable.
Take a look at https://downdetector.com/status/aws-amazon-web-services/ ; scroll down to the comments.
"SSH and Dbconnect stopped on all of my EC2 instances. Anyone else?"
"I can't add a payment method"
The chart shows a big spike this morning, but there was no AWS outage, nor does Amazon use Facebook login.
Again, DownDetector can be a useful "is something unusual happening right now" signal, but it'd be a mistake to take its attribution at face value.
>The chart shows a big spike this morning, but there was no AWS outage
Are you sure? If hundreds of users simultaneously reported there was some sort of outage, particularly a huge spike like we saw, chances are there was an outage.
>Again, DownDetector can be a useful "is something unusual happening right now" signal
Exactly! Specifically, "is something unusual happening right now with my site, in the eyes of my users?" Every site owner should know when that condition is true. What you think about your site "up-ness" isn't as important as what your users think about your site "up-ness". What you attribute your downtime to, isn't as important as what your users attribute your downtime to (you.)
But that's not the case. It's a false positive.
Pick a DownDetector service and open the page every day for a few days. You'll see it most of the time just reflects people waking up in the US timezones.
In other words, we have hundreds of people saying there was an outage, and 1 person saying there wasn't.
That's a problem AWS needs to resolve, regardless of what they think might be the root cause. If the users weren't experiencing any issues with AWS, I doubt they'd be reporting it.
Your comment about timing is a good point: if people are working with AWS early in the day, and AWS is giving them problems, then they will probably report problems with AWS early in the day. I wouldn't expect them to report problems while they're sleeping.
Yes. AWS was not down this morning.
> In other words, we have hundreds of people saying there was an outage, and 1 person saying there wasn't.
We have hundreds of millions using AWS and AWS-backed services successfully this morning.
I'm out.
The fact that some people accessed AWS without reporting issues does not mean that all people did. For those who had issues, AWS is responsible for dealing with those perceptions.
Indeed, it could have been a fault that affected a subset of users, for example 1 service in 1 availability zone. That's still an outage in the eyes of users, which AWS is responsible for managing. It could have been an issue with a route from 1 ISP. That's still an outage in the eyes of users, which AWS is responsible for managing.
An even better example is the DownDetector page for Facebook, with hundreds of thousands of reports. Do we really think there's no correlation between what DownDetector reports and what users experience?
tl;dr: what users think about your site is more important than both what you think about your site and the reality of your site, and you should be tracking it.
Exactly. If you click through down detector when things are _up_ you'll see people still complaining that $site is down. Could be a local power outage or even a flaky connection in their own home.
Down Detector is one of many signal sources and should have a "credibly" score associated with it that's proportional to the number of people complaining that something's down.
Its attribution of what/who is often incorrect. You'll see "maybe it's more than Big Site X!" comments come up on every HN thread like this citing DownDetector; it's almost never the case, and folks on HN should know better.
Yes? That's how all top-level reporting is going to work. It's not going to tell you which part of your service is inaccessible. It's just telling you that people can't access it. You obviously have to do additional investigation to figure out why people are having trouble.
Scroll up the thread a bit; https://news.ycombinator.com/item?id=39605354
Even here on HN, where people should know better, people take its incorrect attribution as useful info. TikTok isn't down. X isn't down. Google isn't down.
Pull the page up tomorrow and you’ll see the same morning spike there as people wake up.
I'd trust Down Detector a lot more if it was filled with Hacker News community -- people who are able to understand that there's "DNS" and "Routing".. and that your phone can have internet access at home while your home PC does not.
I personally hate Down Detector's graphing because it can make it 'look' like there's an issue when there isn't really... Facebook with 500,000 reports looked as down as Google with 1,000 reports... For equally sized / used entities, I would not trust that "Google" is down with 1,000 reports. I had a coworker ask me what was going on with the internet because "everything is down.. Facebook, google, gmail, microsoft!" (when seeing the Down Detector home page)
DD should normalize the graphs against the service history in some way. A service shouldn't spike because it had 30 reports / hour for a day, then suddenly has 100... when it has a history of being out with 100,000+ reports. The 100 reports are probably mis-reporting, but you can't tell until you dig into each service, one by one, with separate page loads.
hell, i'm surprised Down Detector hasnt been outright sued due to the graphs being an actual honest representation of availability that shitty companies cannot hide
Gmail is also on the list. You can't use FB auth to login to Gmail, can you?
A couple hours ago after watching a video I went to my home page, which usually shows recommendations based on what I've recently watched plus a few videos labeled as sponsored that have nothing to do with any of my interests.
Instead everything on the home page was either a sponsored video, or a movie that was free to view with ads, or something from one of their music products.
I tried from an incognito window to see if it had something to do with being logged in. Normally going incognito loses the history-based recommendations but at least recommends user uploaded content. But now it has just like my logged in home page. No user content. Just ads and videos from Google's movie and music services.
Refreshing gave an error that said something went wrong. I then logged in on that page and again got something went wrong. Another refresh got a page with some user content. Another refresh was the ads and Google stuff page.
A little later it seemed to clear up and now my home page is back to normal.
Edit: I found the following, I wonder if it's still the case.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
I looked up the CPU mentioned in the link from your other comment. It looks like HN handles enormous traffic on about 2x the power of the last Celeron chip ever made.
https://www.cpubenchmark.net/compare/2383vs5793/Intel-Xeon-E...
dang, linked in one of the ancestor comments. But I still suspect you are correct.
> Sorry, we're not able to serve your requests this quickly. reload
Note that this only seems to happen for actions. Doesn't seem to be the case if I am just loading a page quickly.
I saw it just a few minutes ago, but I don't remember the exact wording...
I wasn't overestimating anything, but with how easy it is to write concurrently software today, why limit your site to a single core.
It even looks like Arc, the lisp HN is written in has threads now, but Arc is built on top of Racket and uses Racket's green threads, so it only takes advantage of one CPU core. Racket does have OS threads, but Arc does not use them.
These are good for actual business needs, but bad for resume-driven development.
Couldn't use it two nights ago, IDK why.
Both are above their baselines, but I bet some is just mis-reports, or increases in awareness due to more people checking in.
Meta seems to be the only one really affected from what I can tell.
“Yeah so it turns out when Facebook and Instagram goes down so does Google”
I do not envy the SREs at either company. I'm pretty sure all those other ones use Facebook or Google as their OAuth provider which is why they are all being reported as down.
This outage will result in absolutely no ridiculous conspiracy theories.
Actually, if all of them including Xitter went down, maybe things would get better? All the sunlight photons might get sucked in by too many eyeballs though, and there could be grass trampling.
I agree. While I don't think it likely that Facebook or YouTube would enter into it, I'd pretty much bet that DNS being down would cause problems.
And yes, there are bigger issues with that. Much.
For process of elimination, do all of these services do multi-platform logins? Or do some not connect to anyone else?
Interesting to see that all static content was still working during the outage (at least for Instagram). It was still possible to swipe through all reels (I assume the list was cached).
I wouldn't trust it as a single source, but in a case like this where our internal monitoring shows a spike of issues with the Google APIs and we can see a huge spike in reported issues for Google on Downdetector starting at the same time, it's useful to confirm that the issues have an external source.
If I can't login to tiktok because FB is down, then tiktok is effectively down for me. When it comes to technology most people don't care about the trip, they care about the destination.
So yea, tiktok isn't "down" but for a lot of people it might as well be, hence coupling your infrastructure/auth on other providers has side effects like this you must take into account.
Its mention should honestly be banned from this site.
https://twitter.com/elonmusk/status/1765048551023734801
Downdetector is user reports, not automated monitoring. It's... semi-trustworthy.
Even more so when the tweet in question isn't even a direct claim about Twitter, but just a meme making fun of a competitor.
That's an assertion, not a substantiation. A single tweet does not corroborate that, even if you ignore the fact that most outages of large global services (including some of the outages of these Facebook properties mentioned above) are actually partial degradations.
Triggering millions of people to unnecessarily reset their password yet still be unable to login is not a great UX. This seems like one of those cases that's high impact when it does happen, never likely to occur on any given day, but likely to happen at some point; probably just wasn't much focus put on handling a case like this.
Sure you can set up a UX to show that the auth server is somehow down and discourage users from trying to login/reset passwords, but when shit hits the fan, you actually never know the precise error that gets thrown to the client because it could be any layer between the backend and the client that failed...
Invalidating people’s sessions on apps? That’s a HUGE cost. There’s a huge % of users that won’t be able to get back in.
[EDIT] : and if it's a hack, by now half of the world has typed in their password to re-identify ... scary.
> Facebook is not working
Made worse in big corp due to affirmative action + lack of enough qualified candidates meeting diversity criteria.
Which is inevitable when you have coarse criteria applied to such a large industry this way so quickly, as it takes decades for anyone to be qualified for the senior roles, and many years for junior/mid level, even if there were no pipeline issues, which there are.
And unqualified folks in leadership, and mid level == stupid mistakes.
And, with the DOL rules, the company can’t even pay people differently, so no bueno even giving the high performers keeping things afloat better bonuses - unless they happen to meet the diversity criteria and it makes the stats look good.
Which it’s already hard enough to do properly when there is only one dimension, and impossible when there are 2-3.
so the bigger the company, the faster it has to cut its own throat.
Could you let us know where you work? I want to make sure I never apply there.
You don’t think the steady erosion in system reliability and ever increasing outages is unrelated to these pressures do you?
I’ve seen the sausage being made at the middle manager level in big corp for a long time. It’s never any one person/hiring decision, but the pattern and it’s impact has been obvious (and getting unavoidable) for a long time.
That no one seems to want to talk about the actual issues, but doing character assassination and black listing (like this comment) is part and parcel of the problem.
Outages have steadily decreased at major companies. I don't know what you're looking at.
Remember AWS taking out a good chunk of the internet many times a year because their east coast data center kept going down? Remember the fail whale meme-ing because Twitter was so unstable?
Industry site reliability has only gotten better over the years.
I’m sure AT&T, Google, Facebook/Whatsapp/Meta, BofA, Apple, MS, and many others who have had prominent massive outages and embarrassing product launch failures this year will be happy to hear this.
Notably, Amazon is one of the few companies that has managed to avoid a lot of the DEI noise somehow. Perhaps due to their reputation for having such a brutal work culture already?
I can’t wait to hear what you’re going to say next.
Big Corp Software quality improving AND running faster on existing hardware?
HN talks about people in open source holding up major functionality with little to no recognition. That happens within corporations too. Indiscriminate layoffs may directly fire those people, or signal to them that it's better to move elsewhere leaving gaps that only get discovered over time.
And so what happens when you’re required by the gov’t and leadership to also comply with coarse grained population statistics AND you can’t find qualified people that meet those statistics enough? On top of having to make layoffs?
My ex was a reasonably qualified software engineer, and even 4 years ago was getting no-interview offers because she was a woman - as explicitly stated by the recruiters.
It’s only gotten worse since then for hiring managers. She was offended because they literally didn’t seem to care if she was qualified or not.
I can provide links to signed and in force legal agreements between the DOL and Google for instance which formalize the need for this, and can point towards public records of evidence submitted to court of emails (internal) between recruiters which state the same too, btw.
Actual job qualifications (as in skills) did not enter the conversation at all. Just course grained DEI attributes.
So then they end up disproportionately cutting from the non-protected tranches (the groups that DO have to be qualified to stay) first because your stats still have to look good. I’m not saying DEI folks overall have no one qualified or hard working in them - rather, that there are little to no structural incentives for them to be. In many cases, they’re also unfireable/unlayoffable.
And eventually, the non-protected folks leave, burn out, or give up because f-this. Why do so much extra work when you literally can’t even get paid more for it, or be recognized because it will piss everyone else off?
And even if you’re superhuman on that front - everyone burns out eventually. Which is also why you tend to see what you see in Open Source.
It’s never any one decision, but stochastic movement in this direction has been relentless and inevitable.
When I try to access my general purpose account I'm forced to log in again. And when I try to change my password I get the "An unexpected error occurred. Please try logging in again." message.
I suspect that one of the password servers has been compromised.
Also, going to /r/facebook doesn't load, heh, there must be per-subreddit load issues?
My email requires 2FA, as does my facebook account, but when I went there and clicked "forgot password" there was an unknown email address added to my account. That shouldn't be possible.
ISSUE STARTING SESSIONS
We are investigating the issue
https://www.cloudflarestatus.com
I wonder if it's connected to this. Seems like most large players are having issues now.
Facebook, Instagram, WhatsApp outage (downdetector.com)
The Metaverse is temporarily closed for now.
Lets see where the majority of the discussion migrates to. (Twitter / X)
They are not down. (They just don't work for lots of people!)
> Updated Mar 5 2024 10:33 AM EST
and
> Updated Mar 5 2024 10:37 AM EST
My point was just that, of course, the status page is not reflecting reality.
According to them Whatsapp Business on premises solutions has issues since end of February.. But also looks like that WAB is the only product with API issues according to the status page.
Seems about par for the course for big tech these days. There's currently an issue affecting the Google Ads API causing timeouts when sending data to it, but the Google Ads Status Summary page shows nothing [0]. However, there's an incident detail page showing some vague hand-wavy information about incidents [1], which appears to be unreachable from anywhere on the Summary page. Gah.
[0] https://ads.google.com/status/ [1] https://ads.google.com/status/publisher/incidents/gNG1ppoY3y...
p.s.: The actual incident details URLs are available in the "RSS feed" link very transiently and tend to disappear -- the feed, which incidentally for fun reasons, is actually an Atom feed.
I think its a case of bad deployment by Google.
They recently introduced new OAuth flow.
Might be related.
No idea how long it will take to fix it.
Meta has a small collection of tools on AWS to deal with large SEV0 events like these. Another one of them is a basic communication tool that does not use Meta's own servers for anything (including auth), a super basic version of the internal SEV tool.
X-Amzn-Errortype: TooManyRequestsException
X-Amzn-Requestid:
Looks like the endpoints are on aws lambda and its getting rate limited.
Got a weird email with the same code every time from facebook ail.com
https://www.theverge.com/2021/10/4/22709575/facebook-outage-...
Should we assume the teams working to ensure Business Continuity & Applications Resiliency redundancies feel asleep at the wheel?
Also, to assume that no down or outage messaging go out during a fairly routine maintenance based outage?
I call BS
Lol I can't find scenario where this happens, at this scale at a company of Meta's scale
It's very spooky when supposedly the two companies who probably aren't sharing infrastructure seem to go down at the same time...
Ruth (Myha'la): This is for the better.
G.H (Mahershala Ali): For who?
[0] https://en.wikipedia.org/wiki/Leave_the_World_Behind_(film)
--
[0]: https://twitter.com/IvanMontillaM/status/1765036872290681089
Not saying is related, but timing is weird.
Why does Facebook need sharing news media or political ad targeting on its platform from a business point of view? Am I being naive or is it really such an important revenue driver to the business or the core experience of the app?
I think it is the source of enormous reputational damage and risk, that if I was running the company I would even happily trade 10% if not more of the market cap to removing any feature that enables news sharing on the platform.
Actually if you remove these two things (political ad targeting and commenting on news media) I struggle to find any other issue that would make facebook a "political" target, they can literally shutdown the fake news division that employs 10,000 people...
https://www.micahlerner.com/2023/07/23/defcon-preventing-ove...
It's not the DNS, It can't be the DNS, It might be the DNS, It's the DNS
Amphibious assault on Taiwan imminent.
Edit: YouTube seems now to be defaulting to a lower bitrate, leading me to guess it's a demand issue.
WHERE the HELL does Facebook store tracking data on my iPhone?
It shows my previous account even after I delete the app, clear the cache and KeyChain, disable iCloud Drive, AND sign out of iCloud??
Why can't I see where this data is stored? Same for TikTok.
WHY does Apple, parading around as a pompous paragon of privacy, allow this bullshit?
More likely that someone bungled a deploy of user auth. No doubt they are rolling back as we speak.
How would that affect YouTube?
I'm not personally a fan of most social media, but saying "they could go somewhere else" is a pretty naieve/ignorant response.
Of course, in the really long term, we're all dead. Meanwhile, the local mom & pop candy shops keep advertising and selling online and making ends meet.
God bless HN and America!
LOLLLL .. oh dear
me 2 seconds later: nah, i'll check HN
me 5 seconds later: ahh, there it is
As for alternatives, there is Pico, but Quest 3 may be superior in games selection. Or go wired which is of course less portable
There is no way to even access wifi settings or anything else to disconnect the device from the internet. If it's still a problem much later in the day, I'll try turning off my router to see what that does.
Seems like they really did change something in the latest firmwares.
At least from what I've read, there's a bunch of downsides for regular consumers: very expensive ($1000) -- SUPER expensive if you bundle in controllers and tracking points (~$1600), needs external tracking, wired instead of wireless, no built-in audio, can effectively only be used by one person (because each one is built custom to your face), and of course it's not a standalone headset, it has to be hooked up to a gaming PC.
Courage.
Not going to judge anyone, but also nothing to be happy about.
No system changes that, or makes it better, just different.
Standalone wireless headset, reasonably powerful chipset, can optionally stream from a PC either wired or wireless, good optics/resolution, decent controllers/tracking, large game library, large suite of features (including hand tracking and color passthrough), all for a reasonable price. Not sure any other headset really competes on all those things at once.
they did a fantastic job with hardware. I just wish they didn't couple the software so tightly.
I view VR headsets and their peripherals as no different than a mouse, keyboard, and display.
Companies requiring all this nonsense to use your device, put in that light, is ludacris.
Edit: Can we just acknowledge that a lot of the bells and whistles are for the companies benefit at the expense of the user? Thats their right, but it's also our right to want something better.
Correct, it's very similar to game consoles, though it is somewhat more open than those (sideloading is possible, including standard Android apps IIRC, and you can run PC VR games from other stores while tethered).
> Can we just acknowledge that a lot of the bells and whistles are for the companies benefit at the expense of the user?
It's the same model as XBox or Playstation, seems like. They sell the hardware at cost or at a loss, and make it up via software.
A fully open headset with comparable specs would probably cost much more for the hardware. From a business standpoint that would be very stupid for a company like Meta, but this is hacker news, and many commenters here see nothing wrong or silly about asking businesses to commit suicide.
This doesn't explain why its _required_. It just means there is precedent.
Your other point is better, although I think you mean it would cost the consumer more for the hardware, right? The hardware would cost the same to produce, it's just that the company would miss out on surveillance based revenue.
It's a reasonable point, fb would make less money if they made an open headset, possibly to the point that they wouldn't make it all.
But the world where fb doesn't make any headset, and the world where they make an unacceptable headset are basically equivalent to me - the former might even have an edge in that shitty relationships with corporations aren't being encouraged (like they are throughout everything tech related currently). Granted, them blazing the trail has a tiny chance of enabling a reasonable alternative to come along in the future.
But I am a bit of a Luddite, and I know that people want their toys, and they want them now.
More than likely most of Meta's revenue from the Quest series other than hardware is based off of, y'know, selling games. I doubt tracking what games you play to target ads in the OS is more valuable than the money they make when people actually buy games.
In Facebook or Instagram, you're looking at a space that they can shoot lots of ads into, and it's otherwise very hard to monetize. But a gaming-focused VR headset is a different story. Most of the time you're not looking at anything that can have ads in it, but you can actually sell stuff very easily.
Maybe this'll change someday if they actually get social media shit in there that's popular, I'm sure Meta would love that, but so far that hasn't happened.
> But the world where fb doesn't make any headset, and the world where they make an unacceptable headset are basically equivalent to me
Popularizing the format is useful for pushing the tech forward. A big player pushing lots of devices means that the supply chains feeding the manufacture of those devices bulk up too, not to mention other knock-on effects like greater consumer awareness, and "free research" for whoever copies what the market leader does (at least for things that aren't IP-protected).
> But I am a bit of a Luddite, and I know that people want their toys, and they want them now.
I can hear the sneer from over here, yes.
Isn't that a great argument for why they don't need to have such a hard requirement for a logged in session? Consoles didn't have an internet connection for the longest time, though only because it wasnt feasible yet. They moved a lot of games.
> I can hear the sneer from over here, yes.
I don't mean it as judgment, I know I'm the weirdo here. Sorry if that came off rude.
Consoles had physical games. VR headsets don't. Consoles treat digital games the same way Meta is doing them here, I think; if you get logged out, no more games.
The problem here isn't that Meta servers are merely down -- losing connection usually doesn't mean losing access to your library of games on consoles, or Steam. The problem appears to be that authentication is failing such that you're actually being essentially logged out, which would definitely lose you access to digital games on every console as well as Steam.
Which, I mean yeah, that's a big fuck-up on Meta's part.
Again, consoles and steam do this because they want to, because it benefits them, and consumers don't put any meaningful pressure on them for doing so. It's not some kind of fundamental requirement. It's helpful for e.g. anti piracy stuff, but not necessary. It is 100% feasible to sell me a digital copy of a game and then not hang around on my system and watch me play it.
People let triple A PC games basically put rootkits on their systems. It's not like the games wouldn't work just fine (or better even!) without them. It's just that approximately nobody cares, and the companies will do whatatever you let them do.
That could be valid when VR headsets were tethered to a PC via a DisplayPort or HDMI connection and essentially mirrored the display.
The Quest is closer to an iPhone or Android phone or an all-digital handheld gaming device. With integrated compute, display, battery, text input, pointing devices, mic, and speakers, it bears little functional resemblance to peripherals like a mouse, keyboard, or display with no utility unless slaved to another device.
Considering I can use my Quest with no wifi or other network to log in (once initial set up is complete), it seems that the Meta back-end APIs must have broke in some way that confused the headsets into thinking they were available when they weren't.
As you say, there's valid criticism to be made but it's hard to find the signal through the noise.
This makes it a device that's generally capable of using any supported source for its screens, and can pass its peripheral input to other devices, like a PC, not unlike a mouse and keyboard.
VR headsets could treat their "OS" as a minimal experience akin to an OSD on a monitor that lets you switch sources and use the peripherals more generally like a mouse/keyboard with the right drivers on the target machine.
I'm more interested in calling out that Meta missed an opportunity here, and that it's confusing that they offer some semblance of these features (wireless linking for SteamVR...) while coupling that so closely to their OS and online-only experience.
First, the original Rift headsets were as you describe: lightweight, passing through the PC VR image. However, Meta did not miss out on an opportunity. In what was perhaps the most effective A/B test they could run, they released the Rift S (tethered PCVR) and Quest 1 at effectively the same time. The market feedback was resounding: I believe it was a 10-to-1 preference for a standalone experience vs. tied to a PC. Since they doubled down on standalone (or all-in-one if you prefer), well over 20 million headsets have been sold. In fact, they're so popular that even the fraction that connects to Steam is basically tied for market share with the most popular PC VR headset ever, the Index.
Second, even as a PC VR HMD it was a real stretch to call it a monitor equivalent. It's wildly complicated to create compelling VR images. You need two screens at nearly 2Kx2K resolution each, running at 90 frames/second, sustained. Dip below that and you can induce nausea. Not every PC can do that, so you need careful engineering between the client and HMD, with tricks like time warp, space warp, interleaving, compression, prediction, pose estimation, etc. to take up the slack. Creating sub-millimeter precision of location with six degrees of freedom either requires external base stations (cost, complexity) or inside-out tracking with headset-mounted cameras and a processor running realtime simultaneous location and mapping and image recognition code, which implies a CPU and tech stack to support it. Nowadays people also expect passthrough (with real-time depth correction), hand tracking (AI routines for hand posing), and more. All this is to say that significant code must run on the HMD for a modern gaming headset (Meta's target market), as well as on the PC. And if you're investing that much in a custom software stack, you can't make it up on hardware margin - the cost to build an HMD is just too high. So you have to have an app store tie-in, because Valve sure isn't going to share its Steam profits with you.
Now, certainly there have been (and are) HMDs that tried this approach. HP (G2) and HTC (Vive series) both put out quality products leveraging the Steam ecosystem. Neither are sold in volume today, because the economics of selling a headset just aren't good enough.
Immersed and Big Screen are releasing very lightweight fixed-function HMDs for either work or movie watching that do operate the way you describe. Neither are expected to be high volume devices, and both are more expensive than Quest 3.
In short: VR is much, much harder than you may realize. Meta didn't miss an opportunity, the explicitly chose the market-tested, most popular solution that also has an economic model with some potential future payoff. If you want a "minimal experience akin to an OSD" then look at the Big Screen Beyond ($999, https://www.bigscreenvr.com/) or the Immersed Visor ($1,049, https://www.visor.com/). (Note: compare the price of these hardware-model pass-through devices to the Quest 3 ($499) which also includes a CPU, battery, storage, audio, more RAM).
It's also worth noting that Quest 3 is not online-only. It works fine offline once you've logged in once (people use it on planes, in parks, in the car, etc.). But this particular issue at Meta forcibly logged out users, then the API appeared online while failing all future login attempts. Ironically, users that work offline never noticed the outage because the bug couldn't log them out.
Yes.
The current problems sound like a server-side bug while it phones home. But usually it can work fine without internet.
Standalone just means the VR compute is happening on the headset itself, not on a console or gaming PC the headset is tethered to. Of course, most of the people disputing "standalone" already know that, they're just playing definitional games.
It would be cool if Valve came out with a standalone headset, they're one of the few companies I can see that would be in a good position to do that: they already have a good amount of VR experience with one high-end headset + SteamVR APIs + a couple VR games, they have their own highly popular store/platform, they generally have a positive reputation with gamers, and they have a decent amount of hardware experience in general including the recent Steam Deck for mobile gaming hardware specifically.
And of course, a Valve headset would probably be significantly more open than the Quest. The Steam Deck has gotten some good reputation among more FOSS/hacker-oriented people for being fairly open: you can use it in a regular Linux desktop mode, you can install Windows (or presumably other OSes) on it, it's fairly repairable, etc. The default behavior is very console-like, but it's not very locked down if you don't want it to be. Best of both worlds, really.
Yes, because etymologically it's "standalone" vs. "wired"; this is akin to how phones are "mobile" vs. (when I was a kid) "landline".
It's the equivalent of finding out that if Microsoft's auth servers go down no one with a Windows PC can use it since they can't authenticate. I'm fairly displeased.
Standalone just means you don't need to tether to a PC or console.
I am so sick of companies "selling" computers that they continue to control. In what universe does Meta have the right to remotely lock my headset and prevent me from using it to run the software I installed on it? If I were to sell my current desktop computer, or phone, or whatever, on any marketplace, and leave a remote login account on it that I then used to continue to operate the computer as though it were mine remotely, installing software, playing games, and occasionally peeking at what the current owner was doing, that would be obviously criminal. How is this any different? Because I signed away my rights when I "agreed" to their Terms and Conditions box (which I was compelled to do to use the hardware I purchased)?
Something is so fundamentally broken in the current ownership/property landscape. We somehow ended up in a world where people don't own the most critical tools in their lives, companies have managed to recreate feudal fiefdoms within the bounds of the market.
I think these companies need to be reminded they do not own our PCs either.
I'm really starting to like that mantra of "If buying isn't owning then piracy isn't stealing".
As in, gaming subscription services like Xbox’s GamePass won’t succeed if gamers prefer to buy games over paying a monthly sub.
...then Facebook bought Oculus...
...and then required you to have a Meta account to continue using the Oculus drivers.
It's a real "boil the frog" strategy and this is still early days for VR in terms of realized market value. The time to push back on this bullshit is yesterday. As we can all see, nobody can compete with Meta on price with the Quest 3, but the cost to purchase is heavily subsidized by the expected futures.
Of course, OP owns a Quest 3 so its more cut and dry there.
It also fits over / around glasses
Biggest reason not to IMO is of the rumors around an "index2".
More broadly, I find that Valve represents my interests far more than Facebook/Meta does. So I'd much rather send my money to Valve.
Another big reason not to buy a Steam Index is "not available in your country". The only VR headsets I've seen in stores here are the Quest 2/3 and the PSVR2; and the Steam store page for the Valve Index (and the Steam Deck) says "not available in your country".
The difference being that we are discussing platforms, not the things that run on those platforms.
Sidenote, but my experience with the Rockstar launcher has been absolutely atrocious, to the point that I just avoid rockstar at this point even though I'd otherwise be interested because I've been burned so many times. That's a Rockstar issue, not a Steam issue.
Wonder what data they collect from gamers.
(and yes, there are ways if you're devoted enough, to roll your own everything and run Linux on a Framework laptop, and use some kind of custom ROM on your phone without Google anything, 3d print yourself a VR headset, etc. But all of this will cost you several orders of magnitude more time than Meta outages ever would.
A short update from meta with some initial high level technical details:
https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A...
This system that checks for invalid values in the cache looks like a very bad idea in the first place as in my understanding it checks things beyond "is the cache up to date?".
edit: https://www.cnn.com/2010/TECH/social.media/09/24/facebook.ou...
> The cut lines include Asia-Africa-Europe 1, the Europe India Gateway, Seacom and TGN-Gulf, Hong Kong-based HGC Global Communications said. It described the cuts as affecting 25% of the traffic flowing through the Red Sea.
https://apnews.com/article/red-sea-undersea-cables-yemen-hou...
Could be both.
Timelines don't match nicely
Pretty neat if a week after a cable is cut, FB falls over.
Especially when most of the source of truth databases are in the US and Europe, and that sort of data flow doesn't cross the Red Sea. FB has datacenters and points of presence all over, but outside the US/EU it's almost all caching.
that'd be one helluva cache!!
It's not DNS
There's no way it's DNS
It was DNS
Petunia?
Then I saw the news that it was an outage.
It isn't perfect for even that IMO.
> Even Internet search does not help to find out how something trivial is done... The only way is to watch youtube videos.
That says more about where the web is heading than about facebook. Video is easier to monetise ATM⁰, and these days people don't put helpful stuff out there just to be helpful as much as they once did¹, so content creators are making them instead of simple web pages.
--
[0] Everyone wants to be the next big influenza who doesn't need a day job to get by.
[1] That sort of people are still out there, though they are somewhat drowned out as the signal-to-noise ratio heads inexorably towards “WHAT? WHAT?! I can't hear a single thing above the manscaping adverts!”.
Probably not so easy to implement in behemoth apps, consisting of 20'000 source files...
>Statista https://www.statista.com › statistics › facebook-global-dau Feb 9, 2024 — During the fourth quarter of 2023, the number of daily active users on Facebook reached 2.1 billion, a minor increase on the previous quarter.
So roughly 28% of the planet.
You have to be kidding me right?
Should we assume the teams working to ensure Business Continuity & Applications Resiliency redundancies fell asleep at the wheel?
Also, to assume that no down or outage messaging go out during a fairly routine maintenance based outage?
I call BS
Lol I can't find scenario where this happens, at this scale at a company of Meta's scale...
Facebook, WhatsApp, Instagram, Threads are all down. and its time to contact the CEO of Meta to bring them all back up.
At least Twitter / X is still up, so time to complain about it there.
Jest aside, i wonder when/if stuff like that will actually happen.
That being said this year's primaries have got to be historically uncontested that it could not matter less.
Grassroots organizers use FB and messenger and Insta for “get out the vote” communications. People use these services to goad their friends to go and vote. People use FB as a search engine to identify their polling place.
Or you could imagine it as this: your animal brain has evolved to notice patterns. Seeing coincidences like this is akin to seeing faces in the stars. The challenge of evolved being is to override those impulses when they're not logically sound.