Comment by pizza | Hacker News Reader

Comment by pizza | Hacker News Reader

Parent Full thread

> My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.

This imo is the truly alarming takeaway. FB employees were retrieving user passwords? Around two thousand FB employees? How in God's name is Zuckerberg going to perform his usual performative contrition about that one?

I'm just trying to imagine the data structures that were being retrieved from databases. Either they stored something like a big user account data type that contained their password in plaintext, which imo is a really weird design choice, or logs for other services were being mixed in with logs leaking the user/pass combos.

Surely one of the engineers could have noticed and said 'wait a minute... those are logins' over the course of the years? We hear all the time that people want FB to follow a responsible social practices (the debate on what those are rages on, which is great imo), but can't FB at least wrangle its own code base?

On the other hand, we shouldn't take the stance that heads should roll, imo - it would just create a chilling effect that would deter other companies from ever going public about their own security mishaps.

edit: I should probably tone it down in this comment but I'll leave it for posterity

>FB employees were retrieving user passwords? Around two thousand FB employees?

This sounds to me like they were writing it into some log analysis tool, and people happened to pull it up.

I have no knowledge of FB internals, but as an example let's say they have Splunk, which is a popular distributed log analysis tool (or some custom equivalent, because FB is huge). Splunk makes it very easy to pull up tons of logging for any app - that's what it's for. So write a poorly-constrained query and you can get millions of log lines from many different apps.

It's also feasible that everyone (or nearly so) in the company has access to Splunk.

So if somebody, somewhere accidentally writes passwords into a logfile, and it gets indexed into Splunk, a ton of people might theoretically see it, but just skim past it because it's not what they were looking for. Picture trying to hunt down an error, and wading through many screens full of logging from different apps - if there's a password in there you could easily not notice if you're not looking for it.

That's all speculation, but it's also entirely plausible. All it takes is accidentally logging out those passwords to somewhere that Splunk can see.

Also not trying to say this is in any way OK or acceptable, just that I can understand how a seemingly-small mistake could quickly result in thousands of people having access to passwords.

This should be the main takeaway. If true, then 2000 people decided not to raise the alarm. And 9,000,000 queries with results that had passwords. This completely invalidates the other thread of people discussing giving facebook a pass because accidental logging could happen to any company.

> If true, then 2000 people decided not to raise the alarm

Have you ever worked at a software co, discovered <terrible software practice>, raised a flag, and were told "thanks, we've added a ticket to the backlog"?

I'd bet that some of them did flag it.

Explicitly logging a password is one of those practices that doesn't sit on the backlog.

It's probably a bit more complicated than that. Usually the things that I encounter have to do with how HTTP requests are logged.

For example, putting sensitive information in a URL that's loaded over HTTPS is considered insecure because many companies have policies where they log every URL that their employees visit. (Think of a password reset link.)

A lot of inexperienced programmers don't realize this, because they don't realize that you can man-in-the-middle yourself, and that most corporate computers come preconfigured to allow the employer to man-in-the-middle everyone.

So, if a password reset link never expires, it means that some guy in IT can own an account that was reset on a corporate computer.

(This, basically, is how they catch people viewing porn on their work computers.)

Anyway, my point is that the problem is probably something where a junior programmer transmitted a password in a way that they didn't realize was being logged.

> Explicitly logging a password is one of those practices that doesn't sit on the backlog.

If that is your experience, then that's a truly wonderful thing.

Might it be possible that at many companies, teams with deadlines to hit will tend to prioritize feature work over details like this? Perhaps especially so when teams are not rewarded for fixing vulnerabilities and are punished for not meeting deadlines? Particularly when the actual bug at hand is that the full contents of a POST are being logged, and a PM might not read the ticket enough to understand that this includes a password...

Again, you're completely right in every way about what should happen. It just could be possible that this could reflect something other than all experiences all software engineers have had.

> Explicitly logging a password is one of those practices that doesn't sit on the backlog.

For you maybe. For Facebook obviously it's different.

>For Facebook obviously it's different.

What evidence do you have for that? Nowhere in the article does it say facebook "explicity logged" passwords. The logging likely happened through some unintended and roundabout process that is far from explicit.

Why would this be down voted?

I think incompetence over malice is almost certainly the right answer here.

How many times does a company of Facebook's size get to say "oopsie, we're sorry" before you'll stop giving them to benefit of the doubt? I see this as a malicious disregard for the security and privacy of their users, and their history aligns with that view.

That's a valid point of view, of course. The failure is certainly well beneath what one should expect Facebook.

Nonetheless, I stick to incompetence over malice. There's just so much more of the former in the world than the latter!

Back when I was sysadmin, I catched everyone watching porn by suddenly walking into their offices, but I'm sure your method is better.

You're probably right, and I would love to see that confirmed. FB wouldnt have to link to the Jira ticket or give the name or details of the ticket, but verify that was the case and then explain why nothing was done.

Still interesting that not one of them went all out and publicly derided the company. The whistleblower clause would probably have protected them. However our whistleblower laws could use a brush up.

If it's not a feature that makes money why spent developer money on it? -- Every manager ever.

GDPR may levy fines but ultimately it hasn't done anything to stop managers focusing on new features and sales income.

How much do you want to bet that some of those FB employees decided to login by turning off "location detection" flag and in an incognito browser?

I would not be surprised. Some of those 9,000,000+ queries and 2,000+ employees must have had some nefarious use. Statistically speaking...

Judging by the maturity level of the discourse on teamblind this precisely what happened.

Judging by working with hundreds of software engineers over a decade, this seems very unlikely to have happened.

No you're misrepresenting what (probably) happened. They had some system that logged API requests (fine and normal). Some API requests include plaintext passwords (also fine and normal).

The issue is presumably that they had no exclusion to the logging for sensitive information like passwords, which is honestly very easy to overlook.

So two thousand Facebook employees were not "retrieving passwords". They were looking at the API logs, which is a normal thing to do.

This. It's a bit baffling to me that even the HN audience doesn't come to this conclusion immediately - maybe people want to assume the worst because it's Facebook?

Definitely. This always happens when Facebook comes up here. People forget to use their brains and just rant about how evil Facebook is.

> Some API requests include plaintext passwords (also fine and normal)

Wait, no ? How is that fine and normal ? It doesn't seem fine and normal all.

It's completely standard. If you encrypt the password in the browser, potential hackers have access to the encryption code which makes it useless.

Check the network tab in dev-tools when logging in to hn for example and you'll see your password in the request body :)

Maybe normal but still not best practice

What that quote is saying is that the logs that contained the passwords were accessed by 2,000 engineers. Most of those engineers would only be looking at data relevant to their job, only a security engineer would be in a position to notice the passwords, which is what happened.

Isn't it possible that those developers just wanted to look at something harmless like for example failed logins and did not even see or reflect on that the log messages also included the password? But if so it is a bit worrying that 2000 eyes did not spot the bug.

Maybe many people were just using SELECT * FROM ... and got plain text passwords they weren't necessarily looking for.

I read the article and this is what I got from it too. I `SELECT *` all the time and see hashed passwords. It is very rare that I need that hash, usually I'm doing something entirely unrelated, but it's just easy to get all the rows and only use what you need, especially when you are troubleshooting and don't even know what you need yet.

Probably some poor engineers trying to check if their partners were cheating on them. Tools like FB make it easier after all.

Reply on news.ycombinator.com