Hacker News Reader

Comment by hmate9 | Hacker News Reader

"...various errors seem to have caused Facebook’s systems to log some passwords in plain text since as early as 2012."

1) As a software engineer I can't imagine how such errors could possibly have entered production code accidentally, especially after code review. If precise details of these errors are released I am open to have my mind changed.

2) Even if it did, I can't see how it would take facebook's tens of thousands of engineers 7 years to find this "bug".

View on HN

As a mere programmer I can very easily image dozens of ways such errors could easily enter production code accidentally.

With my admin hat on, I don't see how I would miss leaking millions of passwords to logs, since 2012. Unless I never reviewed them.

If they logged complete requests, the password might well be buried somewhere deep in request body, and not readily apparent to whoever is skimming the logs - especially if this doesn't happen for all requests, but only some subset of them.

Wouldn’t it be pretty simple (for FB) to create accounts with super unique password as part of their test process? Then they scan all their storeage for the super unique password? Or am I missing something?

Do you know of any companies doing that?

Yes, I do. They’re called canaries by the security teams. It’s especially useful for when you run automated tests when pushing updates in your pipeline. Scan the logs and database entries for the canaries.

Thank you, didn't know what they're called.

To implement, yes. But you'd have to think about this as a problem first. The very fact that we're having this discussion here now shows that it's not all that obvious, at least for your average dev. How their privacy and/or security review missed it is a more interesting question.

Well combine this with them releasing a "feature" to ask users for their email passwords and you have to wonder.

TIL facebook hires average devs.

That test user also would have to hit every nook and cranny of their code. That’s hard, and may not be repeatably possible (some code may only run at first login, or when making your 1024th post, or if you’re posting a movie before you ever post a photo, or when a single server sees its millionth post, etc)

They also would have to run it in such a way after every code update. Since every code update may introduce a new nook or cranny, that would slow down development too much.

I wouldn’t even try doing that, but instead have a two-pronged defense:

- code review of every single log statement in the code base by individuals whose _only_ job it is to prevent such problems.

- permanent checking of every single line logged for a thousand or so common passwords.

At Facebook’s scale, the second probably has lots of false positives, so I would do that at time of logging, when the location doing that logging is known, so that an alarm only gets triggered if a single log statement repeatedly logs passwords.

If someone logs a full request, the first thing that should come to their minds is the privacy.

Someone who works as a Dev on Facebook should know that HTTP requests can contain sensitive data, session cookies, passwords, credit card information, etc.

Probably because you don't work on systems anywhere near the scale of Facebook. Facebook has 2.3 billion MAUs, so if they had some edge case that hit 0.1% of users, that's already 2.3 million users.

And what does scale have to do with logging any plaintext passwords?

Because it was a bug. They most likely weren't doing this:

log(request.password);

But they had some code that did this:

log(excludeSensitive(request. headers));

But then someone on a different team changed how passwords were sent so excludeSensitive was broken.

Scrubbing passwords and other sensitive information from logs is such a basic requirement. It’s something you write a regression test for and never think about again.

As a manager, I can tell you we would never do such a thing, and if we did, then we never will again, and if we do, never again after that.

Not too far fetch... For example, I've seen people put in place logging systems that logged the whole request body and then someone changed a field name and all the sudden as a side effect the code that sanitizes the body to remove the sensitive information no longer does it's job.

I liked the suggestion of logging known password to detect leaks

It’s a good start at least. You still have to worry about the password being in the middle of a base64 encoded string and weird cases like that but at least you could grep for some known things.

1. You decide that logging passwords is bad.

2. You create a system that detects high entropy content before being logged.

3. You don't want to drop all high entropy content, so you create some rules about where in requests to look for high entropy content.

4. Something about the request structure changes, breaking your log filtering.

5. There is nothing that notices the drop in the amount of content filtered out of logs.

There's oodles of ways this could happen. I'd wager that more than half of all businesses that have a website that handles passwords has logged passwords in plaintext somewhere.

> You create a system that detects high entropy content before being logged.

Unfortunately, as numerous password breaches have shown, most passwords aren't that high entropy.

Perhaps also checking logs for x of the most common passwords could work? On the scale of Facebook this might very likely trigger some positives with such a bug?

"1) As a software engineer I can't imagine how such errors could possibly have entered production code accidentally"

Like i'd love to agree, but frankly, i'm not surprised. I feel like I've seen crazier things happen on production environments...

During a job interview at one mid-size startup one of the interview questions involved handing me a 2 page excerpt of server logs and asking me to identify bugs/issues from the messages and tracebacks in the logs. (neat idea!)

The logs contained user credentials, and they hadn't noticed. I pointed it out to the CTO.

Wow, their interview question (in the hands of the wrong interviewee) could have caused a data breach.

If it's a plaintext password, the minute it hits any human eyes, it is IMO compromised and the user should be required to change it. Employee, Interviewee, Hacker - doesn't matter.

I'd go a step further. IMO the second it hits a hard drive it's compromised so just the act of having that in the log would count.

Hey, in the wild! I've long advocated for companies to use actual bugs (that have been fixed) in testing candidates. Heck, as your anecdote suggests, they might find the candidate provides a fix better than their ninja rockstar did, and in this way could serve as part of a performance review for existing employees. ;)

This happens all the time, likely as the side-effect of some other activity - instrumentation related to exception handling / error monitoring, performance analysis, debugging, etc.

My assumption since the start of this is that it's telemetry / server log data. Honestly, I came really close to doing this exact thing.

My engineer: "Should we log the body of api requests?"

Me: "Yes, of course!....... waaaait a second. No."

The result was manually white listing fields that SHOULD be logged on each end point. It's a pain compared to "just log everything" or even a black list, but it's far safer.

>1) As a software engineer I can't imagine how such errors could possibly have entered production code accidentally, especially after code review. If precise details of these errors are released I am open to have my mind changed.

You can't imagine? Really? Error occurs, request that caused error gets logged.

I think Rails had this issue by default for some time.

Reply on news.ycombinator.com