Hacker News Reader

Comment by EpicEng | Hacker News Reader

I've worked in healthcare / biotech for more than a decade and I can promise you that the FDA would see no difference between the two types of gaffes. As custodians of sensitive information it was our responsibility to ensure that said information didn't leak, period.

I don't know anything about FB's infrastructure, but when I was lead I would have viewed leaks in logs as far worse than something in the DB because our DB's were harder to gain access to.

I get what you're saying, but it's irrelevant. Easy to screw up, hard to screw up, doesn't matter; just don't screw up because the result is the same. This stuff is security 101. If you're logging requests then you need to ensure they don't contain sensitive info.

View on HN

I don't think anyone's arguing that it's ok, just that it's less "easy to screw up" vs "hard to screw up" and more "did something really stupid on purpose" and "did something really stupid by accident."

But yes, either way the end result is the same and you shouldn't do it.

Yes but how many people looked at these logs, saw the passwords, and said nothing? If it was logged presumably the logs were viewed from time to time. The mistake is easy to make, but it's also easy to correct if there is a culture to report such mistakes.

That's true. Kerbs article says:

> My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.

Those numbers are pretty meaningless without some concept of what queries they were making. When I was working with large logs I'd regularly pull huge amounts of logs to only look at maybe a handful of lines. Even then often only scanning for the things I'm specifically working on. Not saying this is what happened just saying those 2,000 employees almost definitely were not doing careful analysis on the complete result of their 9 million queries(And shouldn't have!).

Out of 2000 users, there was not one that had a look at username/password? Impressive. Also developers during dev. Seriously, this is not justifiable, it's pure laziness if we want to be positive.

'Only scanning for the things one is working on' to the point of missing huge security holes or safety risks is exactly that's wrong with the industrial division-of-labor approach. Workers start to assume that even if they do notice a problem, someone else is either going to fix it or has examined and OKed the current procedure. Workers who speak up and assert that something they noticed actually looks more important than whatever task they were originally assigned to are typically punished rather than rewarded, and fired if they make too much of a fuss because otherwise managerial authority might be diminished. Most firms run internally like a dictatorship, and given Facebook's absurd ownership structure (where there are voting and non-voting shares Zuckerberg retains a majority of the voting shares) it's a recipe for dysfunction.

This is true but not very interesting. Like literally every large tech company in the entire industry, Facebook has a specialized team of people that work on security --- it has one of the better versions of that team, for what it's worth. Just like ordinary Apple engineers, ordinary Google engineers, ordinary LinkedIn engineers, and ordinary Airbnb engineers, ordinary Facebook engineers don't have an end-to-end picture of how all of security at Facebook works.

There's no evidence that there was whistleblowing suppression about this issue anywhere at Facebook, is there?

What if one among the 2000 was curious ?

The article indicates it was searchable and used for something by employees, so most of these comments to the effect of ‘they must have been doing this accidentally because Facebook is just so big therefore they are excused’ are invalid. Some group of people did this on purpose and knew it was happening.

You're probably being downvoted for speculation, but that sounds completely reasonable to me. At least one person had to have noticed a password in a log file they were viewing at some time. Most people viewing log files know that passwords should not be there. It would trigger alarm bells, and depending on the person, also excitement -- you have somebody's facebook password.

It's conceivable that someone then told their friend at work, and a few of these 2,000 developers knew of this secret internal stash of passwords they could access whenever they wanted to "prank" someone on facebook...

Why would humans be reading these log files?

For plenty of reasons, I'm sure. From the article:

> My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.

That doesn't sound like a human reading the log. It sounds like automation.

A script counting the number of successful logins, for instance, could read the same data element that unintentionally contained passwords.

We don’t have enough information to make that judgment.

When you’re developing a feature like this, don’t you look at the data that you’re logging in to make sure things are working properly? I would imagine that Facebook has many layers of abstraction, but in somewhere, incompetence or inattention must be involved, sometimes known as negligence, if not outright knowledge of this. The definition of that is a complex thing, but if someone could have reasoned that this was logging plaintext passwords, and either saw it and didn’t bother to change things, or didn’t think about it carefully enough to realize that it was doing this, it would be considered negligent. I know that I would feel that my trust in this organization has been betrayed, as a user.

The logs being in a searchable index makes it more likely that the password storage was inadvertent, not less. It implies that the primary usage model for the logs was targeted queries, not people starting at the top of the logs and reading down in such a way that nothing could have been missed.

My interpretation was that they were using the plaintext password as one of the searchable fields, presumably for development related to authentication.

Zuck doesn't believe in hiring employees over 30 because they're not 'smart' about tech. Great example why that's a mistake.

If he did, someone would have seen that with 10-15 years of corporate experience and would have found that simply unacceptable. They also would have had the gravitas to know how to escalate things to the top and gotten it fixed.

I understand that, but I believe they do not have the correct mentality here. I'm not certain how many of the people saying that have worked on systems as sensitive as FB. They think it's easier to screw this up because, for them, it is, but they don't operate at this level.

This sort of thing should not be any easier to screw up when you operate at the scale of FB. If it happens then you have the wrong procedures in place. In my personal example our logs and logging practices/code were audited in the same way our DB layer was. There was no difference; if a system touches sensitive data, it is a massive vulnerability.

This. If you have two or three engineers at a startup it's a very easy mistake to make. With thousands of engineers it's an impossible mistake to make, so it could have only been done purposely.

And even with a startup of two or three engineers, it's pretty much unforgivable.

It depends on how many users you have and what sort of data you're protecting. If you have a billion users then sending password reset tokens to your analytics provider is a very serious vulnerability. If you have 100 users and you're running a generic message board then it's not really a vulnerability because the difficulty of that method of attacking the assets under protection almost certainly exceeds their value.

> that method of attacking the assets under protection almost certainly exceeds their value

This was most likely the reasoning Sony had used (prior to 2012) when deciding how to safeguard the account info of users who had registered for marketing promotions.

It turned out that user credentials are unlike, say, office furniture, in that the cost of a data breach can vastly exceed the "value" of the protected "asset".

That's the best practice in the security world though. There's no such thing as an unhackable system, only systems where A) the cost of attacking them is more than the value of the assets under protection B) where the time it takes to attack the system is long enough for the attack to be detected.

C.f. Bruce Schneier's book: https://www.schneier.com/books/beyond_fear/

That book was from 2003.

This is Schneier in 2016: https://www.schneier.com/essays/archives/2016/03/data_is_a_t...

Data Is a Toxic Asset, So Why Not Throw It Out?

"All this makes data a toxic asset, and it continues to be toxic as long as it sits in a company's computers and networks. The data is vulnerable, and the company is vulnerable. It's vulnerable to hackers and governments. It's vulnerable to employee error. And when there's a toxic data spill, millions of people can be affected. The 2015 Anthem Health data breach affected 80 million people. The 2013 Target Corp. breach affected 110 million.

If data is toxic, why do organizations save it?

..."

I really could not disagree with this more.

"could only have been done purposely". In this case there's also a chance it was negligence. Who gives a shit. Move fast. A negligent culture can probably do more damage than the odd individual doing something deliberately.

The problem is, you can't ensure that. You can do best effort filtering/scanning only. Let's say you have an API and someone writes a client with a memory corruption bug which results in password and some other field being sent swapped 1 in 10k times. Now you have password logged. You maybe can't even tell exactly where and it's close to impossible to detect automatically. "just don't screw up" is not possible in this case and the chance you'll learn about this soon is really small.

Or if you think a client bug is a stretch: If you're running a large enough website which logs usernames on failed logins, I bet you have at least one password or a concatenated usernamepassword in your logs. Just because someone wasn't paying attention to the text box focus.

You can never ensure anything completely in engineering. Perfection is never the goal, but you can do a much better job than FB has here.

You're talking in hypotheticals. Nowhere did I say "nothing ever goes wrong if you know what you're doing". What about the issue at hand? What's your opinion on passwords being logged on every request for years?

> This stuff is security 101. If you're logging requests then you need to ensure they don't contain sensitive info.

If what you meant by that is "best effort" then sure. The issue at hand is only obvious now that we're reading about it. It could easily be missed depending on how their infrastructure works.

>The issue at hand is only obvious now that we're reading about it

No, it's not. It's completely obvious to anyone who has ever worked on a system which deals with sensitive information. It can only "easily be missed" if you're not auditing your code and logs. This stuff is basic to anyone who works in industries where the data is a liability.

Large orgs could have a central knowledge base of all places someone could type a password (as required by code reviews, and perhaps by automated detection of password-type inputs), and a bot which attempts to fill correct and incorrect passwords for a number of canary users, and scans across all logs for plaintext canary passwords. It's shocking that someone like Facebook, which has so many partner-facing and user-facing interfaces, wouldn't have developed something like this.

Free startup idea: build a SaaS for the bot phase of this, and make the log-scanner simple and open source so enterprise security teams can deploy it freely. Offer a certification process and consulting. Hire lobbyists (and folks with inroads into insurance companies) to paint plaintext passwords as the devil and make yourself a de facto legal requirement. And make sure your log-scanner doesn't do any logging of its own :)

This is exactly how Zuck was able to steal login information in his stories about founding facebook.

I find it hard to believe this is accidental.

Do you have a reference? I was wondering the same but have no reference.

https://www.businessinsider.com/henry-blodget-okay-but-youve...

Yep i remember this story. So clever and yet so dirty.

This is borderline illegal, if not simply illegal. What if he entered into Paypal accounts or bank accounts? I don't want a world where if I enter the wrong password, I have companies pouring over my other accounts trying to breach them. Let alone if I reuse a password by accident.

Wow! That does not look accidental at all!

> I don't know anything about FB's infrastructure

This explains a lot of the comments here. Facebook’s scale is not like anything most engineers have worked on. Facebook probably has logs in the 100s of terabytes. Ensuring that sensitive data isn’t logged takes more than some occasional greps.

My point is that it's irrelevant. Have more data? Need more auditing. Any system which touches sensitive data is subject to security review. Yes, FB systems are massive. They should have massive oversight as well. You may as well be defending a nuclear meltdown because desiging nuclear power plants is hard.

>Ensuring that sensitive data isn’t logged takes more than some occasional greps.

Right; it requires investment into process, requirements, testing, and oversight. Most importantly, it requires a company wide, top down mentality that your customer's privacy and protection is more important than your margins.

If you can't (won't) dedicate the resources required to ensure your customers data is protected then you have no business operating at such a scale.

Exactly. Can every piece of data logged be tied back to a legitimate business purpose? What’s needed here is a mentality change: These logs should be thought of as liabilities rather than assets. You should log only what you need, while you need it, and then turn off the log when you’re done. If your mentality is “log everything, always, because maybe we’ll need it later” then these privacy and security trash fires should be expected.

Log everything: you don't care about privacy.

Treat logs as liabilities: why can't you solve the issue I experienced yesterday?

You can't win. Either you log more than you think you need right now or you can't do engineering investigations on past data. You're going to end up somewhere in the middle realistically.

Everything is a tradeoff, how much cost that fb incure due to this incident, I would guess not so much, at least not big enough to warrant massive resources needed.

Well, that's the problem really; they obviously don't care.

According to the timeline of events in The Fine Article, this story only exists because someone cared. At many companies the story would end at "one diff reviewer noticed passwords getting logged in one diff". All of the numbers in this story come from an internal investigation to see where else they're making the same mistake, _so they stop doing that_. That's not what you do if you don't care.

If FB truly cared it would likely never have happened, let alone gone on for years. I'm not saying no employee at FB cares; I'm saying FB as an organization doesn't, and we have plenty of "I'm sorry, we'll do better" statements to back that up.

This feels like a slightly differently framed version of the "Too big to fail" argument and does pretty much nothing for me.

Collecting data on such massive scales is literally FB's whole business.

But with that also comes a responsibility that shouldn't simply be waved away with "But they are so big, it's so difficult!"

Because when it's about monetizing their massive amounts of, often illegally collected, data then FB seem to have no issues having everything in order and getting stuff to work, regardless of how "difficult" it might be.

Probably has to do with the fact that there's no money in protecting users data properly and FB seems to be pretty much immune from negative PR having any bad consequences.

>takes more than some occasional greps

Of course at FB scale you'd automate this by creating a set of canary accounts with unique passwords that you perform a search for in the ETL pipeline, or some other handy place. This will at least catch inadvertent plaintext password logging.

Facebook ops team: "We're waist-deep in dead canaries."

Only 100s of terabytes?

I work fairly extensively in health care technology and have since ~2005 and am not aware of the particular regulation that would cause the "FDA" to react to a password storage security gap. Could you be more specific about the regulations you're referring to?

Unsafe logging of PHI would be explicitly problematic, but passwords aren't PHI.

I never said the FDA was specifically interested in passwords. The FDA was brought up in reference to my work with PHI. The FDA does care about how you secure access to sensitive data however. If you leak passwords which allow unauthorized access to PHI, you're going to have a problem.

The analogy was intended to focus on practices which should be employed by companies which handle sensitive data, not the specifics of what the FDA is looking for.

These aren't "leaked" passwords; they're logged in plaintext internally, which is not great, but is not the same thing as having leaked them.

"leaked" in the sense that they're somewhere they shouldn't be. I could have chosen a better word. if I were to store database passwords on a file share the entire company had access to we would consider that a "leak" even though they weren't exposed externally.

HIPAA/HITECH would not necessarily; the analysis would have to include compensating controls and who had access. The fact pattern in this Facebook case would probably not support a violation. Again: crappy stored passwords aren't PHI.

I think perhaps I wasn't clear enough on the boundaries of my analogy and where it falls apart.

You work in the industry so you know that the FDA is not highly prescriptive. They give you a general outline of, essentially, process. HIPAA goes into more detail, but still, an organization has quite a bit of leeway as to how the meet regulatory requirements.

As far as passwords go, you're correct. Passwords in logs aren't a direct violation as far as I'm aware. If e.g. an employee gained access to data they shouldn't have access to per internal policy and used that data in an illegal manner, _then_ you have a problem.

Still though, I never meant to imply that what happened at FB would violate FDA regs. What I said was:

>I've worked in healthcare / biotech for more than a decade and I can promise you that the FDA would see no difference between the two types of gaffes.

The "two types" in question here were defined by the GP ("easy to make" mistakes like logging and something something about database design.) I wasn't referring to passwords specifically, and I don't believe that this is as bad as e.g. PHI just sitting around for anyone to see.

The analogy was meant to convey this; in a health care environment we are required to secure (encrypt) PHI and PII at rest. If we were found in violation of that, it wouldn't matter if it got there via logs or poor database design.

Again, I realize this is not an FDA/HIPAA situation. My experience with sensitive data is in that sort of environment, and I believe the same sort of mentality should be taken by FB in regards to the security and privacy of their users.

Wow, didn't expect that to get so long. My thumbs aren't made for this.

This is how professional software engineering and infrastructure implementation is done, and I appreciate you pointing it out so forcefully.

Corollary: Most web development simply lack the rigor of engineering.

Most software engineers aren’t. Engineering requires licensing and exams through a governing body. Can’t have the prestige of the title without the responsibility. Software developers want to have their cake and eat it too (power and respect with no oversight and governance).

Pile on the regulation (GDPR for starts, more PII protection to follow up, extremely painful fines for failures).

I couldn't agree more.

In the US, you only need certifications and testing for a "professional engineer". Most electrical, chemical, mechanical, or civil engineers are not certified PEs. Yet I don't think anyone is claiming they aren't engineers.

I have a PhD in chemical engineering, but not a PE, since I didn't go the industry route. I wouldn't feel comfortable calling myself an Engineer without it, and that was the attitude expressed by my professors and cohort coming out of undergrad.

It's my opinion that the title "Engineer" should be on par with things like "Doctor" or "Lawyer". It should require certification and there should be consequences for claiming to be one if you aren't.

I don't think anyone would get in trouble for job titles like "computer doctor" or "network protocol lawyer". Practicing medicine or practicing law (or in some cases, practicing as an engineer) without the required qualifications, regardless of job title ... different story.

> Most software engineers aren’t.

"Software Engineer" is just a job title.

> Engineering requires licensing and exams through a governing body.

"Software Engineering" doesn't, it's just a job title.

> Can’t have the prestige of the title without the responsibility.

There is no real-world prestige associated with having "Software Engineer" as a job title (as opposed to "Software Developer").

> Software developers want to have their cake and eat it too (power and respect with no oversight and governance).

"Power and respect" is not actually something that is commonly associated with a guy sitting at a desk for a salary.

> Pile on the regulation (GDPR for starts, more PII protection to follow up, extremely painful fine for failures).

You just learned that Facebook stored passwords in plaintext for years and nothing bad happened. In fact, it's hard to conceive that anything particularly disastrous could have happened. Yet, you call for more regulation that will drive up costs for everything. Whole industries could become unprofitable and those high-paid "Software Engineers" would become redundant.

I have the opposite opinion: The fact that computer security is generally poor teaches people how to properly use computers. They need to assume that there are no secrets, because that is the plain reality. No amount of regulation will make computers secure. No certification or best-practice encryption scheme will prevent users from writing their password on post-it notes, or from opening "sexy.jpg.exe", or from answering that call from the "Microsoft Service Center".

> "Software Engineer" is just a job title.

And like the job title "sandwich engineer", it should probably be derided.

Apologies to anybody working in the honorable trade of sandwich crafting. I love sandwiches and appreciate your work. They bring me much more enjoyment than software.

Same with electrical, chemical, mechanical, and civil engineers since almost none of them are PEs?

Seems a bit silly.

I agree. The job title is literally _engine_er. Unless you oversee the operation of engines (as in locomotives), you are not an engineer.

But seriously, the definition of engineer in regular lexicon is someone who designs, builds, or maintains complex systems. I struggle to think how software does not tick all of those boxes. Sandwiches are a different story.

Perhaps you are confusing engineer with professional engineer? I can see how it might be easy to mix them up if you are not paying attention, but professional engineer carries a different meaning. Most software engineers are not professional engineers but are unquestionably engineers.

It becomes a meaningless word if anyone can just call themselves one. There has to be some standard, some governing body, some degree of rigour applied here.

> It becomes a meaningless word if anyone can just call themselves one.

Only to the extent that all words are meaningless. However, we keep a reference known as a dictionary that helps maintain some consistency around meanings of words.

The Oxford Dictionary defines engineer as:

1. A person who designs, builds, or maintains engines, machines, or structures.

2. A person who controls an engine, especially on an aircraft or ship.

3. A skilful contriver or originator of something.

Is software an engine, machine, or structure? That is debatable, although I would suggest that it does meet the definition of machine. Under that suggestion, software engineer is a perfectly appropriate term. Designing, building, and maintaining is exactly what software engineers do.

If you disagree that software falls under the definition of engine, there is still that third definition. Is software created by a skilful contriver or originator of something? I think that is a definite yes.

> There has to be some standard, some governing body, some degree of rigour applied here.

Professional engineers are expected to display those things. That is unrelated to engineers. Different terms with different meanings. Software engineers may not be professional engineers, although they can be.

> 1. A person who [...] maintains engines, machines, or structures.

So we should call car mechanics "mechanical engineers"? And maintaining structures? With all due respect to the janitors out there, their job is vital to society but it's not engineering.

> So we should call car mechanics "mechanical engineers"?

By definition, I don't see why not. I suspect the spirit of maintain may be a little more nuanced, but words are fluid and if someone wants to interpret it that way, then it is so.

> With all due respect to the janitors out there, their job is vital to society but it's not engineering.

If by janitor you are thinking of someone who sweeps the floors, I think that is well beyond the spirit of maintain. That, to me, is cleaning.

Maintenance of a structure is more like addressing a beam that is cracking. I doubt that is a role that an average janitor deals with, and in many cases will legally require a professional engineer to spec the resolution.

But words are fluid, so if you interpret a janitor as someone who maintains a structure, then I guess engineer fits. It really makes no difference either way.

The word engine signifies a product of ingenuity. A siege engine is a clever device to attack fortifications. Engineers don't just have to be engaged with engines in the modern sense to do engineering. The PE bureaucracy is just gate-keeping by the guild masters.

Professional engineers do engineering.

Title only engineers work on complex systems.

This distinction has largely been lost and most "Software Engineers" think they are doing PE Engineering when they are not.

Do you think that if someone calls themselves a sandwich engineer people will begin to believe that designing cars is like makings subs? Do you really think the public can't tell the difference?

I think "Software Engineers" can't tell the difference.

A 'software engineer' need not even have a college degree, the only qualification for calling yourself a software engineer is finding an employer willing to give you that title. Even non-PE mechanical engineers at least have the bare minimum decency to get an undergraduate engineering degree before calling themselves engineers.

Of course, some people get software engineering degrees, or computer science degrees (that term is another can of worms entirely), but that is not the rule.

In Ontario the Professional Engineers (http://www.peo.on.ca/) crack down on unlicensed use of the "Engineer" word in job titles pretty aggressively.

The point is that the public should have confidence in engineers. Software developers who tout themselves as "software engineers" actively piggy-back upon, and undermine this confidence. It's malicious, and you should feel bad for defending the practice.

If you still don't see why, "Lawyer" and "Doctor" are "just job titles" but obviously that's a problem. I trust you can see that.

> In Ontario the Professional Engineers (http://www.peo.on.ca/) crack down on unlicensed use of the "Engineer" word in job titles pretty aggressively.

Yeah well, good for them I guess. I don't think anybody else cares.

> The point is that the public should have confidence in engineers.

Why? "Engineer" is a very broad term used in a variety of occupations for very different things. If you're looking at someone's credentials and you're so uninformed that you can't tell the difference between a licensed engineer in a particular profession and a guy that has "Engineer" written on their business card, you're not qualified to make a decision either way.

> If you still don't see why, "Lawyer" and "Doctor" are "just job titles" but obviously that's a problem. I trust you can see that.

I can see the argument for why a medical doctor or a lawyer should be afforded some amount of protection, because they are directly dealing with laymen. I don't buy the same argument for the word "Engineer", much less "Software Engineer". It just doesn't really mean anything. We already have degrees and certifications for when specifics matter.

I believe there are exceptions like the "Combat Engineer" in the Canadian Forces.

In Canada, the term engineer is protected, but in the states it is specialization dependent. I went to school for software engineering and I have my ring (I'm not a full engineer for a few more years though).

We focus a lot on testing and I just finished a mandatory ethics workshop. Its a different approach to software than I read about elsewhere. Almost no time spent on algorithms either - bad for interviews, but great for being able to write software that works though.

> Engineering requires licensing and exams through a governing body.

Offering engineering services to the public requires licensure. Working at places like Boeing, General Motors, Texas Instruments, and Caterpillar does not (for the vast majority of positions); It just requires an accredited engineering degree.

Note that this is independent of the software industry needing more engineering rigor, which it desperately does.

> Software developers want to have their cake and eat it too (power and respect with no oversight and governance).

This doesn't come from software developers.

This comes from the SV culture of "move fast and break things."

I painted too broad strokes. I agree, and admit my mistake.

Just replace "software developer" with "self-proclaimed software engineer" in your comment and you'd be spot-on.

Nothing about engineering really covers security best principles and practices.

I went through classical engineering school, and I'd summarize with three categories:

* Newtonian Physics & The Mathematics Behind It (i.e. Differential Equations)

* Ethics

* Technical Communications

That's because there is no software engineering program, and other engineering programs don't deal with "security" per se.

My alma mater offers a degree in Software Engineering.

I think that was actually invented at Facebook in the early days, unless I'm mistaken. Its a good mantra unless its security related

A lot of HNers probably have the job title "software engineer" without any of the qualifications of an "engineer"... that's probably why you're being down-voted.

But you're not wrong.

Worked in two data sensitive industries as well. There is no excuse, completely agree. This is amateur and completely indicative of a broken dev process

Reply on news.ycombinator.com