Hacker News Reader

Comment by emtel | Hacker News Reader

I worked at Facebook for most of 2017 and 2018. In the first week, they made it clear that you would be fired instantly for any improper access of user data.

They further said that if you need to access any sensitive personal data, or if you need to log in as a user in order to debug a problem, you need to have approval from your manager _before_ the access, not after.

Also, you are not allowed to access the data of anyone you know personally for any reason whatsoever. You have to find someone else to do that if it needs to be done.

Finally, they really do audit every single access of personal data. I had every reason to believe that if I accessed any data improperly, I would be fired within the week if not the day.

I don’t know how much abuse still exists despite all of the above, but I don’t think this article does a good job of explaining how seriously Facebook takes this.

View on HN

>They further said that if you need to access any sensitive personal data, or if you need to log in as a user in order to debug a problem, you need to have approval from your manager _before_ the access, not after.

But were you still able to just look at the data or login as the user without the permission? I think that's the key question.

Talk is cheap. As a user it's not good enough for me that people are being told internally not to abuse their access. Just remove the permissions from the employees and make them request the permissions for each individual case instead of trusting the employees to follow the rules.

Disclaimer: was at FB in 2014

You could at the time start trying to log in as a user and MULTIPLE red warnings came up that proceeding further would automatically notify your manager and skip of access and a reminder of data policies. Now at that point I did not go further but I did know that content moderation and security teams had special access so I imagine they did both, heavily warn avg FB eng AND restrict access.

How about people with direct database access?

I am close with some people who worked there until recently. All data access is audited; production access is limited via ACLs in both the main data storage system as well as all the others like the warehouse, realtime ingestion, etc.

FB appears to take this extremely seriously. I just pinged my friends and they said the only way people get fired is for sexual harassment or improper data access. And the second is the one that gets audited and monitored every day.

I imagine at Facebook's scale that nobody has direct access to individual database or application instances; and that if someone actually needed to run queries of any kind in production, it'd be as stringent as deploying a code change.

Pretty strict. You don't get direct db access unless in a very specific team/role. You have to request access to tables on a per-table basis.

I believe this is similar to how Google does it.

I believe they dont allow you to access peoples public profile while at work.

I am not sure I agree with this framing. Equivalent framing: separate the genders since rape is possible, even if rape is punishable by death. This is not just talk - FB does have capital punishment for abuse of access, which creates REALLY strong incentives. Of course, this isn’t fool proof.

> Also, you are not allowed to access the data of anyone you know personally for any reason whatsoever.

Which explicitly also includes yourself, because looking yourself up would e.g. let you see who has you blocked.

You're also fairly unlikely to access personal data by accident. You have to explictly go look for it in the internal tooling, which has pretty good signage around interfaces that could potentially expose you to personal data by accident so you know to be careful (I did a couple of tickets for the abuse team and testing that stuff was riddled with interstitials asking if I was sure I wanted to access personal data). "Oops I didn't notice" just doesn't fly.

They're also fairly good at removing the semi-legitimate reasons you'd have for accessing personal data. If you have friends or family that are having some sort of issue, they have a separate priority queue you can submit requests to so they'll look into those issues for you, for example. If you need test data, there's great tools to generate test users with all sorts of weird configurations (so you don't have to rely on finding a live one that meets your criteria)

I'm surprised that this stuff is audit only. At my company, at least in the past five years or so, this type of access has been forbidden to almost all employees. You need to request access to these types of systems and provide justification for why you should have it. Access is controlled on a per-system basis -- it's not blanket access. Many of the most sensitive systems have auto-expiring access for humans.

Nowadays we are seeing many systems switch to a regime where you have to get another engineer to sign off on any access to production, and your access is limited to at most 24h. This isn't merely a policy -- it is enforced by technical controls that forbid ordinary human-user access to production. I literally cannot even send an RPC to services I work with that handle private data without getting a colleague to sign off on it.

> I'm surprised that this stuff is audit only

These days things are mostly working the way you describe - I need to request permission to view my own service’s logs, and I’m working in backend infra not going anywhere near user data (logs are like “did we hit any hardware errors when trying to install the OS on this host?”)

Great to hear. I think a lot of big internet companies are moving in this direction, although I don't know if it affects user trust all that much since it isn't publicized. I guess the one thing is that incidents like the one reported here will be avoided in the future, so in ten or twenty years there will have been less reputational damage than in an alternative universe where these controls don't exist.

How “sensitive” is facebook user data though? All content in a facebook account is already visible to an average of >100 people - their facebook friends.

(Messenger had stronger protections than OP is describing)

Extremely sensitive, by the lights of the organization I work for. The people to whom FB user data is visible are known to the user. Those people have been explicitly authorized by the user to view that data. FB is acting as the user's agent in conveying that data only to authorized recipients, who the user presumably trusts to some degree or another to not further propagate the data. The data is generally not publicly visible, and FB employees are generally not among the list of entities the user intends to convey the data to.

Meanwhile the CEO volunteered early on to dox people at Harvard pretty much for funsies[0].

Yet TFA contains a quote about how abusing personal data is "against Mark's DNA". Horseshit.

Facebook is the enemy.

[0] https://www.esquire.com/uk/latest-news/a19490586/mark-zucker...

While sure I think it’s wise to stay weary of any company you give your data to, Mark said that when he was 19 and Facebook was limited to students. I think it’s disingenuous to use a quote from 2004 to represent his thinking today.

Didn't it start as a hotornot clone using girls pictures without their approval?

Never understood why anyone would trust this guy if that was the case. Pervs are some of the most reliably untrustables on the planet.

They had this rule at America Online when I worked there early 2000s. It was routinely violated by the managers, and was really only in place for the rank and file to cover their butts. I just assume bad management and executives of Facebook routinely violate peoples privacy by digging through their information, it’s there, and Facebook hasn’t exactly shown an interest in protecting privacy.

The big problem with a company as large as Facebook, is it's easy for the reality on the ground and the statements executives make to differ greatly: The company policy may be as stated, but there may be line employees and managers who have no issue with abuse of personal data, and even cover for each other.

The idea that people can just go in and access personal data at Facebook without some sort of actual pre-authorization is insane.

Those policies will only catch someone after the fact. Firing someone is the bare minimum, it prevents a single repeat offender, but they could already do damage.

None of this should even be possible.

Agreed, I was there at the same time, and was taken pretty seriously, and grew progessively more locked down as time went on.

A friend of mine worked at a large bank in customer service and this was also a big part of their training, and there was even a speech trainees were given before going to their desk at the end of training. He said, almost invariably, that at least one person from every class was fired within hours for looking up the accounts of someone they knew or a celebrity.

> almost invariably, that at least one person from every class was fired within hours for looking up the accounts of someone they knew or a celebrity.

While I don't doubt people do this for real, staging something like that might actually be pretty effective.

There's a certain trend in most companies that every bureaucratic rule can be traced back to a specific event where someone caused a problem by doing what the rule was written to forbid - so it's possible that you were indirectly told about four incidents.

If you know the right people, can you be taken off the audit list? I remember in the early days of Facebook, access to everyone’s account was seen as an unofficial perk of the job; the cynic in me would say that this perk still exists but is only given to people who can be trusted to never talk about it.

Just firing is not enough for the cases of personal data abuse. What I would like to see is those employees being reported by Facebook to the authorities to be further legally prosecuted.

We should not rely on the goodwill or internal guidelines of a single company in such a sensitive topic.

So did people actually get fired over this? Or do you have any reason to actually believe that they would have noticed?

Yes. There was a public blog post a few year ago from someone complaining about being fired for this.

There’s a difference between having an audit trail and actually using it. I would be interested to know how often Facebook analyzes this data and actually fires people for improper usage.

> I would be interested to know how often Facebook analyzes this data and actually fires people for improper usage.

From the article: Facebook fired 52 people from 2014 to August 2015 for abusing access to user data

The article does mention more than 50 people being fired for it between 2014 and Aug 2015

I don’t have proof, but we were told that every access of sensitive data was actively audited. It’s very rare to need to do this for your job, so I don’t imagine it’s a huge volume of events to be audited.

The issue is that this is even a possibility. It should not be possible to access user data, even if a manager approves it.

These comment's are all relatively ignorant of the fact that implementing these sorts of privacy controls generally makes your product worse and your engineers miserable.

> Facebook employees were granted user data access in order to “cut away the red tape that slowed down engineers,” the book says.

If we can take a step back, this is a totally reasonable policy. Unfortunately Facebook is facing the reality of the law of large numbers in that once you have 1000+ people the chances of having a bad actor in your system is much higher than 10 people.

Maybe this is a hot take, but I for one prefer that my company trusts me to do the right thing rather than make it hard to do my job. I'm not saying that there isn't a solution for this, but behind the "facebook corporation" there is generally just a bunch of engineers that want to do a good job at work.

This is completely unethical and unreasonable. It's like arguing that police don't need more accountability because it makes it harder for them to do their jobs, and most of them aren't bad people, so who cares about a few bad apples?

Yeah it sucks, but it's part of the job. Start thinking about the people you're supposedly serving instead of yourself first. I'm pretty sure that the overwhelming majority of facebook users want to hear about tighter privacy protections at facebook, not fewer.

They are well audited already. Does every step possible need to be taken to ensure that no data can be leaked ever? No.

You can walk out your door right now and hop on a bus. That driver has a CDL, a good first step. But how do we know that the driver isn't drunk? Through threat of possible audit (breathalyzer) after any incident. We don't test them before handing them the keys every day.

We trust people all the time with things far more critical than a facebook user's data, and we audit them far more loosely, if at all.

"completely unethical and unreasonable" > This seems to be influenced by the belief that tech is some utopia where everything is solvable and the world will be a better place. There is room for good enough in trust.

There is a big difference between throwing guardrails up so people don't do wrong and beating them down with requests for permission over and over all day during their work, driving home the point they can't be trusted. Eight hours a day of being told you can't be trusted is about more than the worker's convenience -- it's about their morale at least and possibly their mental health. It also instills the attitude of "if I can do it, it's legal, because otherwise they would have stopped me from doing it."

This isn't beating anyone up. In any mature development environment, you should almost never have to touch production. When writing new features, you should be running against a test environment without real user data. When investigating and trying to repro bugs, you should be trying against a test environment. If the repro is tricky, the errors returned should have enough information to not need to access prod. And so on and so on.

I work at a competitor to Facebook in a user-facing service and have these kind of restrictions in place (must request access with justification, otherwise I literally don't have ACLs to see the data). It's a non-issue because I run into it at most 1-2 times a month, usually far less.

> Does every step possible need to be taken to ensure that no data can be leaked ever? No.

Which person you're replying to demanded perfect security?

> This seems to be influenced by the belief that tech is some utopia where everything is solvable and the world will be a better place.

I am not the person you're replying to, but the claim has nothing to do with utopianism. It has to do with the claim that reasonable safeguards and auditing when dealing with sensitive data is possible, so that users can have (some degree of) confidence in the operation while workers go about their authorized jobs. This is hardly rocket surgery. Or novel.

What some people seem to be taking issue with is that their company might not trust them as much as they think they should be trusted. My advice to them would be to stay in small companies - if you're below the Dunbar number, you can personally evaluate each other and develop trust that way. In larger orgs, you need policy and enforcement, it is just how people are wired.

If we take your argument further all engineers should be given the root password to all production servers and we should simply trust them (and keep logs) to not use the password?

Access control is something so central to IT systems that I'm frankly dumbstruck that someone would argue against them on HN.

If you don't need access, you shouldn't have it.

If you do need it to do your job, you shouldn't have to run to your manager several times a day to make a request to do it. You should have root or whatever is necessary and it can be audited.

I'm not arguing against access control. I'm arguing for those with responsibility to work to be given the commensurate authority to do their work -- with auditing even.

Who are these hypothetical employees who need to access customer data multiple times per day? If there are more than 5 of them in your O(1000) organization you have serious issues.

For 99.9% of employees, accessing customer data should absolutely be a "talk to your manager" level of occurrence, and each time it happens the manager should ask why it was necessary and what logging you need to add such that you don't need to do it again.

> Maybe this is a hot take, but I for one prefer that my company trusts me to do the right thing rather than make it hard to do my job.

Yes and banks shouldn't lock their vaults or safe deposit boxes because and just trust that all of their employees just want to do their jobs.

Let me be clear here, I'm not advocating for ZERO access control or audit logs here!

Let's take that bank argument, I'm definitely not advocating for not locking vaults or safe deposit boxes. But somebody has access to those, and when they need access they have a process for getting to it. Frankly, it definitely can be abused and banks wouldn't know for the better until after the abuse when the employee would be terminated and taken to court.

That's because they have audit logs in place. And the reason why its part of your contract as an engineer not to abuse your access to customer data.

I think the larger point I'm trying to make here is that its really, really hard to build a system that prevents any type of abuse of data. Now I'm not saying that we shouldn't strive for systems that make it hard to abuse customers data, but bad actors have ways of beating these systems and I have some empathy for a policy that places trust in employees (who need access, by definition of their job!) to not abuse it.

Anyways, these are all good comments made in response to mine. I agree with them!

[EDIT] Okay - I see the incoming point about them not needing access to that data for their job, that's a fair point. But I think most of us have been at a point in our careers where knowing the piece of information about a user that might have gotten them into a certain state is occasionally a valuable debugging tool.

> a point in our careers where knowing the piece of information about a user that might have gotten them into a certain state is occasionally a valuable debugging tool.

Exactly - that's the problem with the mentality in tech right now. Just because something might make your life easier doesn't mean you get to have it. Trading user privacy is not ok because it's "occasionally a valuable debugging tool". That is exactly the problem.

Too many people in tech companies show no responsibility for the data they have privilege to - and treat privacy as absolutely minimal.

You're a parent, you wonder what's going on in your daughter's life and she isn't talking to you about it. You don't then get to break into her diary and read it all because it might give you a tip on being a better parent to her. Yes if she's for example suicidal and there is an urgent situation where reading it might help save her, then of course access it. But corporations don't get to toss aside privacy just because "it might occasionally be a valuable debugging tool".

Write a debug helper tool to clone all of a users state with lorem ipsum.

I don't need to know anything about user123 outside of the fact that they are located in Perth, Australia.

Nothing else matters to me for this, I don't have access to any PII data like email or device (look, I know user ID's can technically be considered PII depending on which infosec person you're talking too).

Is this still a problem?

I understand your metaphors but without knowing that user123, who created a ticket in our system, is in Perth Australia (which for some reason that locality in my own metaphorical example is having issues processing payments) how we're supposed to resolve this.

Maybe I'm just hopelessly optimistic that people aren't as awful as we want them to be, or naive.

Reply on news.ycombinator.com