Hacker News Reader

Comment by nostrademons | Hacker News Reader

"Software engineers on an on-call rotation...data analysts, maybe even some external contractors"

These groups usually get access through a bastion that anonymizes data and logs access. I remember that as a SWE at Google, I could run aggregated & anonymized statistics across query logs, but some info (eg. IPs, user logins) had been scrubbed before any of my code could get access to it, and for things that were more personal (eg. your GMail login) you could only get access to your own account.

There's nothing you can do about SREs who have root access on the box or the SWEs who need to implement & maintain the bastion servers, but that's presumably a more restricted, vetted, and trusted group.

View on HN

> usually get access through a bastion that anonymizes data and logs access

I think this may be something that is unique to a certain size tech company that simply isn't the case at 99.99% of companies with user data in their possession.

It should be the case at Twitter though.

It hasn't been until recently. We are currently locking down all internal systems based on the minimum necessary accesses for various systems. We should have done this a lot sooner.

Good to hear it. Better late than never.

Even Google's policies aren't—or at least weren't—foolproof. In 2010 there was a major incident involving a Google SRE named David Barksdale: https://www.telegraph.co.uk/technology/google/8003925/Google...

Yeah, I was there when it happened (and it was a fairly big deal internally). He was an SRE with pretty high-level access. There's relatively little you can do for this kind of threat - the nature of their jobs usually requires that they have root on the box. (Though a sibling comment suggests that there's even more rigorous procedures now.)

The centralization of infrastructure has really changed the insider risk profile at that company. A decade ago it was common for a service to own their hardware and to have broad authority over those machines but not so much these days. Nobody wants to own their hardware, everything is uniform and it's easier to spot deviants that way.

Of course many of the changes around that time were in response to the breach by the Chinese government, not only in response to embarrassing privacy incidents. Later improvements came about because of things Snowden published.

Why would Google change because of the Snowden Leaks? They knew about it. They were handing over data to the NSA.

There was some spying by the NSA without their knowledge or consent. Network traffic between their internal networks wasn't encrypted, which enabled snooping.

https://www.washingtonpost.com/world/national-security/nsa-i...

IIRC they didn't know about the tap on their international cable and the leaks caused them to prioritize the encryption of traffic on their internal network. Not sure about anything else though.

Its the same for telcos, these days there is much stricter auditing and actual real security vetting - I know team leaders on some systems had to be PV (top secret in US terms)

I suspect that at some stage this might come for some FANG employees

Would it be smart for companies like google and twitter to publish how they vet people for these roles? That’s a seriously tremendous amount of power. I almost think it needs to be regulated like heath info. I get that that would make it more expensive to handle this data at all, and would serve as a significant barrier to entry for startups ... but data breach after data breach is pushing me in the regulation direction.

Pretty much nobody at Google has that kind of access. You can be an SRE of just about anything at Google and never access user data. The "break-glass" means of emergency access is ridiculously booby-trapped. A person wanting to do this thing has to 1) badge into a special room, at which time both production security and privacy incident teams are notified, 2) use a special hardware security device that is used for no other purpose than to activate a VPN box with a hard-line into the production network. By the way if a random Googler just rolls up to a datacenter without a reason to be there, that also triggers privacy incident response, even though physical access to production storage is virtually useless due to all the encryption.

I would say it is much more likely that Google will accidentally lose the organizational ability to become root-in-prod, than it is that a person has done this thing without being noticed.

In short, insider risk cannot be mitigated with hiring practices. You need robust technical measures against insider risk.

Thanks for proving my point, I guess? I’d love to see a formal write up from google about these procedures. It would go a long way to increasing confidence in how they handle this data.

What I would love to see is somebody at Facebook comparing their barriers against accessing user data with these from Google.

I’m actually inclined to think they have similar procedures, if only because we haven’t seen “whistle blower” stories in the news about folks reading texts and looking at other private user data. Maybe I’ve missed them, but because bashing Facebook is kind of a trend one would think there’d be an appetite for “I read illicit group texts for a year here’s what I saw AMA” stories.

If this happens, every incentive (of every party that is aware) is to not whistleblow, as it is a criminal offence for the snooper, and a PR disaster for the company.

When did that start? Here's an account of someone working at Google accessing info to stalk teens.

https://www.businessinsider.com/google-engineer-stalked-teen...

That story dates from 2010.

Snowden's revelations (2013) were a major watershed. There'd been several measures taken since, based on what I heard on the outside, largely through discussions, mostly public, a few direct, with Google staff via G+.

Starting on, of all days, November 9th, 2016, I began regularly posting an image of Jewish shop windows shattered during Krystallnacht, asking whether Google were thinking of brownshirt-proofing their data. That generated responses including from G+'s architect (then in a role with user data safety & privicy), and the data security lead.

It wasn't until some time later that I realised I'd entirely accidentally picked the anniversary of the event for the post. Though the coincidence was useful.

My understanding was that numerous protections were in place by that time. I continue to have concerns.

> I began regularly posting an image of Jewish shop windows shattered during Krystallnacht

This is so incredibly cringey. You're actively building the panopticon and yet you think of yourselves as righteous warriors for justice.

I don't mean this to be a personal attack, but yours is such a revealing comment about the mindset of people inside these surveillance behemoths.

(See also: this "pledge" http://neveragain.tech/ to not build registries for targeting citizens...signed by a bunch of people who work at companies whose entire business is targeting citizens with ads)

> Pretty much nobody at Google has that kind of access.

Sounds like you'd be surprised at what storage, and backup engineers have access to.

Tremendous amounts of ciphertext?

Because encryption keys are never backed up by the same system either?

> Pretty much nobody at Google has that kind of access.

OK, Google.

Even if you managed to get direct access to production through the special room's direct link, you'd still need a special kind of credentials to send RPCs to any service.

There was a video with some of the datacenter security measures, e.g. iris scanning (just to get yours in the DB required approvals from senior people). On the actual floor, to which very few have actual access, you need to badge both on your way in and out, individually. If you badge out without having badged in, the door won't open and an alarm will go off.

> Would it be smart for companies like google and twitter to publish how they vet people for these roles?

I think much like anti-hacking and anti-fraud efforts, publishing information about how they vet candidates would just make it easier for attackers to figure out how to game the system.

Ironically, one of the vetting regimes that has the most publicly available information about it is the US government's security clearance system. https://ogc.osd.mil/doha/isp.html

one of the vetting regimes that has the most publicly available information about it is the US government's

Why wouldn’t the information be public? It’s the same concept as crypto algos being published and peer-reviewed.

The same way defences industries do I would imagine for SF86 etc

Reply on news.ycombinator.com