Hacker News Reader

Comment by mmanfrin | Hacker News Reader

Outside of the case, the most striking thing to me about this is the level of detail that Google has over the logs and actions of their laptop. They were able to tell that a memory card was plugged in for 8 hours a year ago?

I have trouble finding exactly how a customer encountered a 500 10 minutes ago.

View on HN

I was a Windows engineer for Google's Corp Eng PaaS SRE team and have familiarity with this topic.

It's not hard to get this data or set this data collection up on a Windows domain or machine. Anyone can set this up as long as you have the TBs to collect the data and the presentation/searching layer to find it.

Windows has the ability to log everything, including device installs/uninstalls, file opens/closes/creation/deletion, logons/logoffs, you name it. The Windows Auditing library is really, really, really extensive.

Additionally, setting this up in Active Directory is really easy, both manually and magically with Powershell. It takes about ten minutes or so.

As you would iamgine, most of these policies were enabled on the domain to which most users authenticated and the data it collected was siphoned off to essentially a giant cluster of syslog servers.

In fact, just about any domain will audit device plugs/unplugs. Had Anthony known about this (it's easy to find out even if you're not an admin), he would've not plugged in that memory stick :)

In general, it's pretty hard to do stealthy stuff on Google's network. Everything is logged eight ways to Sunday, especially with GAIA and key-based, two factor auth to EVERYTHING. And unlike most other networks, I wouldn't put it past their security engineers to find shady behavior in a moment's notice.

I don't think GAIA has been publicly commented on by the company, in case that matters to you at all. Could be wrong, though, as it seems the term is present in the deposition and un-redacted.

I mean it's in some public docs, e.g. here: https://www.google.com/support/enterprise/static/gsa/docs/ad...

FWIW Google seems to refer to the GAIA ID in stuff like random error messages, documentation for Google Apps, and in data downloaded from Google Takeout. If it's not supposed to be public information then Google isn't doing a very good job.

I've seen references to GAIA online. Also, the deposition makes mention of it, so I think it's okay to mention here

Target Disk Mode?

Yeah I really enjoyed the technical details in Gary Brown's deposition: https://www.documentcloud.org/documents/3515476-Declaration-....

It looks like they could see:

* When the device made check-ins to their management server, and appeared on the Google network

* That the Windows OS was re-installed

* MoMA search queries (some kind of internal system?) for "chauffeur svn login"

* Downloading and installing TortoiseSVN

* Transferring files from SVN

* When a card reader was plugged in

* When the OS was reformatted to Linux

* File downloads from Google drive

Just think of "MoMA" as "the intranet," which is about the best way to explain it. Not super interesting.

Weird that he'd allegedly type "I've never logged into Subversion before, but suddenly I need access" into MoMA, but hey. The whole saga just reminds me that people are really bad at opsec.

The Google Drive/GAIA linkage is the most interesting technical revelation from a quick read. I'm surprised they didn't redact that little trick, because that seems to be the best way to catch the very people doing this. Most people would be forgiven for assuming that a Google consumer product might be a little more obfuscated in terms of forensics (logs access, PII, etc), and I could see that train of thought even though it's dumb. I guess they're willing to burn the technique, because the next person who attempts this will definitely avoid Google Drive.

I now read that deposition in full, this guy is dead and buried six feet under.

If you send your kernel logs to a syslog server you could get that pretty easily. From plugging a USB stick into my computer just now, then running dmesg:

    [14863.014481] usb-storage 2-4:1.0: USB Mass Storage device detected
    ...
    [15045.488440] usb 2-4: USB disconnect, device number 34

They do, though, have quite a bit of pretty impressive logging!

If I read this correctly at this point the machine was running Windows. I have zero doubts, however, that Windows has equivalent impressive logging, apparently the USB vendor and device ID pair was recorded as well.

Absolutely. The Windows Event Viewer logs heaps of stuff, and has heaps more you can enable. When I'm particularly lazy and forget to do my timesheet, I use it to look when I booted/shut/locked/unlocked down my work PC to get an idea of how long I worked.

A cursory google search shows this for USB events: https://dfstream.blogspot.com.au/2014/01/the-windows-7-event...

You're talking about a company whose mission is "to organize the world's information". If you're an employee thinking about wrong-doing, you've gotta have that in the back if your mind.

They have some pretty cool stuff for monitoring their internal systems. For example, this distributed live forensics system:

GRR Rapid Response https://github.com/google/grr http://grr-response.blogspot.com/

Funny note: these screenshots show an AWS instance :)

https://github.com/google/grr/blob/gh-pages/screenshots/Scre...

maybe simply to keep keep all forensic operations as separated from any google operations as possible?

Google has been the target of infiltration by staet actors. At least once we know for certain, with the incident in China, and presumably other times.

Mountable device tracking is very common to log. My company started implementing when we went from 20 to 30 employees. When things like this are at stakes, it's a small price to pay. In fact, it is so well known that this kind of tracking exists that very few people actually do try and actually do insert mountable devices in their computers, which makes the tracking even easier.

Does anyone know if this information was collected as part of a monitoring service or if it was collected by forensically recovering the data from the laptops hard disk?

If the laptop was re-imaged to Windows, I doubt much of the syslogs from linux were left.

Especially since windows is much bigger than a default linux install these days, and this was a rarely used laptop, it seems unlikley any linux was left, even in free sectors.

No, it was running Windows and was reimaged to Linux. It is very strongly implied this was done to cover tracks as it was never switched on again.

This level of logging is easy to come by forensically, even on a stock windows install - except the year time window, probably need to archive logs for that

How many thousands of hours of video is uploaded to youtube _daily_ and stored on google servers forever?

And you are talking about logging actions of your own employees. Let's say 100kb per day per employee (these logs are pretty detailed).

100kb * 50 000 employees = 4.7Gb per day. 1740Gb per year. Probably much, much less then 1 day of youtube videos.

It's obviously not the volume of data that is surprising. It's that they retained system message logs from a random laptop for years. They must regard this as potentially important business information. Does your company do this? Mine sure doesn't.

Data is useful for large companies - especially when you want to make data-driven decisions.

- Can we stop dispensing USB cables in the vending machines?

- Should we deploy the new Macbook Pro without classic USB ports as the new standard?

- Do our conference rooms really still need USB presentation remotes? Does anyone actually use them?

Turns out it was important business information after all.

For a company at Google's scale, figuring out how to cull even a couple TB a year may cost much more in salary than the money saved. Also, it's occasionally really really useful.

What's interesting is that I can see these logs as being a liability as well as an asset; there's a reason most business records retention policies are about getting rid of records, not keeping them. From that perspective it's surprisingly they'd reveal the level of logging in play.

It depends. If you have the ability to use the data, it's a real advantage. It's a flood of data that most folks can't handle.

Reply on news.ycombinator.com