Hacker News Reader

Comment by lhm | Hacker News Reader

I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?

View on HN

Yes, it's more complicated than just SSH'ing into the server.

But on a compromised machine you can't trust anybody, not even the kernel. Assuming the worst, the attacker could have gained root privileges and modified the kernel or the base tools like ls and grep. You also can't trust the log files if they're not stored off-site. The modified kernel or ls could hide the attacker's traces from you.

Thus, the only possibility to really make sure nothing is hidden from you is to (power off the machine and) attach its hard disks to a trusted computer where they're mounted and investigated.

Live forensics are preferred to just reading disk; lots of problems with that and solutions to those problems on various types of machine.

Could you please elaborate on that? How do you do "trusted" live forensics on systems with possibly infected kernels and stuff? Assuming these servers were normal COTS and nothing fancy (thinking of CPU-bypassing memory access...)

DMA memory out using FireWire (if available)? That would be my approach!

Firewire is awesome for the attacker, unfortunately few servers have it, especially not externally exposed ports. Also, smart OSes use some of the newer Intel features (VT-d) to lock down DMA while the OS is running, which usually protects from rogue firewire, and can theoretically help against rogue PCIe, although usually badly implemented in chipset and OS.

Another option is a reboot onto a custom OS which is designed specifically to preserve memory (you get a safe few seconds of holdover). LiveKd is pretty cool (sysinternals)

There are PCIe cards which do processor/network and let you explore main memory -- WindowsSCOPE CaptureGUARD for PCIe or ExpressCard. Probably enough time to pop the case open and throw one in before memory degrades.

Countermeasures are numerous -- everything from doing memory encryption inside the CPU die (putting code in the cache, like TRESOR) and doing hypervisor tricks ("TresorVisor") (http://www1.informatik.uni-erlangen.de/tresor) to using Hardware Security Modules (like the SafeNet or Thales nCipher) to just keeping your servers physically secured from intruders who might memory-analyze them (although a software bootloader and remote-reboot could still be applied).

Forensics as a field seems to be a lot more interested in attacking mobile phones (which is one of the things I'm talking about at RSA 2013), but desktops and servers are still interesting targets.

On the contrary -- you have to at least admit that if you disconnect a compromised machine, the attacker could have installed a script that detects that the machine is disconnected and erases evidence.

If you power off, you can always mount the hard drive read-only and do a forensic analysis.

On the contrary - you have to at least admit that if you power off a compromised machine, the attacker could have installed all his code in RAM only so that powering off erases evidence.

If you disconnect, you can still try to examine the current memory content and do a forensic analysis.

Of course what you really want is a memory dump via a trusted channel while the CPU is halted (hardware hypervisor or something like that) and then immediately power down. This is usually not supported on COTS hardware, so you have to choose the strategy that will erase the least evidence (power off, disconnect, suspend to disk, VM snapshot, whatever) depending on what you suspect the attack to be.

If you don't have the resources to take a snapshot of memory without disturbing the operation, then power off is the right thing to do.

For all you know, the trojan might uninstall itself or otherwise delete all evidence after a short while without network connectivity to a control center ("a dead man's switch").

Or, it patches the kernel so that it is "invisible" from the inside.

What you want to do is take a snapshot of the memory (from outside the kernel, if you can) and then power down.

Reply on news.ycombinator.com