Security Incident on FreeBSD Infrastructure | Hacker News Reader

Security Incident on FreeBSD Infrastructure

freebsd.org

126 pointsdous34 comments

This is how you tell people about a security breach. Inform them soon as you know with what you know and assume the worst with your appraoch to restoring things.

Much respect and defineing the word professional for many.

I have a couple of FreeBSD machines which pulled binary packages between those dates. I'm not overly worried. The packages have been removed and installed again from ports after a fresh portsnap dump and the systems have been verified with "freebsd-update IDS" against known good signatures. Any modified files were manually checked. I use MAC on each machine and pf up front on firewalls so I know what is going in and out as well.

The fact that these mechanisms are available is the reason I use such a system.

Also, if you consider any problems like this happening to a closed source vendor, you may never know it's happened. And don't tell me they don't do it as I've worked for a couple of companies that felt that burying security fuck ups was acceptable practice. It's why I don't work for them any more.

Actually, even with opensource you may never know it happened. Does not matter where it comes from:

- They have to find it out first.

- Then they've to be willing to disclose the incident

- Even if, you still trust the source of the packages, the developers, etc. There's a zillion bugs that look like "just an error" which can also be "just a backdoor".

For these reasons, running mac, checking modified files, etc is ALWAYS good practice (that you seem to follow, don't get me wrong - but that's pretty rare)

I only follow them after I got owned in '97 on the end of a dialup for running an open telnet server (on FreeBSD) with a crappy root password :)

Nobody takes security seriously until they've learned to distrust a computer they know intimately. It's a fact of life.

Take note:

"We unfortunately cannot guarantee the integrity of any packages available for installation between 19th September 2012 and 11th November 2012, or of any ports compiled from trees obtained via any means other than through svn.freebsd.org or one of its mirrors. Although we have no evidence to suggest any tampering took place and believe such interference is unlikely, we have to recommend you consider reinstalling any machine from scratch, using trusted sources."

I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?

On the contrary -- you have to at least admit that if you disconnect a compromised machine, the attacker could have installed a script that detects that the machine is disconnected and erases evidence.

If you power off, you can always mount the hard drive read-only and do a forensic analysis.

On the contrary - you have to at least admit that if you power off a compromised machine, the attacker could have installed all his code in RAM only so that powering off erases evidence.

If you disconnect, you can still try to examine the current memory content and do a forensic analysis.

Of course what you really want is a memory dump via a trusted channel while the CPU is halted (hardware hypervisor or something like that) and then immediately power down. This is usually not supported on COTS hardware, so you have to choose the strategy that will erase the least evidence (power off, disconnect, suspend to disk, VM snapshot, whatever) depending on what you suspect the attack to be.

Yes, it's more complicated than just SSH'ing into the server.

But on a compromised machine you can't trust anybody, not even the kernel. Assuming the worst, the attacker could have gained root privileges and modified the kernel or the base tools like ls and grep. You also can't trust the log files if they're not stored off-site. The modified kernel or ls could hide the attacker's traces from you.

Thus, the only possibility to really make sure nothing is hidden from you is to (power off the machine and) attach its hard disks to a trusted computer where they're mounted and investigated.

Live forensics are preferred to just reading disk; lots of problems with that and solutions to those problems on various types of machine.

Could you please elaborate on that? How do you do "trusted" live forensics on systems with possibly infected kernels and stuff? Assuming these servers were normal COTS and nothing fancy (thinking of CPU-bypassing memory access...)

DMA memory out using FireWire (if available)? That would be my approach!

Firewire is awesome for the attacker, unfortunately few servers have it, especially not externally exposed ports. Also, smart OSes use some of the newer Intel features (VT-d) to lock down DMA while the OS is running, which usually protects from rogue firewire, and can theoretically help against rogue PCIe, although usually badly implemented in chipset and OS.

Another option is a reboot onto a custom OS which is designed specifically to preserve memory (you get a safe few seconds of holdover). LiveKd is pretty cool (sysinternals)

There are PCIe cards which do processor/network and let you explore main memory -- WindowsSCOPE CaptureGUARD for PCIe or ExpressCard. Probably enough time to pop the case open and throw one in before memory degrades.

Countermeasures are numerous -- everything from doing memory encryption inside the CPU die (putting code in the cache, like TRESOR) and doing hypervisor tricks ("TresorVisor") (http://www1.informatik.uni-erlangen.de/tresor) to using Hardware Security Modules (like the SafeNet or Thales nCipher) to just keeping your servers physically secured from intruders who might memory-analyze them (although a software bootloader and remote-reboot could still be applied).

Forensics as a field seems to be a lot more interested in attacking mobile phones (which is one of the things I'm talking about at RSA 2013), but desktops and servers are still interesting targets.

If you don't have the resources to take a snapshot of memory without disturbing the operation, then power off is the right thing to do.

For all you know, the trojan might uninstall itself or otherwise delete all evidence after a short while without network connectivity to a control center ("a dead man's switch").

Or, it patches the kernel so that it is "invisible" from the inside.

What you want to do is take a snapshot of the memory (from outside the kernel, if you can) and then power down.

FreeBSD reports are always extremely professional, I love it.

Interesting choice that some machines will not be reinstalled, only "thoroughly audited".

There are some systems (generally speaking, ones which were installed in the past few weeks) for which we know exactly what files should be installed and what their SHA256 hashes are. Thoroughly audited means "every single bit is correct".

Thanks for clarifying.

I assume if they had root though, they could theoretically install a rootkit in the MBR. Did you SHA256 verify the MBR too? :)

I didn't (I'm not involved in cluster management) but I suspect that it was done by someone.

I'm surprised they weren't more transparent about that reasoning... maybe those machines were running everything within jails?

They're probably one of the few projects with such a track record that we can take them at their word. Thoroughly audited will mean exactly that.

They're most definitely not amateurs, see the back catalogue for examples of 'how to do it right'

Excuse the naive question, but how does one detect intrusion when using bi-key authentication ?

How does one know the offsite repository's are clean if svnsync runs at set intervals? What if part of the attack was to make it look like happened at much late date/time after the malicious code was mirrored and backed up to the offsite repositories?

Scary stuff.

Please use passwords for your keys and allow key access only to a small set of known IP addresses.

Also do share other security techniques you're using besides the ones above.

Passwords are useful, but impossible to enforce. If anyone decides not to use the passwords for their key you are in the same spot. You need better tools, something you can enforce server side. We use two-factor: (note that I founded Authy) http://blog.authy.com/two-factor-ssh-in-thirty-seconds

End-user question: any word on how this affects my pc-bsd laptop recenttly updated to 9.1 RC-2?

They use SVN still?

FreeBSD only finished switching from CVS to SVN for the core repositories a few months ago. (And there's still some legacy systems which rely on CVS.)

What's wrong with svn?

Nothing, if you don't mind using a centralized development model.

The BSD's tend to be ultra-conservative in regard to version control systems - for example OpenBSD is still on CVS and developed their own version: http://www.opencvs.org

I also find the ability to branch and stash easily two super-useful features. Also, I like not having to go to the server to do diffs, merge, or pull files from other branches. bisect is nice, as well.

I use git-svn at work, so I have everything but easy (svn) branching and I find it super useful.

Lacking cryptographic hashes used by most DVCS software, SVN repositories need some external mechanism to be pretty sure the source repository hasn't been compromised. For instance, one might have snapshots of the SVN tree taken every week.

It's much easier with git or hg, even in the single-master-repo model, even if all developers are using the CVS gateway so they don't have local copies of commit history. All such a project would need in order to protect against compromise (ignoring physical redundancy) is a backup git or hg repo, kept offline except for periodic pulls. As long as there are no warnings about upstream rebases, and you trust all visible commits, then everything's fine[1].

With SVN, as long as there's no built-in cryptographically secure way to connect one snapshot to the next when backing up SVN repos, determining the existence of a backdoor requires comparing entire backup SVN repos against the live main repo. Even if sufficient backups exist, the process is slow and requires taking the main SVN repo offline if there's any chance of a root compromise.

[1] Caveat 1: in any RCS, there could be backdoors masquerading as legitimate-looking commits if the attacker had commit access. Caveat 2: The security is subject to the cryptographic security of SHA-1.

In this FreeBSD case, if they're certain the svn repo wasn't compromised, all they have to do is validate the checkout on the two compromised machines against the svn repo.

This is kind of silly. They can simply diff to find changes. All the developers have several checkouts.

There are lots of things that are great about git, but it's not a security cure-all.

Reply on news.ycombinator.com