OS X Auditor | Hacker News Reader

OS X Auditor

github.com

348 pointsevandrix43 comments

OSXAuditor is pretty dope and we used it a bunch at Yelp. Over time, we created what we think is an inspired next version - https://github.com/Yelp/osxcollector

The passive scan approach has the single weakness that it appears to be effective, but mostly is security theatre.

Take for example, when we travelling into Australia, and they asked if we had any criminal convictions - I didn't know that a conviction was still an entry requirement for the island!

I run OSXCollector and I get missing modules error. I'm not a python guy, but looking at the code, it requires:

import Foundation, import calendar, import os, import sys, import shutil

I assuming os, sys, and shutil are part of the system (are they?). What about the other two? Also, which version of Python do I run this under?

...not a python user either. The README mentions 'pip' but I don't seem to have one of those either.

Here's a "non-magical introduction to pip" http://www.dabapps.com/blog/introduction-to-pip-and-virtuale...

Pip is an excellent package management system.

calendar is a builtin as well.

It appears the OSX specific modules are located (on latest Mavericks) in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python

So you need to add that directory (for xattr) as well as that directory + PyObjC (for Foundation) to your PYTHONPATH

Still get the error. Added PYTHONPATH to my bash_profile.

After getting the error again, I looked at that location - it exists but the mods in question (Foundation and Calendar) are not there. I'm on OSX 10.9.4.

It doesn't really matter enough to waste time on though. Thanks for your help.

I just want to point out that in the docs "self contained" doesn't mean the user has to go hunting for mods.

On a reasonably modern Mac, all the dependencies should be setup by default - including modules and the correct version of Python.

I'm on 10.9.4 and I ran into a bunch of problems (which seem to be covered by github issues[1] already).

[1] https://github.com/jipegit/OSXAuditor/issues/27

Foundation is part of the pyobjc package. You'll need to install that.

If you're on a mac using brew or macports usually the paths are usually different. Macports/Brew python doesn't usually have 'pyobjc'. Try running osxcollector as:

sudo /usr/bin/python osxcollector.py

What's up with the blurb on your GH page that says, "Exactly how Yelp uses the output from OSXCollector is a bit of our secret sauce"? How is workstation auditing a part of Yelp's secret sauce? You're a review site. Strong security on Internet-connected machines benefits everybody. There's no competitive advantage here. If you have something awesome, why not share it?

Yeah, that could have been stated better. I guess I meant to say we're not handing out on GH the indicators of compromise for stuff that's currently giving us problems. But we do, in the README and when we talk to folks about OSXCollector, try to share how we do analysis. I'm hoping to release some related tools over the next few weeks that do some amount of parsing and analysis of the output.

There's some neat stuff we've played with (though nothing short of manual analysis is giving really high confidence at this point). Some of the ideas we played with but haven't written about yet: * Feed the output through a parser that finds domains, URLs, and IPs. Feed those into threat feeds and passive DNS APIs. Occasionally this surfaces interesting stuff. * Feed all the hashes to VirusTotal, cymru, or known lists of nasty stuff. Hits are generally nasty. * We've got some known indicators of commodity malware persistence in launch agents. We grep for those cause we know they've hit us before.

One of the recent things we did add to OSXCollector was pulling xattr's from downloads. This allows us to find the source URL - sometimes even the redirect chain - for a download by reading extended attributes of the file. This has been helpful.

How do you read in human-friendly format the 'json' file created? Do you have any extra tool to create HTML, etc. or do I need to roll up my own json parser?

This looks like an interesting project. But, there's one bit of information, which is missing from the description and this discussion, that would help a lot with my understanding.

Is this a tool for a user who wants to learn more about their own machine, or a non-user who wants to know how a given machine has been used?

More information about the use cases of such a tool would be most helpful.

Ditto, in particular would be good to know whether this is only for post--intrusion forensics or also useful ahead of time.

I'd say this isn't quite ready for prime time. The errors I received were annoying and you're not going to get an end user or most IT admins to parse the python just to find the errors.

I've only had 3 major errors but they were significant: 1) I'm on Yosemite so GetAuditedSystemVersion() looks for a PatchVersion variable that simply isn't there. The header reads 10.10 not 10.10.patch as expected. 2) The Safari parsing snafu listed in my previous comment. Opening Safari isn't enough, you have to use the browser quite a bit. The same could likely be said for all browser tests and it would be a good idea to outline precisely what this needs to be. Hint: A new system or install of Yosemite for instance will produce the errors I saw. 3) There's a parsing bug in ParseMailAppAccount() and I just commented out the call completely.

Any number of these could just be Yosemite related but I don't think so. All of the bugs I ran into are variations on index out of bounds due to some hardcoded assumption that mostly works, except in this instance apparently. I'm not the only one with these nagging bugs based on the issues list but mine do seem very specific to Yosemite or how I do(n't) use my system.

Cool idea.

But using raw md5 hashes to verify against a blacklist is kind of useless. Especially now.

You should be using smarter file signatures:

http://hooked-on-mnemonics.blogspot.com/2011/01/intro-to-cre...

so, if i understand it correctly you're saying create AV signatures (the link you point to is how to create ClamAV signatures). to do so in a worthwhile, comprehensive way would required the maintainer to basically become an AV company. possible, but requires a lot of effort to scale. by calling out to various services, the creator of this tool is essentially outsourcing maintaining a file reputation blacklist.

MD5s are, despite their limitations, the lingua franca of the security industry. nearly everyone who provides a file reputation query service supports them (as opposed to SHA1s or other hashes like ssdeep).

so, i think i get what you're saying, but i don't think it's a relevant suggestion here.

I'm pointing out that this tool is essentially useless from security perspective. Most of the payloads will have different hash results due to extensive use of packers.

If you are going to get hit with variant #11929 before the online databases obtains a hash of it, this tool is not going to pick it up but it will tell you that you are secure.

>so, i think i get what you're saying, but i don't think it's a relevant suggestion here.

It's pretty relevant. Without sending the whole file to the third party, the file reputation service isn't 'outsourced' as you suggest. Sending the MD5 will not do any good if the program makes non-deterministic modifications to its binary.

Great idea. Love to run it, but... crash and burn:

~/Library/Safari/LastSession.plist Traceback (most recent call last): File "osxauditor.py", line 1702, in <module> Main() File "osxauditor.py", line 1663, in Main ParseBrowsers() File "osxauditor.py", line 808, in ParseBrowsers ParseSafari() File "osxauditor.py", line 745, in ParseSafari ParseSafariProfile(User, UserSafariProfilePath) File "osxauditor.py", line 717, in ParseSafariProfile LastSession = LastSessionPlist["SessionWindows"][0]["TabStates"][0] File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/objc/_convenience.py", line 451, in __getitem__objectAtIndex_ return container_unwrap(self.objectAtIndex_(idx), RuntimeError) IndexError: NSRangeException - -[__NSCFArray objectAtIndex:]: index (0) beyond bounds (0)

I can't even get pyobjc running on 10.9..

I also have this problem. I'm not sure, but it might be caused by using brew python instead of the systems original python

Looks like you have an empty session list. Have you ever used Safari? Quick workaround might be to pop open Safari once (and browse a little) and then try again.

Why on earth would anyone downvote this comment? The lack of a Safari user profile is the most obvious potential issue here, based on the backtrace.

That's the precise problem I had. Opening Safari got me one step closer and then another failure. I wanted to run all options but got tired of all these errors so I commented out ParseSafari, ParseFirefox, and ParseChrome calls inside the ParseBrowsers function. I could've left Firefox or Chrome but not wanting to shut Chrome down or not using Firefox much, I didn't want to go through this again for each browser.

And that, folks, is why we don't do chained expressions. Variables increase readability and also offer one the opportunity to check a condition that should be true.

This, KeyError, NoneType exceptions and its Java friend NullPointerException drive me batty. They are so simple to defend against, if the author just took the time.

I love Optionals for this.

Chaining itself isn't really the problem here IMO. Creating a variable for each level of the nested data structure and manually checking for the validity of each subsequent one would really, really not produce more readable code. It would turn a "get this value out of a data structure" expression into about 15 lines of code that each need to be individually read and understood. The problem is that Cocoa doesn't provide a great way to say "Access this stuff in this data structure unless it isn't there."

An option type is really the best way to kill this breed of problem, and thankfully Apple is including one in Swift, though it's still somewhat green.

That's a Obj-C problem, because it's being fixed with what Swift does with it's '?' on variables. :)

Not exactly. This program would have the same problem in Swift, because rather than returning optionals, Swift arrays have the same explode-on-invalid-accesses behavior as NSArray.

Maybe monads make this kind of chaining safe.

The accessor variant of the existential operator in CoffeeScript (which Swift apparently has taken influence from) handles this nicely.

This looks nice but there are a ton of issues around pyobjc. I've never been able to successfully install it on 10.9.

This is because you don't need to. Especially for this particular script.

If you're using the natively included python, pyobjc comes free.

As soon as I saw that mentioned in the documentation, I knew this project was going to have issues.

I'm familiar with python bridges at all - how would someone transition something with pyobjc as a dependency to the native alternative you mention?

You misunderstand.

OS X has shipped with python for quite some time.

In addition, OS X python started including a build of pyobjc since OS X 10.5

You literally can fire up python from the terminal on OS X 10.5 and later and type: import Foundation and it will "just work".

No additional installs required.

Very interested to try this out on my own machines to see the results. I know somebody who does computer security at a University and the staff there has been frustrated with the lack of available forensics tools for OSX, so this may nicely fill an empty niche for some industry people as well.

Application path scanning follows symlinks recursively. One upwards link, and it never terminates.

I'm getting an error after running it:

[INFO] Users' LoginItems [INFO] 's LoginItems [INFO] /Users//Library/Preferences/com.apple.loginitems.plist [INFO] Cannot parse /Users//Library/Preferences/com.apple.loginitems.plist (Binary or JSON plist may FAIL)

Traceback (most recent call last): File "osxauditor.py", line 1702, in <module> Main() File "osxauditor.py", line 1651, in Main ParseStartup() File "osxauditor.py", line 550, in ParseStartup if "SessionItems" in LoginItemsPlist: TypeError: argument of type 'bool' is not iterable

I’m guessing that’s probably happening because of the double slash after /Users.

nono, that was originally asterisks but apparently hacker news replaces those with an empty string. that is weird, but unrelated to my problem

Reply on news.ycombinator.com