A blackhat social engineered me by posing as an early adopter of my startup
blog.codeboff.in
blog.codeboff.in
The author of the article had left a default whitelist on the production box that whitelisted /dev/random as being readable. The attacker compiled a program (took a test I'm guessing) that included /dev/random. The author found out based on ip and sheer luck that the attacker was a user of the system that he had corresponded with previously.
The reason I summarized this is because I wasn't totally clear on what codeboff.in was from the article. It makes sense how an attack like this could happen. The author seems to really know his stuff and the attack vector really was a small oversight. It's pretty unsettling to me, though, how persistent/insane this attacker seems.
I've spent the last decade doing mostly IT Security defense work. I don't worry when I see someone port-scan my servers. Those kiddos are almost always benign. I worry when I see that our IDS starts to correlate many subtle attacks over a long period of time from similar places (such as the same school, ISP, etc)
It'll probably be useful to me to keep more verbose access logs and hash IP addresses to track what individual users (at least those not behind a group firewall) are up to.
Thanks for the tip!
Oh crap.
Keep in mind I've played both sides of this game. People who know how attacks work in the wild can usually spot things that might not raise the ire of average sysadmins.
It's worth mentioning that no one is perfectly secure, either. A sufficiently bored, persistent and motivated attacker WILL find some kind of way in. I'll give you that. I've seen it play out enough times. But with centralized logging, transaction shipping for hot-site replication, and sufficient effort placed on separation of duties, an organization can minimize the impact a successful attacker can have, and can ensure that all audit trails remain intact in some way or another.
Anyway, to go on (keep in mind that I am genuinely interested in what you have to say). Isn't it true though that your IDS is only as good as your signature db? So if there was sshd or ftpd 0day that wasn't generic (generic shellcode etc.) that in theory you could spawn a shell without it ever being logged?
I'm not trying to have a pissing contest, but the reason we have defense in depth isn't to get 100% secure. You and I both know it's impossible. It's also there to preserve evidence if the unthinkable occurs. I do genuinely appreciate the forward thinking, so I'm sorry if I was a tad harsh in my reply above. Your responses (and your HN bio) came off a tad skiddie-ish at first blush. Being in infosec, where "Hacker" has a different meaning and stigma than "Hacker" in HN was originally meant to convey, I'm actually glad there are threads where Infosec "hacking" discussions can occur without getting completely buried by haters.
I worked in infosec (on all 3 sides, if you know what I mean) from the mid to late 90s, and then became disillusioned with the entire industry and left for greener pastures. I still attempt to keep on top of things (and have done the odd contract job here and there) but I am not all that up with everything going on.
Wrt the topic, what you are saying is that even with my /bin/sh running in the context of whoever sshd or ftpd are running as, by the time I figure out a local escalation, by then the activity on that shell has already been sent to another machine and onto your phone etc?
As you well know, a careful individual can evade it if they know it's there, but the initial prodding would get logged.
In the enterprise, there are so many IDS/IDP and SIEM products out there, and they're all different. Some are great for mixed environments of Windows, UNIX and infrastructure. Others are better for specific platforms. Your best bet is to get demo licenses and start playing with them instead of relying on word-of-mouth or shiny pamphlets from marketing-types.
Pretty good system to correlate different events to get a better picture of a potential intrusion. If you're familiar with Snort I recommend checking it out.
The first several times I read the term "code evaluator", it never occurred to me that he really meant evaluator as in executing user-supplied code on his server. Holy yikes!
Reading the rest of the article, it does seem that he's at least thinking about how to deal with the ramifications of that decision. It certainly takes some boldness to try this at all. Personally I would have got as far as "executing user-supplied code", thought about it a second, shivered at the implications, and picked a different idea to run with.
The code evaluation runs in a standalone server. I hope to learn what to block over time against a series of successful exploits. Evaluations that failed to compile/run/return-an-expected-result are marked on another server (on which they are not evaluated in any way) so I can study them later.
1. It may not be an insurmountable problem; see eru's and ambition's suggestions below.
2. There is a real business need for the application, and anything worthwhile can have unforeseen difficulties.
The NSA's guide is good enough for low value targets. Besides, these are your tax dollars at work, so you really should make use of them. :-p
[1] http://www.nsa.gov/ia/_files/factsheets/rhel5-pamphlet-i731....
[2] http://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf
On the web application side, you're going to want OWASP. http://www.owasp.org/index.php/Main_Page
EDIT: Just thought of something.... There were several privilege escalation exploits for the kernel made public around this time frame [3]. I'm guessing that's the reason why his kernel was silently upgraded. :-p
An employee of one of our direct competitors even called one of my developers to try and extract information - good thing she caught on quick. We eventually tracked a number of attack vectors to either him or his company. They weren't the brightest attacks though (calling her from his work #, emailing her and other employees from email addresses which he'd used online, running port scanners and SQL Injection tests (etc) on our servers from company IP addresses).
Other times people call pretending to be customers, but they don't pass the smell test. We've had a few people call with tons of questions about our tech, capabilities, existing customers, etc, but won't divulge much of their own information or can't properly answer some of the basic questions that our real customers would know.
Early customers are kind of like a job interview, it has to feel right for both sides.
So right. I consulted for a while as a programmer. It took a few times being burnt before I could distinguish between the good and the bad clients, but after that the difference was night and day.
I don't agree with 'never say no to a customer' in software development work.
I also don't see how it is social engineering. What did the attacker convince you of?
So at the moment you have to leave an email address to take an open test. (This will change soon to no address required). The guy consistently signed in with a fake email address, email to which was bouncing. I blocked that email address, which amounts to not accepting it to let someone sign in for an open test. This is just a soft warning, of course. So the guy appended a character to the address, and left me the message: "Why have you blocked test@example.com? Too bad codeboffin!" I just don't see why they couldn't have discussed it on email on which they were already in mid-conversation with me.
I presume the social engineering was coming. I would probably have told the person anything he wanted to know about my security setup by then. I don't think I would have been as open to anyone else off the bat.
There may be other details that he hasn't shared, but if this is the "attack," it's pretty benign.
I'd yell at my QA team if I had one...
The latter is when any host would do the trick (relay, host platform, whatever).
Disclosure: I am working on XenServer.
With tasks queuing up as they already are, I might not be able to keep a fresh VM pre-warmed every time, either.
edit: The other factor is of course the prospective cost of a brand new configuration. Unless I find I can't prevent the current config from being breached and ephemeral VMs will fix it.
Ach, I see now what eru meant by copy-on-write VMs.
(At least as long as you have a general purpose operating system. A special paravirtualised would probably not need to make much of a difference between booting and resuming. But that's only a theoretical musing for your circumstances.)
This is my new plan of last resort if I can't prevent the current sandbox from failing.
"So of course, the blackhat compiled a C program with #include “/dev/random” in it, and gcc was hosed."