China Mafia-Style Hack Attack Drives California Firm to Brink | Hacker News Reader

China Mafia-Style Hack Attack Drives California Firm to Brink

bloomberg.com

118 pointsssclafani51 comments

"n April 2010, during a 6:30 a.m. check of his servers -- by then part of his daily routine -- Milburn stumbled on a folder buried in an obscure Microsoft directory, one that’s normally unused. What he found inside startled him. The file contained the encrypted versions of all eight passwords in his system -- the keys to the entire network. The hackers could use the passwords to control just about anything he could, from web servers to e-mail."

Ah. He just decrypted the encrypted versions of his passwords by looking at them. I see.

And he "stumbled upon" a directory during his routine 6:30am check. Really? He's manually inspecting servers every morning at 6:30am?

This entire story is several levels of /headdesk.

Maybe he wants to beat the morning rush on his daily drive to the data-center~?

Well, maybe he's just using Xenix...

If your servers were under constant attack, yes checking your servers every morning first thing could well be your routine.

You also assume he immediately recognised them. It could well have been someone one who later told him what he'd found.

Sloppy of the attackers to leave it in plain sight.

>...decrypted the encrypted versions of his passwords by looking at them. I see.

I think this means the hashed passwords. Microsoft have had several vulnerabilities in this area. Just entering a password to browse a server caused Windows (prior to Vista) to store the LM hash. An LM hash can be reversed in a matter of seconds.

A lot of the comments about this presume that the victim could have avoided such trouble with better security knowledge. It's worth noting that the Chinese have hacked Google, Yahoo, Adobe, Lockheed Martin, and over a hundred other companies, which have many more security resources available to them than Solid Oak does. No small business in America would ever be able to defend itself from a sustained attack backed by the resources of one of the most economically powerful nations on earth. For example, does anyone here think they could defend against years of attacks from the NSA?

Your question is not fair. I am sure there are cases where NSA tried an attack and failed. It is a matter of how badly they want the attack to succeed. Just because you are being attacked by "the Chinese" does not mean they will put all their resources behind it.

I'm sure the amount of resources the Chinese government needs to put behind a successful attack on a small US business is minuscule.

The resources united states government would be even less.

As somebody who made dismissive comments about the victim in this case: No, I'm not saying he could have avoided a nation state attack with better security knowledge.

I'm saying that - at least the way TFA describes it - the victim doesn't have enough security knowledge to tell if he's been attacked by script kiddies, a nation state, or a roll of toilet paper.

I'm sure the real story is a bit more nuanced, but nothing about the description gives me any confidence that the allegations are even broadly aimed in the right direction.

Is China a major vector of security attacks? Yes. Does that mean it has anything to do with this particular incident? I wouldn't bet serious money either way, and pocket money against.

"He had no idea who was behind it until last August, when he provided malware samples to a security firm at the request of a Bloomberg reporter. A forensic analysis of the malware by Joe Stewart, a threat expert at Atlanta-based Dell SecureWorks, identified the intruders who rifled Solid Oak’s networks as a team of Shanghai- based hackers involved in a string of sensitive national security-related breaches going back years."

What does that prove? The Bloomberg reporter doesn't show much evidence of being able to discern what is a valid analysis and what is not. What is a malware sample? Is it so easy to tell if a malicious script was actually used by a group, or merely copied by a script kiddie?

To give you an idea of how stunningly bad conclusions can be drawn from technobabble inferences, I present this:

http://www.foxnews.com/tech/2012/05/30/powerful-flame-cyberw...

Powerful ‘Flame’ cyberweapon tied to popular Angry Birds game

The most sophisticated and powerful cyberweapon uncovered to date was written in the LUA computer language, cyber security experts tell Fox News -- the same one used to make the incredibly popular Angry Birds game.

LUA is favored by game programmers because it’s easy to use and easy to embed. Flame is described as enormously powerful and large, containing some 250,000 lines of code, making it far larger than other such cyberweapons. Yet it was built with gamer code, said Cedric Leighton, a retired Air Force Intelligence officer who now consults in the national security arena.

FOX News claims that Angry Birds is a delivery vehicle for a virus because both of them are programmed in Lua. And who can blame the reporter? Her source is a former intelligence officer in the Air Force who is likely paid way more than every developer on HN to hype up such threats.

It doesn't matter. The main problem here is not hacking. It is piracy as the Chinese government stole their software.

I wonder what would happen if Anonymous decided to target China over matter like this. Sure would be impossible to bring a government like that to their knees. But at least the public would gain some awareness on this matter.

Kill that noise right now.

Before you grabbed that little war drum, if you thought of the consequences you really wouldn't want to start such a conflict in the first place.

The quick, unorganized retaliatory response from an anonymous actor is usually such that it puts innocent services and people directly into the crossfire. Not to mention an anonymous moniker allows anyone to act on the behalf of anyone.

Who said it has to be quick and unorganized? While officially the US can't 'engage' China in this way, another NGO could start and perform long term campaigns against Chinese corporations and assets similar to what is detailed in this article. I'm also not naive enough to think that this hasn't already begun.

It doesn't work that way. If the U.S. government condones cybercrime, it cannot allow an NGO to engage in it towards a country that you have diplomatic relations with. I agree with "Kill that noise right now" comment above.

Milburn, after all, had built Solid Oak’s network himself. “I thought they might be able to get around some IT guy, but there’s no way they were going to get around me,” he says.

/Facepalm.

I was going to write that at least he got a bunch of free publicity from the article, but on the other hand the takeaway is that product is probably no more secure than his servers.

I liked "He taught himself how to write code, and eventually mastered complex Internet software protocols."

Seems like a lot of the issues would of been solved by moving to AWS. At least they would of still been making money.

Likely. I don't want to be too hard on the guy because I can recognize a bit of myself in him, but this is a great example of why you should concentrate on your core competency and pay someone else to do the stuff you're not expert at.

If they were able to get into Google, something tells me they would have been able to break into AWS.

Whether they would waste an AWS 0-day on some random stubborn American is another story.

I'll preface this by saying that it is all too easy to criticize from the outside and without all of the relevant information and history.

Once I got to the point in the article where it was obvious that he was under some form of a coordinated attack I almost wanted to send myself back in time to be able to go over to his office and yell "AWS! AWS!" in his ear.

In addition to that, it seems it should have been obvious that bringing a team of experts to help secure the network (or transition it away) and fight the fight would have been the smartest idea. I would think that the quoted losses of some $58K per month would have covered this just fine. And, perhaps what is more important, it is likely that the technology fight wouldn't have lasted three years.

The other thing that struck me --again, don't have perfect data, don't know all the facts-- is the apparent lack of help from the likes of the FBI. You would think that they'd be there in some sustained fashion to help out.

This fellow was out of his league and paid a dear price for it. Hopefully the settlement compensated for some of it. It sounds like he might have ended-up with stress related health issues which are no laughing matter.

It'll be really sad if the Internet becomes just another weapon of war. That should not be permitted. How? Not sure. Is it too late?

>It'll be really sad if the Internet becomes just another weapon of war.

Anything of value can be turned into a weapon of war. Think about what a weapon is - it is a tool to disrupt something delicate. A missile destroys a plane. A knife stops a heart. A virus stops a computer.

It really concerns me that a private US citizen was personally attacked by a sovereign nation. It seems to me that the US has an important duty to protect it's people, whether the attack is virtual or physical.

They attackers didn't just cause downtime for his website. They stole his entire livelyhood and software creation.

The HN reaction to this story is pretty sad, I have to say.

Yes, journalists are poor at providing technical coverage. That's not really their job.

Yes, this guy could have created a more secure network system, but if a government hacking group comes after you, you will likely not do any better.

Ugh...this article reads like a movie about hacking. The story breathlessly conveys how elite Shanghai hackers toyed with a company's lifeblood, wreaking such prolific havoc that the owner literally crawled under the server building to see if a bug had been planted...but the ultimate culprit may have been...

> Examining the script that controlled the payment processing function in November that year, he noticed that a single character was missing from the string -- an apostrophe. That was enough to cause the page to time out, rather than to complete the credit card transaction. Customers were leaving in frustration

Am I right in thinking that this was all hack via SQL injection?

Definitely took them some time to write all that down so pretty.

Sounds like classic Chinese hacking though, this doesn't sound like the work of real pro's.

I get that not every programmer can be well versed in cyber security, but how is it that apparently no one at this software firm apparently practices sound scientific reasoning? Is it possible to go through enough schooling in commuter science and not be able to diagnose a hack with logical reasoning before assuming Neo and the Matrix are real?

  > schooling in commuter science

I think that you found the issue right there. ;-)

Ugh, I think someone just hacked my keyboard...the timing is just too convenient

Just don't start fiddling around in your crawl-space looking for bugs! </pun-intended>

Must be those Chinese hackers!

I think you are missing the point. China will steal your software and higher people who aim to destroy your lively-hood and life.

No, I think you're missing the point. If it's true that rival nations will attempt to hack us for whatever reason, then it benefits us all to have a better understanding of basic cybersecurity than seeing foreign hackers as the Hand of God. Instead of examining the geopolitical problems here, this article takes us through a terrifying cyberstorm whipped up by mystical superhackers when the real perpetrators might as well have been script kiddies.

The problem is that if cybersecurity continues to be framed in this fashion, then all that shit that HN continually complains about -- security theater (via homeland security) and draconian Internet laws (remember SOPA) -- will continue to be status quo.

(I think frozenport was telling a joke.)

Script kiddies or not who cares - the effect on the small business was still effective. Like you say, its another wake up call to us all to have a better knowledge of security.

It seems like it was via an attachment in a phishing email that one of the employees clicked on...

"Milburn contacted Matthew Thomlinson, a Microsoft Corp. (MSFT) threat expert for help. Thomlinson found the malware had downloaded software that burrowed into the company’s Microsoft operating system, automatically uploading more tools the hackers could use to control the network remotely."

It sounds a bit like the site owner was playing wack-a-mole for three years when he should have just wiped out everything. Once you have an intrusion, unfortunately you can't trust anything that a machine has access to and you should start again from a totally clean install. If this was a huge corporate network I could understand that may be impossible. But for a small shop with a few people, there's no reason not to go around and just wipe out every machine in the company.

If this was a government sanctioned job, though, I imagine they could have stepped things up if the script-kiddie stuff didn't work. Why bring out the nuclear weapons if bottle rockets are doing the job just fine.

  > high-tech spies and digital combatants seek to gain
  > a brass-knuckle advantage in the global economy

Digital combatants? Does enlistment in the PRC army now include light cycle[1] training?

[1]: http://en.wikipedia.org/wiki/Tron_(franchise)#Light_cycles

Apart from the unverifiable security blunders (this is a Bloomberg article, not a Full Disclosure post), I have to say I can't feel very sympathetic towards somebody in the censorship business being harassed by fellow censors.

I stopped reading at this point:

Commercial hacker hunters -- who refer to the team as the Comment group, for the hidden program code they use known as “comments” .....

/facepalm

Why? It's seriously what they do. The user installs the malware, it goes to a normal looking web page checks the html comments and receives it's commands. On the face of it everything looks fine to the user but the malware knows then what to do.

I'm going to try to suggest a couple of solutions regarding problems encountered in the article that people might find to be useful to know.

First, some pointers (non-exhaustive).

For workstations: - Antivirus - Firewall - Disk encryption - Password complexity requirements - Updates

For servers: - Key-based authentication - Updates - Firewall rules on host: explicit ingress/egress - Run services as non-privileged users in a chroot jail

Firewall rules for network: explicit ingress/egress

Yes, that's two firewalls for your hosting infrastructure. Attackers would have to escalate privileges from a service user to a root user in order to modify the firewall rules, at which point they'll have access to the other components of your internal network. To get outbound connectivity, they'd have to gain root access to the external firewall as well. Keep in mind they can still upload data if they can execute arbitrary commands as the web service user.

Now, onto more specific items mentioned in the article.

  >realizing only later that the e-mail address was a couple of letters off.

Tell your employees to only accept cryptographically signed email from you. S/MIME is simple to implement, and works across multiple platforms and clients. There is even support for it in iOS, Android, and Mail.app!

http://en.wikipedia.org/wiki/S/MIME

You can get certificates here: http://www.symantec.com/verisign/digital-id

You can even get free ones if you're on a budget! (although they do say for personal use, implement at your own risk if you're a business!) http://www.comodo.com/home/email-security/free-email-certifi...

You can do a Google search on how to get these certificates installed in your mail and mobile clients of choice.

  >clicked on the attachment

Use Google Apps. I'm not sure what the success rate of Gmail's 0-day detection is with attachments, but I'd be willing to bet that they're better if not as equally effective as the average desktop antivirus email scanner.

  >Microsoft operating system

The article doesn't say if they used any sort of antivirus or firewall protection on their workstations. If you absolutely have to have Windows somewhere in your business, you need to have a decent antivirus and firewall solution running. Also, make sure all of the latest updates are applied in a timely manner. IIRC, you can set Domain Policies from Active Directory to do this sort of thing for you.

  >automatically uploading more tools the hackers could use to control the network remotely.

This could have been addressed in a couple of ways. (non-exhaustive list)

1. If the malware was listening for incoming connections, explicit ingress (aka no port forwarding) with a NAT (most residential routers) would have made it more difficult, as another layer must be evaded to connect to the internal machines.

2. If the malware was using a connect-back shell, explicit egress (aka outbound) traffic would make it difficult to get a shell on a compromised machine. As soon as you realize you're under attack, you should go into hardcore lockdown paranoid mode, and only allow network ranges that are absolutely necessary for business operation.

  >Then the company’s e-mail servers began shutting down, sometimes two or three times a week, slowing e-mail traffic, the main way the company provides customer service

Hosting an email service is a ridiculously difficult task security-wise. Postfix and Sendmail both have a history of security vulnerabilities, and the only way to properly host an email server is to harden a *nix system to the extreme, and always be paranoid. But that's hard and most people aren't in the business of hosting email services. That said, I'm pretty sure Google Apps has a great cost/benefit factor when you look at ease of implementation and security, because the Gmail team is more likely better at server hardening than you are. Oh, and make sure you backup your email and support PIN, in case anything goes awry.

  >Similar problems began plaguing the web servers -- a bigger problem since web sales of CYBERsitter supply more than half of Solid Oak’s revenue.

I would have to recommend putting CloudFlare in front of your servers, as (through their blog posts about several incidents) they have demonstrated an ability to successfully defend against a large number of attacks. This also has the benefit of providing a proxy, masking your backend server IPs, which I'll talk about more later.

  >Similar problems began plaguing the web servers
  >figure out how the hackers might be behind it.

Bro, http://en.wikipedia.org/wiki/Host-based_intrusion_detection_...

  >But the agency shed almost no light on the situation, he says, and he was never told if the material was useful.

I'm surprised they didn't offer any advice on how to mitigate attacks such as these. At the very least, some pointers similar to the ones I'm writing now would probably gain some sort of ground.

  >alternating between four different cell phones from three different carriers.

This seems a little extreme, if not excessive. I've used burners (temporary pre-paid cellphones, disposed of after a duration of use) before when I attended DEFCON (hackers like to do sketchy things with RF) but never more than one at a time. I'm sure you could probably use something like Google Voice with multiple Google Accounts or Twilio for temporary numbers while not exposing your actual number.

  >constantly had to reboot servers
  >couldn’t trace the source of the network problems

If you're going to have to do it more than twice, it's probably best to automate it. If reboots are required due to a crash, you should probably identify the cause of the crash (go through logs, etc.) and attempt to rectify it. If you seem to be way in over your head, bringing in an expert should definitely be on your mind.

  >to find that his commercial-grade SonicWALL firewall had failed
  >He spent a good part of the next day on the phone with the manufacturer, who was stumped.

If you have any sort of hosting infrastructure, and your expertise isn't network administration or security, I'd recommend getting (full time, or contracting) a certified network administrator to help you get your networking shit together. If I were to pick equipment, I'd probably go with the big dogs, Cisco or Juniper.

  >He began writing his own software to monitor the connections his computers were making to outside networks, looking for tell-tale signs of the hackers at work.

Bro, http://en.wikipedia.org/wiki/Network_intrusion_detection_sys... Snort: http://en.wikipedia.org/wiki/Snort_(software)

  >servers
  >obscure Microsoft directory

Contrary to popular (or at least some peoples') beliefs, it is possible to harden a Windows server, you just have to really know your shit and be able to tolerate all the graphical clicky stuff.

  >all eight passwords

If you're running a business, chances are you have more than 8 accounts.

1. Use unique passwords for each service/account. 2. Store them encrypted, in a password manager. 3. Use keys/certificates whenever possible. 4. Use two-factor authentication whenever possible.

  >The folder was gone two days later, he says, and in its place were several pieces of software he didn’t recognize.

You generally don't want to leave a compromised server in production rotation. Take it off the network and perform forensics to determine how it got compromised, and how to prevent that in the future. Make sure to wipe the server and reconfigure it before placing it back on the network.

  >Net losses averaged $58,000 a month

(aside) I'd fix the CRAP out of your security problems for that much a month.

  >A hacker could certainly edit the script and break it so it wouldn’t work

This means they have access to your web servers, and/or to your deployment process/service. One of the ways you can mitigate direct access to your web boxes is to use some sort of load balancer (HAProxy) or web application firewall (CloudFlare). Then only allow connections to your web boxes from the proxy, as well as a specified IP address to allow for maintenance.

  >That would be a great way to do it without calling attention to the fact that they were in the system.

If you choose to use git as a tool to deploy your code, you can continuously monitor the codebase for modifications.

I'm quite tired and I think I'm rambling at this point. I hope this is clear enough for some people to get something out of. I'll probably come back later and make some edits for clarity/correctness.

from the Analysis that the article claims identified "thousands of lines copied of code" (sounds like they copied a blacklist):

We found evidence that a number of these blacklists have been taken from the American-made filtering program CyberSitter. In particular, we found an encrypted configuration file, wfileu.dat, that references these blacklists with download URLs at CyberSitter's site. We also found a setup file, xstring.s2g, that appears to date these blacklists to 2006. Finally, csnews.dat is an encrypted 2004 news bulletin by CyberSitter. We conjecture that this file was accidentally included because it has the same file extension as the filters.

So... for how long did the guy keep credit card processing running on compromised servers? And it took them months to realize that?

Hopefully that teaches him to rely on windows.

And that's what I was thinking as I read this - a guy/team who can write net filtering software would surely know what side is up. I was waiting for the part where he set up a GNU/Linix box and just rebuilt his site and perhaps directing his support email to/via GMail, heck how long would that take to setup, a week and cost far less than the lost sales and it would keep the engine running till a more permanent setup was devised. Or just clean installed to all his Windows machines. So many options and he choose the hard one. There has to be more to this story.

The scary thing is indeed that most software companies do not have security experts nor the means to hire ones and are basically defenseless against such attacks. Kind of reminiscent of patent trolls.

tldr version of the article: http://tldr.io/tldrs/50b6e4acbb22039977000f5b

I hope tptacek chimes in. It sounds like a terrifying situation.

Why do we do business with those sleazy bags? Just to save a few bucks on our next gadget? No. I guess we have to thank the Wallstreet bonus whores for dropping this f-bomb on us. Not even Bin Laden would've done better. Thanks boys, you rock!

That's a mighty accurate username.

Thanks I guess. It's not that hard living up to that these days.

Reply on news.ycombinator.com