Kaseya Hack: Incident Overview and Technical Details | Hacker News Reader

Kaseya Hack: Incident Overview and Technical Details

helpdesk.kaseya.com

37 pointsgostsamo31 comments

A former boss used to be a higher up at Kaseya before I worked with him. From his description of their development process, and his general lack of respect for developers, I’m not surprised.

According to him, they outsourced the majority of their development, and he had no clue that there could be any downside to this.

Within a few months of taking over, every single developer in the company left.

https://www.reddit.com/r/msp/comments/ocggbv/comment/h3u5j2e the Reddit comments from Huntress Labs contain more concrete interim technical details from their investigations.

* `/dl.asp` authentication bypass

* `/KUpload.dll` file upload, likely using authenticated session

* `/userFilterTableRpt.asp` command execution on previously uploaded file

From Update 12:

> The userFilterTableRpt.asp file contains a significant amount of potential SQL injection vulnerabilities, which would offer an attack vector for code execution and the ability to compromise the VSA server.

It's been 19 years[0] since Kaseya added this task to their BugZilla: "Migrate from Active Server Pages to ASP.NET".

[0]https://en.wikipedia.org/wiki/Active_Server_Pages

It's also interesting to note that their initial on-prem VSA fix/updates will include disabling some functionality: https://helpdesk.kaseya.com/hc/en-gb/articles/4403440684689-...

> We will be releasing VSA with staged functionality to bring services back online sooner. The first release will prevent access to functionality used by a very small fraction of our user base, including:

> * Classic Ticketing

> * Classic Remote Control (not LiveConnect).

> * User Portal

Without knowing anything about Kaseya, it sounds to me like the exploited endpoints were related to some kind of legacy features with a higher level of technical debt, and fixing all of the identified vulnerabilities in those components will take time.

And apparently one of the vulnerabilities exploited in this attack had been discovered and reported to Kaseya by independent researchers back in April: https://news.ycombinator.com/item?id=27763202

Kaseya doesn't have a CISO from what I can see.

CEO Fred Voccola retreived from https://blog.malwarebytes.com/ransomware/2021/07/kaseya-ceo-...

  We weren’t quite sure exactly what it was, but as third parties, the community, our own monitoring customers, we started noticing some strange behaviors,” Voccola recounted in the video. “Within an hour, we immediately shut down VSA.”

[...]

  “When something happens, it’s how prepared the organization was, how quickly the organization is to admit something happened,” Voccola said. “Seek help from people and try to get focus on the customers and get information out there.”

That makes sense though, their leadership is laser-focused on successful outcomes and a commitment to their customers. It was their incompetent acquisition culture who cut corners on all the boring stuff, like validation, testing, assurance, quality, engagement with for-free security researchers. Their plan was to stumble into the dashboard and trip over the off switch, after someone else told them they'd been wrecked.

This is absurd, incompetent, immorally negligent and contrary to industry SOPs (use bug bounties, have a CISO).

Past related threads:

Up to 1,500 businesses affected by ransomware attack, Kaseya CEO says - https://news.ycombinator.com/item?id=27750040 - July 2021 (290 comments)

Cybercrime REvil gang asks for $70M to decrypt systems locked in Kaseya attack - https://news.ycombinator.com/item?id=27734282 - July 2021 (39 comments)

Supermarket chain Coop closes 800 stores following Kaseya ransomware attack - https://news.ycombinator.com/item?id=27725576 - July 2021 (16 comments)

Major Swedish supermarket chain hit by cyberattack - https://news.ycombinator.com/item?id=27720623 - July 2021 (174 comments)

REvil ransomware executes supply chain attack via malicious Kaseya update - https://news.ycombinator.com/item?id=27716383 - July 2021 (13 comments)

Others?

Not a good sign when your official incident description has a typo [1]. Not that perfect language is important in and of itself. But it indicates that stress is leaking through into the official communication.

[1] (To date, we are aware of fewer than 60 Kaseya customers, all of which were using the VSA on-premises product, who was directly compromised by this attack.

Not very interesting information. They just describe how to see that the attack as happened for those running the system. Not how it happened in order to learn from and avoid the same mistake for the rest of us.

I heard on the grapevine it was an SQL injection, but thats more of a rumour at this point.

The SQL injection theory has certainly been mentioned. But not with any evidence I would trust.

Seems to be a bit more info here with CVE's etc - https://www.tenable.com/blog/cve-2021-30116-multiple-zero-da...

I have basic computer knowledge but no clue on cyber security. Can someone explain what was the negligence that lead to this attack?

They installed a centralized backdoor (remote management software) on all their computers, allowing attackers to use that to run their ransomware on all the computers instead of just one.

Im assuming these softwares are customised for every customer. So the attacker was able to get elevated privileges at the software level that made them run randsomware across multiple customers.

So is this vulnerability basically that they didn't sign and check hashes of their executables and downloads?

> The attackers were able to exploit zero-day vulnerabilities in the VSA product to bypass authentication and run arbitrary command execution.

This is all that we have so far.

This sounds like a 10/10 chained out of a rookie PKI mistake in their (editorializing here) poorly audited critical software infrastructure. Surely nobody steering the ship has been bitten by shitty PKI practices before, right?

These people have no care for the work of others they have trashed. The acquisition culture is so reprehensibly transparent and incompetent, it would blow my mind if it weren't just another day at the office. This company doesn't even has a CISO from what I can see. They must rely on their CTO's outstanding track record of industry-leading assurance technologies in a rapidly evolving landscape (of cash)

Refer to https://www.kaseya.com/company/

A) Their CTO presided over DigiCert, who botched 23,000 orthogonally integrated, outside-of-the-box, innovative security certificates[0] through their bad practices and garbage organization while meeting committed timeframes.

   Dan Timpson leads all product development, security and cloud/SaaS operations teams across all lines of business for all Kaseya companies including Unitrends, Spanning Cloud Apps, Rapidfire Tools, ID Agent, Graphus, RocketCyber and IT Glue. He joins Kaseya with a proven track record of success having led and grown engineering and R&D teams to build architectures focused on security and integrity, data compliance and testing. Through his leadership, Timpson empowers engineering teams to develop solutions that emphasize quality, simplicity and reliability while meeting committed timeframes.

   Prior to joining Kaseya, Mr. Timpson served as CTO of DigiCert, a provider of high-assurance digital certificates that deliver trusted SSL, private and managed PKI deployments and device certificates for the emerging IoT market. There, Timpson was responsible for Digicert’s technology strategy and played a vital role in leading the security industry by driving innovation.

   Prior to joining DigiCert, Mr. Timpson worked for Microsoft Corp leading virtualization technology engineering teams and driving the division’s software security release discipline. Earlier in his career, he managed engineering teams at Novell over the span of nine years building identity and access management systems and Novell’s underlying PKI framework.

   Mr. Timpson holds a Masters of Business Administration and Technology Management from Westminster College, and holds the patents to five technology innovations.

B) Their current CEO was onboarded after presiding over a number of pricey acquisitions in whatever tech fads, driving "unprecedented growth" with his undoubtedly forward-looking vision and commitment to shareholder value.

   Fred Voccola leads the vision, strategy and growth of Kaseya and its family of brands including Unitrends, Spanning Cloud Apps, Rapidfire Tools, IT Glue, Graphus, RocketCyber and ID Agent. Prior to joining Kaseya, Mr. Voccola served as president and general manager of Yodle’s Brand Networks Division, providing digital and market automation solutions to small and medium businesses. In 2016, Yodle successfully was sold to Web.com (NASDAQ: Web). Prior to that, Mr. Voccola was president of Nolio, Inc., a devops SaaS company, where he drove 100% year-over-year growth and successfully sold the business to CA Technologies. Mr. Voccola also co-founded and served as president and CEO of Trust Technology Corp., where he drove significant growth over three years prior to the company’s sale to FGI Global.

   Earlier in his career, Mr. Voccola co-founded and served as COO of Identify Software, substantially growing the business during his five-year tenure, culminating in a sale to BMC Software. He then served as vice president of worldwide sales and services at BMC Software. In addition, Mr. Voccola has held various management and executive roles at Intira (acquired by Divine Systems) and Prism Solutions (acquired by Ardent Software/Informix/IBM).

   Mr. Voccola holds a Bachelor of Science in Finance from the Carroll School of Management at Boston College.

This stuff pisses me off to no end, especially after all the years of being called a crackpot and how nobody would do that or how could they know or whatever. I'm not a genius, this is crayon on the wall.

At least they've "created" a lot of jobs lately. Probably toasted a few, too. They'll be off to the next clown car soon enough, probably driving unprecedented growth in next-generation managed service applications for dialysis machines and pesticide application PLCs. In the cloud.

[0] - https://www.bleepingcomputer.com/news/security/23-000-users-...

> Their CTO presided over DigiCert, who botched 23,000 orthogonally integrated, outside-of-the-box, innovative security certificates[0] through their bad practices and garbage organization while meeting committed timeframes.

I had forgotten about this incident. Somehow I thought this happened before 2018. Anyway, I'm not sure how any of this is DigiCert's fault. Maybe they should have had mandatory audits of resellers to make sure they didn't do stupid things like have the private keys for customer certificates... But given that the reseller sent the private keys to the CA, that's pretty good evidence the keys are compromised, and worthy of revocation.

Indeed, they had one job...

It's not said that they (the vendors code) downloaded that code. If they have a remote code execution elsewhere it can download more.

Does anybody check the signature of all DLLs they are loading (into memory, not downloading) and all executables they are a calling? I have not seen such system in real life. The Linux kernel can check the modules it loads, but that's a different level. And I am not sure how widely it is used in practice. Anyway if you can inject a bad kernel module you are already root and have many more options.

macOS does with notarization.

Good point, I am not a MacOS user. Users can authorize unnotarized executables to be run according to a quick Web search. So what would prevent malware from authorizing its unnotarized binaries?

In the Kaseya case it seemed to have elevated privileges already, because it was able to install bad binaries.

Can anyone share the actual binaries used? (Specifically, the dll cryptor which was sideloaded).

Sadly, not a whole lot of detail here.

Does Kaseya survive this?

Yes, I think. Depends on their management of the crisis. Hacks are part of life nowadays and it takes several examples of negligence before disengaging your users.

In my thinking they should not.

Closed source is just such an inferior development model. Not that bad practices and stupid mistake would not happen in open source. It has happened and will certainly happen again.

But more eyes are always better. And people who value deep technical understanding over just making money are often found in an open source ecosystem. It is incredibly frustrating to have to work with black boxes like Windows where you have inferior choices to explore and fix things.

Obviously there are many decision makers who don't see this. So shops like Kaseya do unfortunately survive.

Is there an open source Remote Monitoring and Management solution though?

My company used to use Kaseya but luckily changed to a competitor a few years ago so dodged this particular bullet, but I’ve always said that these are extremely high value targets to hackers and I’m surprised there haven’t been more incidents like this - though Kaseya was also hacked to deploy crypto miners a few years ago too so this is not a one off event.

Given that the nature of RMM software specifically requires being able to push arbitrary code to large numbers of devices security SHOULD be paramount and having a respected open source solution for this would be ideal

Ansible, Puppet, Chef, Terraform maybe? I am not an expert in that area. I have used Terraform and seen Ansible. Know basicly nothing about the others.

Depends on whether courts hold it accountable for the damage their negligence caused.

If yes, then the damage should bankrupt them.

Reply on news.ycombinator.com