Hacker News Reader

Hacker News

Comment by fadijob | Hacker News Reader

Full thread

fadijob

We analyzed the leaked Delve audit reports and found some wild patterns:

- The same auditor license number (PAC-FIRM-LIC-47383) appears in 487 out of 494 reports

- Every Type II report has identical page numbers: Section 4 at page 30, tests at page 59, Section 5 at page 82

- 220+ "No exceptions noted" per report, across every single client

- The system descriptions were copy-pasted from each company's marketing website

We built tools to check this data:

- Search by company name to see if they're in the leaked database

- Paste any SOC 2 report text to scan for 10 template fingerprints

- A swipe game where you try to tell real audit excerpts from the fakes (harder than you'd think)

455 companies indexed, all free, no signup needed.

I'm also curious what the HN community thinks about the fingerprint detection approach, are there patterns we're missing?

View on HN

I'd find this more compelling if you looked at a few thousand Vanta or Drata reports grouped by auditor. You're going to find the same commonalities with only trivial language differences.

SOC2 reports are private between you and the auditor (that way if you "fail" you can just find another auditor or have a re-do, and no one is the wiser), and basically always gated behind a sales touchpoint (another hint about what utility they provide). I guess the Delve ones leaked which is why they can all be compared.

220 out of 494 "no exceptions" seems quite high to me. Nobody I've ever dealt with allows an exception to make its way into the report.

Genuinely curious: if you just need an independent audit report to check a box, do you really care how good a job the auditor did?

"You" probably don't, but it's not just "you". There's also the counterparty who's asking to see that report. Maybe they're doing it for paper-pushing purposes of their own, but ultimately, somewhere up the chain, there's someone thinking "I can't personally audit all my suppliers, and I can't be sure they're doing the right thing, so I'm going to ask them to get an independent audit".

Of course, this shows that the entire system is a bit of a charade, but the point is that someone cares and they're gonna be annoyed when they find out that the audit appears to be a sham.

Whether they have a good alternative is a separate question. But here's another way to look at it: if we show blatant disregard for self-regulation, the government is eventually going to show up and come up with more onerous rules.

> but the point is that someone cares

Is it true, though? Or has everyone just been psyched into asking for that certification out of a vague fear of "consequences" or of being left behind?

It's not either-or. Companies care about security because of the consequences. If you're a big company contracting a small one, you don't want to get owned through that vendor because you know you'll be the one holding the bag (data loss, reputational damage, regulatory scrutiny, lawsuits).

Small vendors will tell you what you want to hear because they're desperate for your business. Independent auditing is, in theory, a way to get closer to the ground truth. Well, in theory.

Probably not, in fact your auditors not being terribly thorough might be a selling point. But your clients, who are the ones asking for the box to be checked, might.

In my experience, clients don't dig deeply into the report or the auditor, they just want to see that you 1) have the report 2) it doesn’t have any egregious exceptions. Perhaps if this makes big enough news, that’ll change.

As the company? No. In fact, it's likely better for you if they do a bad job. You potentially get shielded from blame, but don't actually have to put in the work.

As a user/customer/potential victim? Yeah, you do.

The swipe game idea is new to me - you have internal testers or some team use that to go through it?

Most of this isn't that damning. SOC IIs are already highly templatized, so pages matching up really isn't meaningful. In fact, an overly detailed or overly verbose template is more likely to have matching pages since you'd never have to add additional content to it.

System descriptions don't necessarily hold much weight. They're often more about giving a general shape of the system to help orient the reader, rather than providing a technically complete picture.

Most of the meat in these is about the controls being tested (which are semi-standardized within an auditor) and the results. Many of these controls are really basic and easy to get "no exceptions noted".

That being said, nearly everyone has at least one exception, even if it's minor. The fact that they didn't find any across all of their clients is a strong indicator they're not diving deeply enough.

The no exceptions noted piece is kinda funny. Most SOC2 auditors at least put in the minimal effort of finding one person who didn’t do their cybersecurity training, so the report’s not total boiler plate.

Reply on news.ycombinator.com