SEC Discloses Edgar Corporate Filing System Was Hacked in 2016 | Hacker News Reader

SEC Discloses Edgar Corporate Filing System Was Hacked in 2016

wsj.com

250 pointsk2enemy57 comments

Sympathy for the SEC on this one. For its faults, EDGAR is an elegant, useful tool for disseminating a massive amount of information. In an effort to be robust in the service of its presumably thousands of poorly-configured clients (academia, hobbyists, day traders, etc.) it's probably trivial to mistake a hostile act for poor programming practices. We've used this system successfully to fuel tons of projects and only on a few occasions been rate-limited despite numerous logic breakdowns, fork bombs, etc. resulting in traffic spikes that would have been easily mistaken by a smaller organization to be DoS attacks. While it's easy to say they should have been more vigilant, they have probably been deliberately permissive in the service of their mandate. Thanks to their patience, our projects have been served. That sort of patience probably leads to overlooking something like this.

In what ways is EDGAR elegant?

Well, for one, it's 34 years old and is simple enough for non-technical people to use but robust enough for institutional use. For another, it's scaled to accommodate 3 decades of exponential growth in public filings and regulation. Did I mention it's 34 years old?

I'm not sure about exponential growth in public filings (and EDGAR does not track regulation, so it's not relevant). For example, the number of publicly traded companies is down about 70% compared to twenty years ago: https://fred.stlouisfed.org/series/DDOM01USA644NWDB

Could you share a link showing exponential increase in the use of EDGAR?

Sure. Although you should be aware that not only public companies are required to make filings with the SEC.

Index - Q1 1994 - 3mb - https://www.sec.gov/Archives/edgar/full-index/1994/QTR1/form...

Index - Q1 2017 - 45mb - https://www.sec.gov/Archives/edgar/full-index/2017/QTR1/form...

File sizes are also increasing because filers are adding structured data[1] to their SEC filings. Also, most SEC Filings are now verbose HTML files instead of plain-text files.

[1] https://www.sec.gov/structureddata/what-is-structured-data

And not really exponential. Something interesting happened 2003-2004 though. We can perhaps observe lead up to dot-bomb (1996-1999 growth) and the 2008 financial crisis.

http://hpics.li/2e78909

    1994,3153191
    1995,4813587
    1996,7539137
    1997,13755958
    1998,16107028
    1999,15935643
    2000,17548021
    2001,16873202
    2002,18904001
    2003,27723307
    2004,47116841
    2005,47982675
    2006,50672891
    2007,51322342
    2008,49635521
    2009,45312542
    2010,45384720
    2011,46455461
    2012,46733150
    2013,45842099
    2014,47065954
    2015,48112988
    2016,46636963
    2017,47037415

Nice breakdown! The SEC introduced some new filing requirements[1] in 2003 and 2004 that may have contributed to this.

[1] https://www.sec.gov/info/edgar/regoverview.htm

It's an index - a '\n'-delimited list of strings indicating the locations of filings. The size of those filings has nothing to do with the size of the index.

Additional regulation increases the number of filings that must be made.

While there may be fewer companies, are there also fewer stock transactions, etc? This system tracks a lot of different types of transactions involving public companies.

EDGAR does not only contain public company filings.

EDGAR is admirable in many ways, but it's age is really showing, especially from a UI and UX perspective. I really wish the government would spend some effort in modernizing that aspect of it. BamSEC.com charges $35/month for essentially just a prettier UI, and it's so much more pleasant to use. Honestly, it's a pity the government never does M&A -- they should just buy BamSEC.

The public filings themselves are also a nightmare of hard to follow formatting.

> EDGAR is admirable in many ways, but it's age is really showing, especially from a UI and UX perspective

I like the bare-to-the-bones simplicity. In contrast to "better designed" portals of some other countries, I can usually point even the most technically-illiterate executive to the filing portal and expect to get something useful out the other end.

Disclaimer: I am not a lawyer. Please consult a securities lawyer before preparing and/or filing anything with the SEC.

I love EDGAR. I use it daily, and agree with the benefits of simplicity in its UX/UI. It loads quickly and you can get the info you need in seconds if you know where to look.

Probably the product of (1) multiple changes in "best practices" data formats over 30 years (XML seems to be the format de jure), (2) optimizing for the filer and (3) diversity of filing data. Agreed it would be ideal if they were all similarly structured, but they have another set of stakeholders on whom they are imposing essentially a tax -- the filers themselves -- who have a legitimate gripe any time the SEC imposes a switching cost. It's easy to dismiss those gripes, but the vast majority of these filers aren't GOOG-sized. Many are very small, and those costs can be non-trivial to them.

Edit: Worth mentioning here that EDGAR's ease of use makes sites like BamSEC simple to create. Probably EDGAR's time is best spent focusing on the institutional intermediary. That allows competition to iterate on the best UX/UI for browsing these filings without imposing the cost on the filers themselves as discussed above.

It's almost as if you are unaware of the website you are posting this comment on.

Thanks for mentioning BamSEC.com. It looks way easier to use, even if it is a paid service.

Try out Last10K.com for SEC Filings. Admittedly it's not as "pretty" but it's free and includes some unique features including sentiment analysis.

Disclosure: Last10K.com is a side project of mine

1. Easy to crawl i.e. simple HTML pages

2. Multiple open data formats, including TXT

3. Easy-to-follow organization (CIK), minimal indirection

4. No Javascript required, no gratuitous frameworks*

5. Bulk downloads of quarterly data (FTP up until 2016)

*not to mention gratuitous iframes (BamSEC)

Thanks to #5 anyone not satisified with the "UI/UX" can take the data and go build their own "UI".

When data is tied up in layer upon layer of indirection, Javascript, iframes, enormous overstuffed URLs, binary formats (e.g. PDF, XLS), then others cannot as easily take the data and build their own interfaces.

Not a good approach for public data nor good use of public expenditure, IMO.

I am grateful to Carl Malamud for creating EDGAR.

Do you know where it's possible to get a list of CIK #s associated with the company names. There used to be a .c file available on the sec site, but I can no longer find it.

What kind of projects?

Check #3 & #4 on his submissions.

Btw, hackers accessing this info could possibly explain something I tweeted to them about 10 days ago:

https://twitter.com/jeffallen6767/status/907171360769662976

Good catch. Spikes in option activity before the release of non-public information is the clearest indicator of insider trading. If it helps, its also trivially easy for the SEC to identify the traders involved so at the very least they are already looking for these guys to trace where they got the information.

Was the SEC notified about the Equifax hack before the public? With enough time for someone to trade on that information?

Yes. Regulation FD says you have to tell everyone material information at the same time.

Right so hacking Edgar wouldn't be a great source of non public information then no?

It would be because it is where the information goes before it goes public. Sane with a printshop.

Breaching EDGAR and getting away with trading on the information -- and we are learning about this now?!

To be fair, I've used EDGAR and it is <cough> very legacy. So no question it was going to be completely compromised.

I'd say the mistake was putting non-public information on it in the first place. The risk assessment for private data exposure was extreme.

Edgar is almost as outdated as the TESS system for patents. I'm surprised someone hasn't "hacked" it long ago. Unfortunately, due to Sarbanes Oxley a lot of that infrination is required to be public.

Shouldn’t that be “Fortunately...” ?

That really depends on how you feel about the disclosure requirement.

You're gonna have to explain why one would feel negatively about public disclosure.

For one it raises the disincentive to go public, likely causing companies to stay private for longer.

why should someone have to explain that simply because you haven't thought about the existence of a possible other perspective.

the united states has protected trade secrets by law far longer than its been on the transparency train.

there are reasons for that, and the corporate disclosure requirements has had no effect on creating a "fair market"

I was just asking for an opinion from someone in that camp.

Your answer does not give faith in your business practices, as a client or as an investor. I suggest you work on communication, even with strangers on the internet.

Congress should look at the SEC obligations for disclosure (the delay seems wrong given the organization's mission of transparency... the SEC would sanction a public company for a similar failure to disclose) and why the SEC decided to wait a year to inform the public. Not disclosing the hack back in August 2016 strikes me as a political decision by the Obama Administration to avoid criticism during the home stretch of the election.

If it was intentionally covered up by Obama's administration, why did Trump's administration wait 9 months to make it public? Not everything has to be politically motivated.

How come SEC only discloses this now? Isn't preventing insider trading a goal of financial regulations? If so, is there a reason SEC didn't disclose it as soon as they gained knowledge of the leak?

I think this goes to show that any data collection is dangerous, even if government thinks they're the good guys (when in fact they're naively collecting data that could be weaponized against them very easily). The EU is on to it as well with MiFID II granular reporting of financial market transactions starting effectively next year.

Though I'm not sure the Brits are going to play along (where the majority of trading happens in Europe); after all, why should they go through the trouble of implementing MiFID/MiFIR when leaving EU anyway?

Edit: preventing insider trading

Well, just because this was disclosed to the public now doesn't mean it wasn't previously disclosed to Treasury, FINRA, the US Equity SROs, and the ESMA. The surveillance systems surrounding almost any market center are substantial, and can seek the sort of behavior having inside information might appear as.

That makes sense, but do we have info on whether that actually happened? Who's overseeing the overseers? The only theory under which it would make sense to withhold that info I can think of are criminal investigation tactics, but there's no actual info on suspect transactions, is there?

Broadly and vaguely trusting in surveillance isn't convincing at all. The usefulness of surveillance must be weighed and re-evaluated against risks, as demonstrated by this incident.

I guess by surveillance I was speaking purely transactional surveillance, and typically investigations into transactional behavior takes time, and under certain circumstances would result in criminal prosecution (on top of any regulatory fines/punishment).

It also takes time to coordinate investigating multiple different asset class transactions, in the event, say, equity derivatives are used.

And no, we don’t have info on what happened and what was taken, and certainly nothing about any regulatory investigations.

"The U.S. Department of Homeland Security detected five “critical” cyber security weaknesses on the Securities and Exchange Commission’s computers as of January"

"The report’s findings raise fresh questions about a 2016 cyber breach into the SEC’s corporate filing system known as “EDGAR.”"

"... it shows that even after the SEC says it patched “promptly” the software vulnerability after the 2016 hack, critical vulnerabilities still plagued the regulator’s systems."

http://www.reuters.com/article/us-sec-cyber-weaknesses-exclu...

I thought EDGAR was only used to disclose public securities filings. What's the big deal?

They likely upload the filings some time before they are released.

I believe IPO applications are also handled there.

I can't see this article as its paywalled. I believe this is the actual statement from the SEC:

https://www.sec.gov/news/public-statement/statement-clayton-...

Thanks. Here is a mirror of the article: http://archive.is/xsniG

Click "web" under the article title above

As much as the UI appears dated, Edgar is consistently one of the fastest websites I use on a daily basis. Punch in a ticker under "Fast Search" here (https://www.sec.gov/edgar/searchedgar/companysearch.html) and click around the filings for a bit to see what I mean.

I wonder if the hackers were from the industry or outsiders who just took advantage of the opportunity.

Probably both. That's a pretty good-sized arbitrage opportunity.

Think about the number of people in the financial world who are looking to get a percentage of a second advantage with high-speed trading. Hard to believe they'd pass this up if they knew about it.

Here's a non-paywalled version:

https://www.nytimes.com/aponline/2017/09/21/us/ap-us-sec-cyb...

> The Securities and Exchange Commission says its corporate filing system was hacked last year and the intruders may have used the nonpublic information they obtained to profit illegally.

This is the most problematic part.

Is EDGAR data publicly available, as in, can I download the archive for a year?

Yes

Facebook Proxy Past Paywall https://www.fullwsj.com/articles/sec-discloses-edgar-corpora...

paywall workaround:

https://www.facebook.com/l.php?u=https://www.wsj.com/article...

Reply on news.ycombinator.com