Comment by rahimnathwani | Hacker News Reader

Comment by rahimnathwani | Hacker News Reader

You could do it step by step. First step is to get a decent amount of material together. Without that you will have nothing to analyse.

If the data is available as a bulk download, you're all set to move to the next step. If not, next best is if they offer an API. In that case, learn how to use Python or something to pull each document using that API, and storing it somewhere (either the filesystem, or in a database). If not, get the book 'Web Scraping with Python' and use that.

Once you have the stuff together, Udacity has a gentle introduction on cleaning document-ish data (using JSON/Python and Mongodb): https://www.udacity.com/course/data-wrangling-with-mongodb--...

Then the analysis starts. If it were me, I might start by splitting documents into chunks (paragraphs?), and classifying them somehow. Maybe use NLTK: http://www.nltk.org/book/ch06.html

Let us know how you get on :)

The first couple of steps are actually the easiest. The last paragraph -- which is also your shortest -- is the hardest part, by far.

OP, you want to basically extract entities from documents without necessarily knowing them a priori ("companies can use different language to describe these things"). This might be even tougher if companies aren't required to report material weaknesses (are they? I'm unsure). If they don't report, then you'd also likely need to manually code training data for companies for which you have filings.

If companies are required to report material weaknesses, then you can likely rely on NLP for this job. Start by building a classifier using material weakness data as your labels and tf-idf vectorized documents as your features (check out parent's link, and pick up a general NLP book while you're at it).

For each of the 50-80 things you mention in your post, OP, you'll need to repeat this: (1) obtain labels and (2) figure out a way to train a machine learning algorithm to handle that task for you. The alternative is to rely on heuristics (eg. "I know from past experience that companies using such-and-such word in filings experienced X, so I'll encode that in business rules"), which is an ugly way to build something but likely faster than learning about machine learning or NLP.

YC-funded Kimono Labs might have the SEC API covered:

http://blog.kimonolabs.com/2015/04/27/unlock-sec-data-sec-ap...

Reply on news.ycombinator.com