Hacker News Reader

Comment by AznHisoka | Hacker News Reader

I used to work at Standard and Poors. They do media monitoring for lots of financial companies.

When I joined, I thought they using state of the art tools to monitor millions of sources.

Turns out they just had a team of thousands in India that manually visited certain websites to check for press releases, transcripts, earnings reports, etc and push them to a database if they found them.

View on HN

Related. I know a company that does analysis for soccer games. I thought it was state of the art too until they told me they just hired a bunch of people from China that logged every move (distance ran, kicks, etc) from eveyone and also put it in the database.

go on...

Don't know about the China angle but the life of a soccer analyst isn't all it's made out to be:

- https://www.pastemagazine.com/articles/2017/01/the-secret-so...

There's plenty of other blogs / articles out there now that people are trying to "Moneyball" soccer a bit more. Like anything that has a tinge of glamour and a lot of money around it, you have to take poor earnings to "pay your dues" in the trenches. Same as other aspects of pro sports, entertainment, etc. Hopefully something motivates you other than money.

I was curious about the logging system for that kind of data. Now that I know the name of the position I made a search and found that most systems use people to input the data. Thank you.

Well, that makes sense, doesn't it? Back in "the day" when certain US companies were trying to convert all the Whitepages and Yellowpages nation-wide into CD-ROMs, they'd ship phonebooks from each zip-code to China to have them triple entered (error correction), and then that's where the nation-wide CD-ROM phone-books came from. This is the same deal, no?

this somehow made me laugh so hard. made my day~

laugh, and then cry

@AznHisoka I would like to hear more about this! This is exactly a problem we are trying to solve (http://evolution.ai)

One cannot even imagine how many people in the world have a boring job of looking at word document and C&P some of the data into excel spreadsheet. Other half have a job of looking at some document and effectively making Yes/No decisions all day.

Disclaimer, I am the CTO

Sounds similar to Scale (https://www.scaleapi.com)

I'm sure they'd rather no job than a dull job... Oh wait.

wonder if that could be crawled and save money.

Accuracy. Every solution I've seen that relies on automatic crawling will eventually have a parsing error when someone changes their sentence structure of a press release.

It's not so obvious when you're looking at the breaking releases for a few stocks or companies, but historical records have at least 1 error per stock per year.

So split your stream:

    1. Data matching expectations (you do have a definition of correct, right?)

    2. Log for manual review -> manual inserts or correction and placed into queue for (1)

Monitor (2). When inserts start trending up, it may be time to update your processing logic.

I came up with a similar idea for a company several years ago where we had a team of people doing data entry from faxed documents. I wanted to build something that would do all the OCR it could and then display it to users to verify, which should have been a 10 times efficiency increase, not to mention speed and accuracy.

The idea was rejected, they wanted either a perfect solution or nothing. I don't know why, but for some reason the idea computers removing humans is acceptable to management, but computers augmenting humans wasn't.

right, it depends on what the reward for high precision vs high recall is

Would writing and maintaining the crawler be less expensive than paying the small army of third world employees to manually check pages all day?

The short answer is that it would depend on the value of any precision loss that occurred and whether or not the shift would disrupt any other systems, be they social or technical, within the company. But with that in mind, there's also the issue of whether or not the money saved would be worth it. It's very possible that S&P simply has enough money to play with that any savings from swapping to an automated solution for media monitoring aren't worth the potential disruption to the company workflow.

or just set google alerts.

financial companies usually use the worst formats to extract info from.

Woah there cowboy

Humans are the cheapest machines.

Reply on news.ycombinator.com