Hacker News Reader

Comment by chollida1 | Hacker News Reader

Many hedge funds have started to consume this type of data.

There is a piece in WSJ that discusses how one fund, two sigma, uses cell phone tracking data, as well as many other sources of data to build trading signals.

http://www.wsj.com/articles/how-computers-trawl-a-sea-of-dat...

In fact, I think the biggest funds are putting more effort into this type of big data exploration from funds than they are into trying to glean more information out of the time series data provided by the exchanges data feeds.

10 years ago, being able to scrape the web was a competitive advantage for funds, 5 years ago it was real time sentiment analysis of news reports.

Today, its being able to consume 100's of disparate data feeds and build alpha generating signals from it.

View on HN

Are they tracking the CEOs of companies in which they invest, or what?

Tracking A, B, and C-level executives should be very profitable. Who meets with whom is valuable info. Especially when you also have their phone call metadata. The activity in advance of a merger should be quite visible.

Tracking elected officials should reveal who influences whom, and who's bribing whom. It may be possible to detect bribery to the level of establishing probable cause for an investigation in that way.

No, it's more about understanding consumer sentiment about a product category or company based on mining consumer information in aggregate. For example, a hedge fund could know how many people went inside a Wal-Mart over the last few months based on cellular location data. If there's an unfavorable downturn in store traffic, the hedge fund can shift its investments based on information that Wal-Mart won't report for another few months.

> Tracking elected officials should reveal who influences whom, and who's bribing whom. It may be possible to detect bribery to the level of establishing probable cause for an investigation in that way.

The way political fundraising is done in the US, most bribes look like legitimate campaign finance money. In some states, it's even legal for a candidate to simply redirect whatever is in his campaign fund directly into his own bank account after the election. And even in states where that is illegal, 501(c)4 / SuperPACs allow politicians to do whatever they want with the money with no oversight.

With laws like that, why even bother to investigate bribery?

>> Are they tracking the CEOs of companies in which they invest, or what?

> No ...

How could you know that? I doubt anyone would reveal it, and I don't see why they wouldn't - for the unscrupulous (there are plenty of them), it could be very valuable. Remember the story about Uber tracking journalists, for example.

They would track CEOs, if only they could. I'm guessing that even in the US, mobile networks only share aggregate or anonymized data about customer location, and don't allow for spying on specific individuals.

If you want to track CEOs, I'm guessing it would be much easier to track their private jets or the license plates of company cars.

they MAY only share aggregate but what's stopping, for example, VZW using the data to their own means? competitors use their towers, potential acquisitions, market/political threats or other interesting entities. Laws and morals? Hahaha!

Most telecoms won't do this because the potential reward simply isn't worth the liability they would have for insider trading (if they get caught spying on corporate execs, then the SEC will assume they spy on every corporate exec). In fact, most big telecoms have a secret "VIP" program that flags accounts of especially wealthy, influential or famous people and restricts access to them unless you're on the "VIP" customer service team.

I'm sure hedge funds track CEOs, but they're not using data from Verizon to do it.

They calculate risk:reward and do whatever nets the most profitable return. Fines and fees are only anathema when the costs exceed the gains gotten from illicit behavior. Otherwise, paying fines without admitting guilt is the corporate way.

There are better ways to track CEOs than through mass data collection -- targeted data collection (aka hiring a private investigator) works much better. Though most CEOs of big companies have security details that keep the wolves at bay.

http://www.cnbc.com/id/38722872

satellite recon of walmart parking lot usage, to estimate business trends

Yeah, I was using that as an example -- but I'm sure you can think of other things you could measure using bulk cell phone data that would give you insight into the future financial performance of a company -- often before the company itself knows.

aren't the companies employing wireless trackers in store? even small shops can get some useful tracking data from "custom" OpenWRT based firmwares on consumer-level wifi/router devices.

> aren't the companies employing wireless trackers in store

Yes. Lookup "beacons"

>In some states, it's even legal for a candidate to simply redirect whatever is in his campaign fund directly into his own bank account after the election

This doesn't sound right. Any sources?

This might have been mentioned on John Oliver - the episode about oil in North Dakota

I thought that kind of tracking was illegal:

https://www.ftc.gov/news-events/press-releases/2015/04/retai...

maybe it's ok as long as it's anonymous.

my initial thought was how many teens are patronizing $MALL_STORE or how many people are stopping at Chevron gas stations versus Shell.

at least that's more demographic/general, in my opinion. what you suggest is bordering "creepy".

Peter Lynch, of Magellan fame, wrote that he'd hang out in shopping malls looking at store traffic.

Just bordering? Seems full on creepy to me.

I'm not privy to what Two Sigma do, but I'm guessing all the tracking data they have is anonymized, did I just make up a word?, and of a more general trend variety.

What. The. F*ck?

People are being tracked for trading advantage and that is not at all illegal? Not inside trading at all? How the hell did we get _here_?

It's not "inside" trading if it's data available in the marketplace.

People are being tracked for trading advantage and that is not at all illegal?

Aggregate data. So it's supposedly anonymized.

As in "We tasked one of our interns, the one with a Russian literature degree, to anonymize the DB ... two hours before we let the clients access it".

> Today, its being able to consume 100's of disparate data feeds and build alpha generating signals from it.

How do they validate that this has any connection to reality? Or those who had budgets to spend doesn't ask such silly questions?

There are a number of traditional statistical methods for validation that can be used to answer this type of question. These sorts of firms have some of the largest private clusters in the world, they are certainly capable to and do answer this question before trading.

Sure. I could answer to this meme by another meme - most of statistics based studies are wrong in all possible way - biased data set, misapplied methods, wishful thinking, untestifiability.

So what's your point? Sure, you can mess up your test -- does that mean you shouldn't trade? Does that mean you shouldn't run validation tests at all?

My point is that there are more faith and wishful thinking than science in all these "models". Moreover, the very nature of the markets are too complex to model, especially using such data sources. The result would be some naive description of reality, like pre-scientific religious cosmology, a wrong but sophisticated enough 'model' based on 'current consensus'.)

Have you worked in a variety of quant funds? What makes you think markets are too complex to model? What evidence do you have that the models used at successful shops are not sufficiently rigorous?

Because all one needs is to bullshit one's boss or sponsor. I gave an example of a religion for a reason. And bullshitting is much more efficient strategy than actual modeling and rigorous research. Group thinking is much easier and cheaper than proper science, especially when even the best economists are full of doubts.

I took Yale's financial markets courses, so I know what ideas they might have. This is the Nobel laureate's course.

The bosses in the typical quant firm are also quants themselves, and have no trouble following the rigor used to justify the validity of a trading strategy. Sure, at a certain point in the chain, you're talking with someone (eg. COO) who isn't allowed to see the details of the modeling procedure, but this is the point of scaling up and having risk/pnl limits -- no HFT firm, for example, is at risk of blowing out from lack of statistical rigor.

I doubt taking Yale's financial markets courses is sufficient to understand or be exposed to what is currently being used in quantitative finance. I took Stanford's financial markets and statistics classes, and those are far behind and unaware of what's currently being used in the industry (I've worked in two highly successful HFT groups since Stanford).

I find this hard to believe: "consume 100's of disparate data feeds and build alpha generating signals from it". Can you point to an example?

Well there are 40 different exchanges with which to trade in the US alone, Canada adds another 10.

So there are 50 different signals before you even start up your servers for the day.

Since there is no one true currency market most HFT funds make their own currency signal from anywhere from 5 - 20 different feeds, lets call it 10 to bring our list of data feeds to 60 before we get into any futures and bond markets or to Europe and Asia.

Consuming the above mentioned brings us close to 100 already.

Add in feeds from

- google trends

- analytic s about consumer trends from say 5 different providers each covering say 5 demographics across 5 sectors

- twitter data feeds aplenty, following 2-300 different hash tags representing stock #APPL

- twitter following for news feeds, like AP, Reuters, etc to do sentament analysis.

- don't forget your machine readable news feeds

- now lets get into government data. FRED has 100's of data sets to parse to use as model inputs. See: https://research.stlouisfed.org/fred2/

What are we up to, 200-300 different signals and we haven't really broken a sweat yet.

What sort of source would you like me to provide other than what I see on my computer screen and do for a living?

I know of one company that takes photos of retail parking lots using nanosats and then sells that data to hedge funds. They're trying to put up more satellites so they can get even more photos per unit time.

Reply on news.ycombinator.com