High Frequency Trading Tech For Non-Financial Problems. | Hacker News Reader

High Frequency Trading Tech For Non-Financial Problems.

blogs.wsj.com

38 pointsaditiyaa118 comments

The area is also called stream computing. I work on a project in this area, which made HN a bit ago: http://news.ycombinator.com/item?id=2442977

Note that the above article erroneously ties the Streams project with the Watson project - aside from coming from the same research lab, we have nothing to do with each other. My comment on Streams from that thread:

Why I find working on Streams exciting: we're designing a language and runtime system for a new programming model. Not just a new language (which we do have), but a new programming model. The way you write programs changes when you have essentially infinite streams of data coming through your system - you have to think in terms of operators that transform or filter individual pieces, and you have to keep in mind its inherent distributed nature. That is, you may write a chain of operators to process your data, but each operator can run in parallel with the others, even though you will probably design your application thinking of one particular piece of data marching through the system. The runtime system itself is, of course, distributed and fast. To get an overview of this programming model, take a look at this paper: "SPADE: The System S Declarative Stream Processing Engine": http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.160.... Just keep in mind two things: the newest iteration of the language is called Streams Processing Language (SPL), and that it is a complete rewrite of Spade.

If you're interested in getting a feel for the programming model, here is a public getting started guide: http://publib.boulder.ibm.com/infocenter/streams/v2r0/index....

And the language spec: http://publib.boulder.ibm.com/infocenter/streams/v2r0/index....

I suppose I should put in the disclaimer that these are my views, and I do not represent IBM.

I'm all for burning less fuel and making shipping more efficient, but I think it is a bit of a stretch to say this particular use case is not related to finance. Ports becoming more efficient ultimately saves oil, natural gas, and other commodities companies money. Commodities traders are very tuned into port congestion and global movement of ships and make lots of money trading upon the flow of commodities (mostly oil/gas).

If this particular case is not related to finance, then how come the Bloomberg terminal tells me there are exactly 401 ships in Rotterdam port as of this morning, the 5 day moving average is creeping up, and 8 of the top 10 ships by tonnage are crude oil tankers?

In a global economy, everything is a finite number of steps away from finance. That does not make everything in service of finance.

Finance is just a way of doing resource allocation in a quantifiable way. The ideal scenario is that the cheapest option is also the most efficient (or vice versa, depending on how you think about it). The finance industry is a highly abstract way of finding the methods for creating the most wealth (for any given definition) while consuming the fewest resources. That's called "return on investment".

I don't disagree with what you said. My comment was in response to what I saw as the claim that just because something impacts the financial industry, it's related to the financial industry in a meaningful way. That reasoning could be applied to just about everything, which means it's a meaningless distinction.

Believe me, I don't disagree. I'm just talking about the example of ports/shipping because of the huge amount of money moved around in commodity trading. If the example was applying this to say, trunk congestion/scheduling on the nation's highways, it would be a completely different story since traders don't care about (i.e. trade on) that information.

Well, I think the distinction is that the techniques were originally designed to improve the response to the state of, for example, shipping in Rotterdam. That is, the traders didn't particularly care if the shipping patterns were the most efficient possible, but that they were making the best trades based on observed patterns.

I believe the point of the article is that now these same techniques are being used to instruct the markets they were originally designed to simply monitor.

I guess what got me thinking about it is because physical oil/gas traders have demurrage clauses in the contracts which kick in when a tanker is unable to offload its cargo by a certain time. This translates port congestion directly into hard cash that has to change hands, so for this specific problem that class of traders do very much care about the efficiency of the shipping patterns.

Idly I've wondered if there was a way to turn the efficient markets model on it's head. Rather than gathering a ton of predictive information to try and get another 2% efficiency, instead use the already efficient market activity to predict other things. For instance if oil prices drop GM should start ordering more V8 engine parts instead of I4. I'm sure this already happens at some level.

Yes. The Economist often quotes the prices of insuring against default of a creditor (like Greece) as an indicator of how likely they are to go bust.

Nice to see all the resources invested by the market makers in savage zero-sum competition having some actual use.

You know, if you think in terms of the technologies they fund the development of that eventually everyone can use, the three most virtuous industries are banking, porn and the military.

You have to look at the opportunity costs, though. Lots of things get invented for the military in (or in preparation for) war. But more things would probably get invented in peacetime, just not as concentrated on one sector.

Please don't spout vitriol when you don't understand what you're talking about. Trading is obviously zero-sum in wealth, but the important thing is that it is not zero-sum in utility. That is the fundamental reason that people trade: to transfer risk. American Airlines does this every time they want to hedge against the price of oil. The fact that there are competitive market makers makes this cheaper for the airlines to do, and that savings in cost is transferred to you every time you sit on an airplane, go to the gas pump, buy orange juice or wheat products, buy a car, or almost anything else.

The fact that market makers don't service you directly by selling you an apple does not mean that what they do is useless.

I absolutely agree that trading is socially useful, and markets would be much less liquid if we didn't have big companies ready to stand as counter-party to whatever buying and selling smaller market players want to do. It just makes me a bit sad that the profits that rightfully accrue to the sector are then seemingly wasted on HFT infrastructure, a competition that doesn't seem to provide anything useful above the standard market-making role.

* It just makes me a bit sad that the profits that rightfully accrue to the sector are then seemingly wasted on HFT infrastructure, a competition that doesn't seem to provide anything useful above the standard market-making role.

I think that's the point though: the immediate execution that liquidity offers you is the price you pay to hedge risk. The other option is for these companies to set up their own trading desks. For instance, suppose an airline wanted to buy oil at a price it wanted. If the market is very illiquid, they're less willing to cross the bid/ask spread to hit whatever wide offer someone else is posting. The longer they have to wait, the more risk they're at. The "standard market-making role" solves this problem exactly. It allows them to hedge their risk at lesser cost. That is a valuable service, and that is why market makeres exist.

Oh really.

"Vitriol" in the rose-shaded eye of the beholder aside, methinks absent the WSJ pom-poms we will agree that a much more fundamental reason that people trade is: to make money without having to worry about consequences; and that particularly in light of events over the last decade, the fact that highly-interdependent, dubiously competitive "markets" have sprouted in many sectors means that the costs of mistakes, incompetence, fraud, theft, etc is transferred to you every time you sit on an airplane, go to the gas pump, buy orange juice, blah blah.

As such, one supposes that dependence on many of those markets indeed resulted, as you say, in something other than zero-sum in terms of utility - they resulted in huge losses for some and huge inefficiencies resulting from unquestioning confidence in the supposed "efficiency" and beneficence of "markets".

http://en.wikipedia.org/wiki/Externality

I will absolutely agree that traders want money. I may not be motivated by societal welfare, but it's naive to think most people are. Founders start companies because they want to hit it big. Sure, they do want to change the world, and I genuinely believe that, but I think if the huge upside potential wasn't there, the startup industry wouldn't be as competitive as it is.

Can you be more specific about how these markets transfer the costs of mistakes to you? I'm not entirely disagreeing, I just don't know of any specific examples (although that may be a symptom of my ignorance).

The fact that markets cause some to have huge losses doesn't particularly bother me. I'm talking specifically about US equities, futures, and options markets which, as far as I can tell, are extremely liquid and competitive. I don't see any negative externalities caused by HFT market makers in these markets, but again, if you have examples, I'm glad to be corrected.

You also cite "huge inefficiencies" from "markets". Maybe we're talking about different markets, but I don't see these inefficiencies in the markets I mentioned. I know nothing about derivatives or anything other than HFT, so maybe that's what you're talking about.

Reply on news.ycombinator.com