HFT in my backyard – II | Hacker News Reader

HFT in my backyard – II

sniperinmahwah.wordpress.com

197 pointsliotier65 comments

HFT profits have fallen, and will probably continue to do so. If at one point some of the hardware becomes superfluous because it is too expensive to maintain, what is a good use of a low-latency 100MBit link? Tele-medicine?

People are lamenting the ill-directed use of capital to build all this stuff, but maybe that becomes one day a valuable by-product.

The towers have been repurposed once and the same can be done again. Use them for BASE jumping as one in the article already is.

But think of the network equipment--how many advancements in the Ethernet, TCP, and UDP that we all know and love have been partially funded or prompted by people willing to pay to ride the bleeding edge? Several years ago, UDP multicast in Linux suffered a very serious performance regression. Who pushed to get it fixed? An HFT firm. Now everyone has well-performing UDP multicast in Linux.

Or how about Arista Networks? Back when they were called Arastra they got their start largely by selling US $25K switches to HFT folks. Now they're selling for "the cloud," and lots of people can benefit from higher performance networking.

it's almost as if capitalism works...

Indeed. If you look at the advances in technology, HFT industry is one of very few that can afford to engage in long-payoff expensive projects. For example, things that we've worked on (balloons) are similar to what military has been trying to do for 20+ years, and what Google is doing with Project Loon. see our PR: http://www.prnewswire.com/news-releases/innovator-windy-appl...

Anything related to wireless modulation/demodulation also has implications for 5G networks and satellite broadcasting (most of 'delay' of satellite video signal actually is from encoding/decoding buffering, and is reducible).

Does anyone know why it makes sense to make the base of the tower diminish to a point?

Seems like an odd thing to do unless there was a good reason for doing it!

I'm not an engineer, but it seems like having a single pivot at the bottom of a large mast would be favourable -- it'd allow the mast to freely flex in winds while being constrained by the support cables without causing damage, whereas a small base might just crumple if flexed too far.

A rigid base would have to be extremely strong to supply the promised rigidity, small base means big force to supply the turning moment. You can think in terms of it being a Dirichlet (fixed but flexible) vs. Cauchy (fixed and unflexible) boundary condition. Not to mention it being unnecessary; the only advantage I can think of is that it might exclude some undesirable oscillation modes, which is probably easier to achieve by attaching support cables at various points along its length.

Are you sure you didn't study structural engineering while sleepwalking ? :-)

Just playing lots of Pontifex [1] ;)

(ok, and a physics degree, but that's less relevant)

[1] http://www.chroniclogic.com/pontifex.htm

That is to allow the tower to be flexible to wind induced vibration. If it would have been a standard three legged foundation then the constant vibration would have introduced cracks leading to unnecessary stiffening.

I wonder if they take the extra time and latency to encrypt their traffic.

Is it possible to intercept a message by flying a drone in the path of the microwaves?

Interception is one thing, but I'm pretty sure that covertly interfering with the link (injecting data both ways, for a long-ish time) would be pretty interesting...

Yes, it could be encrypted using low-latency HW implementations. However, I'd be surprised if these actually perform all the necessary steps to establish a truly secure link (like use message authentication besides encryption, and such).

There's no point in encrypting the traffic. It's publicly-available data. The purpose of the towers is merely to ensure that their owners receive the data a couple of milliseconds before everyone else.

Then how for they stop free riders?

You could probably do the encryption very cheaply from a latency perspective (use AES to generate a key stream that you XOR with the output data, which could easily fit on a sending/receiving FGPA without introducing any significant delay), but in general I would think that it is far more trouble than it is worth.

All these features would require testing, and availability is pretty important for this kind of stuff.

The encryption would take too long.

I don't think the data is going to be very useful. Your interception would block the signal, so this would be a very temporary exercise.

They are using the towers to transmit information from the exchanges (already publicly available) and their own transactions (soon to be publicly available).

One use for this information might be to beat the low-latency masters at their own game, and act on their trades before they arrive.

It's theoretically possible you could design a system which would relay exchange data from the drone at such low latency you could compete with or beat other data feeds for a given location. That would be a valuable commodity, but hard to capitalise on in the short period of time between the drone blocking the signal and the data being re-routed.

It sounds as if anyone with the capability to use the intercepted data would do better using their efforts elsewhere.

I would be pretty simple. You use a high-latency channel with "proper" encryption and authentication to stream a one-time pad key; then encryption and decryption would be a simple XOR, plus a bit of synchronization (to know where in the key stream you are). Doesn't sound like it would introduce much latency.

One-time pads are only useful if they're not re-used.

Or are you suggesting that the high-latency link is higher throughput, and provides constant gigabytes of pad material? (Eg, stream cipher)

A microwave link is much lower bandwidth than the optical fiber that already exists to connect the two points (you can't rely solely on microwave, you need fiber to back it up when it rains or is cloudy, etc.).

Also, depending on the market, you're only trading 8 or so hours a day. You can just send the pad during the 16 hours of the day while the link is otherwise unused.

There is already a second fibre link in these setups that has higher throughput and not much more latency.

That method is missing any sort of message authentication. Knowing the framing, one could easily flip a buy to a sell, etc.

I don't think the one-time pad is necessary here, since the aspect that makes it low-latency is reducing the critical path impact to an XOR at each end. I believe there are various existing encryption schemes which already have this property.

Legal problems and technical problems aside, what are you going to gain by intercepting the traffic? Most of the traffic is going to be market data and order data which are public information (well, order data is public once it hits the exchange). You can take advantage of knowing about order data before it hits the exchange but then you'd have to transmit that information extremely fast, which using a drone (and against those huge radio tower) is going to be pretty much impossible.

You could intercept and then re-emit false market data, then trade against whatever position this puts the users of that data in.

Whether you could do this, and do it without being noticed, I'm not so sure.

You're going to use ... what ... to re-remit false market data? A model airplane, compared to a giant tower designed to send microwaves over vast distances. Sure, that sounds possible. As for intercepting the signal for profit ... what are you going to do with it? The microwaves are already the shortest path.

I agree, it would be technically difficult.

My point is that there is a mechanism by which you could profit: false market data. Whether you could remain undetected and whether this could actually be implemented technically, legally or logistically seems uncertain.

I like the drone idea, but it wouldn't have enough power... Unless it was stationed right by the receiver, where it wouldn't need too much?

This is the secret radio tower which the government used to intercept thousands of trunk phone lines running through Britain to the Republic of Ireland.

http://www.lamont.me.uk/capenhurst/original.html

Fantastic article, even better than the first. This is exactly what I enjoy discovering on HN.

I hope he suffers no repercussions with respect to his trespassing, although it seems the worker who left the tower open and unattended might have some explaining to do. You think since another tower had been attacked earlier this year they might have installed security cameras. I wonder why these towers seem to lack security given their importance to their owners.

I know, HFT is insanely competitive in developed markets and there is no way an amateur can enter this segment but what about emerging markets like that of India. Is it possible to exploit HFT without expensive infrastructure, or million dollar code?

If you're not competing with anyone then sure. The algorithms can be found in economics textbooks, the implementation isn't hard if you don't need insanely low latency. The trouble is that if you get your bids in slower than everyone else (with penny spreads there is no way to compete on price) then no-one's going to trade with you.

In India the tick sizes tend to be 1-5 paise << 1 US penny. Over there it's actually a price competition game rather than a speed game.

http://asiaetrading.com/resource/securities-tick-sizes/

Incidentally, for anyone who wants to stop HFT, all you need to do is copy the Indian tick size.

http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html

I think you are oversimplifying. In the US you can compete on price at a subpenny level by posting at an exchange with inverted (taker/maker) pricing. Shares posted at BYX are really $0.004/share cheaper than BZX, and most smart order routers will take at the inverted exchanges first. Also, changing the tick size won't eliminate the race to 0 (or c) because there are other advantages to low latency execution beyond just getting good queue priority at the inside price. One obvious example is quickly canceling or modifying your orders when the price of a related instrument moves.

I am oversimplifying. You are right that speed still matters for cancels.

It's true that you can post hidden midpoint orders. That's a far cry from posting a public bid at $42.2837 in order to jump the queue ahead of someone posting $42.2836.

Another option is to match orders with frequent batch auctions (instead of continuous double).[1]

1. http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...

Batch auctions already exist already and still have HFT algo trading.

Further, this paper also assumes a very particular assumption, that there is 1 and only 1 exchange. Even if we wanted that in 1 legal venue (history says we don't) we can't enforce it globally. So venue arbitrage will still happen and therefore there is still a speed advantage.

Finally, this paper glosses over with 1 phrase "it may be necessary to ration one side of the market to enable market clearing" the whole crux of the problem. How do you remediate ties in price that can't be fully filled? IE, what is the matching algorithm? If it is still FIFO, then you still have a race.

Do you think the $.05 tick size experiment will demonstrate the disadvantages you claim for larger ticks?

Do you work in HFT? Because as someone who used to work in HFT, I can say with 100% confidence that the algorithms are not found in economics textbooks.

I was about to say the same thing. I've studied economics, and worked in latency sensitive finance, though not for one of these firms, and I've never come across anything that was obvious. Perhaps you could enlighten us about was sort of thing is used, because time constraints seem to imply the algos must be quite short.

Does anyone know if the author's book was ever translated? http://www.lekti-ecriture.com/editeurs/Cinq-Six.html

Its odd title---"5+6"---makes Googling for it very difficult. In fact, does anyone know the author's name?

I believe it's http://www.amazon.co.uk/6-5-Alexandre-Laumonier/dp/293060110... (although that's still the french version)

And the author is https://twitter.com/SniperInMahwah

I do hope there is an english version around, it's definitely something I'd like to read.

New post: https://news.ycombinator.com/item?id=8374613

As someone who has little to no experience in trading the technical side is very interesting to me, but the overall concept of all this is just insane and frightening.

Do you care to explain why this is "insane and frightening"? Are you frequently terrified by telecommunications?

I can't speak for him, but it terrifies me that we expend so much energy and capital to completely remove any sort of human reflection from the operations of our global economy.

So you're on hacker news, but you're terrified about automation and telecommunications, because it "removes any sort of human reflection". But only in this one particular application no doubt. Fitting that the towers are in Holland, maybe you can go and throw some clogs at them.

Well said, but Belgium isn't in Holland ;) For some reason a relatively large number of players in this space are Dutch though.

Oops, my mistake. I tend to lump the Flemish in with the Dutch without thinking.

I don't think reading some news and HFT have the same impact on the global economy.

http://en.wikipedia.org/wiki/Knight_Capital_Group

What do you believe was the impact of KCG on the global economy?

Take these graphs into account when answering: http://www.chrisstucchio.com/blog/2012/flash_crash_flash_in_...

Great link, but I would say still more than reading HN ;)

Human reflection used to play a major part in trade execution, and people used to pay a per-share spread tax measured in whole dollars as a result; all the way into the 1990s, that tax was maintained by the collusion of faceless computer programs no wait that's wrong I meant to say humans.

Automated trading has probably been an unqualified win for normal investors.

>I can't speak for him, but it terrifies me that we expend so much energy and capital to completely remove any sort of human reflection from the operations of our global economy.

The mistake you're making is thinking that human reflection is an inherently "good" thing to have. Humans have cost their companies orders of magnitude more money than algorithms have.

http://en.wikipedia.org/wiki/Category:Rogue_traders

We aren't removing human reflection from the operations of our economy. High frequency traders don't make decisions about the allocation of capital. They _can't_, because they don't hold positions overnight; they sell their inventory by the end of the day. Actual investment decisions -- which bonds or stocks to buy -- are still almost entirely made by human beings. HFT and algos only comes up when it's time to implement these decisions, and even then play a very minor role in the financial system.

If you want to understand how minor, compare the cash piles HFT companies manage with cash piles real economic players manage: The biggest HFT firms I know of are Virtu & Knight/GETCO, which both have a bit over $1 billion under management. Compare that the CALPERS pension fund's $300 billion, or with Goldman Sachs or Norways' sovereign wealth fund, both of which have about $900 billion in assets, or with PIMCO's bond fund ($2000 billion) or BlackRock's total AUM ($4500 billion).

Would the problem that if causes be easily solveble by doing somthing like 50% of your trades bluffs.

There is evidence that this sort of thing is already happening. Google 'quote stuffing'. There are those who say quote stuffing in HFT is a bug; my personal opinion is that it is done by design....

Can you explain the design concretely (i.e., the mechanics), or is this just a gut feeling?

For now I'm going to offer gut feeling. Take this or leave it - I wrote two paragraphs here that essentially came down to an argument to an authority re: my personal experience and the opinions people who know who trade. That's shitty logic, so let's go with gut feeling for now...

I don't believe that the same specific bug has manifested itself across many many different HFT algorithms run by many different actors over a long period of time, in a way that loses money consistently for everyone involved whenever the bug is triggered, without ever being addressed by anyone.

To argue otherwise is contrary to my current understanding of software engineering best practices - imagine another industry where every service provider has the same bug manifest in the same way, across all their different software stacks - some running Java, some Python, some even with their own proprietary languages - and no one ever catches it even though it's losing money and it's occurence is easily identifiable on a time-series graph (so very easy to be matched up with all available logging data)? No one has the resources to throw a team of devs at this for 6 months to identify the bug, or they do and they're just willing to lose money and not make every last nickel they can? Not a single person is trying to reach for that lost money with a bug fix or a novel tweak to their own existing algo? For years?

I believe it's more likely that either the actors who are quote stuffing believe they are benefiting by tipping their hand early with false information, or they actually are benefiting.

It's hard to imagine how these actors could be benefiting; perhaps they believe that someone is watching their quotes in transit or has visibility into their quotes before trades are executed, and the quote-stuffers are trying to fake these malicious actors out?

The alternative scenario where quote-stuffers are not benefiting from their actions - or rather, that they aren't even operating under the irrational assumption that they are benefiting (because the quote stuffers could be stuffing intentionally but incorrectly) - seems much more implausible.

It seems to involve a whole lot more things going wrong in a very specific very atypical way,from a software development standpoint, with no one every interested in picking up the money they're dropping on the floor due to this bug when things go wrong, over the course of years, even though all the data to pinpoint the source of the bug is public, charted, and discussed openly.

The first scenario just requires a quote stuffer to hold a possibly incorrect belief about the other actors in the market. The second requires multiple specific distinct software development failures to line up like dominoes and stay lined up without anyone ever addressing their root causes.

But who knows :) <shrug> Trading is chaos, after all.

The thing is, the bugs manifest in many different ways. Sometimes you get ramp&drop, sometimes you get on-off flicker, there are a wide variety of things that can happen. Take a look at the graphs from Nanex - lots of different things are happening. I realize that to an outsider they may all look like "graphs going wonky", but similarly all computer bugs look like "omfg I just want my email" to a marketer.

It's not a programming language issue - it's a math issue that would happen in any programming language. Whether you use Python or Haskell, you'll run into problems if you call SOLVE(A,x) on an ill conditioned matrix A. So you cook up a special routine that works well for most of the inputs you expect to see and it screws up for the tiny fraction of cases where you get a bad matrix.

Throwing money and a team of devs won't get around this. Similarly, you can't put together a team of crack devs (each earning $500k/year) and ask them to solve the CAP Theorem in 6 months. All you can do is choose your tradeoffs and hope that most of the time your network puts you on the good side of those tradeoffs.

Chris: you need to identify yourself if you're going to push your viewpoint here. You're the only author of a controversial blog post that's pushing what I'll gently describe as an interesting theory - not a casual HN observer just bringing up some talking points. If I hadn't caught your CAP theorem quote ("solve the CAP theorem", wha?) and the Nanex reference, I would have missed this and not drilled down to your user details.

Next time, feel free to add a </disclaimer> of some sort.

I'm not an outsider unless you've got one intense No True Scotsmen definition of the word, and I'll leave it at that. Let's just agree to disagree, and I'll say that you are certainly entitled to your opinion. Thank God, it's a free country and a free market :) Good luck.

Really, are you that naive to how the markets work? Intenalizers - Citadel for example, give retail traders prices based off the SIP and buy/sell on direct. Whenever there's a pricing difference, it's instant risk-less profits. You can induce that pricing difference by quote stuffing.

The problem with every allegation of quote stuffing I have ever seen is that it always looks like this:

1) Blast quotes

2) ???

3) Profit

Same problem in this thread. It would be a much more convincing argument if you could spell out the actual trade.

Internalizers (Citadel) are the beneficiaries - as well as those competing for the top of the queue on every price shift. It would disappear quickly if it were just a bug.

Then of course, is that phone conversation I had where it was explained to me (see above)

Technically that is against the law. It already happens though and enforcement is difficult.

On a poker game that would help.

I wish I had something more relevant to add but, hello from a former Mahwah resident

Reply on news.ycombinator.com