Thoughts on low latency trading if exchanges went full cloud | Hacker News Reader

Thoughts on low latency trading if exchanges went full cloud

blog.abctaylor.com

189 pointsarcza177 comments

The biggest current limitation with cloud providers when it comes to exchange tech is the lack of real multicast support. It is rare outside of exchanges, but extremely low latency L1 multicast market data has become the backbone of exchanges, both for fairness and for scalability.

Knowing you can saturate your entire network with 10G traffic and every participant will get the same market data packets at the same time[0], and there will be zero queuing or bottlenecks is very hard to do otherwise. There is a pretty good podcast episode about it out of Jane Street[1].

I know AWS have 'multicast support' but last time I tested it, it was clearly just uni-cast traffic with a software switch doing fan-out/copying, I assume using the same tech as their transit gateway, I think it was called hyperplane or something.

[0]: for some definition of the same time, at least low enough that you can't measure it without equidistant optical splitters or White Rabbit synced devices.

[1]: https://signalsandthreads.com/multicast-and-the-markets/

> lack of real multicast support

Yup, this is a problem for us in GCP today even outside of trading. I don't know how Pub/Sub works for them.

Pub/sub systems in unicast-only environments are very complex distributed systems to handle the load involved in fan-out routing while maintaining a global order. I had an interviewer once get annoyed with me for suggesting using multicast to solve the fan-out part of a pub/sub system, which made the global ordering part small and simple.

We lost a lot by thinking of HTTP as the one true level of network abstraction.

A reliable multicast network that preserves global order even during maintenance and doesn’t drop packets is not something you will find off the shelf.

A reliable multi-tenant multicast network also appears to be a rare beast. I’ve only heard of it in finance, and that’s only because it’s private and expensive and all the participants need to be generally nice to each other because it’s a repeated game and the operator can literally pull the plug if the rules are broken.

Do you regularly allow untrusted machines onto your private pub/sub instances? I'm not sure the "operator pulling the plug" part is unique to the finance industry.

Also, yeah, you have to do some engineering around your multicast distribution to make a pub/sub system, but multicast pretty much solves the data rate scaling problem - you are now basically O(1) in the number of connected subscribers.

There was a time in the 00’s where a lot of server hardware had 3 NICs and you could use those for redundancy but a better use was to create three networks: inbound, service and database calls, and administrative.

You had more control over your services talking to each other and control plane tech, thus could make some more guarantees than with inbound data. Don’t cross the streams.

Unless you build your network for it, multicast is a huge pain in the ass to administer. None of the big cloud providers built for it at the scale that traders use it, and I think they prefer things that way. When customers want it, they all just fake it by doing fan-out unicast.

All hyperscalers have an SDN that essentially spoofs local ARP/DHCP inside the hypervisor and does not support broadcast or multicast by design (there are some caveats here, since some telco protocols that require them can be made to work).

I find it sad that equal access between the entities doing HFT and regular Joes is not required for fairness, but god forbid one HFT having some milisecond advantage over another. That would be unfair. Can't have that.

Because average Joes don't do algorithmic trading, and if they do it not at the level that HFT does. Not even all the big financial players care about HFT and millisecond timing, so they're in the same boat.

What do you think an Average Joe is going to do with that extra millisecond available to them?

I think Average Joe has quite a lot of disadvantages compared to some hedge fund or whatever it is that does HFT. Do you really think the only difference between them is a millisecond? That's only between HFT traders.

My point was that other advantages/disadvantages are not being cared about, not that we should provide milisecond access to Average Joe.

Of course the Average Joe has disadvantages compared to a hedge fund or people who are experts in a field and spend the bulk of their life dedicated to some aspect that the Average Joe is not dedicated towards.

If Average Joe wants returns comparable to these hedge funds, then they should stop trying to time to market and instead stick to diversified ETFs and stop worrying about millisecond differences in the stock market.

Believe it or not, if Average Joe does that they can actually beat most hedge funds over a long time horizon [1].

https://www.cnbc.com/2018/02/16/warren-buffett-won-2-point-2...

I mean, you could also write "of course whoever has servers closer to the exchange can do HFT better". "Of course companies that invested a lot in having servers closer will reap the advantages."

No, expertise is not the difference. If you're a private person with 100 years experience in trading, you still can't do HFT. You need to be an instutition, have lots of capital to invest in servers, software development maintainance etc. As a private person I think you don't even get access to the API.

"Of course whoever has more capital has advantages in the market"? Of course they do, but I don't think "of course they should".

For this discussion, that funds don't beat the market average over a long term is irrelevant. Why not say "who cares you get more latency than the other bank, if you want money just invest in S&P500 and long term you'll beat them". But you don't apply that to banks against banks, only to banks against Joe. Why?

>of course whoever has servers closer to the exchange can do HFT better

Can do better at what? Can get their trade in the order book faster? Yes they can. But does that automatically mean they will make more money? No it does not.

>If you're a private person with 100 years experience in trading, you still can't do HFT.

Of course not, 100 years ago there were no computers. Having 100 years of experience in trading on the pit would not give you any expertise in software development.

Someone with 10 thousand years of experience plowing can't compete against someone with a tractor. That's kind of the point of the tractor...

I'm sorry that it disturbs you that the Average Joe sitting at home with his discount online brokerage account is unable to gain the same kind of benefits putting out individual orders here and there on speculative stocks that he likely knows nothing about, that hedge funds, institutions, and other highly specialized and skilled professionals are able to gain by doing this for a living.

The Average Joe does have access to highly diversified and low fee ETFs, and as I said the Average Joe can reap almost all of the rewards that the best hedge funds and banks do by sticking to those instead of trying to play the market.

I think it's just beautiful how you made this point and other people couldn't even understand the concept of equal access to data and trading platforms. Yes, that's your point exactly.

It seems you're confused about how competition works among hft shops. There is no regulated, same latency for us. We compete for faster access just like everyone else.

Likewise, banks chronologically rearrange the transactions in checking accounts to maximize overdraft fees. Yet when I suggest batching and chronologically randomizing the transactions on exchanges to reduce the benefits of low latency / centrality, people behave as though I have transgressed against Moloch.

Regular Joe is so bad at trading stocks that hedge funds literally give him a discount on the national best bid/offer for the privilege of being allowed to trade with him.

Giving retail traders access to the "actual market" would most likely result in worse execution on average, according to some studies.

In AWS you don't even do neighbour discovery through ARP. Or that's a lie, you do, you get a arp reply, but it's not from any of your devices on the network. And traffic is authenticated and authorized at both the source and destination, so you can't do fun things like manipulating arp tables. You get a lot of nice features when you have a fully software defined network, but it comes with a couple of caveats, like you mentioned here. I doubt we'll ever see "real multicast support" in the sense that network engineers are used to.

> Knowing you can saturate your entire network with 10G traffic and every participant will get the same market data packets at the same time[0]

Hold on a second. Multicast is nifty, but it does not perform miracles. If you operate a 10G multicast network and actually saturate it, you will experience drops and buffering-induced delays. Perhaps you can play games with time-synchronous networking, but as far as I know the exchanges don’t do this, and it likely needs special hardware.

The point of 10G multicast is to use simple, standard (but complex to configure!) equipment to distribute much less than 10Gbps simultaneously.

I'm curious if you know what, at a switch level, would actually cause drops and buffering for a 1:N (near-) saturated multicast flow. If all the packets are coming from the same source machine at (perhaps) 9.9Gbps and flowing into the switch, I would expect the switch to robustly redirect all that data with near-zero latency or packet drops to all its output ports. I don't think 10G Ethernet has "backpressure" in a way that would allow some output ports to get slowed down.

If there are other data flows also going through the switch, that could obviously change things, and the sending computer could drop packets if there's jitter in how quickly the application produces them, but it seems impossible for the sending computer to burst packets into the switch any faster than it can handle because all the incoming packets are coming over the same 10G link.

Not an expert here, legitimately curious.

Periodic background traffic like DHCP and background noise causing packet loss.

You can’t run a queue at 100% and have any expectations of latency. In fact the rule of thumb from queueing theory is 50% to avoid latency spikes.

I mean it's not that hard to eliminate all other traffic on a closed network like that, at least where there's millions of dollars at stake.

Must be nice to open Wireshark and see _nothing_.

That would be highly uninteresting to the rest of us don’t you think?

The nature of this kind of traffic is that it's pretty bursty. Think 100x-200x the normal packet rate in the span of a millisecond. Perfect opportunity for drops. Ultra low latency switches have tiny buffers.

A modern cut thru switch typically has one ASIC for a group of ports, and that ASIC handles all the traffic. If the traffic for all of those ports is greater than what the ASIC can handle, you'll have buffering and/or drops.

That being said, the ASIC can typically handle line rate on all the ports. You could have 10G input and fan it out to 10G output on all the ports with no drops, but if there is other cross port traffic, something could get dropped.

Are people still using 10G in PROD? I thought 40G and 100G had generally replaced that. I have 10G cards in my homelab that are a decade old and cost less than $100.

We are 400G to the machines and 800G on the splines.

Cool! Even faster than I realized. What application(s) can actually push 400G through a machine though?

Data transfer for training. It goes directly to the GPUs via RoCE. That said, we also stack the boxes with a bunch of NVMe as well, so you can cache there first, if you want so that you don't have to worry about network bandwidth as much. We're flexible on customers needs.

When 800G nic's come out next year (along with PCIe6), we will start buying those as well so that it is 800G everywhere. Let's see how long that lasts before it is considered slow... heh.

10G serialisation delay is lower than 40G or 100G.

Most markets can disseminate their feeds on 10G effectively. This isn’t true of the major US exchanges.

People get weird about the word “saturate”. I think GP is switching to “max sustainable” and expecting everyone else to come along for the ride.

Queuing theory has many many bad things to say about actual saturation.

add "carrier sense multiple access" packet creation

Almost all exchanges disseminate market data over multicast these days. If you miss a tick it doesn't matter because by the time a tcp retransmission completes this is old, useless data.

Generally speaking how multicast is used in trading situations[1] is you have two networks. On the primary network you do most of your normal IP traffic between applications etc. Then you have a seperate marketdata network that has most of the multicast traffic and it's exclusively used for marketdata. Marketdata generally is delivered on an "As fast as possible" basis[2]. So you don't care too much about occasional drops although fewer is obviously better.

[1] At least in my time in the front office.

[2] For example a very common pattern at the very low level for a marketdata subscription is when you subscribe to marketdata for some symbol the system will actually have a double buffer where it writes into one slot and you read from another slot and every time you read it switches the slots around. This means you can generally accept marketdata as fast as it arrives and process it when you can and you will always get the most recent packet when you ask for the next packet.

I’ve seen some multicast market data protocols with remarkably poor ability to detect or recover from drops. And they are very much not of the form where a newer datagram supersedes the older one.

Yes, although people have been doing marketdata networks the way I said above using IP multicast for at least 20 years now, so in general they choose protocols and network architectures carefully to minimise problems. You do see problems from time to time but they are somewhat rare. Some of the restrictions are interesting. For example IP multicast was basically completely banned on the trading floor where I worked except for marketdata, because of an IP multicast snafu from some random application that took out the whole network once.

One thing to realise about marketdata specifically is it's really different from other low-latency situations most people are familiar with (netcode in a game for example). As I mentioned before, it's not that big a deal generally to miss a few packets - the thing that is a big deal is to make decisions based on stale data. So you're not generally trying to reconstruct the full state after a drop- you just want the freshest current packet as fast as possible. If/when you need to reconstruct state you can make specific requests if needed.

Out of curiosity, what specific protocol are you taking about? I’m very familiar with a couple of these protocols, and the issue has nothing to do with wanting to know the history of the data — the issue is that they use what is, in effect, an ad-hoc delta encoding, and you can’t reliably reconstruct the complete current state if you’re missing a packet. On top of this, sometimes the packet sequence numbering is designed creatively, to be polite about it, so you don’t necessarily find out that you missed a packet as soon as you would like to be able to.

I ran into this problem a while back working at a company that was working to distribute video streams with low latency (lower than Low-Latency HLS) to a large number of viewers. Initially a prototype was built on top of AWS with fan-out/copying and it was terrible. This was partially due to inefficiency, but also due to each link being a reliable stream, meaning dropped packets were re-broadcast even though that isn't really useful to live video.

Moving to our own multicast hardware not only greatly improved performance, but also greatly simplified the design of the system. We required specialized expertise, but the overall project was reasonably straightforward. The biggest issue was that now we had a really efficient packet-machine-gun which we could accidentally point at ourselves, or worse, can be pointed at a target by a malicious attacker.

This 1-N behavior of multicast is both a benefit and a significant risk. I really think there is opportunity for cloud providers to step in and provide a packaged solution which mitigates the downsides (i.e. makes it very difficult to misconfigure where the packet-machine-gun is pointing). My guess is that this hasn't happened yet because there aren't enough use-cases for this to be a priority (the aforementioned video use case might be better served by a more specialized offering), but exchanges could be a really interesting market for such a product.

It would be pretty efficient to multi-cast market state in an unreliable way, and have a fallback mechanism to "fill in" gaps where packets are dropped that is out-of-band (and potentially distributed, i.e. asking your neighbors if they got that packet)

Here is a research paper I recently wrote about a fair and scalable multicast in the cloud: https://arxiv.org/abs/2402.09527

I would love some feedback!

One key consideration is “provable fairness”. It’s my understanding that exchanges use techniques like long, same length fiber optic cables to all racks within the exchange datacenter to convince customers that everyone is on a fair playing field.

This is a lot harder to do when a server is virtualized somewhere on some rack on EC2. Exactly as mentioned, people will try to optimize by spinning up/down instances as close to the exchange server as possible. Customers will be unhappy because they can’t prove that it’s fair, even if they have the closest server.

Overall great, thought provoking writing btw

> This is a lot harder to do when a server is virtualized somewhere on some rack on EC2.

There are bare metal EC2 instances.

It's about the interconnect and the proximity.

and not having a for() loop doing multicast fanout in software

which sounds like what AWS Transit Gateway is

At some point, someone has the shortest route connecting to the exchange's bare metal EC2 instance, and that organisation has a significant advantage in high frequency trading.

It's provable that it's not fair. AWS multicast is software based, not hardware based.

I think the lines between software and hardware-based are a little blurred these days with accelerator cards and whatnot. It's just a lot harder to come with the same level of guarantees when you're basically running a hypervisor on top of it.

The only exchange I know that does the cable in a box trick is IEX. Everyone else is based on "the closer, the better". Colocation is king.

Nice article. Wondering though why trading is not done in discrete batches, e.g. 5 second intervals? Trades in the same interval get filled equally or stochastically? Info about trades with that same 5 second batch delay? Is there some (theoretical) market efficiency thing at play? All this HFT feels wasteful and bad for 'regular' human investors.

I guess the argument would be that this limits price discovery? I hear this proposed a lot and haven't heard super compelling arguments against it

The reason the GP is proposing it is because it limits price discovery.

IMO, if you have a problem with limiting it to 5 seconds long quanta, you are doing something wrong.

Shareholders would fume hearing this

This is how the Taiwan exchange used to do matching, and I still think it's the best system I've seen.

I don't think the reason has anything to do with price discovery, it's just because exchanges want to maximise their trading fees. Continuous order book trading leads to more trades and hence more profit for the exchange.

They also charge differently (extortionately, some might say) for different speeds of data feed, although I'm not sure if they have tiers just for HFTs.

The HFT is wasteful but isn't bad for human traders, they tend to get better prices. It's bad (sometimes) for huge investors (VHNW individuals, hedge funds, pension funds which I guess represent regular people) that want to make large trades without moving the market but there are also winners here - e.g. if Johnny the day trader buys a stock that Texas Teachers Fund is selling huge batches of, he's better off if HFTs are causing price changes to propagate more quickly.

If you think about it you can never eliminate the advantage of being faster. If you do 5 seconds batches it just means the edges of the batches become the time-sensitive points.

If you want to kill HFT you can do it directly via very very small transaction fees. But guess how popular that is...

You can certainly alleviate the disadvantage of being slower though - and that’s exactly what literature in batch trading argues.

> If you think about it you can never eliminate the advantage of being faster. If you do 5 seconds batches it just means the edges of the batches become the time-sensitive points.

You mean you can never completely eliminate the advantage? But mostly eliminating it might still be useful?

Suppose the rule is that if you get your request in by 01:23:45 then it gets handled in the following 5-second period and the response is sent out at 01:23:50. Does someone (A) who finalises their request at 01:23:44.9999 and gets the result back at 01:23:50.0001 have an advantage over someone else (B) who has to finalise their request by 01:23:44.8 and gets their result back at 01:23:50.2? Yes, certainly, but it doesn't seem to be much of an advantage ... So person A can take account of exciting news that arrives at 01:23:44.9, while person B can't, true, but when it comes to reacting to other trades, person A has 4.9998 seconds to think about the news, while person B has 4.6 seconds to think about it, which doesn't seem like a huge difference. Compared to how things work today.

It's funny that you say that, with the crypto transaction fees still a big problem. Feels like HFT and crypto are on a convergence towards that concern.

Exchanges already extract per order commissions. You do not pay per message (so add, cancels and amends are free, you only pay when you get traded [1]).

A per-message would probably significantly affect existing strategies and greatly increase spreads, but I don't think it would prevent all forms of ULL trading.

[1] But even there exchanges offer rebates, if not outright incentives, for market makers to provide liquidity.

How do you tie break? If there are more sellers than buyers (or vice versa) at the clearing price?

Flip a coin.

The same way you would without a clock I guess?

You could match what you can distributed equally and leave the rest unsettled.

You could let people decide whether to roll-over the partial bid into a new bid on the next clock or to cancel unsettled.

You could clock to something both very fast on a human scale (50ms), quick enough it'd still feel instant but slow enough that it could reduce HFT silliness and need for extreme low latencies.

> You could match what you can distributed equally and leave the rest unsettled.

Equally per market participant? Do large participant like banks trade same amount as retail investor one trade at a time? Per quantity? HFT will time the end of the interval and decide to place a large order or not.

It would be weighted by bid size. If there's $10m of bids one side and $5m of offers on the other, you match up the $5m on that side and every bid gets 50% settled.

I'm not sure I understand the problem with "waiting" for the end of the clock. The pool wouldn't be public so you couldn't get knowledge inspecting the pool. All bids and offers would be published on the clock and settled by weighing all the bids and offers against each other and matching by volume.

The trickier issue is what happens in this scenario (assuming limit orders):

Person A bids for 500 units @62

Person B offers 100 units @61 Person C offers 400 units @60

Clearly there needs to be full settlement, we have a bidder who wants to buy 500 units at a price which sellers are happy to sell at.

Correct me if I'm wrong, but in a traditional market it would depend on the order they came in.

Here we would need a formula to work out the correct settlement price. Intuitively this ought to be somewhere just above 61. ( If it were just two people, a bid at 62 and an offer at 60, you could intuit a fair settlement would be 61. )

I'm sure fair formulae can be derived however.

You trade latency arbitrage with statistical arbitrage were participants try to estimate the market and overbid to try to capture as much of the market as possible. That seems dangerous and unstable.

The general rule is HFT will have the elite level mathematicians to figure out what is the most optimal strategy is, and the most exotic hardware to implement it to the extreme. Other party will fall further behind given the more complex and unconventional exchange rule.

This is all gameable. Like he said, you just time it with an over-large order and let the remainder expire. Everyone gets 10% of their order but my order is 10x what I actually want so I actually get 100%.

Per rata matching is already a thing in some financial markets. They tend to be _more_ latency sensitive as size gets inflated to game the matching algo, thus risk being inflated and thus the value of timely cancels.

They already do this for the opening and closing auctions. You can have a market-on-close order or limit-on-close order for example. The market on close orders are guaranteed to fill. The limit orders are filled using price-time priority, so best prices submitted earliest fill first, after the market price orders.

I guess it is possible that there are remaining marketable orders that never fill because of an imbalance one way or the other, but I doubt that ever happens in practice.

If you keep price/time priorities you still get a race to pile into new levels after the previous batch.

That's what happens with the opening and closing auctions currently.

Yep. They are more latency sensitive than the continuous portion of the day.

> All this HFT feels wasteful and bad for 'regular' human investors.

Quite the opposite, thanks to the tough competition the market makers are setting the bid/asks spreads as minimal as possible. Which leads to less costs for human investors, pension funds, insurance companies etc.

I used to be a market maker in the 90's before HFT took off. The margins we kept sometimes felt like a rip off but customers had no other choice but to accept them.

People who ask for transaction fees, forced delays in executing or whatever, tend to forget that these force market makers to increase their spreads, which means customers eventually pay the price.

>Quite the opposite, thanks to the tough competition the market makers are setting the bid/asks spreads as minimal as possible. Which leads to less costs for human investors, pension funds, insurance companies etc.

It's not automatically the case that the disappeared margins & thinning of bid/asks have been shared equitably between the trading firms and customers.

Take two exaggerated markets for example:

1) No HFTs: The customer wants 100 shares in Company A. The shares are available on two exchanges, one at $100, and another at $105. A market maker charges the customer $5 to access the 100 shares at $1 each. The customer pays $105. The market maker earns $5.

2) With HFTs: The customer wants 100 shares in Company A. The shares are available on two exchanges, one at $100, and another at $105. The customer clicks "buy" on their trading platform, the HFT races to the $100 shares, and purchases them, then fulfills the order at $105. The customer pays $105. The HFT firm earns $5.

For the end-customer, all that's happened is the margin goes to another firm. The consumer still has no other choice but to accept these transaction fees. There was arguably a need for HFTs to reduce the market-makers exorbitant fees in the 2000's, but that requirement has been served, and the technology now exists to remove both from the market entirely.

HFTs are a rent-seeking entity interjecting in a market which, at least in theory, exists to most efficiently allocate capital to the productive benefit of all.

This is entirely false and ignores Reg NMS. Everyone must execute at the NBBO. As well customer orders are often given better and tighter prices than other market participants. HFT firms will often offer them better than NBBO prices. As well, none of these prices you quote would not exist without market makers, its just now the fact that to be a market maker you must be an HFT firm as well due to the scale that is now required.

In the United States at least both scenarios you mentioned are illegal. Market makers are not just sitting in the middle of orders. They buy without a seller lined up and then fill orders from their own inventory (or route orders to an exchange in the case where they can't fill a buy order from their own inventory). In cases where they route to an exchange they are required by law to fill the order at the lowest price available. Typically they fill orders at better prices than what you can get on an exchange. So you, as a retail investor, are actually getting better prices than you would if your broker just filled orders on an exchange.

How the price improvement gets allocated is complicated. Some of the price improvement goes to the broker (in the form a payment-for-order-flow) and some goes to the actual investor (you). But in either case the retail investors are strictly better off.

Latency Arbitrage still exists in a world with NBBO regulations. Research consistently finds that not only does the strategy work in theory, but that it is consistently put into practice by HFT firms to the detriment of other market participants. If a firm can calculate the NBBO ahead of other market participants and the market regulator, it can still legally front-run the market, and risklessly extract rents from end-customers. The NBBO formula is not computationally expensive, and its underlying data is necessarily publicly available to all trading firms. This occurs in the real world, in the order of $billions annually.

The UK's Financial Conduct Authority:

>We use stock exchange message data to quantify the negative aspect of high-frequency trading, known as “latency arbitrage.” The key difference between message data and widely-familiar limit order book data is that message data contain attempts to trade or cancel that fail. This allows the researcher to observe both winners and losers in a race, whereas in limit order book data you cannot see the losers, so you cannot directly see the races. We find that latency-arbitrage races are very frequent (one per minute for FTSE 100 stocks), extremely fast (the modal race lasts 5-10 millionths of a second), and account for a large portion of overall trading volume (about 20%). Race participation is concentrated, with the top-3 firms accounting for over half of all race wins and losses. Our main estimates suggest that eliminating latency arbitrage would reduce the cost of trading by 17% and that the total sums at stake are on the order of $5 billion annually in global equity markets

https://www.fca.org.uk/publication/occasional-papers/occasio...

The University of Michigan's Economics department:

>We illustrate this process and the potential for latency arbitrage in Figure 1. Given order information from exchanges, the SIP takes some finite time, say δ milliseconds, to compute and disseminate the NBBO. A computationally advantaged trader who can process the order stream in less than δ milliseconds can simply out-compute the SIP to derive NBBO,a projection of the future NBBO that will be seen by the public. By anticipating future NBBO, an HFT algorithm can capitalize on cross-market disparities before they are reflected in the public price quote, in effect jumping ahead of incoming orders to pocket a small but sure profit. Naturally this precipitates an arms race, as an even faster trader can calculate an NBBO* to see the future of NBBO, and so on.

http://strategicreasoning.org/wp-content/uploads/2013/02/ec3...

The Bank for International Settlements:

>Conservative estimates suggest that at least 4% of dark trading occurs at stale reference prices. High-frequency trading firms (HFTs) almost always benefit from such stale prices, being on the profitable side of the trades between 96 and 99% of the time. Furthermore, stale trading does not happen at random but is driven by the behaviour of HFTs. HFTs as a group almost never provide marketable liquidity in the dark and rather behave strategically to exploit their speed advantage by submitting marketable orders to execute against stale quotes.

https://www.bis.org/publ/work1115.htm

What you described is not latency arbitrage.

I described the canonical front-running example in my first comment, as it gives most non-finance readers a quick overview of how HFTs work, without needing to describe regulations, strategies, NBBO, SIP, etc.

The specific strategy currently employed by HFTs is somewhat immaterial in the broader context of a discussion about front-running. For as long as a firm can legally front-run the market with any strategy, it can undermine the market and risklessly extract profits.

The cause of latency arbitrage is not HFT, it is the fragmentation of liquidity.

This is how wholesale electricity is traded, although for unrelated technical reasons.

Bids and offers are collected for auctions that happen at regular known intervals, for example every 15min.

Any article that talk this in depth?

I'm afraid I do not know, but the Wikipedia page is pretty good:

https://en.wikipedia.org/wiki/Electricity_market

Those batched trades are called “auctions” and they are a part of many exchanges.

I think it’s pretty uncommon to do them every N seconds.

A common pattern is to collect quotes before the market open, do an “opening auction” to set the opening price, and then switch to continuous trading for the rest of the day. If trading in a stock ever pauses (which can happen for a variety of reasons) then another auction occurs when trading is restarted.

> All these people in tech optimising clicks, ads and engagment feels wasteful and bad for 'regular' humans.

IIRC they have done trials of this on some exchanges. It didn't make a big difference either way. The opening and closing auctions are already sort of done the way you describe.

Why would the exchanges want to move their colos into the cloud. They've spent the capex and they can charge lots of money to rent data center space, cross connects and other services to their customers. If they moved to the cloud, all that revenue would go from them to AWS/GCP/etc.

Doesn't seem like a profitable move for the exchanges. They make more $$$ with on prem setups.

Somebody needs a project to get promoted …

Nothing to add but I found this to be a well written thought exercise on a space I realize I know very little technical detail of. Thanks for writing this!

Thank you. I'm only happy to hear this was helpful. I'd also love comments from any HDL quants about how portable FPGA code would be to virtualized environments and what other languages might stand out particularly well for a VM stack.

There are cloud FPGAs, but they are offered as a compute accelerators, and have no access to the network. Trading FPGAs need network access for latency.

Im also sure that if theres enough customer demand (from people willing to spend $M), AWS will make network connected FPGA happen.

I'm sure they won't, at least without a lot of development. FPGA networking used for trading is borderline abusive of networking protocols, and I assume that Amazon doesn't want that on their production network.

If a big exchange goes to the cloud it won’t look like a regular company setting up an aws account and getting a bunch of ec2 instances in us-east-1. They would at least have dedicated racks.

I suspect the provider would end up with a plan where traders can get servers that all have the same network distance from the exchange’s nics (down to the same length of fiber).

Honestly for something this niche I wonder if AWS would make a new region, like gov-cloud or secret. The goal would be to use tech that can accommodate ULL deployments.

This seems to the most likely, and it's not unheard of, they have 'local regions' like osaka in Japan.

Alternatively they could just stick a bunch of Outpost racks in the NYSE/NASDAQ data center and create an 'eXn.8xlarge' instance type and charge 100$ an hour.

I think and exchange would use AWS Outposts to get hardware in their data center.

But at that point it would be just another colo, right?

I used to work in machine maintenance, so I always think about what will happen when the machines involved fail.

NYSE machines hosting the trading server fail: presumably they have hot backups they're ready to switch to but that takes time and will interrupt trading during the cut over. Not to mention that not all failures are hard failures, what if the NIC is downtrained to a lower speed, RAM is slower than it should be, or a single hard drive storing important data crashes? Lots of interesting failure modes. When the NYSE owns their own machines they can handle these cases directly. When they don't and Amazon is responsible for repairing these machines it might take a lot longer to get things fixed. I hope NYSE is thinking about hardware failures and building a system to check performance of their trading servers before letting them become the active host.

Thinking about failures on the side of the traders: basically if they get unlucky then there could be delays as Amazon rerprovisions them replacement servers in the case of failures. This likely impacts what trading strategies are viable, and could cause them to lose money if machines fail at unlucky times.

They literally have triple hardware redundancy. They can afford it.

This is an interesting read but I think it leaves outside what kind of trading is the one that would benefit from ULL.

ULL and currently HFT seems to be very useful for market making (buying the ask and selling the bid and profiting from the bid-ask spread making parts of a cent per transaction, done a few million times a day), but there are other uses for HFT. One of them would be to execute very big orders over time to instead of drastically rising the price of the security they can get a better cost basis by performing a set of trades, letting the market absorb the impact and continuing with the order.

The thought of having the market in a cloud provider like AWS scares me! Although I’m sure that AWS might have pitched the idea already. If the markets could be controlled by a private company that could schedule “maintenance” at convenient times for them, that sounds like a recipe for market manipulation bay trillion dollar company. Sounds like something the SEC wouldn’t stand for.

Your description of market making is backwards. If you're buying the ask and selling the bid, you're paying the spread not collecting it.

Thanks for the correction!

About the market data & monitoring element, beyond the usual suspects Clickhouse and kdb+ - worth noting that Aquis mentioned in this article uses QuestDB ( https://questdb.io/case-study/aquis/)

Why on earth would exchanges go full cloud when running the trading infrastructure reliably and predictably is their whole reason for existing?

I like the assumption that NYSE would just grab some EC2 instances and run an exchange on them, and that AMZN wouldn't bend several directions at once to deliver new products that just happen to exactly replicate the environment they're 'leaving'.

Orders should have some durability and it would probably change behaviors enough to make hft go away.

If you list a buy or sell order it just has to be in force for some period of time, say a minute or something.

HFT shops will say this would reduce liquidity, but it would only make clear what real liquidity was in the first place.

Why would anyone want that? Everyone would just quote a lot wider to make up for the potential volatility of the next minute, probably leading to worse trades for retail orders.

If orders has to been good for atleast a minute it would massively increase the spread (by like over a 1000x probably)

Also, what's wrong with hft?

Will you put an order 1 minutes before financial statements, then? If so, at what price?

If no, knowing that there will be no liquidity in the very last minute before financial statements, will you put an order 2 minutes before?

This won't propagate forever, but is going to bring very complicated changes.

Interesting read, thanks! Some points:

> Hogging instances might pay, if this stops competitors getting good hosts. This would eat into PnL and is also wasteful on energy.

Aren’t reserved instances cheaper than spot?

> Bad players could do a ping test to many thousands of EC2 instances, find those which are also at very low latency to their good boxes (assuming these are competitors), and DDoS them during trading hours to hammer the hypervisor’s NIC. This would result in critical overhead occurring for competitors sending orders out.

Leaving out the logistics of how someone could do this (why are your instances reachable from the internet?), wouldn’t you have a good case with your exchange to get them kicked out?

> wouldn’t you have a good case with your exchange to get them kicked out?

Would you know who is doing the pinging? The cloud provider would know what account was pinging, but somebody doing this as a trading tactic would have the resources to churn through aws accounts as quickly as they are banned.

It's my opinion that stock exchanges should batch trades every 30 seconds, or longer (depending on the market), so that millisecond arbitrage becomes impossible.

Front running the market in any manner should be illegal.

Yes. There is actually some research into this idea where markets would effectively conduct rolling auctions, but I'm struggling to find it at the moment because I'm in a work meeting. Iirc the evidence suggests this would reduce market dislocations when news comes out etc so would generally improve price discovery.

Markets already conduct an opening and closing auctions and conduct an auction to resume after a volatility break (what people often call a "circuit breaker" in the press although it's a volatility break) so this would not be as much of a technological lift to implement this as it may appear.

How it works from a practical perspective is the exchange suspends matching for a period (so say 30mins) but order placement still works. Then when the market comes out of suspension a single print runs to uncross the order book, and everyone who submitted an order which matched gets executed at a single price. So as you say timing arbitrages of the current kind are effectively impossible. So in the case of a rolling auction you would do that print and then immediately suspend matching again and do another auction.

Here's some background on how auctions work in financial markets in general but it's not the specific paper I was referring to https://www.princeton.edu/~jkastl/auctions_finance.pdf

For example this is a proposal for continous auctions in power markets. https://www.deutsche-boerse.com/resource/blob/1458710/717470...

10-100 millisecond auctions have been proposed seriously. 30 seconds is pretty long. If you would like a counter-argument, the options markets essentially trade on an auction basis: every time someone* sends an order that crosses a spread, there's an auction that does price discovery of the security. That does not promote having narrow spreads or transparent prices - options markets have huge spreads even for very liquid products.

*someone who is not a market maker

That works in theory but as soon as you have multiple stocks batch auctions become worse than continuous. How are you can you implement a long/short strategy with auctions? If prices are transparent during the auctions then the stocks that runs the latest auction becomes susceptible to latency arbitrage so you're back to square one. If prices are not transparent you can't see prices during the auctions and you can't be sure of the relative prices for your long and short leg which complicates risk management considerably

Fun thought exercise, thanks! My question is, what advantage would a large exchange find in moving to cloud? They’ve already got the personnel capable of managing their environment. They’re not a rapidly-growing startup in need of flexibility. They’re large enough to get at least decent deals purchasing gear. “The cloud” will naturally expect to make a profit on the deal, which likely eats up (and then some) any savings which might otherwise be delivered.

I “get” cloud in a lot of circumstances but it doesn’t seem to make much sense here.

Without reductions in personnel, then none.

That's essentially what you're buying from a cloud provider. Most of the time its not so much renting the hardware as renting their labor in maintenance.

That is assuming your hardware needs don't have a wide enough variance from time to time (scale up/scale down)

I work at a prop firm/mm. You mention cloud and you'll be taken back behind the chemical shed.

The author of this article clearly knows a lot about the subject, but I think this would have been better titled "Low Latency Trading isn't Going To The Cloud and Here's Why". Or to put it a different way, infra peeps within exchanges have a very specialized skill set and priorities. General cloud infra peeps don't. No shade, but there's always going to be some business that doesn't make sense to switch to the standardized solution.

At first glance, it seems this would even the playing field. However, large players will allocate resources to spinning up instances and overloading machines. Similar to how we're seeing the DDOS shenanigans going on in crypto.

Net-net, it still benefits startup quant shops and sophisticated independents. Most retail isn't doing HFT or really any quant. But for people wanting to have their own shops, this is a better version than having to build hardware and colo.

I'm a total trading noob. Can you explain why a low latency is worth so much? And how these traders "exploit" that advantage to make a profit?

Reaction time to events, be it news, securities movements, etc. If you’re early, you can open positions before large movement has taken place; you can also close those positions precisely when you want.

Sometimes this may be making pennies (x N shares), other times it may be quite substantial.

A delay could impact the price. Say you wanted it at $5.00 but it becomes $5.01 by the time your bid comes in you either miss out or pay more.

These traders do it at high frequency. Imagine $0.01 x millions at a time.

What would you do if you knew the price of something 1 day before everyone else?

How would you make money off that?

In any market, a seller or buyer willing to close the transaction quicker is a benefit for the opposing seller or buyer.

Quicker can be on the scale of years to milliseconds, depending on what is being exchanged and amongst whom.

This is extremely simplifying the nuance, but imagine there are two traders who want to buy what a single trader is willing to sell at a given price? Well, the first one to get to the exchange and "lift the offer" will get the price, while the second trader will have to pay a higher price.

High-frequency trading is essentially a low-margin, high-volume play that exploits small price differences for profit. The most obvious example would be arbitraging the same security on different exchanges (buying low on one and selling pennies higher on another). Similarly, algorithmic models could exploit price volatility for individual securities on the same exchange. Under such conditions, fractions of a second can determine whether a given trade is a winning or losing one.

Most exchanges prioritize their order book by price, then time.

If you're a market maker, you (usually) want your orders to be selected. A common market making strategy is to issue a buy order a bit lower than the last executed price and a sell order a bit higher than the last executed price with the assumption that there's a lot of random and small price motion up and down. If you can consistently process order fills and update/replace your orders in the book faster than the other traders, you'll get more of the trading volume, and other traders will have to compete with you on price. For some stocks where the minimum price increment is large relative to share price, most market making traders will converge on the same buy and sell prices, so latency is it.

There's also value in responding to filled orders in one venue at other venues. Many stocks have a 'home' exchange, but trade at many exchanges, if there's a significant price movement at one exchange, other venues will quickly follow, but if you can follow quicker than most, you can execute against the now mispriced orders on the book, etc.

Showing that I don't understand economics while also telling that I don't understand economics: It would probably do the world more good to tweak the structures making ULL trading profitable anyway, it's not like the trading in and of itself brings any value to the broader world, while consuming enormous amounts of resources that could have been spent on actually improving systems that create real value.

> it's not like the trading in and of itself brings any value to the broader world

This is a common sentiment, but the reality is that increasing market participation is good for everyone. Yes, even retirement funds benefit from the presence of market-makers. Liquid markets allow for better price discovery and cheaper transaction costs.

I specifically the high-speed trading. I definitely agree that actual investment and trading has some benefit.

Nobody was helped by the 200 nanosecond thing that the machines did when the marked opened (except the owners of said machines, of course)

Plenty you people were, you just don't notice it.

What if collectively those ULL and HFT orders help stabilise the market. Then I would say, yes, "we, the world" are being helped.

Riddle me this. If you got what you wanted, and these value destroying people went away, what would they be replaced with?

I don't want them to go away, I want their talents put to use towards creating broader value.

It's amazing to me that regulators have not required a minimum latency, or random latency dispersion in orders/trades to level the playing field.

Should they also require a F1-like cap on FLOPS for offline model fittings? A limit on bandwidth? What about quant compensation?

> The NYSE runs out of a public data centre (called NY4) which is run by Equinix.

No. NY4 is in Secaucus. NYSE operates out of an ICE (NYSE parent co) owned facility in Mahwah about 25 miles north of there. They managed to pick out the one big US equities exchange operator _not_ running in an equinix facility.

Sorry but this whole post sounds like someone who is sort of HFT adjacent but doesn't really know what they are talking about. Sending orders at "09:29:59.9999971 at the hope your order arrives at 100ns past 9.30am." What?

> Sending orders at "09:29:59.9999971 at the hope your order arrives at 100ns past 9.30am." What?

This literally does happen, though. One of the things the hyperscalers have convinced the world is that precise time is hard. Precise time is easy if you are willing to pay extra for your hardware. Sub-10-ns precision is unremarkable when you use PTP.

It doesn't happen. All the exchanges have a "Day" order type that you can send before 9:30 that will be live on the book when it opens at 9:30 (or transitions to the "core" session at 9:30, most US exchanges have a premarket session prior to that). The idea of having some sophisticated strategy that sends 100ns before 9:30 is nonsense.

As far as I know, you're correct that this exact trade probably doesn't happen on the US exchanges - day orders do have a matching phase before market open, so it may be advantageous to slide in right afterward, but you likely wouldn't do it without knowledge of the state of the opening auction.

However, sending things just a hair early for scheduled events to catch an exact time is a pretty well-known trick at this point. I remember complaining to the exchange that their clocks weren't precise enough for this to be reliable.

Lets start with exchanges allowing for trades during roman catholic holidays

IEX Exchange is building a cloud-first stock exchange that uses the concept of "slowed trading" to eliminate some of the worst practices of HFT. They even use a 38 mile loop of fiber to slow down connections that are "too close".

https://en.m.wikipedia.org/wiki/IEX

I'm not sure IEX is cloud-first, that would be a recent development. Their 38 mile fiber gimmick is also kind of silly because they have to provide data to a consolidated feed with no delay.

How would any cloud offering deal with something like, say, the full options data feed, which is close to 40 Gb/s of binary packed goodies? You need both a very fat pipe and ultra-low latency: does the cloud, any cloud, offer that?

Also: how often have you guys seen the stock market being down? What's the "x nines" availability of, say, the US stock market and US options feed?

Now: do we wanna talk about the various cloud outages that made the news? Sometimes lasting hours?

Also what I've seen with the cloud is websites are now displaying spinners everywhere, for the myriad of not-low-latency-at-all microservices often taking seconds to respond. And that'd be on an ultra low latency fiber to the home setup, with 2 Gb/s down (and the ISP really supporting that).

Why the heck do I have to wait seconds for oh-so-many things to display in my browser, on a last gen Ryzen ultra-speedy machine, with a super fat and low-latency Internet pipe? The worst offenders being all those banking websites showing a balance of 0 instead of "-" or "n/a" while fetching my info: nearly gives me heart attack every single time.

I take it it has to do with micro-services all contacting shitloads of other micro-services, all living in the not-low-latency cloud. The problem being compounded by an army, a generation, of programmers who have never learned anything about optimization or latency and who solve every problem they have with the only hammer they have: the cloud. All these programmers know are JSON (or, worse, XML)...

I mean: JSON vs 40 Gbit/s of interrupted bit-packed binary feeds? How could these two world ever reconcile?

Now I don't do HFT but I do trade options and I do it through a desktop app and that app also offers an API through which I can fetch prices, send orders, etc. It's a good old Java app. And it's more advanced than any website I've ever used.

Can we please not enshittify everything with countless micro-services and JSON files in the cloud?

Only someone that both misunderstands microservices and HFT architectures could have this opinion.

100 Gb/s is possible on AWS via Direct Connect.

What does cloud have to do with micro services and json? Crypto exchanges already run on the cloud and it works.. mostly fine?

> Also: how often have you guys seen the stock market being down? What's the "x nines" availability of, say, the US stock market and US options feed?

Well, it goes down every afternoon :p and it's down on the weekends. More like seven sevens than nine nines.

It's been a while since I noticed a story about a significant stock exchange disruption, but there's a lot of things going on there. Tickers are largely independent, so it's easy to shard, and exchanges do shard them; (operational) trading outages often affect only a single stock, or rarely a set of stocks. There are procedures for administrative trading halts on individual stocks or the whole market and procedures to resume trading during the market day. There are also procedures for resuming trading after an operational error halted trading; I'm sure most traders don't like brief outages, and exchanges do their best to avoid them, but they can happen and be resolved with out a lot of confusion because the procedures are known and manageable.

There's also redundancy. If one exchange is having difficulty, there are many others that likely still work. Whole system events are usually not operations issues, but trading issues --- one or several participants placed weird orders, the exchanges processed them, and weird things resulted. 'Circuit breakers' have been designed in to pause trading when this happens. This is an intentional outage, and it's ok because it's intentional and the parameters are known.

If an exchange goes to AWS, would it suddenly look worse than its competitors? And would that hurt the exchange’s revenue?

There's no directly colocated hosting that I know of, but "direct connect" gets you close: https://www.equinix.co.uk/partners/aws

See Morningstar's choice of being among the first customers of AWS Outpost.

Outpost is literally a AWS branded server rack that is installed into a data center, I don't think it's considered "cloud" in modern stack.

The end-to-end infrastructure (and parts of PaaS delivered within Outpost, from within the set of AWServices) is in fact a cloud "extension", with an on-prem leg, NOT an on-prem solution connected to the cloud. There are a lot of constructs which force the AWS cloud products/services usage, not the traditional on-prem ones.

The outpost deployment in financial services that I've seen is the opposite in cloud extension. These companies want to say they are cloud-enabled, rather than push a full stack onto the AWS regions, they essentially buy AWS outpost rack as a way to extend their on-prem environment to the cloud (mainly S3).

... is there some movement toward "upload the strategy and let the exchange run it"? which would provide a more level playing field, reduce energy and hardware costs, etc?

You mean enabling arbitrary code execution from a third party when you can lose billions of people's money in half a second? Also if two people want to make the same trade, who gets it?

Exchanges do provide very limited special conditional execution instructions such as peg orders or stop orders, but it seems like a hard problem for them to support anything more sophisticated and general.

I agree with your sentiment but I would clarify that historically some exchanges did indeed create custom order types for larger clients that were not always public knowledge. I think that has mostly been eliminated but there are still a range of unique order types you can utilize depending on the exchange.

It doesn't have to be arbitrary machine code. eBPF / WASM coupled with a standard library supplied by the exchange. (Plus the exchange can run it in a VM.)

> Also if two people want to make the same trade, who gets it?

Whoever pays more currently, right? So it can be uniform random and folks can pay for better than random chance, etc.

Or we could just.. not do that? It provides no benefit and a host of downsides

It seems a lot more elegant/efficient/sane to me than trying to squeeze more and more racks into one building. So that's why I'm asking, as I think the benefits are clear, much better scalability, fairness (or getting as close to it as the exchange wants), cheaper (no need for fancy hardware), probably it would attract more market participants (lower barriers to entry).

It's the same hypothetical "EC2 model" without the meta-game of trying to get closer to the cores the exchange runs at a given time.

Can you elaborate on the downsides besides security?

What you're trying to eliminate is just a small part of the infra arms race. But it's not even effective at that. You say one of the goals is to not have to squeeze many racks into a building, but your proposal assumes the servers placed in the racks are running at low capacity, which is completely not true. On the other hand, if you trade say GOOG, the matching engine for that is going to mainly sit in one place, how do you stuff everyone's program to run on the same server, and how do you make sure they each have fair access?

I wonder if that’d make them a broker and not an exchange. Different set of rules, regulations, and licenses.

If you manage your own rack, all you are doing is uploading a strategy and letting it run, returning the executed trades to your own inhouse systems for booking.

If a third party provider (there are several which offer this service) manages the rack, router, host infra, and provides you with the virtual guest OS, or going one step further with the application into which you load the strategies, then it's much easier to "right size" the infrastructure you actually need.

As others have pointed out, the pain point is multicast into the VM. Drop a packet = you lose money.

All that said, some trading is going the other way, RFQ based flows with 30 second quote lifetime, which is perfectly suited to the cloud and it could well be the nature of what is traded moves away from volatility based products (options) to value driven (bonds, funds).

Low latency/rapid trading seems to have added no value to the stock market and eroded a large portion of the fair market evaluation of companies. Whether it's technically possible or not it's likely we'll need to artificially add delay into the system - ideally that can be done in a way that makes it fair to traders that have a naturally high latency connection to even the playing field.

The claim is they play marketmaker (they hold stocks for short periods of time so your trades execute faster), they obviously want a profit incentive for doing so.

That begs the question by assuming that trades executing faster are a good thing to optimize for.

Is the stock market for investing in a company, or for extracting money from momentary fluctuations? Those seem to be mutually exclusive.

> Is the stock market for investing in a company

With the exception of the IPO and buy-backs, the stock market is NOT for/about investing in the company.

When you buy IBM on NYSE, IBM doesn't get anything. Similarly for selling stock. If IBM doesn't see any money from a trade, how is that trade an "investment" in IBM?

Stock trading is trading partial ownership. That's very different from investing.

In other news, there's no money in the stock market. The money that you pay for IBM stock does not go to the "stock market". It goes to whomever owned the stock that you bought.

Low latency trading should be forbidden.

The exchanges should add random amounts of latency to every trade.

Why? Historically spreads have been much wider, I for one appreciate how close they trade now. Instead of a guy on the floor taking dollars from you, you have a bot taking pennies.

Even large funds like Vanguard have said that the current system has reduced costs in net.

> Instead of a guy on the floor taking dollars from you, you have a bot taking pennies.

^ excellent way to put it

They do it's called exchange jitter. And you still have HFT

Add more jitter.

Cloud of rapidly expanding ionized gasses, yeah.

Reply on news.ycombinator.com