Goldman Sachs is spending $100M to shave milliseconds off stock trades | Hacker News Reader

Goldman Sachs is spending $100M to shave milliseconds off stock trades

cnbc.com

195 pointsgscott314 comments

Markets around the world are determining prices on a massive variety of instruments that derive value from the current and future value of products such as currencies, interest rates, equities, grains, livestock, metals, oil, gasoline, natural gas, and electricity. These prices allow us to prioritize resources, make fair transactions, and manage risk (i.e. buy insurance on the value of critical products so that we can plan and invest more effectively). The value of all these instruments are related to one another, as well as to realtime world events, in complex ways that no one entity has a perfect view of. Consequently, markets work by polling the expertise of many different parties who all understand a piece of how things should be valued. This results in millions, if not billions, of interconnected price-discovery feedback loops. The markets are kind of like a massive, distributed, realtime, ensemble, recursive predictor that performs much better than any one of its individual component algorithms could. The reason why shaving a few milliseconds (or even microseconds) can be beneficial is because the price discovery feedback loops get faster, which allows the system to determine a giant pricing vector that is more self-consistent, stable, and beneficial to the economy. It's similar to how increasing the sample rate of a feedback control system improves performance and stability. Providers of such benefits to the markets get rewarded through profit.

Meh, the short term price fluctuations are almost pure noise. Faster trading just incase the frequency of the noise. Increasing frequency only works if the feedback loops are stable.

Over very short time scales, the price fluctuations vary from almost pure noise to almost pure signal. The almost pure noise situation occurs far more often, but even then, with good estimation techniques, you can extract a signal component that can improve pricing a small amount. Improving many interrelated prices by small amounts can lead to a significant overall improvement to the markets. The almost pure signal situation occurs far less often, but it is almost always extremely important to overall stability. Handling it well can minimize market overreactions and extreme events such as flash crashes and market panics.

The faster you can trade, the faster you can flash crash. The current state is that bots can crash the market faster than humans can react, how does that help stability?

The faster you trade, the faster you can detect and help limit or prevent a flash crash caused by others trading poorly.

This system is the only one unbiased estimator / decision maker humans ever found. Though it has quite high variance, in the long-term its results are just astounding

If it's unbiased, that's the same as saying it's random?

If it leads towards efficiency, productivity, most beneficial allocation, as suggested, those are all biases.

An unbiased estimator is a technical term. It means that its errors in estimation are equally distributed above and below the true value.

Edit to add: I am not sure GP was using the term accurately, either.

I second this.

An issue I have found with technical minutiae having,for lack of a better term, general names is that they are prone to being used wrong. And once a critical mass of persons start using it wrongly, there's no going back.

An unbiased estimator is one whose expected value equals the true value. Its sampling distribution can be asymmetrical with median not equal to the true value. You seem to be referring to a median-unbiased estimator.

You are correct, of course. It’s too late for me to correct my comment; I slipped up while trying to de-technicalize the definition. Thanks for the correction.

> If it's unbiased, that's the same as saying it's random?

No, its deviations from the true value are essentially random.

>> The markets are kind of like a massive, distributed, realtime, ensemble, recursive predictor that performs much better than any one of its individual component algorithms could.

That's really interesting, I never thought of it that way.

>> markets work by polling the expertise of many different parties who all understand a piece of how things should be valued.

Does the whole picture ever become apparent to all of the interested parties after the fact? Or do market movements remain subject to a high degree of interpretation even after they happen?

Nobody has the whole picture. I would say that the more extreme an individual event is, the easier it tends to be to understand in retrospect. This depends greatly upon one's analytical sophistication, level of market data, fundamental product understanding, and professional network (to know what happened in other firms).

If nobody has the whole picture, how can we be sure if it's beneficial?

edit: Or asked differently: What do we have now with high frequency trade established compared to the situation before?

It's enormously cheaper to do business on markets nowadays. Back in the old days, spreads were sometimes multiple dollars on human-made markets with a lot of inventory.

With HFT, people compete to offer the best market, and spreads are in the pennies.

That argument is basically Hayek's (and Mises's) answer to the economic calculation problem. Hayek argued (in "The Use of Knowledge in Society") that free markets are better than centralised planning, because of dispersed knowledge.

https://en.wikipedia.org/wiki/Friedrich_Hayek#The_economic_c...

https://en.wikipedia.org/wiki/The_Use_of_Knowledge_in_Societ...

> The reason why shaving a few milliseconds (or even microseconds) can be beneficial is because the price discovery feedback loops get faster

Berkshire Hathway has couple of trades every minute and difference in bid/ask prices is huge around $1000+, yet you dont see people complaining about that.

The bid/ask spread is a separate issue from latency (Edit: although it should tend to be less with better price discovery). To appreciate the benefit of lower latency, I think you have to consider the bigger picture of many interrelated price discovery feedback loops involving many instruments. Many small speedups can result in a much more stable and beneficial system. In the case of Berkshire Hathaway, the value of their stock is dependent on the value of many other equities, interest rates, energies, raw materials, etc.

> the value of their stock is dependent on the value of many other equities, interest rates, energies, raw materials, etc.

Exactly, its the same with other stocks.

> The bid/ask spread is a separate issue from latency.

Yes but it also the second point made by HFT's that they reduce the spread.

Any market maker's bid-ask spread will be dictated by a combination of factors, including aspects like latency (relative to competitors), expectations on holding time, historical volatility and spread (that one is somewhat circular), availability of information from correlated markets, ability to hedge into correlated markets (and what are _those_ spreads, as the market maker will be crossing that spread to hedge), trading fees, etc.

In markets where many of those aspects can be minimized, market makers will end up very tight and the dominant factor becomes latency. a fast market maker can offer a tighter spread to counterparties and will see a virtuous cycle of increased trading opportunities leading to revenue to stay fast.

If the other factors remain significant risks, then the benefits of latency optimization fall off. So, spread and latency are related, but other market fundamentals do play a part.

> will see a virtuous cycle of increased trading opportunities leading to revenue to stay fast.

Not sure whose revenue you assume to be fast.

Anyway I think you did not understand the point I was making, Berkshire's spread is huge and has been huge for decades, yet you dont see lot of people complaining.

Is there any empirical evidence that these perceived benefits to society actually ever materialize? It's clear that there is a benefit to a trader from knowing something milliseconds before the rest of the market (otherwise Goldman wouldn't be doing this), but it's not clear at all to me that it helps the rest of us.

We know, empirically, that a lack of liquidity increases trading costs, which in turn is directly channeled to the prices of goods and services that rely on this liquidity (more or less everything in the world, even more indirectly ones like education).

It's difficult to say 'things would be X% more expensive' because of the interconnected complexity the GP was talking about, but there is definitely a very apparent benefit.

> a lack of liquidity increases trading costs

Sure but, does a couple of milliseconds to or from affect that?

You have to consider some form of multiplier effect that comes into play. Because the prices of assets are derivatives of others (incl. many, many other variables, of course) these millisecond savings end up turning into seconds, even minutes en masse.

Yes, and I'll tell you exactly why. Markets consist of informed and uninformed traders.

For example your typical Joe Sixpack rebalancing his 401k is uninformed. His trading does not tell you anything about the underlying value of the stocks. The typical informed trader is a hedge fund manager, who's investing tons of resources in gaining an informational edge. If he's buying a stock at a particular time that is in and of itself a signal that said stock is worth more than you thought it was otherwise.

Liquidity providers love trading with uninformed traders. The problem with informed traders is that you don't want to be on the other side of their trades. Since liquidity providers are the immediate counterparties to most order flow, they're the ones that primarily eat this cost.

To counter this, liquidity providers invest enormous resources in profiling order flow to try to identify when to what degree its informed. This allows them to provide lower costs and more liquidity to uninformed traders, like Joe Sixpack. It's analogous to how requiring a checkup allows life insurance companies to provide lower premiums, particularly to those who are healthy.

One of the most important ways to profile order flow is to quickly adjust quotes as market conditions evolve. For example said hedge fund manager may be trading a thesis that the chipmakers are all undervalued. He may come in and buy Intel, AMD and Nvidia in one swoop. If an HFT sees a huge buy hit Intel, it can bump its quotes on AMD for a few milliseconds.

If its a cigar-chomping hedge fund manager executing a basket algorithm, it's quite likely that he'll try to hit AMD and thus pay the higher price. But if its Joe Sixpack the probability that his trade just coincidentally lands in a 5 millisecond time window is vanishingly small.

I think the question is at what point does liquidity have diminishing returns?

If a security could only be traded once per 10 years then it's obvious that its lack of liquidity would make it less valuable. Holding it would tie up your capital quite significantly.

However, if you had a turn-based market where every trade got cleared at the top of the minute it's not clear to me at all whether that would effectively be less liquid than what we have now.

It seems to me that a model where traders all compete for how many nanoseconds away their HFT servers are from the action doesn't really benefit the market as a whole. If anything it just makes things like flash crashes more likely.

In reality the way that markets are cleared wasn't necessarily so much more different than a turn-based one, even in one where the clearing happens by the minute. The issue is, indeed, mostly with HFT. However, you shouldn't consider HFTs to be market participants in the traditional sense. Most of them focus solely on moving stuff around very fast instead of actually trying to purchase or sell things for a separate economic goal (e.g. production, hedging, short- and long-term investments).

Once you disregard the rapid transactions that do not have a significant effect on the price, your average human investor is probably putting down some fill-or-kills or limit orders and actually benefits from HF liquidity trading.

It's hard for me to understand, let alone explain, why milliseconds would matter for the human investor, but I what I can tell you is that GS is not investing 100M USD just for buying derivatives every other minute.

I think part of the problem is that "liquidity" is an imprecise term. It implies both velocity and flexibility. HFT definitely increases velocity, but it can make the market either more or less rigid depending on the circumstances.

> I think the question is at what point does liquidity have diminishing returns?

The general consensus at this point is - no one actually knows - and it's up for serious debate. We know lack of liquidity absolutely has negative effects (because we've experienced it), but we don't how much liquidity is "too much".

Why don't we install a knob that we can turn that effectively limits trading to X seconds precision?

One day we may decide to turn that knob from the millisecond range to 1 second and see what happens. If it's bad, we can always turn the knob back.

We kind of already are doing this: https://www.bloomberg.com/opinion/articles/2019-08-01/why-ex...

> If it's bad, we can always turn the knob back. Sure, but by then maybe some people or organizations have made/lost millions. So we can't realistically experiment with that (even though I would absolutely love to)

If a businessmodel is questionable from a societal viewpoint, then they shouldn't be surprised that some regulation hits them. See for instance Airbnb, where the businessmodel has become impossible in many cities already.

it would make more sense to first get rid of the penny rule.

I feel like we do know how much liquidity is too much, in the sense that the HFT traders are siphoning profits from value-seekers to rent-seekers. The total of their profits is the amount by which there is too much liquidity.

It's not necessarily clear whether it could be reduced to seconds of liquidity, or minutes, but it's clear that milliseconds is too short. The money isn't remaining in the market long enough to provide value -- especially when the market has already been made and they're just trying to figure out who gets the surplus between ask and bid.

A turn based system seems like a horrible idea. First the major players would have a legitimate excuse to make sure their trades were first in the queue. Second they would immediately game this so that everyone in the line behind them had to watch as they triggered market changes one minute they were then in a perfect position to take advantage of the next minute.

Wouldn't that be blatant market manipulation? The SEC would fine the hell out of anyone who did that.

In a turn based market your place in the queue is determined only by the price you bid. Everyone trades at the same price, except for those who bid too high/low, they don't trade.

These systems are already running in many markets, typically at a rate of one trade round per day, but using them is optional.

> However, if you had a turn-based market where every trade got cleared at the top of the minute it's not clear to me at all whether that would effectively be less liquid than what we have now.

The primary problem with that is that it pools order flow into a homogenous, undistinguished pool.

HFT heavily rely on profiling order flow into the informed and uninformed. A typical uninformed trader is Joe Sixpack who's rebalancing his 401k. A typical informed trader is a hotshot hedge fund manager, who's invested enormous resources in gaining an informational edge. If Joe's buying a stock that doesn't tell you anything about the value of the stock. If hotshot hedge fund manager is buying a stock, that in and of itself is a credible signal that the stock's worth more than you thought it was.

Liquidity providers love being the counterparts to Joe, and hate being on the other side of the hotshot hedge fund managers. The more you can profile the order flow, the better prices and more liquidity you can offer to Joe, by charging the hotshots more. Think of how life insurance companies can offer better premiums, particularly to the healthy, if they require a physical exam before underwriting a policy.

Even on a millisecond by millisecond basis, there's a ton of distinguishing characteristics regarding the informational content of order flow. Uninformed flow basically looks like a bunch of small, randomly spaced trades. Informed flow is more likely to cluster together in small time windows, move sequentially in the same direction, try to sweep liquidity with huge trades, and immediately follow similar moves in other securities among other things.

If you pool all orders into a homogenous one minute pool, HFTs would lose much of their ability to segment order flow. The end result would mean that the hotshot hedge fund manager would see his trading costs reduced, and Joe Sixpack would see his trading costs increase.

Nice explanation.

However, you postulate that the introduction of a turn-based system would effectively transfer money from Joe Sixpack (who'd face higher costs) to hotshot hedge fund manager (who'd face even lower costs).

Maybe, though, we'd see HFT shops go out of business (and not building micro wave towers between Chicago and NY anymore), and see a transfer from HFT shops to hotshot hedge fund manager and Joe Sixpack, both facing lower costs.

How do you know it's not this second scenario?

Forgive me for asking but what does GP here stand for?

Grandparent comment, i.e., two links up in the comment tree.

(It's a common term on internet forums -- I learned it from Slashdot years ago)

Thank you very much. I thought of searching the web but GP is too generic a term.

GrandParent comment.

The comment directly above is the parent, and the comment above the parent is the grandparent—in this case the GP they’re referring to was 1e-9’s comment.

GP grand parent https://news.ycombinator.com/item?id=8363625

Not all liquidity is the same. I've seen calculations that stated that HFT doesn't actually add valuable liquidity to markets because the moments you need that extra liquidity are price uncertainty moments where HFT traders disappear. I don't have the reference on hand though.

And yet Vanguard say that HFT helps lower their costs by adding market liquidity and therefore reducing spreads:

https://www.forbes.com/sites/timworstall/2014/04/28/bill-mcn...

Not a hostile question, I'm genuinely trying to understand this: How is liquidity provided by, for example, someone interjecting themselves into a trade that was already going to happen?

If you want to move a large amount of money into an asset, you either have to place bids and wait for people to take your order at a particular price, or you have to pay a slight premium to dig into the ask side of the book and pay more money as you consume orders and liquidity.

If you place an ask and the price moves down it might not fill, so you have to move it over time while the price slips.

People provide liquidity by seeing your new sell order and filling it quickly, or by leaving a large number sell orders that people can take immediately, which results in money moving more quickly and consistently.

Willingness to accept a smaller spread, but more volume?

> lack of liquidity increases trading costs

Sure, that's almost tautologically true, but that does not mean that the benefits to society materialise:

* Traders getting faster access to an exchange have an advantage against other traders (and might make more money), but that does not imply at all that the market will be more liquid.

* The volume of actual utilitarian trades (investing, borrowing, asset exchanging, hedging) is relatively small compared to overall trading activity. A pension fund investing, someone taking out a mortgage, people exchanging currency for the holidays, the often cited farmer hedging his crop - they trade infrequently, and certainly don't care about some milliseconds.

* The dynamics are those of an arms race. Arms races are wasteful, by and large. Building a "straighter" fibre glass connection between NY and Chicago, and then building a series of micro wave towers (because the speed of light in the air is greater than in the fibre) - how does that benefit society? Just postulating higher liquidity and lower costs is not enough to justify it, I think.

(I am reminded of the earlier discussion about advertising - once Cola does it, Pepsi has to do it, too. Are we thus better off, or would we get cheaper sodas if both didn't?)

Business produces goods and services.

Finance is an online multiplayer game.

They are two weakly-connected systems.

>Business produces goods and services. Finance is an online multiplayer game.

But finance is also products & services.

Think of a corn farmer. It's easier to think of him adding tangible value to society because "corn == food".

In contrast, finance just seems like useless office workers copy pasting numbers around in Excel spreadsheets. (This is probably true in many cases.)

But the farmer often wants to sell "futures" which is a product & service provided by the financial industry. Instead of using the jargon of "futures", we can just say the farmer wants a product/service to give him a "guaranteed-selling-price-regardless-of-future-volatility-of-corn-prices-so-I-can-sleep-at-night".

Other examples of desirable finance products that farmers want include crop insurance and equipment loans/leases.

>They are two weakly-connected systems.

Farmers' food crops and the financial products/services of of futures is an example of how businesses making tangible products and the finance industry are strongly connected.

Those are the weak connections I referred to. Last I looked, the volume of financial transactions was about two orders of magnitude larger than the volume of purchases of goods and services. So maybe a couple percent of futures transactions involve actual buyers and sellers of the commodity, while most trades are among financial system players.

But for example the crypto market has no goods or services - only the market trading aspect. Yet it's value swings wildly anyway.

finance is the meta-game. "All problems in computer science can be solved by another level of indirection"

If a market can't be cleared using a spreadsheet with 2^16 rows and 2^8 columns then the market is not efficient.

Largely because most of the trade strategies live in ancient xls files no one dares edit.

Nope, it's pure loss to everyone else. Every dollar that an HFT makes by front-running your trade is a dollar you don't make as an investor. The money isn't coming out of nowhere.

HFT's are purely parasitic unless you're another HFT benefitting from the faster trades. But what's a solution that doesn't add so much friction the house of cards falls down?

The theory sounds great. But why then, our streets are lined with homeless, and our nations are stricken with poverty? Could it be that the only real aim and motivation of market traders is to earn money? One day, maybe.... when these are replaced with DAOs on the blockchain. But until then it's the Wolf of Wall Street.

The markets on which financial instruments are traded are effectively separate from the ones where policy decisions are being made on, such as labour, vocational education, and healthcare. It's not the derivatives trading that is problematic, it's the market inefficiencies not being resolved by governments.

Because having homeless people lining our streets on our commutes to/from our jobs is a daily reminder that if we don't work hard enough to increase corporate profits, then our bosses might lay us off and we'll end up like them.

That or moral apathy. At some point in the 80s we decided that markets driven by business profits should dictate every aspect of society.

I imagine 100 years from now they'll look back at today in disgust.

100 years after Communism burst on the scene, we currently look at that development with disgust.

Right now most people have access to abundant food, cellphone in every pocket, access to a wealth of information, access to transportation, incredible medical advances.

I can't imagine that the progress we've made would be scorned. Like other market driven forces, bad players will not be rewarded as information about them increases.

However, if information is not increased because of something like a company buying a newspaper so investigative threats can be used against politicians to avoid information gathering, then we have problems. This is more crony capitalism than just capitalism.

"What we have now is the bad kind of capitalism! There's a different, good kind of capitalism which in theory does all these great things!" is essentially the "communism works great in theory" argument.

100%, both systems are great theories. However, the last few hundred years of actually trying to implement capitalism has "most people have access to abundant food, cellphone in every pocket, access to a wealth of information, access to transportation, incredible medical advances.", if by "most people" you mean "possible a majority of people in the richest countries in the world". Unfortunately, the cost of that is that we've done irreparable damage to our environment, are causing the worst Great Extinction ever, and have caused a climate crisis that may cause us to go extinct.

Hundreds of millions of people have been brought out of poverty, outside the United States.

That's due to technology; not capitalism, they aren't the same thing.

A huge amount of R&D happens in academia which isn't capitalism; but then is monetised by capitalism (but doesn't reinvest it back into the academia)

Converting that R&D into actual products and services, I'd wager, is quite difficult without a profit incentive. It's like the difference between having a great idea for a startup and actually turning it into a functioning company.

Without invoking too many absolutes, there's so much bullshit involved in the latter that people doing the former aren't willing to put up with. It's mostly two different kinds of people with two different skillsets.

Doesn't capitalism either directly or indirectly fund that academic research? When I was in a university lab, all of my work was funded by private companies

Nearly every country on Earth that has bought into the Capitalist system has become far better off than they were 100 years ago, not just the richest countries in the world.

I agree that "No true Capitalism" is just as bad a fallacy as "No true Communism". But our real life imperfect Capitalism has still had incredible results whereas real life imperfect communism has lagged significantly and failed more brutally.

You are right that the world has gotten immensely better in recent decades, pulling hundreds of millions out of poverty.

If you compare real median wage growth in the West in the 60s and 70s with the last two decades though, it doesn't look so great. Maybe we can do better.

Woa who said anything about communism? All I suggested is that we should take care of our citizens. We could do this for example via Universal Basic Income, which is very much capitalist (unless you abide by the common American false notion that "helping people" = communism).

> most people have access to...

Why is "most" a good enough metric? If most people have homes but my commute to work is littered with tents of homeless people, is that adequate? Should politicians just throw in the towel then and call it a day because "most" people have houses?

I think we can do better.

I find it funny that people scorn the pursuit of economic profits, but complain that the other "better" activities aren't given economic rewards.

Ultimately "Corporate Profits" produce the economic value that people desperately want, and participating in the creation of something people want _should_ be a prerequisite for getting economic value in return.

I'm not really sure what you're arguing here. All I'm saying is that corporate profits shouldn't dictate every aspect of society. That's why sane countries have implemented things like universal healthcare and free primary/secondary education.

I'm also proposing that in a wealthy first world country, perhaps nobody should have to go homeless. Crazy idea, I know.

What have these "corporate profits" you worship ever done for homeless people?

Yes, the real aim and motivation of traders is to earn money and respect. What else should it be?

Why pick on traders? I know a lot of developers making well into six figures. I hear them talking about getting 3 new graphics cards for their gaming rigs instead of how they worked at a soup kitchen.

What is your point? That GS should be donating 100m to charity instead of reinvesting into their business?

Where do you think that money goes? Workers will be paid to implement their plan and taxes will be paid on those wages. In fact about 40% of that 100m will eventually end up being paid in taxes.

I don't mean to pick on traders. Nominally it's a hard and honest job and money made is money deserved.

The problem is that if I have a company [and this is a systemic example, no exceptions, see #1], and I allow (central-)bank "friendly" people on its board, so that we can receive as many low- or zero-interest loans (with open due date or refinancing at will, i.e. free money, printed freshly from thin air) from the bank, so that we can under-price, destroy and acquire all our competition, become a monopoly AND finance a massive lobbying power in the DC so that we can get laws passed which increase our profits (at the disadvantage of the citizen), you can bet all your savings that such system's demise is written in the fabric of space and time, because the most essential feedback loops (and the ones that you mention, the ones in the market, work in exactly the opposite way) in that system have been disabled and its just a runaway train without brakes.

Buying laws starts and finances wars, relaxes food, water and environmental toxicity limits, enables false advertising, eventually raises taxes, enables trading of derivatives so detached from reality that a computer game pales in comparison, you name it. The days of this system are numbered and we should really speed up the development of trustless alternatives based on blockchain, or we're going to hit the wall really hard.

#1 https://www.newscientist.com/article/mg21228354-500-revealed...

The upshot of this argument is that this is valuable activity. We need markets to price tradable assets and provide liquidity.

The counterargument is that there are diminishing and/or negative returns to increased liquidity and velocity.

Take just stocks. Liquidity is not a problem. You have liquidity whether trades take minutes or milliseconds. Pricing? I'd say we have pricing covered too, at least the pricing that more/faster algorithmic trading will contribute.

Meanwhile, all this stuff costs money, people, resources that aren't available for actual productive work instead of overhead.

My view is that current liquidity and pricing are far from perfect, particularly when it matters most (e.g. a market panic) and particularly in the global context of the vast universe of interrelated instruments that need better relative pricing. Given that we benefit from realtime pricing, milliseconds matter when you must determine a large vector of prices with complex dependencies using the ensemble recursive system that is the global market.

> My view is that current liquidity and pricing are far from perfect, particularly when it matters most (e.g. a market panic)

And in a market panic, your friendly HFT shop next door is there and offering to buy and sell to stabilise the price?

> Given that we benefit from realtime pricing

Yeah, if you assume the conclusion that we benefit from it, then we do. But have you shown this?

In which market do we benefit from milliseconds pricing, compared to, say, an auction every minute?

> And in a market panic, your friendly HFT shop next door is there and offering to buy and sell to stabilise the price?

Some are. It's not that they are friendly, it's just that doing so can be highly profitable if one can estimate the mispricing with a sufficient degree of certainty.

> In which market do we benefit from milliseconds pricing, compared to, say, an auction every minute?

I argue that all of the significant ones benefit. The global market is huge and interrelated in complex ways. There are many trading entities with a variety of specialties. They communicate their expertise through the markets by placing orders. It's an iterative process and a lot of information must be conveyed. The faster the entities can communicate back and forth, the more accurately the prices can represent a weighted consensus. Constraining the trading to 1 minute auctions would reduce the communication bandwidth.

>that aren't available for actual productive work instead of overhead.

That’s a very biased view. Another view would be that improving the efficiency of the largest markets in the world have a much larger positive impact on society than the vast majority of the “productive” work you refer to.

It's biased to the premise, which is that "day trading's" contribution to price efficiency & liquidity are of diminishing value, and that the marginal value is essentially 0 or negative.

> Meanwhile, all this stuff costs money, people, resources that aren't available for actual productive work instead of overhead.

That's a very high standard. What's productive? What's productive enough, in your book, to be worth the effort used here?

Median productiveness is plenty, assuming this is not productive activity.

But what's "productive?". Farming? Manufacturing? What if the farmed food is unhealthy? If the manufactured good is wasteful?

Yes, that's the model. So, tell me. Would any profit be possible in an ideally functioning market? If so, how is any item worth more than the total cost of all inputs including externalities? If not, how is such a dynamic system attracted to an ideal state, given that those who would make the market ideally functional are best placed to gain from market inefficiency and dysfunction? Given an answer, do you have a sound game theory explanation for this?

Additionally, what is the nyquist limit for such an ideally realised market, if it is indeed to be modelled as a recursive sampled approximator and how is this derived? Given an infinitely recursive network of arbitrarily connected market agents, is any such calculation convergent? If so, why? If not, how does the market ever converge to any appropriate price - a price which accurately reflects the market conditions excluding pricing operations and market costs which aren't directly related to the production of the instrument in question?

Keep in mind that, if the market is functioning ideally no market participant will exceed the nyquist rate as all participants knowledge of market conditions converges to zero. How is any sampling rate, excluding zero, convergent? If not, how is any such market realisable? If so, what is the loss function between ideal model and realisable, perfectly imperfect real world implementation? What is the minimum profit, if not zero, and why?

However, it does seem that arbitrage opportunities decrease when such high-speed trading is occurring and, does so even more quickly the faster trading speed and market sampling are increased. How can we account for this, if not by increased market efficiency?

I conjecture that, by ever increasing the sampling rate and the speed at which transactions complete, markets are not being made more efficient. Instead, I hypothesise that, as markets directly effect the price of the instrument reflexively, the feedback latency produced creates relative local pockets of perceived value - which are only profitable trades in relation to local information asymmetry. As the vast majority of high-speed trading holds market positions on extremely short time scales, shifting exposure constantly, this profit is immediately realised locally resulting in the gradual diffuision of this inefficiency as the increase in price of all instruments. This is a direct result of the cost of trading being factored directly into the agent's local acceptable sale price of held instruments. Every local agent trading action is ideal, but the global market is a divergently inefficient one. Indeed, it is a market in which its pricing inefficiency is maximally concealed from all market participants.

In a sense, I conjecture that the estimator is not functioning to increase market efficiency but is, instead amplifying local inefficiency globally, in effect, much like a charge pump would operate in a voltage multiplier circuit. In essence such a scheme acts to conceal increased market cost and overhead (including the profit of market participants) into market instrument pricing. However, it does so in an extremely small and diffuse way so as to make the rise in price of a single instrument, as a result of this activity, extremely difficult to detect as all instruments increase similarly on the same time scale.

This behaviour appears to be similar in nature to 'salami slicing', an often effective embezzlement technique - except that, instead of exploiting an information asymmetry created by lack of interest in small quantities in the part of auditing accountants, it exploits the information asymmetry created by the speed of light itself.

Of course, the faster the sampling rate, the more efficient the described amplification process would take place. Does this effect correlate between markets with differing but estimable information asymmetry? If there is no correlation, this hypothesis is invalid. It would seem to be an area ripe for research and analysis of market data.

Do you see any technical issue with this conjecture by which we may discount it immediately?

A perfect market where all participants get information at the same time does not exist.

>This profit is immediately realised by the increase in price of all commodities globally.

This would only hold if there weren’t profitable short trades. The profit can also be realized by the decrease in global commodities that would have been slower before.

I'm sorry for the confusion, I should've reviewed my post before submitting - I have edited that section extensively for, hopefully, greater clarity of thought.

In a perfect market, there would be no economic profit. Your entire comment is very interesting; sorry for the short response!

No worries at all. I know it's an extremely long comment on a comment on an article - thanks for reading it.

On further thought, the conjecture's behavioural outcome is actually not quite so analogous to 'salami slicing' as it is analogous to monetary policy caused inflation. In effect, the amplification effect would serve to create profit by creating an apparent valuable trade where none actually exists - such trades essentially print money. This activity would function much like the "profit" realised by a central bank when it chooses to print additional currency for redistribution at government prerogative.

However, monetary policy induced inflation is merely limited in effect to those exposed to any one central bank's monetary policy domain - and generally only occurs when the money supply is permitted to rise for all participants. The type of inflation produced by the activity outlined by the conjecture is inherently global - and would exert a pressure on all existing markets which permit this type of trading; and it is not governments, which are ideally responsible to those they represent, which benefit from this inflation - it is private market participants, in the profit they realise from each trade.

This would certainly seem to account for the new behaviour of central banks having to cut their interest rates to near or at zero to compensate for this asymmetric inflation to drive slowing market activity outside of the financial sector... if the conjecture holds - they appear to have entirely lost control of monetary policy to the global market - and those who are best placed to capture value in those markets as a gestalt - via this mechanism.

If the conjecture holds - and central banks and regulators are unable to reign in the behaviour globally - the economy will experience hyperinflation of Weimarian proportions. Unfortunately, such inflation will have vastly asymmetric effect - benefiting only those best positioned to participate in and drive the amplification behaviour itself.

Indeed, it appears to be a naturally occurring divergent state in a market permitting ever higher sampling and clearing rates. Such behaviours are increasingly profitable - seemingly without end - and so it will attract a geometrically accelerating amount of market activity until such activity is no longer profitable due to market collapse.

The analogy is a fascinating, and scary, thing. It's a bit like considering someone nucleating the economic equivalent of a false-vacuum collapse - or someone already having done so. I need to think about it more and find some way of formally stating and ideally disproving the conjecture.

We might disprove the conjecture by looking for anti-correlations in the growth, availability and capacity of high-speed trading and clearing in markets controlling for the returns of financial institutions instruments and portfolios and the changing monetary policies of various central banks under whose jurisdiction they fall. Simulation of economic systems with and without these elements might also yield some insights, when compared to market conditions at large.

Is anyone aware of any other similar research, work, and/or thought in regards to this concept?

I'm an Econ grad student currently not paying attention in a Maths class (something about the Implicit Function Theorem) and I'm absolutely intrigued by your two comments. Kudos.

Also, to sidetrack a little bit, may I ask how long it took you to gather these thoughts and post them? I'm trying to get a sense of how far along I am about gaining a holistic understanding of markets and trading.

While I don't think any time taken is necessarily a great indicator of general understanding, I was struck by a great feeling of unease while reading 1e-9's comment. Initially I sought to understand the apparent contradiction in the measurable reduction in arbitrage opportunity as a generally accepted proxy measure for increasing market efficiencies with my general understanding that HFT generally massively increases volatility and does not generally appear to result in the reduction of instrument pricing due to lower trading overheads and losses. I also sought to explain the seemingly unending profit stream made possible by such strategies - when, paradoxically the more efficient the market becomes, the less profitable all HFT stratigies should become globally - and yet, it does not seem they do.

All in all, about 10 minutes or so of consideration, followed by about an half an hour of editing.

That said, I'll likely spend much more time looking for existing models of financial markets under the information relativistic conditions created by HFT activity - it occurs to me that as trading moves closer to speed of causality in the market the models underlying market understanding may need to be adapted, perhaps using relativity as a prototype. With any luck they already have and I can elaborate from those to solve for conditions of such markets with information asymmetry and agents capturing value. If such markets are inherently volatile and that volatility increases geometrically nearing the speed of causality - presumably, the speed of light, then this may provide the mechanism for the apparent global inflation of instrument prices via distributed profitable high-speed trading activity while preserving lessened arbitrage opportunity and other visible market behaviours.

I really must formalise this so that it may be thoroughly and logically evaluated - both symbolically and under simulation. However, I'm at a disadvantage in that I am merely a dabbler in the field of economics and game theory. I also have no formal background in stochastic finance or physics. I am but a Systems Engineer. So, fun challenges ahead.

I don't know, I see many problems with this comment (akin to the writing of postmodern continental philosophers). Maybe you can restate your conjecture again?

> In effect, the amplification effect would serve to create profit by creating an apparent valuable trade where none actually exists - such trades essentially print money.

Trading is a zero sum game (notwithstanding the allocative function enabled by proper price signals), so I don't see how it would engender inflation.

> the new behaviour of central banks having to cut their interest rates to near or at zero to compensate for this asymmetric inflation

There are many theories about the persistent low rates ("secular stagnation") etc., but I've _never_ heard that particular problem linked to HFT.

> I conjecture that, by ever increasing the sampling rate and the speed at which transactions complete, markets are not being made more efficient.

That's fairly clear, and I can agree with that.

> Instead, I hypothesise that, as markets directly effect the price of the instrument reflexively, the feedback latency produced creates relative local pockets of perceived value - which are only profitable trades in relation to local information asymmetry.

What?

> As the vast majority of high-speed trading holds market positions on extremely short time scales, shifting exposure constantly, this profit is immediately realised locally resulting in the gradual diffuision of this inefficiency as the increase in price of all instruments.

Not sure what you're saying there, but of course the idea is that traders with superior information can realise trading profits, and via such trading, information spreads through the market, until no such trading opportunities persist. However, HFT does not necessarily follow this kind of Hayekian vision, but is maybe more insightfully analysed in a game-theoretic framework.

> This is a direct result of the cost of trading being factored directly into the agent's local acceptable sale price of held instruments. Every local agent trading action is ideal, but the global market is a divergently inefficient one.

> Indeed, it is a market in which its pricing inefficiency is maximally concealed from all market participants.

What?

> In a sense, I conjecture that the estimator is not functioning to increase market efficiency ...

Possibly, yes.

> ... but is, instead amplifying local inefficiency globally, in effect, much like a charge pump would operate in a voltage multiplier circuit.

How is it amplifying it? Yes, we have seen flash crashes, sure, resulting in some transfer of wealth. But this does not explain or predict inflation, geometrically accelerating market activity, nucleated false-vacuum collapse, or any such things.

All excellent points and questions. I'll consider appropriate answers and restatements and respond soon. I'm rarely satisfied with my ability to communicate ideas like these. I really must express this conjecture formally - in the language of mathematics - so that it may be unambiguously communicated in the appropriate contexts.

That gives these mechanisms way too much credit. Markets can benefit from a more accurate valuation but ms-response times are not needed for this.

The real reason are competitors. You have an advantage if you have the faster line. In the name of fairness there are lines of the exact same length* in many trade centers precisely for this reason.

There are bots that feign transaction so that others react in a specific way. In the last moment these are canceled again, too late for competitors to still react. This is an example for when you need a faster line.

To suggest this arms race in high frequency trade has a serious economic benefit is ridiculous in my opinion.

* I do literally mean cable length. Yes, they have become that crazy

> There are bots that feign transaction so that others react in a specific way. In the last moment these are canceled again, too late for competitors to still react.

In stock exchanges at least, what you describe is called spoofing. The regulators are not friendly towards it, because spoofing enables to skew price discovery.

It's generally hard to detect and hard to prove, but there have been cases where the regulators have proved sufficiently well that in certain cases, there were orders never intended to be executed. And yes, if you get caught, there are sanctions.

https://en.wikipedia.org/wiki/Spoofing_(finance)

I very seriously doubt you'd lose any of those benefits by just trading on 10 second or even 1 minute increments. Everyone has a bunch of time to submit orders, the auction is ran, and then everyone gets the result back. I'd really like to see a use case in any of those real-world markets for pricing with sub-second granularity. It seems much more likely that trading on sub-second granularity is just extracting money from the actual participants in those markets into the pockets of financial firms with no value returned.

All that'd happen if you did that would be to increase the risk of any market maker having the wrong price at execution time (they have less data to make an informed price discovery), so you'd get wider spreads to compensate.

Wider spreads just makes it more expensive for everyone. Personally, I'd like my pension money going towards the actual investment rather than paying for a wider spread, but I'm just strange.

If you're holding for a pension, wouldn't a wider spread affect you less than any single other market participant? Like a 1 cent different due to spread isn't going to matter in 30 years?

If I'm holding a pension, I'm paying in regularly to a fund that's managed over a long period of time. If they manage it actively and trade it a lot over 30 years, 1 cent a time will add up.

That's not what they're doing, though. The costs will likely be minimal.

Spreads used to be much higher, sure, but that was not because there were no HFT shops around back then.

I don't know. Building and running fast trading systems sure creates more work and requires additional infrastructure and energy. So many smart people must be busy doing that, instead of something else. Do I understand this correctly: the alternative is that somebody else would randomly pocket this money, without working for it? Doesn't sound so bad to me. A bit less fair, but more human lifetime would be available to work on other problems.

"If only all these smart people were curing cancer instead of designing trading systems!"

Presumably you also object to academics working on entirely abstract problems? After all, they could be doing something else much more useful.

Who gets to decide what the most useful allocation of resources is?

> Do I understand this correctly: the alternative is that somebody else would randomly pocket this money, without working for it? Doesn't sound so bad to me

If that doesn't sound so bad to you, I suggest you haven't thought it through enough. We've been there, in the past. It was worse then.

> Presumably you also object to academics working on entirely abstract problems?

Yes I do. But at least there isn't a large monetary incentive to push people into it. They push themselves.

> After all, they could be doing something else much more useful.

I doubt it. Maybe.

> Who gets to decide what the most useful allocation of resources is?

Interesting question, but unrelated. We were discussing about the overhead versus benefits of high frequency trading. It's about the efficiency of the system itself, not about how to act within this system or where to direct the resources you get out of it. There is no conflict of values, I think.

My feeling is: it's an arms race. The profits that used to be randomly allocated are now either lost to you (if you decide not to participate in the race) or they are predictably spent on the race itself. Nobody wins, except those who enjoy the race for its own sake. Before that, someone was just winning randomly, and got to decide what to do with the profit.

Do those fast trades actually increase the overall efficiency of the system by more than their cost?

Is there any evidence defining where the speed of information reaches a diminishing return?

To dilate on your feedback loop comment, physical systems may benefit from a higher sampling rate but usually only to a point. This point is often related to the physical dynamics of the system (e.g., natural frequencies). For example, a small thruster may benefit much more from increasing sampling rates from 1000Hz to 10kHz than a large rocket engine. I assume/wonder if there's a similar analogy to diminished returns in stock information systems, like more volatile markets benefiting more from higher frequency of data. It would be interesting to see where the diminishing returns are.

I just have a nitpick with your analogy to feedback loops.

Assuming the market can be said to have a Nyquist rate, then once you hit that you have all relevant information. Increasing the sample rate past Nyquist does not make a system more stable unless you have a very specific system designed specifically to take advantage of that. More typically, it just increases your noise-bandwidth product and can decrease total system stability and accuracy.

I agree with your comment but am curious on how far this analogy goes. What do you suppose defines the Nyquist frequency in a market?

To stretch the analogy a little further, to take an observation of the market, you have to buy or sell which in turn affects the price. It's very much like the effect of measuring a quantum particle, the impact of a photon is enough to change to observation so you have an inherent uncertainty to everything. Because of this, I don't think you can ever distinguish between noise and meaningful trading with respect to a Nyquist limit in the market. If you tried to for instance, look at the price of all stocks over time and find the 2-D frequency function required to represent that, it is going to creep up based on your measurement rate. Another way to describe it might be that your observability and controllability vectors are not orthogonal.

This also skates around the issue of defining what a Nyquist rate of the market even means in real terms. But, if you want to use control theory to model the market, it's important to know that faster does not inherently mean more accurate. In the simple analogy, increasing trading frequency will improve measurement results, up to a point, after which it will actually likely result in decreasing accuracy.

I've also ignored the whole conflation of frequency and group delay in these analogies to keep things simpler as well.

I don't want to oversimplify your stance, I'm just trying to distill it to something I understand. In the analogy above, would you think this tipping point where the signal is no longer improved is related to the volume of the buyer compared to the overall volume?

In other words, Warren Buffet buying a huge portion of a penny stock will drive past that frequency tipping point easily while lil ol' me buying a few shares of an index fund will have effectively little to no impact?

The (global) market is the whole world itself. I very much doubt you can identify an useful Nyquist frequency.

I think if you could, you would have one of those fancy gold medals.

Is this sarcasm?

This is such a confident statement, and I don't mean that as a compliment.

For starters, is the evidence behind this Hayekian market efficiency really so strong as to warrant this kind of absolute confidence in the wisdom of markets?

> markets work by polling the expertise of many different parties who all understand a piece of how things should be valued.

…as well as orders of magnitude more people who do not understand how things should be valued. → noise, which is fine ("excess volatility"), but which can also become highly persistent in the presence of correlated expectations ("bubbles")

> This results in millions, if not billions, of interconnected price-discovery feedback loops.

Well, there are negative and positive feedback loops, only one of which is stabilizing!

> beneficial […] because the price discovery feedback loops get faster

This can also backfire. In fact, this is why a number of stock markets have instituted a trading stop if an asset moves "too fast". Slowing things down / reducing liquidity can stabilize a situation. Actually, this reminds me of

[1] W. A. Brock, C. H. Hommes, and F. O. Wagener. More hedging instruments may destabilize markets. Journal of Economic Dynamics and Control, 33:1912–1928, 2009.

where you have a similar counterintuitive argument.

The history of the idea of market efficiency is long and the idea remains controversial or contested. See e.g. Philip Mirowski's writings.

You misread my post. I never said the markets were efficient. On the contrary, they are generally far from it. What I said was that the markets combine the efforts of many different entities in order to determine prices in a way that is superior to what any one entity could do.

> Well, there are negative and positive feedback loops, only one of which is stabilizing!

Absolutely. Entities that consistently contribute positive feedback cause harm to markets and they are generally doing something that is either prohibited or foolish. I don't consider either a good long term profit strategy. The market regulation departments work to remove one and large losses tend to remove the other.

> This can also backfire. In fact, this is why a number of stock markets have instituted a trading stop if an asset moves "too fast". Slowing things down / reducing liquidity can stabilize a situation.

Sure, exchanges use a variety of market integrity controls, including limits on rapid and/or large price changes that can trigger order rejections or trading halts. These controls can be beneficial when the price fluctuation was due to poor trading, but can be damaging to a market when the fluctuation was due to significant new information or because there is a natural high volatility situation such as a derivative that is about to expire or is rarely traded. Consequently, the exchanges have to be careful about how and when halts are invoked. Some exchanges often get it wrong.

The main point I was making is that lowering the latency of the multitude of price discovery feedback loops making up the global market can be very beneficial because it allows the pricing dependencies to be more fully determined.

This is an accurate description of the game theoretic argument.

It is not very precise about the premisses, "the market", "the reward" or "the critical products"--variables in a non-linear equation, so to speak, that do not necessarily have a unique solution, or no solution.

For what it's worth, I used to work for GS and was in the algorithmic brokerage business for 2009-2010. One thing a lot of commenters on this thread are missing here is this isn't the same as an HFT or hedge fund, this is the brokerage business - executing orders on behalf of GSs clients, who are largely institutional investors, hedge and pension funds etc.

The real-world impact of increased speed of execution by brokers is less money left on the table for HFTs to snap up and less slippage to the actual economic beneficiary (ie the actual retail investor or pension fund investing people's money gets a better trade).

Does GS do HFT itself? Is there a conflict of interest?

I'm not a current employee, but at the time that I was, they did have a small unit who did pseudo-HFT on a prop basis and various electronic market-making businesses, but they were all firewalled off from the people who did brokerage business. We didn't share code or information with them and for obvious reasons they had no ability to view client order flow, unexecuted orders etc. Likewise people in the high-touch (ie voice) execution business couldn't see anything in the GSAT business even though they could use our algos to execute orders if they want to.

My understanding (could be incorrect) is that their quant trading business (what they called their 'HFT' shop, although it wasn't really high-frequency compared to real HFTs like Winton, Knight, Jump, Citadel or whatever) was probably going to get shuttered as they moved out of proprietary risk-taking generally, but I don't have any information either way.

In GSAT at the time, compared to others on the street our tech was pretty sophisticated intellectually but not fast (eg we didn't have ultrafast marketdata, our exchange latency was quite high and our execution algo speed was pretty slow) and so we had to do a lot of smart coding to prevent ourselves being ripped off by actual HFTs given they could move so much faster than we could.

There's a lot more to HFT than reg NMS by the way, I was working in London, so we did all the GSAT trading on European exchanges none of which has anything crazy like reg NMS and there was still HFT shenanigans of various kinds that people would try (eg timing arbitrages if they could see that you had different execution speeds on different venues etc).

Yes https://en.wikipedia.org/wiki/Chinese_wall

The concept of the Chinese wall is not really relevant to this kind of conflict of interest.

Yeah, this here is all on the public side of the Chinese wall.

"Doing HFT" is not a thing... it's just called trading. With computer automation, trading by definition becomes high-frequency.

The better distinction is whether those trades are on a principal or agency basis.

It depends on what you mean by HFT, if you mean prop trading, that was made illegal by Dodd Frank for large cap businesses to engage in with investor funds. If you mean market making HFT (which some prop firms also do), that's what this article is about.

People love to rail on HFT, but at this point, its really not that profitable. It's just a reality of trading in the markets. There was a blip of time between 2008 and 2014 when HFT was extremely profitable. Those inefficiencies have been gone from the market for years. People were whooped into anger about how much money was being made, at this point its a complete non issue and needs to be removed from the highlight reels aimed at generating anger in the public. Lets move on from discussing the boogey man that is HFT, its really nothing.

As always, some profit is to be made from some trading practice - the market removes it within a few years of "mass discovery". All working as intended.

Adding to this, HFT is a product of rule 612 of Reg NMS (the sub-penny rule). Markets are not allowed to show quotes in increments of less than $0.01 for most names. Since traders cannot compete on price, they have been forced to compete exclusively on speed.

The impact of such regulation was tested by the SEC recently with the 'tick size' program. Instead of reducing the minimum increment, some names saw it increased to $0.05. The hope was to increase liquidity while decreasing volatility in these names. In fact, those names experienced decreased liquidity with no decrease in volatility.

HFT is a result of regulation.

[0] https://www.sec.gov/divisions/marketreg/subpenny612faq.htm

[1] https://www.benzinga.com/general/education/18/04/11517027/th...

On the other hand, the Intercontinental Exchange reduced the tick size for sterling interest rate futures towards the end of 2018, and the result was ... decreased liquidity! And resulting increased volatility.

I'm not sure anyone knows for sure why this happened, but the best theory i've heard is that the reduction in tick size reduced the expected profits of market makers, because they are collecting less spread on every contract they turn around, while not affecting their potential losses, because external factors which cause the market to jump three basis points will still cause it to jump three basis points. Halved regular profits divided by constant occasional losses equals no longer worth bothering with.

Or there was adverse selection: only kooky traders go into kooky trading rules.

you have some firms that know how to quote a product and can make enough money doing it to be worthwhile. They've done a lot of research and implemented systems to do that.

Change the rules on them for arbitrary reasons, the firms that were there leave. At least long enough to build new systems and trading strategies. Who replaces them, anyone? Why?

Increase the tick size, liquidity drops. Decrease the tick size, liquidity drops. The moral there is know why you are changing the rules in the market, how you are doing it and the implementation details and side effects that will result in getting the result you want or just don't do it. This could be better is garbage, know it is. Change is not good for its own sake if you want people to quote.

Without having read the regulation, why don't institutional investor just make a private market place where they can trade for sub-penny values?

Usually it boils down to consolidation of liquidity. For the same reason there are well over 20 ATS venues but only a few are successful, ultimately new venues have difficulty naturally drawing resting order flow.

Market fragmentation aside, adding more price levels to set orders to disaggregates liquidity in the book, usually resulting in lower execution quantity (which increases your overall transaction costs if you’re trying to space trades out).

What’s the minimum lag for a packet to reach around the world?

Multiply x2 and add an extra 10%.

Make that the minimum order placement tick duration.

There would be 1 single global price and no arbitrage between markets possible.

Now we just need the teleport

What about order cancels - would you similarly limit those?

Yes, the ability to cancel an order now vs 1 second from now is not that meaningful. Or as the general public is concerned ‘If you’re directly placing orders on the stock market you can ware your big boy paints.’

How is the packet traveling? Through undersea fiber? Through Starlink? Through the Earth by neutrinos?

I imagine that no one with any clout is petitioning for a more equal platform.

Orders are filled in the order that they arrive to the exchange (if there's more than one order at the same price). A global mis-pricing would still be subject to a race to exploit it - whoever submits first gets the fill, even if it happens in the future (in the next tick).

What you're proposing is turning continuous trading into a fast series of auctions, like what happens for every ticker on every exchange at the opening. This would have the disadvantage of no clear bid/ask - how can you be sure that the parties do not withdraw their offers before the next tick? And surely you must allow for offer withdrawals.

Why? It’s just like a clocked bus; you leave enough time for the levels to propagate along the transmission line and settle before closing the register gates.

All orders are placed at the previously known tick price and later orders will occur at the price declared on the next tick.

Of course no trader should access the “ghost price” before it is announced on the tick.

Propagation delays remain limited to local data centers and in any case it’s about globally known prices.

Where’s the arbitrage here?

not to mention risky. remember Knight Capital, the kings of HFT?

on August 1 no less, their new software deployment essentially annihilated half a billion dollars, all due to - you guessed it - a refactored command line flag! can't make this stuff up.

Yea, I was one of them. I mixed up trading ahead from side-channel knowledge which breaks insider-trading rules, with having a fast engine which can just move bits faster than the other guy.

I don't "like" HFT the same way I don't "like" the market at all, but HFT is not actually stealing grannies money.

Side channel knowledge doesn’t break insider trading rules. You need to have a fiduciary duty to someone to betray to be doing insider trading. There needs to be someone who has the right to that knowledge, who you’re supposed to act on behalf of, who doesn’t want you trading on it.

There are many types of “side channel” non public information that aren’t insider trading.

Depends. This is the case in the US, but in the EU, “insider trading” is more absolute, just “trading on non-public information” (regardless of how it was obtained) and can happen even without fiduciary duty.

As an outsider with admittedly limited knowledge. What would happen if you limited movement on a stock to be on the second?

Would that not prevent this never ending race for faster and closer access. Something that doesn’t really seem to be adding value to society or the market.

who gets the priority in order fulfillment placed in that second? otherwise you have the same issue. the brokerage might also be tempted to make money by front running those trades since that have all the trades in front of them for a second before needing to be fulfilled

Let’s say Priority is made random. In other words It doesn’t pay to play at the second level.

As for bad actors you are 100% right but assume they can be trusted or regulated to behave for now.

I’m just interested if removing HFT at less than minute scope as it would have been before PCs would actually impact the world negatively.

would it impact it positively? why?

What if you limited it to be on the day ? The week ?

What if stock markets where about the long term ?

people aren't mad only because tons of money is made on hft. It's also because money is _wasterd_ on hft. That's $100 million dollars spent on something that has 0 use to society. It's just rich people playing weird games.

Think about the social benefits of $100 million invested in nyc transit infrastructure.

The economy's incentive structure is broken and this is a prime example.

Who’s mad about HFT? 100m out of a what 40tn dollar world GDP?

I get it, just seems... unlikely. Who cares?

Maybe mad is a strong word, but, it isn't just $100m, That's one investment by one company. It's billions of dollars across the world to gain arbitrary milliseconds.

Restructuring financial markets, to use periodic auctions for example or by adding speed bumps rather than continuous trading would probably do away with an entire wasteful industry and benefit the actually relevant parties in the financial system, people who have money, and people who need money, at the cost of the unnecessary middlemen.

HFT has more benefit than huge villas and cars and private jets. It can bring benefit to software and hardware. I'm okay.

What sort of benefits has it brought to software and hardware?

edit: honest question :)

I don't know, I didn't use past tense. I imagine that investment into technology is going to bring more benefit than pointless consumption. Also - what about effective markets, that's not a benefit?

right, you used present tense. You meant hft "could possible have" more benefit? HFT is much more the product of the current regulatory structure than a feature of efficient markets.

Instead of reading Flash Boys, which is good but not really very academic, I'd recommend "The problem of HFT" by Haim Bodek. It explains how he set out to build a sohpisticated trading shop with modern technology and realized all of his competitiors were just gaming the market structure and being handed advantages by exchanges desparate for trading volume.

HFT, being technology, has more benefit than pointless consumption, by definition - at the very least someone is creating something new and that is good for the economy. What are the actual benefits of this project in the market is unknown, especially because it does not exist yet.

Faster networks. Arista exists because hft was a big target market and now everyone using networks with Arista switches in them has benefited.

Faster networks would come for other more useful endeavors too, like high quality video conferencing/telepresence stuff. People say how important it is to be working in the same place because conferencing is "just not the same", not realizing it's mostly a technical problem, shitty ISPs, slow routes with high jitter or loss, and insufficient mics, speakers, and software.

But nobody is willing to put up the money to fund the research required to improve those speeds like HFT customers. A million dollars per switch is not a problem for them.

SolarFlare as well. HFT shops have been early adopter of kernel bypass setups which now is becoming more common place.

On a related note, the architecture of something like ScyllaDB is very similar to some of the HFT systems I have seen.

Those 100m are not destroyed by burning them in an HFT furnace but rather used to pay developers, hardware, factory workers etc. Sure, it's not going directly into infrastructure but it is not lost.

In fact, it's quite possible that if it wasn't invested into HFT it would be held as cash by the company or paid out as a dividend (which is fine as well).

That is false. The 100m could have been spent on something which increases the productivity of people. It would still go to devs, factories, etc but at the end there is something from which society benefits.

So paying hardware manufacturers and programmers for their work doesn't benefit society? It is not like firing these people would see more cancer research or other stuff you might regard as more beneficial for society.

It does, but it doesn't create anything which can be leveraged to improve things further. Might as well just give the people the money directly, with no expectations of deliverables.

It's important to make money but,how much you're paid for your work is an exceptionally bad way to measure that work's value in our society.

Tim O'Reilly said "Create more value than you capture." I would argue that hft is the definition of people capturing value that they didn't create.

Firstly, this article has nothing to do with HFT. Goldman is executing on behalf of clients who are not HFT

Secondly, your statement makes an implicit assumption that there is no value in providing liquidity to capital markets. This assumption is false. Think of your local grocery store. Sure, you could drive down to the distribution center and buy stuff there. But instead you go to the store where it’s conveniently laid out for you. Same with HFT. If it didn’t exist, you would still be able to buy and sell but markets would be a lot less liquid and buying and selling less convenient.

An article titled "GS spending millions to shave millisecconds off stock trades" has nothing to do with HFT?

It isn't at all clear to me buying and selling at an auction even only once a day, let alone once a minute or second would make financial market end user worse off.

Grocery logistics is a terrible analogy for financial markets.

The "F" in in "HFT" stands for "Frequency". This is a separate concept from latency. GS are spending money so they can execute their clients' orders on the market faster, to reduce the risk of the price shifting between order and execution.

> It isn't at all clear to me buying and selling at an auction even only once a day, let alone once a minute or second would make financial market end user worse off.

Go look at the history. Back before the 80s, trading was done by hand, sub-second anything was impossible. Guess what! Spreads were enormous, and the cost of doing business was huge. As a financial market end user (I have a pension), I want the smallest spreads possible.

GS is spending the money on this project to reduce high frequency traders ability to capture some of the value of their trades on behalf of clients. Why do you think the price moves away from them? It isn't random. It has everything to do with hft, which has everything to do with market structure.

I'm highly familiar with the history, no one is advocating for a return to open outcry.

Your pension is almost certainly a GS client and footing the bill for this project through execution costs. A periodic auction model would make latency much less of a problem and society's time and energy could be put into solving real problems.

Actually he probably meant that 100 million are being used to redirect efforts and resources destructively. Your argument can be used just as well to defend an "investment" of 100m into efforts to dig up aluminium, turn it into dust and throw it into oceans.

This makes the assumption that time in and of itself has no relevance. Sure the money is reused, but ultimately what is valuable is labour output, and usually that has time relevance. Further, if we consider the activity as steady state, that now means a % of the labour force is semi-permanently unable to be used for something else.

Your definition of 'waste' is awfully narrow. I'm not sure anything but destruction of the cash is waste in that view - except that its not even then a total waste because you are helping the money burning industry and furnace industry to thrive.

Burning money is not that bad, even burning 99% of all the money in the world.

...what?

Is this the finance edition of broken windows fallacy?

Broken window fallacy. The work those people are doing could be used for something else, so if they are doing something useless, it is a waste.

If having people work on something useless was beneficial, you could just pay them to dig holes in the ground and fill them up again.

>People love to rail on HFT

probably because it's difficult to see any actual value that this provides to society.

> probably because it's difficult to see any actual value that this provides to society.

Doesn't this apply to a majority of activities in the financial sector?

No, a large fraction of modern tech and companies would not exist without the 'activities in the financial sector'.

Essentially any endeavor requiring capital beyond your means would have to be bootstrapped or required borrowing money at exorbitant rates. There is a reason the financial sector exists. It makes money by selling convenience and taking over quantified risk.

My point is not that the financial sector is unnecessary, it's that the majority of the activities are not of benefit to society, which admittedly is something hard to define.

Following on from that line of thought, it could be argued that a lot of companies don't provide any benefit to society, so the financial sector is just enabling these firms and thus of no benefit to society.

The sector exists because people willingly give it money in exchange for services. If you can’t imagine why that’s happening, maybe read up on why people pay for financial services rather than assuming something as stupid as most of financial services not providing benefit to society.

No one's confused why it's happening. It's legal to make money through financial services, and it reliably makes money. The investments into fintech that divert more cash one way or another until others catch up also provide very little and ever diminishing value, mostly just sideways and upward redistribution of wealth. The actual value generating sectors of the economy are always getting more anemic, and they're due for collapse because of how much everything depends on oil, cheap 3rd world labor, and creating external costs we've avoided facing.

>No one's confused why it's happening. It's legal to make money through financial services, and it reliably makes money.

No, you’re misunderstanding me. People are willingly using the services offered by the financial services sector. It’s the reason companies can quickly raise billions through IPOs, the reason you can get a million dollars for a mortgage and pay it back over 30 years, etc. Market participants that enable better price discovery and subsequently narrow bid/ask spreads provide immense value to society.

An average of just a quarter percent lower interest on mortgages is billions of dollars kept in people’s pockets. More efficient markets enable that and it’s these traders you loath that are making it more efficient. If it were just up to the banks they would love nice slow markets with huge spreads so they can line their pockets with your money.

>The actual value generating sectors of the economy

Sigh, that statement makes no sense already because finance generates massive value. It’s the reason people can retire. It’s the reason normal people can buy houses. It’s the reason normal people can start capital intensive businesses.

Efficient allocation of capital is one of the largest force multipliers of any modern economy. You lament that other sectors are anemic, but many could not even exist if it weren’t for financial instruments that allow them to control costs, raise capital, etc.

> Sigh, that statement makes no sense already because finance generates massive value.

I think it's not clear what I meant by value generation, and that's on me as I'm sure there's an established meaning that differs from mine. In my eyes, moving money from one person to another is not value generation. Only work that improves the net quality of life is generating value.

As an example, someone who spends all day digging holes and filling them in for money has destroyed value, because their work helps no one and the money transfer is almost neutral overall. Being a facilitator of mutually beneficial trade has value, but work that only extracts wealth destroys value by using labor to no net benefit --they could have been enjoying their time instead.

It's not something you can easily measure, but through this lens you can see how much of what we allocate human effort to is a waste.

>In my eyes, moving money from one person to another is not value generation. Only work that improves the net quality of life is generating value.

Right, and that’s naive at best. There isn’t an unlimited supply of money. Choosing where to place money can result in massive value creation or destruction.

Labor (or physical work by anything) and value have no implicit or explicit relationship. That line of thinking has been discredited so many times (even in your own ditch digging example) that it’s not really worth getting into here.

Doesn't this apply to a majority of activities, period? (See recent discussions on "bullshit jobs")?

Its just a cost of doing business, something you have to deal with when trading securities electronically.

So what? Poker doesn’t provide any value to society, but if other people want to play it, how is that hurting me?

I think you’ve proven the point here? Poker does not hurt anyone else. Wall St does. See: 2008.

The financial crisis was not caused by high-frequency trading. I don’t see how it’s related.

The reason this provides little value to society now is because there’s no easy way to leverage price information to make inferences about the world-state. HFT on a prediction market would provide lots of value to society.

> People love to rail on HFT, but at this point, its really not that profitable.

But it is a huge barrier to entry now; it also is a waste of resources.

...actually it doesn't sound to me like this article is about HFT-based prop trading at all (prop trading would mean Goldman Sachs taking positions onto their own books), but about the business unit called GSAT (Goldman Sachs algorithmic trading) who execute trades on behalf of clients, so never taking any positions onto their own books.

The traditional market model used to be that at every point in time, a market maker would offer a price to buy/sell at. The bid would, of course, be lower than the ask, the difference being called the spread. When a pension fund wanted to execute a trade, they would have had to cross the spread, and, statistically speaking, half the spread would immediately accrue to the market maker as profit. ..so this is money that YOU, the holder of a pension, are losing, and that THEY, the rich folks acting as market makers that the public likes to get mad at, are taking away from you.

The business model of GSAT is that a pension fund can ask Goldman Sachs to use algorithms to do things on their behalf that are less naive than what I just described above and end up giving less money to the market maker. For example, GSAT can become a market maker on your behalf, but make a market on only one side, i.e. only offer a price to buy at or only offer a price to sell at, with the resulting trades being executed on your behalf, so it is now you who makes money off those trades, in the same way as it would traditionally be a market maker's privilege to do.

The public loves to get mad at Wall Street. But they should please get their facts straight about who the good guys are and who the bad guys are.

Have to agree.

Catering to HFT shops was already in 100 microsecond latency a few years ago using commodity hardware and software (cannot speak of Goldman Sachs but another one of similar ilk, can't imagine GS were much far away if not even better)

Worked in an HFT shop and we were down to optimizing nanoseconds. Lots of folks had war stories about spending months to shave off tens of nanoseconds off pieces used millions of times per day.

Yep, I have heard such stories. "Folks" were FPGA engineers and one of their primary target was to be able to cache "stuff" in SRAM as much as possible as that is the part which yields nanosecond latency.

My experience was more along the "man in the middle" setups which HFT shops used to hide their trading in the shorter term. The man in the middle got better prices at the exchange due to increased volumes and the HFT shop got a fairly good opportunity to hide it's activity for a short while.

At a later stage at some venues, the HFT shop did not even need to route their orders via MITM, the MITM simply got a firehose of HFT's execution reports to reverse build the order books !

Smoke and mirrors...(and nonsense)

> When a pension fund wanted to execute a trade, they would have had to cross the spread, and, statistically speaking, half the spread would immediately accrue to the market maker as profit. ..so this is money that YOU, the holder of a pension, are losing, and that THEY, the rich folks acting as market makers that the public likes to get mad at, are taking away from you.

This is totally incorrect. Let's say a pension fund wants to sell 1000 shares of APPL. The bid is $99, the ask is $101. They sell to the bidders and get $99,000 (minus some fees to the exchange probably). Your pension scheme just successfully liquadated their position.

At this point, contrary to what you say, the market maker has made 0 profit. What they've done, is taken a position in APPL which they think theoretically is profitable, but they aren't in the business of speculating on APPL. So now they have to hedge that risk by buying negatively correlated products and slowly trying to offload that position either by letting the market fill their offers on the ask, or hoping the bid improves. Once they have paid for their hedging and closed out their position according to their strategy over a period of time, whatever they have left is their profit.

What you paid that market maker for was for taking on the risk of holding the product whilst spending time to offload it to the rest of the market.

What GSAT do is say "Hey, don't sell this to the market, let us take care of that for you"- which is exactly the same service a market maker provides except they don't quote publicly, and in fact they probably do this by working with market makers.

Don't start a reply with "this is totally incorrect", when you're merely pointing out a simplification or approximation.

I was making a point about the business model of a proptrader marketmaker versus the business model of an algorithmically sophisticated broker and needed to establish some preliminaries and it would have served zero purpose to go into the particulars of the costs related to risk warehousing.

Having worked for an equities highfrequency marketmaking business myself: Mark-to-market at mid-price is the benchmark that those traders will use for figuring out, at the end of the day/week/month, whether it was a good or a bad day/week/month, and at the end of the year for negotiating their bonuses, even when they are left holding some positions with uncertain future. Everybody knows that it's a simplification/approximation, but, due to the efficient markets hypothesis, it's a very good one.

It's the kind of approximation where it's being taken for granted that people understand that it's not ACTUALLY the trader's profit. Because otherwise one would need to include in the discussion the fact that the receptionist at the proptrader marketmaker's office building is also a cost factor eating into their margins.

I didn't mention the risk warehousing for the same reason I didn't mention the receptionist. And I'm not going to go into a rebuttal about how it's definitely not necessary to hedge every single trade, for the same reason: because it's not the topic under discussion here.

It's called the "maxim of quantity" and it is a generally-accepted maxim of conversation. (see https://en.wikipedia.org/wiki/Cooperative_principle)

Maxim of quantity: * Make your contribution as informative as is required (for the current purposes of the exchange). * Do not make your contribution more informative than is required.

So, to conclude: Your conversational move in this language game was a pretty weak one. The sentence "this is totally incorrect" however sounds like someone trying to establish dominance. Weakness and trying to establish dominance is a bad combo.

It's not a simplification. It completely misrepresents how market makers make money. They are taking on risk and they have to handle that risk. If you're trying to make a nuanced point, then you need to get at least the basic fundamentals of the business model right.

You have written something that doesn't actually say what you mean. Frankly, the fact you think that market makers' management of risk is as relevant to their business model as a receptionist means you either don't have a good handle on what market makers do, or you're MASSIVELY over-paying your receptionists.

Yes. Many people use "algorithmic trading" as a synonym for "automated trading" or even HFT, when it is used (in the industry) basically for "optimal execution, computer aided, of a client order".

It seems like we could save a lot of pointless expenditure on an ultimately meaningless arms race in flash trading if we imposed reasonable limits on the time required to hold an equity in order for a trade to be legally recognized.

we should also not allow the president to engage in trade wars, but i possibly digress

please digress.

Allowing trading of fractional pennies has been theorized to be another way to break up flash trading as you need to accurately guess the real price not just get your exact price up before anyone else.

> pointless expenditure on an ultimately meaningless arms race

What about price discovery is pointless? Would you prefer that prices update only once a day? Once a week? Once a month? Realtime pricing of securities and derivatives is critical for an efficiently functioning economy.

> if we imposed reasonable limits on the time required to hold an equity in order for a trade to be legally recognized

This would damage the ability of market makers to function, ultimately driving up the cost of offering pensions, 401k plans, retail investing, low fee ETFs, etc. If these companies are willing to spend their money competing for the ability to offer you a better, faster price, why is this upsetting?

If you're opposed to the emphasis on latency in equity markets, focus on rule 612 of Reg NMS, which prohibits showing more competitive prices.

Nanosecond pricing updates might be a little more than anybody needs. Maybe if the exchange cleared once every second, that would serve anyone's purposes. Remember the market is ultimately about allocating capital between businesses and governments, and for economic purposes it doesn't need to run any faster than they can.

> Nanosecond pricing updates might be a little more than anybody needs

That may be the case, but the 'problem' is caused because market makers aren't legally allowed to offer better prices than they might otherwise desire. It doesn't seem prudent to layer flawed regulation on top of flawed regulation when there's a simpler solution.

> Remember the market is ultimately about allocating capital between businesses and governments

Equity markets have many purposes. Companies indeed raise money by issuing equity (IPOs and secondaries), and there are cases where governments participate for monetary purposes (BoJ & SNB's equity purchases). But markets also allow:

* employees to sell equity compensation they've received whenever they want

* retail investors to diversify their asset allocations

* sovereign wealth funds, pension funds, endowments, and other real money sources return on their portfolios

* for hedging financial risks

* companies to return money to shareholders in the form of buy backs

> for economic purposes it doesn't need to run any faster than they can

All the previously mentioned functions happen all the time, and the marks (prices) generated from this activity helps inform many other economic functions. Everything from central bank policy to insurance pricing depends on well functioning markets. In light of this, why should we make things less efficient by slowing things down and driving the cost up?

> Nanosecond pricing updates might be a little more than anybody needs.

This statement reveals that you do not understand what is happening when a transaction occurs. The price is simply the market clearing price. It is not as if updates are being published, simply that the correct price is being discovered more rapidly.

If you think of the price erroneously as something that has been published, then of course there is nothing beneficial about speeding up the transaction turnaround time.

But if you think of each transaction (and every participant willing to transact for close to the clearing price) as a vote that the price being transacted is close to accurate, then the more participants and volume available amount to significantly more information than was previously available.

Imagine if trades were available only once per hour. Consider the kind of spread would a market making firm have to utilize to avoid losing money!

Speeding up the market adds additional efficiency and reduces inventory risk for market making firms, increasing information and liquidity for all.

> as a vote that the price being transacted is close to accurate

These votes are not an indication of accuracy though are they? they are gambles about future price discovery. As in they don't care if they think the trade is worth $5 if they think it might go up before crashing to what they really think it is worth.

I'm not very knowledgable about HFT so I'm just trying to reason about how it works. I guess it's the concept that without the higher frequency everything must be more unstable which is hard to grasp. It reads like we have to make it easier for these firms to make more money so everyone else can enjoy the 3rd order effects of the process.

Most of these efforts are aimed at taking advantage of lags in information flow, often within the very trading systems on which the trades are occurring. They are exploits, not essential market-making. Noise, not signal. What's the right timeframe? Something based on the time it takes for humans to reason about a price. Not a day, but certainly not milliseconds, either.

If they were noise, they would not be profitable strategies.

Noise compared to the signal actually needed for a market to function. Markets functioned nicely on much, much longer timeframes in the past.

They also had much higher transaction and information costs.

I don't want to get caught up in arguing what is noise in the mathematical sense. I think the parent is refering to the fact that the arbitrage traders are not adding any real value by making these trades this fast. The market would adjust in miliseconds. Who really wants to let some guy who built a 825 mile cable be a rent seeker on every trade for all eternity? https://www.businessinsider.com/chicago-stock-exchange-respo...

He's not a rent seeker. Absent his existence, the people on both sides of the trade would be harmed by crossing a bigger spread.

Is the human allowed to use pencil & paper when reasoning about the stock? A graphing calculator? A spreadsheet program? A numerical SDE solver?

By what rule do you propose limiting other people’s free choice to use computing resources to accelerate the time scale of reasoning about a price?

The "free choice" argument is a charade: the only people with the "freedom" (ie, wealth) to win that race are "people" like Goldman Sachs. On the whole, it's just another legal mechanism for pumping money from the poor to the wealthy. Modern markets could not exist without regulation, which by definition limits freedoms. We're just talking about a sensible regulation that eliminates a pointless misdirection of resources, not unlike laws against gambling.

A lot of HTF shops are comparatively tiny companies (in the order of hundreds or less employees). Large hedge funds and investment banks are relatively late players to the game.

You didn’t address my comment at all. This just seems like some unfounded normative assertions about some other thing entirely.

By “taking advantage of lags in information flow” you’re actually disseminating information and reducing the lag.

The social value or reducing the price dissemination lag from a microsecond to 110 ns is zero. Hence why the billions of resources going into it are a greater waste than even bitcoin.

>Would you prefer that prices update only once a day?

Obviously you are being hyperbolic, but some people have proposed literally laying excess cable to slow down the speed of automatic trades, which can be highly volatile. That's not damaging market makers at all. It's smoothing out the supply and demand to prevent micro-crashes and other arbitrage.

> What about price discovery is pointless? Would you prefer that prices update only once a day? Once a week? Once a month? Realtime pricing of securities and derivatives is critical for an efficiently functioning economy.

Pretty much everything at sub-second resolution is pointless.

I'd like to hear a coherent argument how realtime or even sub-second pricing of securities and derivatives is critical for an efficiently functioning economy, yet the largest markets in the world are closed 2/3rd of the day.

The fact that most markets are closed on all weekends plus over 10 holidays per year suggests that even an update once per day wouldn't make much of a difference.

This. A million times this. All possible arguments for price discovery are totally invalidated by regular market closures.

Conversely, if sub-second resolution is somehow "a good", then by extension sub-millisecond price discovery is "even better". There are some insane people that state this kind of gibberish with a straight face.

If millisecond are good, then surely microseconds are even better! Next... nanosecond resolution price discovery for the uuuuuultimate liquidity.

They’ll only stop when they get to Planck time resolution price discovery

If they achieved Plank Time Resolution I don't think they would need to be in the finance industry to make money :P

>Conversely, if sub-second resolution is somehow "a good", then by extension sub-millisecond price discovery is "even better". There are some insane people that state this kind of gibberish with a straight face.

if sub-second Internet latency is somehow "a good", then by extension sub-millisecond Internet latency is "even better". There are some insane people that state this kind of gibberish with a straight face.

If millisecond are good, then surely microseconds are even better! Next... nanosecond resolution Internet latency for the uuuuuultimate speed of information.

Surely this is crazy talk!

Would you be happy if regulators mandated that airplane control systems can only sample their input at most every second?

HFT systems look a lot like feedback driven control loops. Mandating a minimum resolution would be ridiculous.

If what you're sampling and responding to is noise, then yes, it's pointless.

If what you're sampling and responding to is noise, and the responses themselves generate self-perpetuating expanding feedbacks, it's worse than useless: it's actively harmful.

> This. A million times this. All possible arguments for price discovery are totally invalidated by regular market closures.

As has been pointed out elsewhere, the really big markets are OTC.

> If millisecond are good, then surely microseconds are even better! Next... nanosecond resolution price discovery for the uuuuuultimate liquidity.

If I go to market to get a price, I'd quite like that price "now", not at some arbitrary point in the future. Increasing the resolution reduces the amount of time I have to wait.

> Pretty much everything at sub-second resolution is pointless.

Should your credit card network only allow transactions once a minute? Price data comes from transactions (trades) as they occur. There’s more utility derived from a market where people can transact on demand.

> yet the largest markets in the world are closed 2/3rd of the day

Not correct. Equity markets are far from the largest in the world, yet it’s possible to trade 24 hours a day, 5 days a week (actually thanks to HFT). Currency markets operate around the clock. Rates markets operate ~22-23 hours a day, 5 days a week.

> The fact that most markets are closed on all weekends plus over 10 holidays per year

Markets exist for their participants. Corporations looking to hedge commodities exposure, trade currencies, manage interest rate risk, or buy/sell stock have employees who work during business hours. That doesn’t diminish the need for real-time pricing. And most holidays are domestic, so major markets still operate internationally.

I wouldn't mind if my credit card only allowed transactions once a minute. I don't think I ever had to do two transactions in a minute.

> I don’t think I ever had to do two transactions in a minute.

If the network only clears once a minute, you need to wait at least a minute for each person ahead of you in line at the store. Alternatively, you need to be willing to pay more to cut them in line.

Oh I see what you mean. That wouldn't be too terrible either I suppose, but I see your point now.

Wait, isn't the correct analogy for the stock market that there is an (almost) infinite amount of registers, but each of them can only clear a customer at the first second of each minute?

This is where the analogy certainly gets stretched. The network and the registers are really both the exchange here.

Equity trading only happens at a small number of venues (inclusive of OTC, dark pools, and internalizers probably not more than in the hundreds, potentially low thousands). These would be the registers. In order to buy a stock, either you choose to ‘cut the line’ by paying the price someone tells you they are willing to sell at, or you wait for someone to sell at the price you announce you’re willing to pay. These transactions are processed serially (albeit quickly) at the exchange.

Allowing trades only once a minute (or some other period) is akin to allowing the credit card network to clear once a minute. Trades cannot occur more frequently, so the ability to enter and exit positions on demand is diminished. Obviously someone who wants to trade immediately for one reason or another is harmed by having to wait. Additionally, everyone (even those not trading) is harmed by not having up to date valuations for the positions they hold.

Sorry, conflating frequency of price discovery for efficient investment allocation with payments (as you here) or with internet speed or aircraft control systems (as someone else above) is really disingenuous.

Being closed on weekends is just fine for the market (unlike credit cards, the internet, or aircraft control systems).

> I'd like to hear a coherent argument how realtime or even sub-second pricing of securities and derivatives

I won't touch equities, but surely it's obvious why derivatives have to be priced quickly? When the underlying moves, you have to re-price the derivative, otherwise you're giving away money!

Well, anything that limits the complexity of derivatives is probably a good thing!

Obligatory caveat: I know derivatives aren’t evil, and are very useful to the world, but we all know what happens when they get so complex that almost nobody knows what they are.

Artificially stale derivative prices are unquestionably more complex than up to date ones.

Pretty much everything at sub-second resolution is pointless.

Some people think so, but the truth is you will simply shift the competition from “as fast as possible” to “within as few picoseconds after exactly one second” or whatever the limit is.

Unless you add random delays to each trade.

That uncertainty will only increase the spread and retail investors - ordinary folks like you or I - will pay for it.

Ad long as you can model the latency statistically people will still try to play the game. It will be much riskier of course, and I'm not sure why that would be a better outcome.

How is it different to any other competition between companies, what makes this one specifically pointless? Your idea will just put more money in the pocket of the bank traders rather than spreading the wealth to all the IT engineers and equipment manufacturers which supply this slight advantage, what's wrong with that?

Even if the business functions were pointless, the research and execution going into these projects is valuable knowledge that likely has broad application.

I wouldn't say it's meaningless. These types of technologies are usually arbitraged out; ending up with a few large players that are squeezing out a return.

It means high frequency volatility is taken out of the equation for the rest of us -- Which generally is a benefit for other players in the market.

FX has (effectively) been that way for a long time.

A simpler solution might be to just increase the transaction tax

But why? Who is the victim here?

Who pays the $100M? Yes, the customer of the bank, of course. Which usually means you and me are paying in the end.

You are a client of Goldman Sachs? I think you won't worry about a share of $100M then.

"Using so-called microservices to break complicated problems into easy-to-solve ones"

I would love to see microservices which actually solve problems and reduce complexity! :(

I am curious how that will help latency here. It must be pretty bad if refactoring and adding more network connections is an improvement.

Yeah I know people preferring microservices for its horizontal scalability advantage but never for reducing latency. Maybe something was lost in the translation to the author.

Only thing that I can think of is that it would make it easier to detect the bottlenecks. Other than that, certainly in high-frequency trading - I only see it adding latency.

They were likely running old Tibco/RV systems (network) distributed across the network (common for 1990s to early 2000 trading systems), and replaced the system (and hardware) with multi-core boxes, and use shared memory for message passing. Reduces internal latency from milliseconds to sub-microsecond.

Makes sense in this case. I guess you can just deploy the microservices as containers on the same machine or what have you.

Complexity is never reduced, but made more manageable. Microservices make a good architecture a lot harder, certainly if you have to start with a blank slate and don't really have an idea what you're up against.

However, things that are virtually impossible to enforce or even very hard to implement in a monolith, are made manageable if you have the proper setup. That includes central standardised logging, tracing, metrics and monitoring. If you have these in place and can enforce them, you're off to a good start.

These things rarely happen in a v1 though - which usually is a POC that ends up in production, and if you have a full-blown microservice architecture from the start, this will probably grow into something a lot worse than huge PHP monolith. With microservices you need to design a 'platform', and ad-hoc POC development never results in a good design, but just something that functionally works.

Agreed. You missed db syncing, container orchestration, deployments, sharding, DEBUGGING and some sweet sweet anxiety attacks!

Most exchanges have an auction process that sets opening and closing prices. They let everyone get their orders in and then run an algorithm to find the price that will execute the most volume.

They could do the same process every 5 minutes and only allow stocks to trade in the auction. Then all of the resources used on pointless HFT could be used on something economically productive.

even every 1s. If we can do auctions for web ads, we can do auctions for the stock exchange.

> even every 1s. If we can do auctions for web ads, we can do auctions for the stock exchange.

There was ~$107 billion in online advertising in 2018. There was ~$145 billion traded in Nasdaq listed equities yesterday.

Global financial markets and online advertising have different requirements.

[0] https://www.marketingcharts.com/advertising-trends/spending-...

[1] http://www.nasdaqtrader.com/Trader.aspx?id=DailyMarketSummar...

How is the 145 billion calculated? If I buy and sell with a 100$ 100 times does that count as $10,000 or $100?

The number quoted is total volume, so it would be $10k

> There was ~$107 billion in online advertising in 2018. There was ~$145 billion traded in Nasdaq listed equities yesterday.

That is a pointless comparison. That $107b was all revenue for somebody (Google, Facebook, etc), while the $145b was just the nominal value of shares traded.

(edited typo)

> That is a pointless comparison.

I was highlighting the difference between online advertising and financial markets. They operate at different scales.

> That $107b was all revenue for somebody (Google, Facebook, etc), while the $145b was just the nominal value of shares traded.

These are cash instruments, not derivatives, so that is indeed $145b of cash changing hands each day.

not necessarily, money has velocity; market participants don't have $145B to spend, they spend a fraction of that and then reconcile balances at the end of the day.

see https://en.wikipedia.org/wiki/Clearing_(finance)

I’ve worked in both. For the purpose of the earlier conversation, the auctions for ad exchanges happen at about 1 order of magnitude more than for exchange tradable symbols.

That is to say there is no tech reason that financial exchanges couldn’t run auctions.

Aside: personally I think that continuous exchanges are great & people who have problems with them usually don’t know what they are talking about.

Preventing HFT would mean exchanges make less money, meaning they'd either have to charge more or find a new way to subsidize market activity. It's akin to how Robinhood sells data to trading firms to subsidize trades for long-term retail investors. "Economically productive" isn't always black and white.

Most exchange open/close auctions use time priority to deal with order imbalances. If there are buy orders for 300 shares and sell orders for 500 at the same price, then only first 300 shares on the sell side will be filled.

The speed incentive is a consequence of time priority, not the auction frequency. Switching from continuous auctions to open/close style auctions every 5 minutes would not remove the incentive to be fast.

You're talking about a "Periodic Auction" market, Cboe have one[1]. Another alternative is "Auction on Demand"[1].

MiFID 2 regulation was a real push to move people away from using dark pools.

0 - https://markets.cboe.com/europe/equities/trading/periodic_au... 1 - https://business.nasdaq.com/auction-on-demand/index.html

Can someone explain to me what is gained by processing the trades in real time vs. batching the processing into say 1 second increments? What does GS gain by being able to get their trade there a few milliseconds before the competition and what do I as a consumer gain from this?

You don't gain anything.

GS is in an arms race with other fintech firms to be first in line to act on new information.

That's not right, you (we) get accurately priced securities. We also get tighter bid-ask spreads.

Accurately priced securities 100ms faster (which have had any profit opportunity already arbitraged). Grea.t.

The market is closed so much of the time available for us means we don't need these milliseconds for accurate prices/price-establishing?

suppose "real time" means order of magnitude milliseconds for example's sake.

the price could change a bunch in that interval of 1000ms. you may think "well only slightly" - fractions of cents - but if GS can make fractions of pennies on those events, scaled up to all seconds that the market is open, you can see why that's potentially attractive.

for whatever it's worth, GS in 2009 claimed that HFT generated <1% of their profits. whether you believe that is another thing.

Ok that explains why they want to do it, but how does it benefit the market to allow that? How does it benefit the consumers and corporations? In other words, what is the argument against creating a law that requires a minimum of 1 second batches for example?

Exchange traded funds depend on special traders who buy the underlying stocks (like AMZN and MSFT), and sell them in exchange for the ETFs (like SPY and VOO). If the individual stocks are more valuable the special traders can buy the ETF shares, exchange them for the individuals, and then sell those for cash. If the ETF is more valuable, they can buy the individual stocks, exchange them for the ETF, and sell the ETF for cash. This is what keeps Index funds matching the underlying index.

Whenever the prices are out of whack, one of the assets can be exchanged for the other. However, there is the concept of tracking error, where by the value of the ETF doesn't track its index. High frequency traders are constantly trying to take advantage of this mismatch (i.e. arbitrage), and as a result the fund's tracking error is moved very close to 0 (i.e. it achieves the goal stated in the prospectus).

Retail investors (like you and me) benefit from this because the price of the index fund is kept in line, and HFT people get to make a little money being keeping it that way. Who would want to make a law that hurts everyone involved by preventing them from entering a mutually beneficial relationship?

my question would be how/who does it harm? what is the argument FOR setting that law? other than you think it’s greedy or something?

I don't care about the greediness, I just care about fairness. Why should GS get the best trades just because they can make a deal to get the shortest ethernet cables?

Think about it this way: When you get new information that according to your trading algo might move the price of a stock and you want to act on it, once you have made the decision you want the time for your order to reach the exchange to be as low as possible as you assume others can have the same info and act to it accordingly.

Not just anyone can have a direct market access connection to an exchange's order book. This is usually reserved to the members of the exchange, and many rules and regulations apply. So even large volume traders use Sponsored Access, transacting directly with the exchange through the access platform of a sponsoring member that ensures not just technical service but most often also some risk and regulatory compliance controls. This service is not provided for free.

GS competes with a few others to provide such an access platform. Reducing the overhead latency of the platform itself in the trade loop makes them more attractive and allows them to attract HFT clients and/or maintain healthy margins.

Batching won't solve any problems you might think it would solve. The real world is continuous. Because of this, everyone is incentivized to wait until the last possible millisecond (or nanosecond) before the auction to submit bids so the incentive to build fast acting systems remains.

There are other problems as well, like how to deal with bid/ask imbalances at auction time.

Anyone here working at Goldman?

I’ve heard very mixed things about working there. Would love some first hand stories!

Have things changed from some years ago where the technology didn’t really get any respect and everyone was second class to the front office traders?

How’s the work environment? Is there still a dress code? Is the system kept running by heroics and sleepless nights, or is there a mature understanding of human factors?

While not related to high frequency trading, per se, the Google research on the gap between nanosecond and millisecond latencies (AKA "attack of the killer microseconds") springs to mind with this article. Typically the hardware bottleneck in HFT, I assume, is network latency, which leads to colocating trading servers near the exchanges themselves...so how do you improve on that? Smart NICs/FPGAs? Better non-volatile storage (there's hardware on the market today that's getting us to read latencies in the single digits on PCIe bus at least)...? ASICs for HFT specifically? This is obviously a marketing piece put out to attract quants to Goldman Sachs given the lack of detail, but I naturally wonder where that $100M is going to be invested.

https://cacm.acm.org/magazines/2017/4/215032-attack-of-the-k...

It's certainly possible to do it in FPGA, I was involved in this project a few years ago: http://www.argondesign.com/news/2014/jun/23/ultra-low-latenc...

Although it limits the complexity of the model and makes it harder to iterate on the development of the model.

I know a couple of the folks doing this sort of coding and while I'll tweak my APIs to be in the 10s of milliseconds they are tuning for nanosecond optimization. Crazy to see code where the speed of light is becoming a bottleneck.

Tom Scott did a nice video [1] on IEX's setup with 38 miles of fiber to add enough latency as a mechanism of fairness. Purportedly, this benefits all exchanges after it was successfully deployed. It seems like the incentives are not there for larger, more established exchanges to implement such blanket latency. So, firms like GS see a benefit to this kind of investment.

[1] https://www.youtube.com/watch?v=d8BcCLLX4N4

A great read on high frequency trading: Barbarians at the Gate

https://queue.acm.org/detail.cfm?id=2536492

Title is actually Barbarians at the Gateways which is, of course, as play on words on the actual Barbarians at the Gate:

https://en.wikipedia.org/wiki/Barbarians_at_the_Gate:_The_Fa...

Another great tail of finance that operated at much slower speeds.

I would highly recommend the move Hummingbird Project which does a pretty good job of what it takes to build this kind of networks: https://www.youtube.com/watch?v=U8DweHM0Gnc

To me this is incredibly ridiculous, millions more dumped into the brain draining maw of finance.

Forgive the AMP link but Bloomberg has a paywall of sorts now.

https://www.google.com.au/amp/s/www.bloomberg.com/amp/opinio...

> And while Scientel clearly has high-frequency trading in mind—“the name that keeps coming up among industry sources is Citadel”—that’s not all it has in mind. It’s a tower that can send signals for lots of stuff. “Scientel has said it will equip its Aurora tower with 28 antennas—24 for public safety and municipal use and four for ‘private’ purposes.” People often complain that high-frequency trading encourages a socially wasteful arms race, and you certainly see some of that here, as lots of trading firms compete to put their antennas as close as possible to the CME servers. But another way to interpret this story is that high-frequency trading is subsidizing high-tech communications infrastructure for everyone else, building towers to send high-speed signals for public safety and municipal use just to justify a couple of high-frequency trading antennas.

Reminds me of this comic

https://smbc-comics.com/index.php?id=3890

I really wish we would curb this kind of behavior. Even if this speed is related to the brokerage business and not HFT, it is unfair to retail investors that large businesses have better market access. My solution: exchanges are required by law to process contracts in pulses with strict execution ordering rules. E.g., every 15 seconds process all buy and sells, processing from highest bid and lowest ask first for each security. Basically a turn-based stock market.

their stock isn't doing so great, it seems to track the rest of the financial markets (bac, jpm, c, etc)

Profiting in this way seems to be a bastardization of the purpose of the market and provides no value.

This is called rent seeking. Just siphoning money from society without any benefit.

Tangentially related, but I wonder how long before HFT harnesses neutrino transmission to send market data directly through the earth? The MINOS [1] project is underway to send neutrinos from Fermilab in Illinois to detectors in northern Minnesota. Straight through the earth's crust.

Detecting neutrinos is hard, and modulating data on them will probably be even more difficult, but if it shaves a few micro/milliseconds off trading time, I'm sure someone will make it happen eventually. HFT have already added dedicated undersea fiber-optic links [can't find source, NYC->LDN is obvious, I think there was talk or they began one from CHI/NYC->TOK but might be wrong, it involved the melting icecaps to easily sink the fiber cable], shortwave radio broadcasts [3], and microwave links [4]. I think HFT is frivolous, personally, but if it helps networking I think it would be cool to someday talk "on neutrinos" through the center of the earth!

[1] https://en.wikipedia.org/wiki/MINOS

[3] https://sniperinmahwah.wordpress.com/2018/05/07/shortwave-tr...

[4] https://arstechnica.com/information-technology/2016/11/priva...

Very interesting. Physical wars brought pointless spending that prompted innovation.

I wonder if the mini-financial wars being waged will result in some cool tech coming out of all the craziness.

I had never though of this when thinking about the rent seeking nature of many parts of the financial industry.

Already has. The first notable example that springs to mind is "message queues" which were pioneered for use in finance. Companies like TIBCO grew fat on such software and now they are commonplace.

So some quick Googling shows that the neutrino beam fires every 1.3 seconds[1] and the 'near' detector on average captures 20 neutrinos, the 'far' detector sees <1 detection [2]. Note that the latter paper is from 2005 so this may have been substantially upgraded.

Each 'spill' takes 8.67 milliseconds, so if you simply vary the firing time to send a signal using coordinated clocks, naively you could send up to ln(1000 / 8.67) = 6.8 bits per second, but at the cost of the very latency that you're trying to minimize.

I'm sure there are smarter schemes, like also varying the neutrino beam density, but at 20 neutrinos per detection there's not much channel capacity in the amplitude either.

It looks like the main bottleneck is in the neutrino generation though, you can use more accelerator beams without needing to build more detectors right away.

Note also that the 'far' detector is only 500 miles away, so depending on how collimated the neutrino beam is you may be losing a lot of signal to get all the way through the Earth.

[1] http://nusoft.fnal.gov/nova/public/neutrinos.html

[2] http://neutron.physics.ucsb.edu/docs/MINOS_info.pdf

They got this far with public funding, that's enough for proof of concept. Let the billion-dollar budgets of institutional/prop trading desks see what they can do with this technology.

That doesn't seem like it would be actually faster or closer for NYC to HK/Shanghai, close but not a latency improvement at the speed of light? Well I guess if a direct line can be made itll shave off some milliseconds

Maybe less infrastructure required.

yay more control to bankers!

Vamipre Squid...

They've taken greed into stellar territory! Well done...

Amazing to me how much impassioned defence there is of HFT. I assume a lot of the HN readership works in HFT.

Would be nice to read more about technical sides of this transition. Also getting rid of Java code, that is so popular among financial institutions, will increase the speed. Something like Rust is a perfect fit[1] for such industries.

[1] https://internals.rust-lang.org/t/proposal-business-applicat...

I was pretty impressed by Rust's latest showing in the techempower benchmarks. The Actix framwork blew the ceiling off of previous benchmarks. SIMD code, lock free, static dispatch, pipelined postgres driver, and other performance enhancements.[1] As fast or faster than the C/C++ frameworks.

[1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/48...

Does Goldman Sachs use Rust/actix for quick trading?

People that are very serious about it use ASICs/FPGAs. I'm sure Goldman does too.

For slightly slower stuff it's the garbage collector and OS scheduling that's the problem (as both can unpredictably inject many milliseconds pauses). No GC and good usage of low level kernel primitives is the game.

Reply on news.ycombinator.com