traK6Dcm

HFT may not, but trading itself definitely provides positive value. People's utility function is not purely monetary over an infinite time horizon. Trading allows you to trade off price and risk over multiple time horizons.

For example, a trade where someone needs to convert his assets into cash due to a family emergency benefits both sides. The person with the emergency takes liquidity from the market and pays a premium because the trade is time-sensitive - he needs cash the next day. Other liquidity traders may profit from such "uninformed" flow in the long term, but both parties are happy because they got what they want.

Another example is trading off risk and hedging against certain changes in the world that would affect you.

Yeah, fair enough. The truth is that I'm just a little frustrated by people touting the horn of "making the world a better place" when in reality most end up doing nothing of the sort but then criticize people working in Finance :)

Some people argue that HFT provides liquidity for retail investors, but it's debatable whether that liquidity is real or not since it'll be gone during black swan events. At the same time HFT profit from uninformed/retail flow. So it's debatable wether the actual activity provides values. I'd say probably not.

But there can be indirect value in working in HFT, just like there is with other demanding jobs. There's interesting systems and ML research going on in the field - that's knowledge you can take with you and share with others. Some time ago, HFT was driving a lot innovation in networking, infrastructure, and systems.

Also, over time I've come to realize that "providing value" is really difficult to quantify. Does working in Academia provide value? Maybe if you get lucky and publish something that actually turns out useful. The majority of academics publish papers that are nothing more than noise - negative value. Does working at FAANG provide positive value to society? Again arguable. You may inadvertently end up promoting ads and misinformation. Not everyone does - but it's not guaranteed that you don't without realizing.

"Providing value" is subjective and most of the time it's nothing more than a story people tell themselves so they can feel good about themselves.

I think he means smart as in "how long your model prediction takes". If your neural net (haven't actually met anyone who uses these in trading) takes 5ms to make a prediction that'll lock you out of a whole lot of trading opportunities/strategies.

Speed always matters, no matter where on the smartness spectrum you are, but it's relative. If your model prediction takes 5ms you're not getting much ROI out of investing $1M into shaving off 50ns in your data processing. But if your end-to-end latency without prediction is 1ms, you better invest in getting that down.

> I understand they are not making the world a better place, but are they making it worse?

Most people would argue that they are not directly making the world a worse place, but rather that they are sucking in talent that could have used their skills in other places to make the world a better place. For example, many talented Physicists ended up in trading companies a few decades ago. Today, you can make a similar argument for companies such as Google or FB who probably have tons of extremely intelligent scientists working on ads.

> Something that I have heard is that they are providing a service by providing liquidity to the market.

It's unquestionable that they provide liquidity, at least most of the time. That's good. What's questionable is whether they are doing it in an efficient way. Their primary goal is not to provide liquidity - it's to make money. Liquidity is just kind of a side effect of their activities, which is an elaborate race to "beat" other firms through speed and algorithms. Most of the resources go towards that, but don't result in additional liquidity, most of which is probably not even needed. It's like saying "FB provides an important service by allowing you to message your friends" - sure, that's not wrong, but that's not their primary goal. And you don't need a trillion dollar to make a messaging app. The difference is that FB also does lots of other things, so it's unclear how much of their resources are actually wasted vs. potentially used for "making the world a better place" - it's not quite as obvious that they are wasting a lot of talent and capital.

The other question is what the alternative would be. Markets run by human traders were significantly more unfair than anything we have now, but HFT was never meant to happen, it was just result of markets becoming electronic and transparent, and people trying to extract more money.

Even if that was true (it's difficult to prove either way since there is little data on black swan events), the fact that they provide liquidity during stable times can make up for not providing liquidity during black swans. The impact could still be net-positive.

Also, it's unclear what the alternative is. While HFT may not be needed, markets run by human traders were significantly more unfair than anything we have now. It's just that fewer people complained about it because nothing was transparent. Or do you believe human traders don't pull quotes?

It hasn't really, at least not in production. Academics are now publishing a lot of papers using Deep Learning or RL, but you won't usually see those in live systems.

In live systems, latency is usually more important than a "better" model - A model that takes milliseconds to make slightly better predictions is too slow when you're working on nano- to microsecond scales, often on specialized hardware. Really, the "AI" part is less important in HFT than you may think. It's often more system/infrastructure.

This is for HFT specifically, perhaps it has had more impact on longer time horizons, or something like portfolio management. My impression is (but I may be wrong) that there aren't that many people doing something in between HFT and much longer (minutes to days) time horizons, something like milliseconds to seconds. Maybe there is an opportunity there for some of the newer AI techniques.

I can recommend this [0] book. It's focused on financial time series and trading, but the techniques covered in the book are generic enough to apply to all kinds of time series, you can just ignore the finance parts. If you search hard enough you can find the PDF for free online. The way they treat convolution operators and efficiently approximate them with fixed-size EMAs was quite interesting to me. It's definitely a bit dated, but that's some of its charm.

[0] https://www.amazon.com/Introduction-High-Frequency-Finance-R...

What people are complaining about is not the overfitting, but the unrealistic assumptions in the backtest. In the real world there is slippage, latencies/jitter, special market open regimes, hidden orders, market impact, front-running, variable fees, and all kinds of other complexities. Their transaction costs are apparently also an unreasonable assumption. Sophisticated simulators used in professional trading firms can account for such things to some extent, but most academic papers conveniently ignore these complexities and just assume they can trade at whatever price the data tells them. It's completely unrealistic.

To answer your original question about overfitting, they can still overfit to test data by running a lot of experiments with different hyperparameters, architectures and parts of the data, and only report what has worked. There are also more complex ways that test data can leak into training data (see the book Advances in financial ML for a good overview). You can already see this is likely the case just from the variance in their results and trades. They also don't compare to baselines. It's not unlikely that the results are just random and they fail to report those experiments that didn't work. Of course, you cannot prove this without having an exact log of all things they ever did to the data. But again, that's not the main issue here.

Data distribution shift. The market changes over time and your current data does not come from the same distribution as old data. That limits the amount of data you can use for training and testing. You need to be very careful not to overfit. That's especially true for something like daily or hourly data - there isn't much data to begin with and you won't have much left if you look at only a few weeks or months. Market data already has a low signal/noise ratio to begin with, so you need a good chunk of data to learn from.

As you go to shorter time scales you get more usable data, but then you also need to deal with other issues such as latencies/jitter, market impact, complex order types, order book queues, etc. It becomes a different game.

I don't think daily OHLCV data is a good data source. First of all, it's too little because of the data distribution shift over time. It's also driven significantly by outliers and events outside of the data (news, etc). There's way too much noise in daily prices that most of the signal is drowned out (longer time horizons = more uncertainty). I don't believe you can find any edge looking at daily data. This kind of data is would be equivalent to what MNIST is in ML. Nice for some playing around, but nobody who is serious would use it for production or benchmarking, at least not by itself.

There is a good reason trading firms pay a lot of money (sometimes millions) for fine-grained historical data from exchanges. It's not only about speed. For interesting experiments you IMO need L2 or L3 order book data, ideally somewhere on second or sub-second scales. That's not HFT (which is nano and micros), but somewhere in the "middle" - it's a different world than what you are talking about.

By simulators he means market simulators for L2/L3 data with a matching engine, latencies, queue positions, jitter, complex order types, etc. You can't simulate other market participants (at least not fully, but there are techniques to even estimate this based on live trading feedback), but there are still many things left that you can simulate in a realistic way during training and backtesting. Trading companies typically have their own high-performance simulators built in house. Some of these are incredibly complex. Good simulators can give you a huge edge and are absolutely necessary.

Is 200-500k still true? It used to be, but I think it has decreased significantly over the last decade. I'd say most junior people in this field are making about the same or less than software engineers these days.

But like you said, the range here is incredibly wide and largely depends on how well your strategies do and if you have your own desk/fund.

In this simple example, yes, but you are assuming that monetary value = utility. That's not always the case. People have all kinds of different incentives for participating in the markets.

Let's say I am a market maker offering to buy Apple shares at $99 and sell them at $100. Let's take an ex-Apple employee who owns some shares. He just had a family emergency and wants to liquidate his shares to get cash, and he needs it quickly. He doesn't care about paying a few dollars extra in exchange for a quick trade because he needs to pay a bill tomorrow. I buy his shares for $99. He is happy because he immediately got his cash.

On the other side, there is a a retail investor doing long-term investment and wants to add Apple to their portfolio. They also don't care about a few cents because they're holding the stock for a decade and love the new CEO. They buy my Apple shares from me for 100.0. They are happy because I can guarantee them a stable price for a decent number of shares.

All participants are happy. I just made $1 from the spread for providing liquidity, the investor got the long-term investment they wanted, and the ex-Apple employee got his cash.

Sure, both sides of the market could have made more optimal trades if they had put in more effort and "optimized" their trades with algos and somehow skipped the middle-man, but they would've sacrificed convenience and time, which may be worth more to them than the little bit of extra $ they paid. Aren't we all winners?

When you go buy bananas in your grocery store you also don't complain about them taking a cut for providing liquidity. You don't say the farmer has "lost" money because the consumer paid more than what the farmer originally sold for to the grocery store. The farmer is happy because otherwise he may not have traded at all or his bananas may have gone bad (= needs to trade quickly). This is no different.

You make a good point. I've also gotten into trading because I enjoy the algorithmic and mathematical aspects, and I would love to share more of what has been working for me and write extensively about it. And there are probably more people like that out there. However, trading has such a bad reputation and uncertain future that I am not sure that's a good career move. I'm torn.

You're right that there are probably some gems and people writing up good posts and articles. However, 99% of what comes to my inbox, which is certain newsletters and arXiv subscriptions, is clearly BS. I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. In trading the opposite seems to be true. Seeing a good trading paper on arXiv is incredibly rare. I would even go as far as saying that reddit is a significantly better source of information than arXiv for this field.

Yup, it's probably just the output of an optimizer and then tested on held-out future data. Not overfitting is the key here and what's really hard. You need to be careful about the number of parameters and the amount of validation data you have.

These alphas will likely be only profitable for a short time period as long as the market data distribution (i.e. strategies of other market participants) doesn't change. So you would need to continually optimize and update them.

The way I think about it is that you are essentially finding the right parameters to "exploit" the combination of algorithms of all other participants, where algorithm could also be a human looking at charts and following certain rules, with a lot of random noise from retail traders thrown in.

Why do you think it's a weird subject of study, but games like Chess, Go, and Starcraft are not considered weird? Aren't both studied for purely their benchmark potential as opposed to the problem itself? Why are games widely accepted, but trading is "weird"?

That's overly simplistic. While the overall system may be zero-system over an infinitely long time horizon, this doesn't typically matter in practice. It can be positive sum for participants over some time horizon they care about.

For example, an HFT trader make pennies from each trade by exploiting tiny price inefficiencies. He essentially takes money from a "stupid" retail investor who does not know how to optimize his trades. However, the retail investor may not actually care about optimizing trades and just wants to liquidate assets or make a long-term (10+ years) bet. He is totally fine with throwing away a few dollars because optimizing his trades through complex algorithms would be too much work. Here, both parties win, the HFT trades gets paid because he provides convenience, or liquidity, to the retail trader. The same would apply to any human market maker, it doesn't have to be HFT.

And yes, HFT liquidity may disappear during HUGE market movements due to risk, but it doesn't disappear as long as both parties get what they want and the risk is manageable, which is "most of the time". Of course, HFT has other issues such as the race to zero and unfair advantages for a few central players, and I don't want to defend HFT. But saying that "it's all zero sum" is not correct.

An analogy is your nearest grocery store. They're a market maker because they buy from the manufacturer and sell to the consumer and profit from the spread. Do you also argue that these are all zero-sum and we should cut them all out and connect all consumers and farmers directly? And their liquidity also disappears when black swans (corona) happens :)

I don't claim that all the science that isn't actively being applied yet is BS, but this kind of science typically happens within trading firms, tested on real-world data, and is not being published on arXiv.

As a side note, what this specific paper here did is neither novel not innovative, so it's very fair to criticize it. A3C is 4 years old, and they just take it and run it on some data. It's like downloading a convnet and running it on MNIST. There have been hundreds of papers on RL + Trading. I see them in my arXiv emails every other day and they all do the same thing.

I actually believe that trading is an interesting problem that should be studied more in Academia and Machine Learning. It has many aspects (sparse rewards, long-time horizons, simulation-to-real-world transfer, non-stationary data distributions, etc) that current ML algorithms struggle with.

Unfortunately it seem like most ML people are not really interested in trading, perhaps because it has such a bad reputation (which is IMO unjustified) - so they work on games instead :)

In general, books are a much better source of information than papers or blog posts when it comes to trading. I haven't read the one you posted, but a few I can recommend:

[0] is okay. I disagree with a lot in there, but it's pretty well written and one of the better books on the subject. [1] Is very old, but it's one of my favorites. It's very mathematical. The ideas still apply today. [2] Is a good introduction overview

[0] https://www.amazon.com/Advances-Financial-Machine-Learning-M...

[1] https://www.amazon.com/Introduction-High-Frequency-Finance-R...

[2] https://www.amazon.com/Trading-Exchanges-Microstructure-Prac...

I don't think it necessarily has to be true. I also built a profitable system and wrote about it, but I didn't share all the details. Not even close. There are just too many small details that must be "just right" that they would fill a whole book. It's kind of like building an operating system from scratch. It's not something you can put into a single post or paper. There isn't one "trick" that suddenly makes it all profitable - it's a combination of so many small details.

Then there is the cultural aspect. People who are working in trading are just not used to sharing openly. They don't write online, or anywhere. They are not even allowed to write due to their employers. And people who work in academia are naturally not working on "production" systems - their only job is to write, not to build. So you almost never see people in the intersection of: writes-online & understands-trading & is-not-in-academia

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system.

Not only is it (very!) easy to overfit backtests (especially with so little data they are using here), but backtests are nothing like the real world. In the real world there are HFT traders front-running you, latency, jitter, fees, hidden order types, slippage, and a lot of other complexities that don't fit into a short HN post. Whenever you see a paper ending with a backtest you can already assume it's BS.

It's similar to training a robot in an extremely simplified 2D simulation environment without physics or other interactions, and then claiming one has built a real robot. A mistake many people make is believing that trading is all about AI. But in reality, the model often matters less than infrastructure/latency/system/data issues.

In addition to that, people who are actually "good" at trading don't publish papers, they silently make money. Papers are typically published by academics or students who have never built anything profitable but would like to put a paper on their resume. I have yet to see a single good academic paper about trading.

[0] https://www.tradientblog.com/2019/11/lessons-learned-buildin...

If someone wants to buy at $100 and someone else wants to sell at $90 it depends on who came first. To really answer this question, you need to understand the difference between market maker and taker and you need to understand how the limit order book [1] works.

Assume there are no other orders in the order book.

Scenario 1: Seller submits a limit sell order for $90. Since there are no buyers, this order goes into the book. Then a buyer submits a limit buy order for $100. The order would be filled at $90 (the best ask) and the buyer only pays $90. Here, the seller is the maker and the buyer is the taker.

Scenario 2: Buyer submits a limit buy order for $100. Since there are no sellers, this order goes into the book. Then a seller submits a limit sell order for $90. The order will be filled at $100 (the best bid) and the seller gets $100. Here, the buyer is the maker and the seller is the taker.

Market makers are responsible for setting prices and providing liquidity. If you want to understand this in more detail, check out this post [1] I wrote up a while ago.

[1] https://www.tradientblog.com/2020/03/understanding-the-limit...

The short answer: What exactly the "price" shown on the exchange website is depends on the exchange. Typically it's the last trade price or the mid price (average of best bid and offer). There really is no such thing as a single "price" because the price will depend on the quantity and direction you are transacting in.

Long answer: You need to understand how the Limit Order Book works. I wrote up something about this here [1]. It also goes into different definitions of price.

> If a very low-volume stock is listed at $4, and then I offer to buy a share for $100, does the NYSE suddenly start listing its price at $100?

If you trade actually absorbs the order book and pushes the asks to $100 then yes, that could be case depending on the exchange, but I'm not sure about NYSE specifically. Most likely that could never happen due to various hidden order types and HFT market makers though.

[1] https://www.tradientblog.com/2020/03/understanding-the-limit...

Thanks a lot for the detail. I've always wondered what the actual costs look like.

Author here. Personally, I just don't. I tried doing it but it was too complicated. So I end up just hiring a tax accountant specializing in crypto, send them all the data I have, and pay a few $k. In case something goes wrong, it's their fault and they take the risk.

You can't go short but you can still hedge your exposure by going on short on something very correlated - like Bitcoin futures. I do this manually depending on how much capital I'm trading.

I actually have no idea what my Sharpe ratio is, sorry. With the system constantly changing, data formats changing, exchange balance apis changing, and accurate monitoring already being a challenge in itself, it's very difficult to keep track of exact returns and historical data. I could probably calculate it if I spent several days (weeks?) trying to process all my historical trade data, but that would be a waste of time for me personally.

Or to pay the exchange a large sum to get privileged access :)

Many HFT systems provide liquidity. If there is no liquidity, retail investors like you or me cannot buy or sell.

Just imagine you want to exchange a currency because you go traveling and the exchange tells you "Sorry, nothing available right now, gotta come back in a few weeks". That's what would happen if there is no liquidity.

Author here. Actually, the system is mostly taking liquidity from the market so it's not even doing that :) Perhaps, "opportunity for others to create more liquidity"

Jokes aside, it's actually something I am thinking about a lot. Such systems don't create value, but they extremely intellectually interesting and I've learned a lot. You can say the same for many other projects, for example academic research in many fields. Most of it is just noise to promote the author and does not create value in the world. But it's intellectually interesting, so people work on it.

Other people write compilers for fun to learn something new without creating value. I don't think this is fundamentally different.