Ask HN: I am interested in reading about techniques HFT firms use | Hacker News Reader

Ask HN: I am interested in reading about techniques HFT firms use

100 pointsgk2k0873 comments

I am an aspiring technologist who is interested in understanding how the tech and techniques in High frequency trading systems. Can you recommend me some reading resources?

View on HN

I was in this area from 2003 to 2013. Can't think of any reading materials, but it all boils down to ultra low latency (think < 100 microseconds within the local network) and high volumes (think > 30,000 transactions per second). The state of the art may have moved since then, but:

1) Network layer: raw TCP sockets with binary packed messages (sometimes structs directly cast to and from raw buffers, keeping network/host byte order in mind); separate message data dictionary to avoid metadata overhead inside messages.

2) Processing: optimised C++ code; pointers; multi-threaded processes that utilise all available cores; minimal locking and contention between threads (queues with spin locks and bulk transfers); threads directly bound to CPU cores (prevent context switching overhead); prevent heap locks by allocating large object pools at startup;

3) Disk I/O: minimise disk I/O in logic-heavy, low latency threads; application level in-memory disk write queues; prefer append mode writes; application level in-memory caches (basically hash maps of reference data loaded at startup)

4) None of the "trendy" programming practices: no OOP design patterns; no dependency injection; functional programming, but without fluent APIs;

5) No popular frameworks: internally developed solutions optimised for our specific use cases, instead of off the shelf libraries or frameworks (e.g. the messaging middleware was in-house)

There is more, but this is what I could think of, off the top of my head.

To add to this, I think nowadays even C++ is too slow for some stuff. Many places use FPGAs

Having fast C++ code is always good even if its not directly responsible for execution, but yes typically order entry and trade parsing at really serious HFT firms is done with an FPGA/ASIC. Sometimes a trade packet wont even hit the software and instead be processed entirely on the FPGA.

Some neat tricks people have done is to actually process packet bytes in order as they come in on the FPGA, and if it has some actionable header they just send an action before the entire trade packet is received. With that you can go down into the few ns for your w2w latency.

Is C++ slow even if you don't use any language features that add runtime overhead, e.g. runtime exceptions?

It’s not so much C++ is slow as FPGAs are so incredibly fast.

The system programming languages (C / C++ / Rust) will always have more overhead than FPGAs.

A microsecond is fast for C++. It’s like an eternity for an FPGA.

Not C++, general CPUs are “slow”.

When I left, they were experimenting with an FPGA implementation of the TCP stack (!). Obviously a very challenging undertaking, not sure how far they got.

Interestingly, other than direct cabled network connections and trying to get ground stations physically closer to the satellites, which is obviously impossible, this is almost exactly what we did for first-level ground processing of geointelligence data for the NRO. Major difference would be data from the sensor platform doesn't even bother with TCP. It's just pure packed structs, but also encrypted with decryption handled by dedicated hardware devices. No need for PKI since it isn't public. The collection metadata from the tasking station still uses TCP.

re (2) for execution code, its actually the opposite. Typically the execution hotpath is done on a single core and some places go as far as disabling hyperthreading to eke out as much single core performance as possible.

yes, no points of contention at all

userspace networking with one thread busy polling, all power state management + ht disabled with receive packet steering

You can troll through the job postings of most of these companies and learn about their org structures and tech stacks. When I looked almost nothing is public other than older documentaries. Nanex had some neat wiring diagrams and a few videos clips if you google them. Don't get too excited you're not going to learn about some super secret strategy or anything.

This book was good https://www.amazon.ca/Advances-Financial-Machine-Learning-Ma... in that is sort of explains at a high level how you'd build a company to look at something like this (think it was one of the last chapters). You'll need extremely deep/skilled hardware/networking groups, data acquisitions (think packet capture to data) folks, people to clean and maintenance massive the data repos, internal tool teams, quantitative analysts, traders, management/admin, etc. At this level just getting the fiber to each exchange likely costs many many millions. When you go and look through those job postings you can sort of slot these into your mental model of this flow and start to see how it might be put together.

That book is good but if you are new to the topic it will be almost unreadable. At least it was for me. I had to read it 3-4 times over the years and still find new things as my knowledge grows.

I don't know if rentec is a HFT firm or what strategies they use but this was a really good interview with one of the founders https://www.youtube.com/watch?v=QNznD9hMEh0.

> just getting the fiber to each exchange likely costs many many millions.

I've never worked in US financial markets, but that's def not true in many countries.

< 400 USD / month can set you a server + connection within a B3 datacenter (Brazil's largest and only exchange).

Worse fiber is a terrible choice over what’s typically used in low latency trading (which is different than but related than high frequency) - typically microwave networks are used because you get good line of sight advantages and latencies are better. HRT, stryker, and others are heavily dependent on their microwave advantage.

Just to clarify this for people who don't know the EE details of this. Even direct point-to-point fiber suffers compared to microwave because of the velocity factor of the cables. What?! Inside a glass fiber, the speed of light is only about 70% the speed of light in free-space! Microwave in air, though, travels very very close to the free-space velocity of light.

I assume from the comment they mean from one exchange location to another, but even then it’s not true. Those routes are _overbuilt_ precisely because of HFT so getting on dark fiber for that is cheaper than other backhaul fiber.

I’ve been out of HFT for a long time but when I was in it you could get onto the state of the art microwave networks between Chicago and the east coast for ~25k a month, which isn’t cheap but also not the biggest expense an HFT firm would have.

I think some US firms may have dug tunnels beneath the Hudson river to be able to locate in Jersey City rather than Manhattan. That probably cost millions.

They would probably use microwave vs tunneled fiber.

That’s a hard question to answer. The question is far too broad.

HFT is a combination of lots of things: low-level computer architecture knowledge, hardware engineering, computer networking, numerical computing, machine learning, super-computing, market simulation, market microstructure knowledge, risk management, financing, psychology/game-theory, etc etc.

Usually no one person is in expert in all of those areas. Usually, a firm is successful because they put together a world class team with all of those skillsets AND they’re willing to invest heavily in the right things.

And even then, it’s so competitive that you can still fail. It’s quite challenging to get everything right. There are far more ways to fail than to succeed.

I’d suggest you pick some sub-area you’re interested and focus heavily on that.

I don't know very much about HFT, and these books are old, so take this with a cow lick of salt, but I've seen these mentioned on a blog I found here on HN (by 'yummyfajitas), and I found them illuminating:

Algorithmic Trading & DMA by Barry Johnson

Trading and Exchanges: Market Microstructure for Practitioners by Larry Harris

You might also be interested in 'yummyfajitas blog:

https://www.chrisstucchio.com/blog/2012/hft_apology.html

If people have up to date replacements for these books, I'd be super interested.

If you keep asking around, people are gunnuh recommend Reminiscences of a Stock Operator, and it's an enjoyable book but keep in mind that it's basically the fictionalized biography of a problem gambler who happened to operate in the stock market, he traded recklessly and died destitute (by suicide - when you reread it, you start to notice how he never mentions enjoying anything, or talks about the people in his life in more than passing). Don't take it too seriously.

These are still fairly canonical, and what I got recommended when I started recently. Trading at the Speed of Light by MacKenzie is very good too- a lighter read, but more recent.

The LMax architecture: https://martinfowler.com/articles/lmax.html

Also read up on branchless programming, lock-free data structures and avoiding heap-allocations.

This was a fun read I stumbled upon a while back: https://web.archive.org/web/20210205014443/https://meanderfu...

second this

I went down this road a few years ago. Reading up on algorithmic trading is a great start. HFT often employs normal algorithmic trading techniques optimized for low latency. There are some algorithms that only work when you have a temporal edge. That is the software side. The hardware side is really interesting as well. If a connection to an exchange can be reduced by a few milliseconds it gives a competitive advantage. There is a lot of money spent to be the fastest because for some types of algorithmic trading being fastest almost means free money. Because of how lucrative a minor edge can be you are not likely to find many details on the bleeding edge of HFT.

Beware, the more you read about the practices of HFT firms the more some of it looks like fraud on a grand scale. For example, front running orders is a common practice that goes unpunished for HFT firms but will land you or I in jail.

I got into algorithmic trading and HFT when I was dabbling in bot trading cryptocurrencies. The wild west of crypto bot trading is pretty much gone now but man it was fun while it lasted.

In addition to the algorithmic trading books suggested I found these links helpful: https://www.quantstart.com/articles/Beginners-Guide-to-Quant...

https://www.youtube.com/watch?v=rB5jJuMP84E - just neat

https://www.youtube.com/watch?v=oUVchh37AO8

https://www.youtube.com/watch?v=bo-F61ZuBDg - 2010 flash crash

Do you work at/have worked at a trading firm? Frontrunning is indeed illegal and several firms get fined for it.

I was at a top trading shop and we and other places take compliance pretty seriously nowadays. Otherwise (if you aren't a prop firm) you'll lose clients, get fined, lose reputation, etc.

Also fyi being the very fastest has diminishing returns. You are at the whim of the exchange's matching engine processing order, which may be nondeterministic.

There are places nowadays that derive their edge from fast hw/sw, but increasingly, the most successful places are ones with the smarter strategies, higher volume, and more products traded.

I found Headland's blog post about quant trading useful [0] and as well as their other blog posts [1].

Jane Street's podcast and blog are insightful too [2]. I read somewhere that Dr. Simons (one of the founds of RenTech) has this book in his office [3].

0. https://blog.headlandstech.com/2017/08/03/quantitative-tradi...

1. https://blog.headlandstech.com/

2. https://blog.janestreet.com/ and https://signalsandthreads.com/

3. The Econometrics of Financial Markets, https://www.amazon.com/gp/product/0691043019/

KDB+ and Q: https://kx.com/academy/ Julia: https://docs.julialang.org/en/v1/ The C Programming Language: https://www.amazon.com/Programming-Language-2nd-Brian-Kernig...

I don’t know if any HFT that use KDB and q or Julia, those are more quant funds. HFT tends to be heavily C++ not C.

Alright, well there's at least one that uses KDB. Agreed on C++ over C, but given that we're talking to someone that's calling themselves a 'technologist' thought I would recommend the best book to get started.

I've had a lot of people on Hacker News very confidently tell me that I have no idea what I'm talking about on this topic. For a long time I thought I was just dumb, but still employed years later and have realized that most of the people that 'correct' me have exactly 0 experience.

I think you raise good points. Please consider whether your comment would be better if it looked something like: "I know of at least one which does. I have relevant professional experience in this regard. I agree re C++, I thought these were the best books to get started, given they described themselves as a technologist."

(From someone who also gets defensive in online comments. I don't think they per say told you that you were wrong, but rather described their experience in relation to what you'd said. It seemed to me like commentary more than criticism. I don't have a crystal ball to tell me when people are exaggerating their credentials on the internet but they seem to be credibly claiming experience in this thread.)

Your message tells me that the tone that I intended was properly conveyed.

Hey I’m sorry, I’m not telling you you don’t know what you’re talking about. I do know what I’m talking about, it’s my field, I was just pointing out that often people mix quant and HFT which have different needs. Julia and KDB and Q tend to be more about statistical arbitrage or risk neutral pricing based model trading or whatever. They’re definitely widely used in funds, and a lot of traditional HFT have branches out from traditional HFT to include fast statarb and other things to juice margins and find a place for capital investment, so I wouldn’t be surprised if many of these shops use these tools somewhere but they’re not using Julia say in the kernel of the traditional HFT strategy.

I think if you feel like that’s happening to you the best strategy is to ignore it, but the second best is to prove your understanding. Explain with depth that can only be gotten from direct experience. If you know you know, either be confident in that or demonstrate it. I didn’t feel like you didn’t know what you were talking about though, I just thought you might be missing some nuance on the topic.

HFT is relative. But here's one thing that is actually used in many firms: https://github.com/Xilinx-CNS/onload (and correspondingly these network cards: https://www.xilinx.com/products/boards-and-kits/alveo/x3.htm...)

The sniper in mahwah blog is a great reading series on Microwave towers and other information you can glean https://sniperinmahwah.wordpress.com/2019/03/26/4-les-moeres...

A good resource I have been reading is from a former HFT firm founder, Christina Qi

Here you go:

Part 1: https://twitter.com/christinaqi/status/1670221886587047941

Part 2: https://twitter.com/christinaqi/status/1670910949921005568

Part 3: https://twitter.com/christinaqi/status/1673621133885849600

Part 4: https://twitter.com/christinaqi/status/1676135720417910784

Part 5: https://twitter.com/christinaqi/status/1677359287172231168

Max Dama used to have a blog. Don't know if it's still up or archived anywhere.

The main aspects are:

On the trading side, HFT is mainly about what's called "market microstructure." These are relatively simple things that are quirks or features of individual markets. Things that most people don't care about, but can make a large difference in HFT profits include when an order comes in, who gets to trade with it first? How are orders routed between different exchanges? What are the popular types of algorithms that people who place large orders use?

On the tech side:

The main is minimizing tick to trade latency. This includes using techniques as basic as colocation at the exchange's (very expensive) data centers and as involved as designing custom ASICs and anything in between like bypassing the normal network stack.

Unfortunately the real state of the art things are all closely held trade secrets, but I'd start reading the exchange data specs to start getting an understanding of the microstructure side, and you can build some of your own experiments to see how quickly you can go from tick to trade. Can you get it sub 1ms? sub 100 us? 10us?

As of my involvement in high end HFT firms a few years ago no one was yet doing ASICS and the state of the art was FPGA. I would say people have a mythology about HFT that’s both deserved and overblown. They tend to be very sophisticated in many ways, but also surprisingly immature. Likely because they’re so private and insular, and grew up very non traditionally. They’re often also long term participants so their stacks are fairly aged at this point. Most of their advantage is the barrier to entry is high.

I know of a few places actively looking into ASICs/hiring xilinx/intel/amd hw engineers to work on them, but from what I can tell the majority just use FPGAs.

So if you can't colocate with the exchange, it's pretty much futile?

Exchange collocation is pretty much table stakes.

I've learned a lot from Matt Godbolt's blog https://xania.org/ . He's the author of Compiler Explorer, also works at https://aquatic.com/.

I would note afaik aquatic isn’t HFT per se but quant trading.

Strategy-wise, the fundamental HFT strategy is to detect large orders from banks or hedge funds as they are filled on one exchange, conjecturing that the order may have been split between multiple exchanges, and the using their speed advantage to front-run the order on other exchanges by a few milliseconds. This is what is describe in Michael Lewis' book Flashboys.

Most of these strategies are simple in concept but extremely difficult in execution. Many of them also rely on fine-grained understanding of the exchange's internal mechanisms, things that most traders would not know or be able to figure out.

I enjoyed my time in HFT, it is a fun game to play for a while, but stressful and ultimately unsatisfying as a career trajectory unless you are making serious cash.

This is fascinating, but is focused on the engineering side and not the modeling: https://www.youtube.com/watch?v=b1e4t2k2KJY

Check out their podcast too.

https://signalsandthreads.com/

Great to see Nigito. Brian, if you're reading this you still owe me a dollar from our bet.

While they are out of date and not too technical (and arguably, quite biased), the two books that I've enjoyed the most on this topic are:

1. Dark Pools, by Scott Patterson, and 2. Flash Boys, by Michael Lewis.

Both are highly recommended. The field has moved on, but they still explain the history of it in a very entertaining fashion.

Flash Boys is nonsense. Largely a placed ad for a specific exchange. You should skip it.

Dark Pools is better but doesn’t particularly touch on technical approaches.

You might find some useful recommendations in this recent thread:

https://news.ycombinator.com/item?id=34766728

Check out books Dark Pools and Flash Boys for some stories about HFT...

https://www.amazon.com/Dark-Pools-Scott-Patterson-audiobook/...

https://www.amazon.com/Flash-Boys-Michael-Lewis-audiobook/dp...

For deterministic low latency infrastructure concerns read the docs for red hat’s real time distribution.

https://access.redhat.com/documentation/en-us/red_hat_enterp...

Last I heard from a finance CTO (years ago), the type of companies that would engage in HFT have a direct line and latency SLA with the specific markets they would trade on. My impression is this is not something you can emulate on a small scale in a consumer setup.

The variety of answers here is telling - doesn't seem to be much consensus. Not sure whether that means various people have very different approaches, or just "those who know don't tell, those who tell don't know".

Isn’t that the entire moat of these companies? Very curious what is public

Tech is mostly FPGA order placer interfacing with a C++ market data decision router and a high end Melanox card. Their kit is inserted as physically close to the exchange matching system as possible. One of the big firms famously reversed the box insertion into the rack and trimmed their Ethernet cables down to minimize the physical length signal had to go (probably theatre but that’s the mentality). Some strategies require good low latency inter exchange connectivity so a comprehensive and well organized microwave network is another bit of the stack. Otherwise it’s largely what you would expect. There’s some mythologizing about these companies but a few notes: they’re often highly insular and their code and stacks can be 10 or more years behind the times, but the kernel of execution will be highly current within that stack. A large part of their moat is that mythology, most people would quit and do some other segment of finance like quant trading or stat arb rather than try to squeeze out some profit in HFT. My view is it’s not a winner takes all market though and there’s a lot of room at the table, but it’s a very challenging market overall. I think bringing in a more modern tech stack that cuts the cable back a little more would be an advantage too.

On the techniques, there’s not a ton explicitly written unlike quant and statarb that have almost too much written about them. The essence is very simple though, you basically use techniques you would use in board games or casino games - look at the structure of the game itself and try to find little gaps in the rules that let you insert yourself. For instance if you can see the order being placed into the book before any march is processed you can try to insert yourself into the match engine before anyone else. That’s low latency trading more than HFT though. Likewise price discrepancies between exchanges (microwaves) is low latency.

HFT would typically be something like a market making strategy where you place lots of small orders around where you believe the market is going playing long and shorts. You place bazillions of orders all the time to pick up small variations in a way that on average is better than random trading. This is high frequency trading because the frequency of the trading is high.

Often they get conflated because they complement each other. Being able to do one often implies you can do the other, so why not? They also sometimes help each other synergistically.

Source: I was there (and I’m building my own as a side project with some long time coworkers)

>A large part of their moat is that mythology, most people would quit and do some other segment of finance like quant trading or stat arb rather than try to squeeze out some profit in HFT.

I find it silly that math types gravitate so much towards HFT/quant trading when there's so much money to be made just by studying companies and becoming a traditional trader/investor. Why go to the bother going down the route of having to invest millions of funds to build an uncertain HFT business when there's no cost in opening a retail brokerage account.

If one lucks into getting a high paid HFT/quant job straight out of university it's great to save some money but otherwise I wouldn't recommend it except for pure engineers.

Well, most people don’t have much alpha as retail traders either. Technical trading is voodoo mumbo jumbo. HFT and quant trading actually produces reliable returns. But I agree don’t try to do it yourself unless you’ve been doing this a long time. Me and my partners each have 30 more years doing this and know the weaknesses and our strengths and basically what we need to do to get a toe in the game. The reason it’s worth it is generally there are no customers, no investors, nothing but you and your partners and you can make a lot of money and be independent, then grow that into a firm. That’s attractive, and frankly with my background I feel it’s easier than trying to make a tech/retail/b2b startup again.

Who said technical trading? If one reads the financial statements properly it will take less than 30 years to become profitable.

30 years - I want to be profitable now. :-)

I got paid for the last 30 years. I didn’t do it as a way of investing. I did it because it’s some of the hardest technical challenges out there. I did a tour at FAANG, built my own startups, among other things. Now it’s my turn to do for myself.

Value trading works but it’s a slow slog and returns are bounded and tight. In many ways stat arb and similar quant trading can be viewed as value trading with power tools. It’s good stuff. But I like to make things fast and powerful. For me it’s not about being profitable, although profit is the primary metric. It’s about doing awesome stuff with awesome stuff no one has ever seen before.

Are you interested in the tech, the trading techniques or both? Do you have any experience with software development or trading in general, or are you starting completely from scratch?

You probably won't find good resources unless you work in a HFT firm where you can gain the inside knowledge.

There are three techniques:

1. Market making 2. Directional trading 3. Arbitrage

The tech is an enabler of these.

Flash Boys by Michael Lewis was a good read, IMHO.

high end network cards with insane packet buffers and short, fast, expensive fiber runs to their hedgies servers.

Has HFT any value for the society in general?

Yes. It creates liquidity in markets which causes markets to be far more stable and accurately priced. I would put it at worst as morally neutral.

I'd say it depends. A lot of HFT is able to manipulate prices

It technically helps increase market liquidity (enabling someone to easily sell or buy new positions) and there's research that it decreases the bid-ask spread, but I don't buy it. If you believe having a small portion of society be able to easily trade stocks, then yes, but imo, I'd say it generally doesn't have any value for society in general

Seconding that. Nothing in this discussion convinces me that HFT is not simply a money-skimming operation.

They add liquidity to the market, so when you (or someone indirectly on your behalf) buy or sell stocks, you (they) get a better price through narrower bid-ask spreads. They also sometimes generate comedy[0].

[0]: https://www.henricodolfing.com/2019/06/project-failure-case-...

Does a stock market have any value to society in general?

If so, should there just be one stock market in the whole world, listing a small set of uncorrelated instruments, or is some level of competition/ variety healthy?

We haven't found any way to have those things without either HFT or something more expensive.

It's consistent and reasonable to take some kind of fringe anti-capitalist position and answer no on one of the first two questions, but if you answer yes, you end up with HFT providing value.

Vanguard's Chief Investment Officer said that HFTs are beneficial for them[0].

[0] https://www.sec.gov/comments/s7-02-10/s70210-122.pdf

Reply on news.ycombinator.com