That list has Avengers Infinity War and Endgame on it by the way.
198 karma · joined December 20, 2012
Also your comment on the "Bankrupt MTA". Public transit is just that - PUBLIC transit. It is not supposed to make a profit, the same way highways for cars are a public good. When is the last time you complained that your taxes are too high because of all the road maintenance? Those damn highways better make a profit!
Look at the data before you gobble up right-wing misinformation. Do you even live in NYC?
- Sean M Carroll's work, in particular his Biggest Ideas in the Universe books: https://www.preposterousuniverse.com/biggestideas/
- Artur Ekert, basically the father of Quantum Cryptography has an amazing course for free on youtube: https://www.youtube.com/@ArturEkert . It's a very precise and understandable explanation of quantum computing, and some of the math that is involved with quantum mechanics.
- If you have hours to spare, watch Richard Behiel's videos on Youtube. He's like the 3Blue1Brown of Quantum Physics. His latest video on superconductivity and the Higgs Field is almost 5 hours long (!!!) https://youtu.be/DkH1citHtgs?si=-yQNYDu9TlTpE1A0 . It builds on his other videos, so I'd recommend starting at the beginning.
Watch the whole video (posted months ago predicting all these actions), but here is the relevant section: https://youtu.be/5RpPTRcz1no?t=1201
NYT interview: https://www.nytimes.com/video/podcasts/100000009910862/curti...
Gil Duran did a lot of the reporting on this. https://www.thenerdreich.com/the-network-state-coup-is-happe...
There's `perf stat` that uses CPU performance counters to give you a high-level view if your workload is stalling due to waiting on memory: https://stackoverflow.com/questions/22165299/what-are-stalle.... However it won't tell you exactly where the problem is, just that there is a problem. You can do `perf record` on your process, and then running `perf report` on the generated data. You'll see what functions and what lines/instructions are taking the most time. Most of the time it will be pretty obvious that it's a memory bottleneck because it will be some kind of assignment or lookup.
If you're using an intel processor, VTune is extremely detailed. Here's a nice article from Intel on using it: https://www.intel.com/content/www/us/en/docs/vtune-profiler/... . You'll see one of the tables in the articles lists functions as "memory bound" - most time is spent waiting on memory, as opposed to executing computations.
A buy market order would try to match with the "best price" which in a deeply crossed book would mean matching with a really low priced sell order. Exchanges match orders in price-time priority. Similar is true for a market sell order - would match at an extreme high price.
Besides the midpoint of the order book, another metric for a "current price of the stock" people use, is the "last trade price". In the situation above you would get "swings" in the price because market orders would be trading very high and very low if they alternate between buying and selling. The data structure on the exchange itself isn't "swinging", it's just the overlapping region being slowly eroded by market orders. The "last trade price" metric looks really insane in this situation.
The price is usually calculated algorithmically by the DMM firm and sent to the person at NYSE to approve. Pretty arcane. Also somewhat shady, as the DMM firm can be and is part of the auction themselves. DMM firms can analyze the order book to see what the imbalance is in the overlapping region, and place an order of their own to correct the imbalance and then set the opening price. I can see how one can profit from this in certain situations
Now, as a result, there needs to be a way to set the opening price and closing price, like a bootstrap process. A smaller version of this process actually happens every time a stock gets halted and resumed.
An exchange has an order book - orders of things people want to buy and sell at different prices. During normal operation the buy and sell orders don't overlap in the order book - if two people want to buy and sell at the same overlapping price, they just get matched by the exchange at that moment. Unmatched orders stay in the order book data structure until a matching order comes along. The "price" you see in charts is just the midpoint between the highest buy and lowest sell price in the order book.
Now, if the order book is empty, what the heck is the price? That's what the opening auction needs to solve. The way it works is that people can start placing orders ahead of the opening bell, but they won't get matched until the open. So before the open, the order book is getting filled with orders, but crucially the _orders will overlap_. This "crossed" order book is a no no during normal trading, but ok before the opening auction. When the auction comes, a price is picked which maximizes the amount of orders filled (it's more nuanced than that, but bear with me). Imagine you pick a price in the overlapping region of the order book - every buy order that has a higher price than that will match with every sell orders that has a price lower than that. They will get matched and executed at the opening price, and BAM, you have an uncrossed order book, full of orders.
If the auction doesn't happen, and you just open the stock, then all hell breaks loose. Many things can go wrong here. Firms connected to the exchange may have code that assumes a book is not crossed (or at least not as crossed as it would be during an auction) causing wild behavior. The exchange itself could start matching orders haphazardly in the overlapping region, causing those "price swings" that the article talked about.
Can't imagine the panic that day haha.
One small inaccuracy is the claim that there is only one place for each stock. That has not been true for many years. In the US that was changed by https://en.wikipedia.org/wiki/Regulation_NMS . NASDAQ is the primary listing exchange for MSFT, which means they will hold the opening and closing auctions, but it can be traded on any equities exchange, NYSE, IEX, BATS, EDGE-A, EDGE-X, you name it. RegNMS also has rules that if there is a better price at another exchange, the order must be routed there. This establishes the "NBBO" - National Best Bid and Offer, so in a way there is always one best bid and one best ask, but it's an aggregate over all the exchanges.
- Michael Lewis really only got one side of the story - that of Brad Katsuyama, who had a vested interest in casting HFT players in a bad light to promote his own business - building the new exchange IEX.
- Brad also blamed HFTs for systems at RBC failing to make massive trades like they used to. There was nothing nefarious here - RBC had just fallen behind the time in technology, like trying to send a Fax in a world where everyone already uses Email. If Brad, or RBC, or RBC software engineers picked up the phone and called any of the exchanges, they would probably gladly update them on the industry and save them all the work of re-discovering it themselves.
- The claims about front-running are completely false. Front running would mean that a market maker somehow knows someone's orders at two different exchanges and somehow is able to "get in front of the line" or even know that those orders belong to the same person. This would mean the exchanges leak information or allow certain users "ahead of the queue". None of this is true. What Michael Lewis called front-running, was HFT firms reducing their risk on other exchanges when they would get traded against on one exchange. They did this without any knowledge that Brad Katsuyama was on the other end, or that he was just late trying to make the same trade at another exchange at a later time. There are no guarantees that you can make the same trade at different exchanges - the same rules apply to everybody.
- Unsurprisingly, IEX as an exchange is no different from others, in that they need market makers (a.k.a. HFTs) to provide liquidity on their exchange. I wrote the code for the FIX gateways to connect our firm to IEX, and it was all business as usual.
The human readability argument doesn't really hold any water because if you have a structured description of a protocol (e.g. a C struct), you can always write simple tools to inspect the protocol and make it just as humanly readable as JSON is. This is even easier if a language has reflection to generate all this code.
The part I disagree with relates to "relying on the optimizer for placement". Even in C++ using the above factory pattern, you are returning the constructed object from a function - and there is no problem if it is ultimately part of some larger object. The C++ standard specifies copy-elision very precisely so you don't have to hope the optimizer does it - it is required to. To demonstrate you can do stuff like this even if you object contains non-moveable members, like std::mutex
class Foo
{
private:
std::mutex mutex_;
SomeComplexSubObject sub_;
Foo(SomeComplexSubObject sub) noexcept
: sub_{std::move(sub)}
{ }
public:
static std::optional<Foo> make(SomeParams params) noexcept
{
try {
return Foo{SomeComplexSubObject{params}};
}
catch (std::exception const& e) {
return std::nullopt;
}
}
};
I'd say starting a "pure" HFT firm is incredibly hard nowadays. There's a big upfront investment and huge operating costs, and there's a lot of consolidation happening in the industry.
In general, most hardware and software is tailored to high-throughput, and a huge part of the effort of HFT is fighting all that to prioritize latency. If you want to compete with the very best firms, that means renting rackspace in a colocation with the exchanges, overclocked CPUs, expensive network cards, FPGAs, paying out the nose for the most accurate market-data feeds, and many order-entry connections. Then you have to tweak the BIOS, kernel settings, isolate your processes on cores, write your own userspace networking stack etc. etc. etc. On top of that if you want to trade between different colocations, you might need a network of microwave towers because fiber is too slow (light travels about 2/3 slower in fiber because it effectively "bounces" along the fiber). At the end of the day though, if your algorithm or strategy doesn't work, you won't make any money - speed is only a part of it. At the previous firm, we had a market making desk that was very profitable, but was quite "slow" by HFT standards (triple digit microseconds vs sub-microsecond response times). You still need to be smart about it.
Having said all that, I think HFT gets a bad rap, about being a waste of resources/talent, and somehow "stealing" from the common man. The truth is, they are only "stealing" from each other - they compete to remove the tiniest inefficiencies in pricing. They ensure the cost of trading is actually as low as it can possibly be. By themselves they don't have enough capital to move the market in any meaningful way. In fact, most flash crashes occur when HFT firms step out of the way! They move out of the way of huge market movements because they can't cushion the blow without taking an unnecessary risk as a business. It's like asking your plumber to fix your sink while your house is on fire. Market orders, stop orders and panic are most of the problem when it comes to flash crashes. PSA: Please use limit orders :)
I might ruffle some feathers with the comments, but I think overall HFTs aren't the bogeyman most people think them to be. Sure there are some bad apples, but that's true of any industry, and it comes down to the people - not the system itself.
I think you're on the right track regarding value investment, if you're thinking about it long-term, and it's for personal investments. Passive index funds like SPY are the best bet for most people. Finding companies or following trends you in particular have some insight on can also help. Things like "having worked in industry X, and having read their whitepapers, this new company is clearly overbought and is all marketing hype".
If you want to get more analytical, look at portfolio theory, different hedging strategies, and try and find "alpha" - roughly meaning the extra factors explaining the price of an instrument that gives you an edge over others. Check out Quantopian - they have some nice articles too. The key is coming up with a model that accurately captures the risk of your assets which would allow you to properly allocate your money amongst them.