jnordwick

The definitive source on how to spell "chili":

https://redhotchilipeppers.com/

The Fucking Article, from RTFA - Read the Fucking Article - and RTFM - Read the Fucking Manual/Manpage

“I notice that you use plain, simple language, short words and brief sentences. That is the way to write English―it is the modern way and the best way. Stick to it; don't let fluff and flowers and verbosity creep in. When you catch an adjective, kill it. No, I don't mean utterly, but kill most of them―then the rest will be valuable. They weaken when they are close together. They give strength when they are wide apart. An adjective habit, or a wordy, diffuse, flowery habit, once fastened upon a person, is as hard to get rid of as any other vice.” -- Mark Twain

disappointed i had to scroll so far down to find the org mode reference. The text straight looks like an org mode doc too.

emacs org mode tables can even have formulas embedded in it and do rotations and automatic recalculation.

https://orgmode.org/manual/Advanced-features.html

Ender's Game IRL

smoke signals is a non-standard extension. however over avian carrier has its own rfc and is entirely legit:

https://www.rfc-editor.org/rfc/rfc2549.html

I was working in K2 in 1999, and it was public then. This is back when it had the built in GUI and dependency graph. I remember the first time somebody showed it to me, and I didn't think it was very impressive. It took about 2-3 months of messing with it before I realized how very wrong I was.

I control-F'ed and didn't find it before I wrote that comment. I searched for KDB and APL, but found nothing.

EDIT: yeah that sentence is new. I think some of the sentences around Influx being the "first" have been adjusted too now saying it is the first "mainstream" and called other db's niche. it's also a little off. KDB+ is more popular in large banks and trading firms than HFT places.

If you are writing an article about time series databases, and you don't mention KDB - straight to jail. It is the grandfather of time series and predates influx by about a decade. It is still the fastest out there too. It is used by about every major financial and trading institution in the US and Europe.

Everybody thinks TSDB are something new-ish, but they've been around since the days of APL. All you youngins disappointment me every time you write about time series, vector languages, or data-oriented programming and entirely neglect all the work that comes under the APL/Vector umbrella. SOA and DOD have been around for 50+ years, and they didn't start with c++ or Pandas.

Now the creator, Arthur Whitney, has a new one out called Shakti that is even faster (but has also ditched from the "niceties" of Q.

https://shakti.com/

Per Fog's tables, Ice Lake shows a reg-to-reg cmov decoding to 1 uop with a latency of 1 and throughput of 0.5. *mont all have latency 2. P4 had a latency of 6. AMD's K10 latency 4.

Now do the Erdos number: defined the same way but for coauthors of papers linked to the absurdly production (and always high on amphetamines) great Paul Erdos. The most glorious of all, the Erdos-Bacon number, is defined as the sum of your Bacon number and Erdos number.

Mathematicians Daniel kleitman and Bruce Reznik both tie for top spot with a number of 3 (Erdos 1s and Bacon 2s). Danica McKellar and Elon both in there with 6s. And Mayim Bialik, Natalie Portman, and Kristen Stewart all up there with 7s.

https://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93Bacon_numbe...

you don't really wait on L1 cache writes though. The store buffer absorbs it, and if the data is needed it can be forwarded from there before the write to cache even happens.

Most x64 L1d caches have a 4-6 cycle latency depending on CPU, that 1 to 2 ns depending on frequency.

since you aren't going to be writing a message per nanosecond, you can always just do `last nanos = max(last nanos + 1, cur nanos)` and then use last nanos for the timestamp. you can even do it in rdtsc ticks and get 1/3 of nano values. Obv the clock isn't nearly that accurate, but it lets you use those fractional nanos to ensure a strictly increasing ordering.

cmov turns a control dependency into a data dependency. It is still there just in a different way. (its like the Fourier Transform of instructions - lolol).

I haven't seen cmov lose to a conditional in a few years (on very recent Intel hardware). Maybe there might be some circumstances where in creates a loop carried dependency that wrecks the optimizer, but I haven't seen one in a minute.

To be fair - cmov versions often involve some extra compution to get things in a form where cmov can be used, and I often just opt for the well predicted conditional since those couple extra instructions to set up the value to move can be avoided.

The background thread can just sleep if no data in the queue.

Since you are preprocessing the log messages and you know the minimum bounds. If the thread sleeps for 1 ms. Be generous and say you can at most push to the queue in 5ns, and if you know the largest messages you push will be 200 bytes, you can statically determine a 40M ring buffer will ensure enough space to sleep even at max rate with the largest messages.

And that's just a simplstic scheme. If you have a pool logging structs so your queue is just a pointer to one of those, you can get away with much less even.

And I can think of faster ways too. It isn't that difficult to get around using a semaphore

that's what the vdso version of clock_gettime does. If you use one of the *_COARSE clocks it will only update periodically and be much faster, but that means like 15 milliseconds of log messages will all have the same timestamp.

The fastest for nanosecond precision (bonus is this is even sub nanosecond) is just to store the return value of RDTSC and let the background thread figure it all out. You don't even need to precalcuate the freq or epoch offset. Just write a couple logging messages of the rdtsc value and CLOCK_REALTIME and let the post processing figure it out.

To cut down on I/O each log message's timestamp can just be an offset from the last even.

If you are willing to push a lot of work to the background thread and even more to the post processsing step, you really don't need to do very much.

as long as you are just using static strings and native types it amounts to a pointer/index bump and a load/store per item. Lets imagine you have the format string, priority number, system id, and 7 pieces of data in the payload. That would be 10 items, so like 40 cycles? I can see the 18ns the paper gets.

I had no doubt the 7ns number is heavily cooked.

In my logging code I wrote that is basically a SPSC ring buffer, I use some RDTSC assembly and at startup I calculate the frequency and epoch offset. It has a throughput of around 30 cycles.

That's already ~10 ns, so I'm not sure how they are getting their numbers. If they are timestamping the data when the background thread gets to it that would be pushing even more work to it. It guessing they do or else they could potentially be logging out of order data with multiple threads.

It uses a background thread to do most of the work, and it appears the 7ns latency numbers are a little cooked:

1. The paper's 7ns like number is 8ns for microbenchmarks but 18ns in applications. The 7ns number I'm guessing is microbenchmarks, and the true application level number is prob more in the 17ns range.

2. It isn't precisely clear what that is measuring. The says that is the invocation time of the logging thread. Considering the thread making the call to log just passes most of the work to a background threads through a multi-producer single consumer queue of some sort, this is likely the time to dump it in the queue.

So you really aren't logging in 7ns. The way I'm reading this is you're dumping on a queue in 17ns and letting a background thread do the actual work.

The workload is cut down by preprocessing the creating a dictionary of static elements do reduce the I/O cost of the thread doing the actual writing (I assume this just means take the format strings and index them, which you could build at runtime, so i'm not sure the pre-processing step is really needed).

My logger than dumps binary blobs onto a ring buffer for another process to log might be able to beat this invocation latency. This isn't really groundbreaking. I know a few place that log the binary blobs and format them later. None of them do the dictionary part, but when that is going to a background thread, I'm not sure how much that matters.

When Linus made that comment cmov was like a 6 cycle latency. For the last decade it has been 1 cycle, and I don't think there is any scenario where cmov is now slower than a branch.

I absolutely loved Thompson's blog. He hasn't posted in a decade, and that makes me sad. I learned so much from that. I used the term "mechanical sympathy" all the time now when speaking to starting developers and trying to explain how to write fast code.

you have a group of machines, and each usually has specific obligations: logging message to file and db, one connects and gets prices and broadcasts them into the ring, one does order checks, one sends orders to the outside, one does strategies. Except there are many for each responsibility (like one price feed per venue). Plus these all have back ups.

The multicast ring every machine multicasts their packets, and if a machine wants that data it just subscribes to that channel. So the logger might subscribe to every channel, the price feeds will send to a specific group of channels, the order routing systems will listen on a particular channel, and core might subscribe to the strategy channels, and the strategies will subscribe to the price feeds and core and send to the order channels.

This allows you to have hundreds of machines without any slowdown, and you can literally just drop a new machine on the right and it just works. No need to change anything else. It highly decouples the systems. You just need to have a very tight protocol and extremely well written network code (these are always custom implementations - I've written custom SSL/TLS, websocket, webservers, TCP itself, etc. for these things.

This also allows close to perfect replayability. The machine logging logs everything, and then when you want to analyze something, you generally have tools to replace the message back into the ring and get the exact same state.

And all these active machines also have backs up, so you have two machines taking orders and routing them for a venue: one is the currently active and doing all the work and heartbeating, but there is another that is just reading all the messages keeping the same state as the primary, but when the primary goes down, the secondary can seemlessly pick up and none of the other machines need to even know what happened. It makes for a very resiliant system.

Plus you can use any language you want. We wrote all the trading systems in C++, but we also had nodes on the ring in python, java, and even shell scripts listing for particular messages. I remember one hack I did when the futures market keeping hitting its limit down a couple days in a row, so we made a shell script that listened for a particular limit down message from the price feel to be broadcast into the ring, and when it got it, it send another message to each algo telling them to get out of anything and pause. took less than a day to write using bash.

Its a really nice way to handle event systems, just it is sometimes difficult to get others to understand the benefits and it requires a lot of implementation skill and experience.

In finance, I know it goes back to Island ECN (which became part of NASDAQ). They had a small group of heavy talent that spread out to a number of high profile banks and prop shops. I've worked at a few places with the same architecture and it always seems to lead back to those guys at Island by a couple degrees.

I wouldn't be surprised if there was some cross pollination. I regularly read game dev blogs for ideas and information. While trading tends to have higher latency requirements and stronger networking requireemnts, games have other issues (the PCI bus and graphics card), but in the end there are a lot of architectural similarities and both tend to be a small group of highly skilled developers so you have a lot more freedom to chose complex index and advanced tooling.

purchasing protection and fraud need to be a percent. And "necessary" isn't the correct term for prices, "can" is.

> federal debit card

I don't see how this could possibly be abused by law enforcement or politicians. I'm in!

> The goods in stores that take only cash do not tend to be cheaper than those that do.

In NYC they most definitely do. A lot of the corner stores will change you less with cash. I'm not sure it is a the card payment or that they are keeping the sale off the books, but something that might cost me $18.50, I'll pay $18 for.

The big win is how it allows bulk processing while taking serialization dependencies into account.

So you algos reading market data messages and yank all that is there and apply them in bunk instead of trying to do each one individually.

And your logger that is storing all the messages to a file and/or db knows what messages the algo has processed and can grab all the messages up to that point so it is always writing as much as it can and current.

This isn't even close to unique. I['ve worked at a few HFT/MM places that all had a similar architecture in place before LMAX wrote this. This, multicast rings, and other patters are copied a lot in the industry because they work well for the problem space.

Levine is a national treasure. He's always so balanced, funny, and informative.

It makes sense for interest rates to go down when the reserve rate is cut. That means there are more dollars to buy bonds with pushing up the price and lowering the yield (the inverse relationship between price and yield is a mathematical identity). Eurodollar rates are routinely lower than us dollar rates, and they don't have any reserve requirements.