The $440M software error at Knight Capital (2019) | Hacker News Reader

The $440M software error at Knight Capital (2019)

henricodolfing.com

294 pointsbfm172 comments

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012.

Some of the issues mentioned include:

  - Keeping synthetic test data generation as part of a production build.
  - Keeping dead code for years.
  - Re-purposing a feature flag.
  - Refactoring without regression tests.
  - Manual deployments without peer reviews. They forgot to update one of their servers with the new code.
  - Automated alerts sent via email were ignored.
  - Rolled back to a version of the code running on the server they forgot to update, making things worse.
  - Rushing out a release without proper software engineering hygiene.

The article suggests improvements that could have prevented the chain of events.

For those here who are in HFT circles, have things improved after the Knight Capital Group debacle?

edit: formatting

Hard to speak for HFT in general. Like in software, different firms have different levels of hygiene. About half of your bullet points were true of my previous employer, at my time of leaving.

Some of this is unforgivable, but reflecting on it I also realized that software engineering at quant firms has an almost impossible mandate. You want something akin to the extreme rigor of mission critical software (airplanes, cars, NASA, etc), while also remaining nimble enough to modify strategies as market conditions rapidly evolve.

Same is true for blockchain smartcontracts, which have similar catastrophic consequences.

That truly is scary to me. I can easily* write advanced Solidity and could try to make something big. But I won't, because I know I would not be able to handle the stress and responsibility. One tiny logic error and millions lost. Thanks but no thanks.

*The fact I believe I could easily do it is probably exactly why I'd end up making some huge mistake. ;)

I’m sure that any big contracts are written only by CMM Level 5 organizations using formal methods and provably correct with certification that less than 1 line in a million contains an error. It’s all spelled out in the SOW attached to the RFP.

I don't think I could, since the language was designed by amateurs after no research into safe programming, and is only being improved one gigantic loss at a time.

Maybe if you could translate from Coq.

Why unforgivable? It's only numbers in an account. No one died.

It is challenging, although, with financial markets, it seems like it would be simpler to have some automatic anomaly detection mechanism to unplug or slow things down to prevent further damage.

There are a lot of preventative measures they could have taken, starting with just not leaving in dead code and paying attention to automated alerting. But the moral of the story is that they got away with it for so long that nobody cared about it anymore. After all, if it were truly a big deal why hadn't it broken years earlier. Then when the technical debt finally got called it bankrupted the entire firm in one go.

Most of us (hopefully) have less devastating technical debt to deal with, but it is still a cautionary tale about what could happen if you ignore it for too long.

That's a weird statement.

The extreme rigor on the one hand seems to require a value judgement of the real benefits to HTF that I'm not willing to make. The remaining nimble'ity, on the other hand, is an odd word to use over agility or old fashioned responsibility. The benefit is proportional to it, but not exclusively.

The rapidly evolving market conditions concern regular trade too. Swift reactions are expected in any other systems application. "almost impossible" is a weasel word. It's almost impossible to win except for the last man standing, is that it? And there's no practical upper limit to nimble'y, though conservative estimates indicate that less work is more.

What's missing is the perverse incentives, corrupt policies, sociopathic leadership, ...

I worked in algo trading for years, eventually got out because quite frankly the level of risk I was carrying on my shoulders everyday for what I was being paid were just way out of whack, I at least personally never got the huge pay days that people talked about until after I left finance for more pure tech. Interestingly, I worked at Knight and my team pioneered trying to blow up the firm, but that was in 2004, and things were much friendlier- instead of front page news, it was a small blurb on page 3 of the markets section of the WSJ.

Anyway, I still have friends in that business. It hasn't really changed, they have too few people covering systems that are quite complex and while there are checks and such, no one really understands things entirely from end to end in detail that can prevent all problems.

I will never invest directly in an investment bank- either through carelessness or maliciousness I could have easily caused a 9 figure loss, if not more, and there were probably a thousand other people in the same position.

When I read the detailed writeup around this a few years back, I think by far the biggest issue was reusing a tag that had been previously used to denote which strategy to use. I understand why they may have chosen to do so, at the Big Bank I was working at, getting a new fix tag to be passed through all the layers properly would involve at least two other teams and coordinating releases and probably several weeks worth of meetings. If you just reuse an old value you can avoid all that since everything is already set up.

I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or whatever, which is based on fund performance and yada yada. But it feels way off. I had heard so much about the staggering paydays at these places but it seems you need an ML PHD or some trading chops to be part of that.

The attitude that finance pays more is a leftover from a previous era. 10-15 years ago it was true: the profits from HFT were so also way, way bigger and split up amongst a much smaller group of firms.

Now those firms are all in a completely competitive industry squeezing each other for basis points.

Meanwhile the definition of a FAANG is that it has an effective monopoly, and these companies are taking in way more money than the HFT industry. (Netflix is losing its monopoly but we can’t really drop N from the acronym without a replacement..)

> but we can’t really drop N from the acronym without a replacement

Huh, yeah. That would be quite a GAAF. Gotta come up with something before Netflix is forced out of the FAANG club.

I’ve seen MAAM being used.

When FB took it's huge hit I saw people using MANGA. But again if we drop the N...not great.

FAGMAN is still my favorite.

Typical edge lord naming from r/wallstreet bets, thats where this originated from wasn't it?

Same as FAANG to FAAG, shocking but a touch funny.

MANAMA is fine, you just get MAAMA in case of Netflix sinking.

GAMMA works if you add back Microsoft, who are doing better lately.

Tbf, most of us don't really prefer to be called as HFT's but as Market Makers. Different name, but we still use the same ultra low latency techniques to get the job done.

“The job” being trading securities at high frequencies?

I worked in option AMM when the term HFT started getting thrown around in some articles, and was discussed as this uber secret hush hush thing, and I was interested. I was then astonished when I kept reading and found out that it was what I had been doing- it was just another name for market making really...

Yep. At the end it is market making, in markets where speed is of the essence to be competitive. Funny thing is, now firms in the industry want to be termed as Market Makers rather than HFT since the media has turned HFT into a very negative term.

Back when Flash Boys came out, we all started calling each other Flash Boys. There was then a serious proposal to rename one of the strategies (the hedger, for added irony) Grandma Annihilator 3000.

When the KCG thing came out, one of the new grads was renamed “Power Peg” for a few weeks.

I miss the 2010s. It was a lot more fun when people leaned into stuff instead of being scared all the time.

Hahaha, that is pretty cool. I don't think we'd do something like that, especially the "Power Peg" one, at my firm. Although, it'd be fun.

> I miss the 2010s

Ah, so true. I was in college and grad school during that period, would have been more fun to work in the industry back then.

I cannot confirm this. ~300k is pay (excluding sign-on) fresh out of college at a big HFT -- sufficiently senior devs make 7 figures.

At our firm, the numbers are similar: $600k TC for new grads ($200k base, $100k minimum first year bonus, $300k signing bonus)

WTF! I am at an HFT firm in Chicago, this is insane. This seems to be a lot like an offer from Radix, or Headland, or maybe Algo Dev at HRT.

There is certainly much variance between the firms, especially the sign-on IME. People I know have turned down HRT core dev for big tech because their offers were unimpressive.

I think an interesting target for comparison with big tech is Jane Street, since their culture and WLB are good, so the main QoL drawbacks of finance don't apply. A new grad will get ~300k at Jane Street, though probably not with this large a sign-on.

This is interesting, didn't know this about HRT Core Dev where offers were below FAANG. My understanding is that core devs are basically the folks who work on all the low latency stuff, so they'd be pretty well.

Jane Street, from what I recall, is 300K (base + bonus) and 125K sign-on, and also it is non-negotiable. No idea what their numbers are like for experienced hires from competitors.

Honestly, that depends on the firm. There are same that do pay very well like this, eg: HRT, Jane Street (they are not an HFT though), Headlands, Radix. Some others like Jump, Optiver the pay varies depending on whether it's front office or back office.

Where I work at, the new grad offers are slightly better than FAANG, but the growth is very good based on performance, we also pay very well to people coming in from a competitor.

> sufficiently senior devs make 7 figures

I can confirm, known firm, manager role, offer was $1,3M full cash.

Could anyone (successfully) apply for such a position with no referral but CV only?

Not sure as the positions were not even public in my personal situation.

Yeah, pay at the big banks is shit really, especially when you consider the utter lack of work/life balance. I left in 2013 making 150k, which was supposed to be supplemented by a ~40% bonus for the level I was at, but each year was "well its been a tough year..." and after getting a token amount one year, and then zeroes the next 2, after working 50-60 hour weeks, I was like I am not only done with this place, but this industry, and left for a 50% pay raise, my TC is now 4x where it was in those days. A neighbor of mine is more or less sitting in my exact seat there, and is somewhere in the 200-250k range.

That said, I went back to finance to work at one of the premier hedge funds out there, and they actually lived up to their expectations in terms of comp, that place was more like a tech firm though than any other firm I have ever worked at aside for maybe Knight. 8% annual increases were normal there. You can look in my post history back to 2018 if you want the name, I recently left after 5 years there and just want to stay out of their crosshairs- they monitor social media aggressively and there is deferred comp at stake.

At big banks, there are really only a very small number of people who are in tech that are getting paid- you have to know which questions to ask- where is the bonus pool coming from- are you "in the business" or the tech pool, which is a second class of citizen. I would have to be in a pretty bad place to ever consider going back to a bank, it was borderline abusive... always dangling the prospect of that big check that would make it worth it

Most quantitative hedge funds and prop trading firms are now following a very tech like culture since they realize now that technology is just as important as the strategies. To get the best engineers, especially from FAANG, they need to have a similar culture otherwise they'll have a hard time getting new hires.

It depends what the strategy is.

Frankly, a lot of what people refer to as "prop trading firms" and "quant hedge funds" on here are just market-makers...they are taking very little to no risk, a lot are just riding the wave of ETF growth. Even the ones that are running alpha, I have heard of a few strategies, and they are largely what you would expect: low-edge, crowded trades (frankly, a lot of it is LTCM-style stuff).

That is why the business has become more tech-like, because actually taking risk is...quite risky. If you are going for alpha, you are trying to hire one guy out of thousands, that person knows what they are worth, etc. It is far easier to hire lots of devs on low wages and do the grunt work jobs that are less lucrative, but don't require being able to actually work out if someone is a decent earner.

Man Group has their own department at Oxford Uni, I think Winton hires people out of their department at Cambridge...Man Group's shaky record is legendary (they are doing everything now that blew them up in 2008), Winton was top tier...not anymore. AQR is another one, although a more traditionally finance quant approach...it all turned out to be pure beta. The hybrid approach of hiring devs or ML PHds to generate alpha has only ever worked accidently.

The firms that have been printing money from quant investing over the past five years have all been traditional hedge funds that incorporated quant methods into a fundamental process. And I expect that will continue because, bluntly, these firms know what the price of a security should be better than some busted factor model.

Interesting, would you add RenTech and TGS into the "traditional hedge funds that incorporated quant methods into a fundamental process" bucket?

No. Because that isn't what they do (I am talking about fundamental investors: Marshall Wace, Point72, etc.). I have no real idea what they do or do not do but they trade over shorter time frames where there is no change in fundamental information (i.e. they are trading based on liquidity signals which is, essentially, what market-making is now).

I spent a few years at an investment bank, not in the US, and the only people on serious money were some of the top managers. But my overall opinion of these people were that they did very little but the lower downs I met were some of the most talented people I ever worked with.

The top ones would spend all their time traveling the world to the offices and meeting with staff in each location and the sending emails to the rest of the department about what the staff in that location were working on. They would harvest ideas from the staff as they went and then present that as their own or approve projects that staff have suggested to them. I really didn't see how they were worth the $5M they were earning since they didn't come up with the ideas for what would be done and didn't do any real work.

Probably there were many ideas.

>> Then when we get to talking numbers they're like "300k a year".

> Yeah, pay at the big banks is shit really

I would say that a $300,000/year salary is really quite far from "shit". Many jobs are stressful, demanding or unrewarding but rarely do they provide such a salary, and quite often they offer hourly wages hovering around minimum wage with poor benefits. It's worth taking a step back once in a while to think about this, because it sounds like some people are a little out of touch with how the majority of the world lives

This is a welcome bit of perspective, so thank you for that.

I often find myself complaining about things at work like competing priorities or wearing too many hats and it helps me to ground myself by thinking about past jobs at minimum wage or even watching what people making less than a fifth of my salary have to put up with on a daily basis.

I believe that technology and other workers who are paid these salaries are actually just the only workers lucky enough to be getting a fraction of what is their fair share. You shouldn’t let a CEO walk with 300x their workers wage and say “well at least I’m not in poverty like those other workers”. You should almost always fight for more.

I would agree, but I really don't think many people are making that kind of money at IBs. Things may have changed, but as I mentioned, a neighbor who is pretty much in my seat- well technically one level up as a director (though title inflation has just made pretty much all titles below MD worthless), and was not pulling 300k. The new guys on the desk were not clearing 100k, and even "senior" guys like me were making a wage that just barely put us in the ownership class, and we were sacrificing a good chunk of our lives for it.

And again- these are the cream of the crop, this is the team most people wanted to work. I was one of the first to leave, but all those guys are now making multiples elsewhere that I am aware of. The point is that there are just better opportunities out there where you get to actually enjoy your life. I hear you on other jobs being stressful and demanding, but I have not really worked at a job where I was wearing that stress 24/7- well typically nothing traded on Saturday night, so nothing would ruin that, but outside of that, I could expect a call at literally any time of day- the markets were trading 23x6.

Sounds like I had a similar experience to you, except many years prior. I actually worked on a desk where there was a front page-headline scandal, partly caused by lack of investment in tech. I would never recommend anyone work in finance based on my experiences, but most people get very excited and assume it was fun and lucrative work. It was neither.

I heard, on multiple occasions from senior leadership, that industry practice was to categorize all dev work as IT so they could justify lower compensation. Pay was, just as you pointed out, not the main problem with this setup. There was no respect for other people and individuals were frequently berated or humiliated for no reason.

One interesting observation: despite the fact that I am very well qualified to work in the sector, it's very rare for fintech recruiters to reach out to me. They must know that I'm far out of their pay grade now and will never go back.

Why pay more than market rate of a replaceable ML person?

Is there any realistic path for a demonstrably smart and hardworking person into that 7+zeros club? Evidence suggests no: leetcode grinders and FAANGers are not in that club, and most of them will never even make it into the 6+zeros club. Net wealth -- sure, but not income.

It's all about making $$ for the firm. If the strategies developed are very profitable then 7-figures is definitely reachable for the researchers at a prop trading firm.

7 figs is 6 zeroes in his lingo, right? My man is looking for 8 figures.

Yes, exactly, and nomenclature borrowed from the parent. (This is another throwaway, and I am responding.)

> > "...all about making $$ for the firm. If the strategies developed..."

The topic here is on offers to new hires -- not value delivered after time and work invested. Offers at market rate -- even in the FAANG world -- are not at 7 figures (6 zeros), definitely not at 10x that, and certainly not before the new employee has done anything worthwhile for the firm. There are individuals who do get such offers, but they are rare, they have done remarkable things previously, and they have a reputation they can bring to any conversation about compensation. Their market is not representative.

I am sorry, I misread it as getting to those numbers as an employee at a trading firm rather than as an applicant. I agree with you. The very, very rare cases where this has happened is for people with exceptional records elsewhere, and many times they are joining new places as partners rather than a regular employee.

> I understand why they may have chosen to do so, at the Big Bank I was working at, getting a new fix tag to be passed through all the layers properly would involve at least two other teams and coordinating releases and probably several weeks worth of meetings. If you just reuse an old value you can avoid all that since everything is already set up.

There's a good meta-lesson here which is that smart people will do dumb things if you make the smart thing require too much red tape or process.

Processes can block stupidity, but they can also block intelligence if not well designed.

Weirdly a market maker near me paid less than competitive wages for a dev. I think they rest on laurels that people want to work in finance badly.

I remember the Knight Cap event, I was working on order routing at the time.

Things have changed a lot since 2012, and at the same time haven’t. Circuit breakers and position monitoring are no.1 in any sane market making firm. What happened then I can’t imagine happening now (accumulating a huge position for, what was it, 30 minutes? With nobody killing the algos within a couple of minutes?). On the other hand, the perfect world of “code hygiene” and 100% test coverage will never exist in this world, things will slip and they do frequently. What’s better, externally, is the availability of good tools for development and change reviews (bitbucket taking hold, for example), automated deployments, containers, testing frameworks and similar. This type of software, end to end, is incredibly complex and difficult to reason about when unexpected happens (there was a TTL misconfig for multicast and we never got such and such update? Well, no one thought of that!), esp these days with the influx of ML algos for price generation.

I worked in a lot of front office groups in investment banking. The short spell I did in HFT had great software development and DevOps practices.

This is all basic stuff I look to set up in every team, and it's crazy given how these firms work directly with tons of money that they don't have an even higher standard. Guess I wasn't wrong turning down these roles.

I used to work in HFT. I have seen highly variable practices in this case, including a "mini-knight" incident in the single-digit millions due to tech debt and poor test coverage. However, the most useful change that has resulted from the KCG debacle was adding several layers of kill switches, a dedicated ops team to watch trading and flip the kill switches, and embracing devops automation.

There is a much more serious focus on having a defense in depth, and making sure that problems like this are noticed before they become an issue. Rollbacks are no longer the first action when something goes wrong: the kill switch comes first.

Dead code, tech debt, repurposed flags, and spotty test coverage are everywhere still.

> poor test coverage

Yet you don't have to hang around here long to be told that "Unit Testing is Overrated": https://tyrrrz.me/blog/unit-testing-is-overrated

I’m curious about the “repurposed flags” part.

I wouldn’t think of flags as expensive / effortful to make more of, but clearly they must be if people are tempted to reuse them. Can you help me understand what is meant by a flag in this context, and why it would be repurposed?

Repurposing flags not always well-motivated, but one legitimate reason to do this is the memory (and particularly cache) footprint.

Often flags are local to a particular object. If there are lots of such objects, you want each to take as little space as possible. You should check out the contortions linux devs go through to make struct page small [0]. This is important, because there is one such struct per page of physical memory. The memory use is a near-constant percentage of your total memory, and you wouldn't want it to be any larger than necessary.

Even when there are not a lot of these objects, in low-latency software it's important to hit the cache. Your program should always just be as compact in memory as possible.

Semantically flags are booleans (is proposition P true of this object). They are stored compactly as bitsets, often implicitly, say:

    #define FLAG_1 0x01
    #define FLAG_2 0x02
    /* ... */
    #define FLAG_8 0x80

   struct order {
       u32 qty;
       u16 id;
       u8  type;
       u8  flags;
   };

This struct will fit into 8 bytes. This is great, as you probably won't waste space to alignment in many cases -- 8 is a good multiple. But if you wanted to add FLAG_9 here, your flags would become a u16, and your struct would, frustratingly, stop fitting into 8 bytes. To avoid this, one might repurpose flags.

Another example of this is intrustive flagging, using, for example, the high or low bits of a pointer aligned to 2^n bytes. If you run out of bits there, not much you can do.

[0] https://github.com/torvalds/linux/blob/master/include/linux/...

This is pretty much why flags get repurposed. It's also important to mention that things like JSON and protobufs are too expensive for HFT, so you are likely going to be sending structs over the wire. Repurposing flags lets you change a wire format with a lot less friction than adding a byte to a struct. Essentially, it lets you change the minor version number on a protocol and only recompile the endpoints without changing the major version number and recompiling everything.

Thank you both!

This is what I thought as well, and I'm in the field. It's legit to try to fit everything in cache.

However, you can still do something about making this safe. For instance the program could do some sort of version check on startup and panic if things weren't correct.

There's a bunch of stuff that needs to be done before the program reaches its low latency steady state, where speed doesn't matter. Might as well add checks there to make sure things are correct.

Currently work at an HFT firm. Most of the firms invest well into good DevOps, Trading Systems and SRE teams, to ensure that everything from installing a trading server at the colocation facility, to CI/CD and making changes to the systems configs, is done well. There are also guards in place to ensure that if the system seems to make trades that are way too odd then pull the plug and go down immediately.

Also, any code that does not need to be there, is promptly removed right away.

Where I work at, we have a few people from KCG i.e what was formed after Knight Capital merged with GETCO, after this incident. Sometimes this incident is bought up, although none of them I think ever worked for Knight Capital before this incident.

Repurposing feature flags is some kind of next dimension horror for me. We've got quite a few of these to deal with, and if someone started changing what they mean we'd be fucked super fast. Simply suggesting that we alter the meaning of an existing FF would result in the resignation of a non-zero number of project managers on my team.

Rolling back code is another thing I have no tolerance for anymore. The only option we entertain these days is a roll-forward. If your software takes so long to iterate/build that you need to go back to and old version in an emergency, you need to review your languages/tools/frameworks/processes. We maintain a contractual obligation to our customers for same-day code updates (in cases of production/regulatory emergencies) because we have enough confidence in our processes.

You’re likely using human-readable names for the flags and shipping a multi-MB payload of JS and JSON. An HFT firm is likely bit-packing flags so they can send an 8 byte payload rather than 10 and might be using an FPGA hanging right off the PHY to figure out “is this message even interesting to me?”

Your feature flag might be “SHOW_STRIKETHROUGH_PRICING_ON_CROSSSELL_OFFERS”; theirs is a bit mask macro to pick off the 5th bit from the 7th byte. (Why do they care? Because if they allow themselves to get fat and slow, a competitor will take the money.)

Roll-forward only, same day SLA is probably right for your business, but isn’t for a company that could have their systems dusting off $1M every handful of seconds that bad code is running.

Different business problems call for different technical approaches. You should no more adopt theirs than they should yours.

Rolling back isn't about avoiding rebuilds, it's about restoring to a known-good state. Making an emergency patch is typically far riskier than going back to last week's build. We always favor rollbacks, unless there were critical fixes in the current release we absolutely cannot afford to lose.

The only thing worse that comes to mind than repurposing flags is re-using UUIDs, which I have seen some production DB do (no, not mine, thank you very much!).

Reusing the RAM occupied by flags might be do-able in a clean way using guards like the structured enum in Rust, which permits unions (objects that occupy the same space) that always have the right type (i.e., the compiler has knowledge what is in there at each point in time). This mechanism could in theory be extended beyond type-safety to accommodate other contexts in systems programming use cases where memory usage is extremely important.

Rushing out a release without proper software engineering hygiene.

Sounds familiar. From everything listed above, it would sounds like this must have been yet another one of those "Just F-ing push the change the server now, dweeb, the traders are going crazy" environments. That is to say: the real problems were most likely cultural, and not about the sum of a certain set of bad practices.

Random, but I interviewed at Knight Capital for a software engineering position a few weeks before this all went down. I was in London, so the interview was done over the phone. Picture me in the evening, handwriting C to solve some problem (the fog of time too thick to remember what that problem was), then reading out what I'd written, semicolons and all, to the interviewer. Because of course there was no shared doc. I did very badly. But then, so did they.

Same! Although I interviewed in person in their NYC office. I was very junior at the time and the team I interviewed with was awesome. I (luckily) didn't get the job. I did a few more interviews and accepted an offer from another company where I met my wife!

Do you return the wife when you leave the job?

In all seriousness, you dodged a bullet. Do you remember what problem or set of problems they asked you to solve?

Also, what made the interviewing team awesome in your opinion?

The interview was incredibly basic compared to modern FANG style full day interviews. The coding portion was basically a fizzbuzz question. They had a one page printout of a Java method and I needed to give the output. I answered it correctly nearly immediately as it was simply testing pass by reference vs pass by value. The main interview was all verbal q&a core java questions.

> and accepted an offer from another company where I met my wife!

That really cheered me up this morning, I met my wife at work as well.

> handwriting C /…/ reading out what I'd written, semicolons and all, to the interviewer.

I had to re-read it to make sure you are not joking. The fact that you were not made me laugh harder.

I'm now just saying "retuuurn" in various exaggerated accents.

I'm picturing them arguing vigorously over what character the word "bracket" specifies in American versus British English.

Are brackets different in American English? I'd say the Queen should order a second burning down of Washington but, the way things are going, the locals will probably get to it soon.

parens: ()

brackets: []

braces:{}

Versus:

brackets: ()

square brackets: []

curly brackets: {}

Still drives me less mad than "inverted commas" though.

But don’t many (most?) US programmers call () parentheses? I assume non-programmers in the US call them brackets?

1.9M results: lisp "parentheses" -brackets

800k results: lisp "brackets" -parentheses

I admit result counts are more similar if lisp is replaced with “python” or “golang”, but square brackets come up more often for those languages.

Disclaimer: I am searching from google.co.nz, and I presume search result counts are not a great proxy measurement for actual usage!

No. Most US non-programmers call () parentheses, not brackets.

In my lifetime, “parentheses” has been the canonical way to refer to () in the US. “Brackets” is an umbrella term that can describe one or all of (), {}, and [].

If you’re in a context where only one of the bracket types would normally be used, you can just call them “brackets.” Since many people never really use [] and {}, “brackets” casually refers to ().

To disambiguate, you can call () “open brackets”, [] “square brackets”, and {} “curly brackets.”

And for the cherry on top, {} can be called “curly braces.” That’s my preferred term for them. But it’s awkward to call [] “square braces” and () “open braces” :)

I've always called the curly ones brackets and the square ones braces.

> Since many people never really use [] and {}, “brackets” casually refers to ().

I have never heard anyone call () "brackets". Is it more regional than just the US maybe?

Same. I've seen {} and [] both called braces and brackets, though usually in that respective order. The ambiguity is precisely why they are also called curly braces and square brackets. The double naming is more concrete.

I've never seen or heard any programmer call () brackets or braces. It's always parentheses. I may have heard a math or science teacher call them all brackets or braces, however, since they are synonymous there outside of odd exceptions like interval notation.

Fascinating! While I’ve definitely heard “brackets” used in the midwest occasionally, “parenthesis” is the standard term by far.

British English here: "(" is "open bracket" and ")" is "close bracket".

Together they are just brackets, not open brackets.

As someone who has lived on the West Coast of the US for decades, I have never heard any American call (), brackets.

Chiming in as someone who has lived decades in both South East US and North East US. I've also never heard () called "brackets" only "parentheses". This includes in and out of programming circles.

I assumed the comment saying "parens: ()" was talking about US, but the replies seem to be interpreting the opposite.

Never? You guys don't use the BEDMAS acronym in grade school for teaching order of operations?

I've only ever called them brackets.

> BEDMAS

Was PEMDAS for us.

> You guys don't use the BEDMAS acronym in grade school for teaching order of operations?

No, didn't use any acronym for that in grade school, and when I did encounter one used in grade schools, it was PEMDAS, not BEDMAS.

BODMAS for us...

BODMAS for me too, although the "O" always confused me, the "E" standing for "exponent" always made more sense to me than the O" standing for "[to the power] of"

Yeah, never. PEMDAS for sure. What part of the country were you in where you got BEDMAS?

FWIW, I did my schooling on the West coast of British Columbia in the early 90's and it was BEDMAS.

BEDMAS in Canada (Southern Ontario) circa 2000.

Here is what Clang uses: https://github.com/llvm/llvm-project/blob/589b9df4e15131348b...

parens: ()

squares: []

braces: {}

basically a mixed of British and American English

There seems to be some confusion. I am saying the top is American - at least as I was taught it in school. The bottom is the more typically British variant. But I think the reality is the both America and Britain have variations. To an American ear, bracket to mean parenthesis is not just odd, but wrong. But someone already said they consider bracket to be a catchall term, so obviously this is not universal.

Edit: okay, apparently there is just complete bracket anarchy across America and words have no fixed meaning.

One difference is that we don’t have “round brackets” over here, only parentheses.

(We do have “square brackets” however.)

I've heard "lumala" or so but most people just say "paren."

People who say paren are monsters.

brackets are for squares.

Peace, man!

squares are for brackets.

I believed it. Amazon tried to make me do the same thing. I stopped the interview.

this is hilarious, apparently someone on their end thought it was an acceptable interview format, thanks for sharing.

If Yegge's blog is to be believed this is exactly how they used to do it at Amazon too, many moons ago. I think this just used to be common practice. Before my time though.

I would have started them off with a couple of preprocessor macros.

I may well be dense, because I am just not seeing why email would not have been a viable option for sharing code..

Before all these code sharing sites were popular, that's how all the phone screens were done. I had to write my code on paper and dictate on phone. this was before 2011-2012

Did they not have email in 2011? Seems really difficult to dictate code

I also interviewed at knight a few weeks before everything blew up, didn’t go well, interviewer was one of those guys who finally got to hire someone and figured now it’s his turn to be a jerk. Really smug and arrogant. When I heard how badly things had gone it did bring a smile to my face knowing somewhere that guy was having a Sunday dinner sized serving of humble pie.

> I did very badly. But then, so did they.

Thank you for sharing, this started of my day very well, had a hearty chuckle.

The incident happened after a technician forgot to copy the new Retail Liquidity Program (RLP) code to one of the eight SMARS computer servers, which was Knight's automated routing system for equity orders. RLP code repurposed a flag that was formerly used to activate an old function known as 'Power Peg'. Power Peg was designed to move stock prices higher and lower in order to verify the behavior of trading algorithms in a controlled environment. Therefore, orders sent with the repurposed flag to the eighth server triggered the defective Power Peg code still present on that server [1]

> Power Peg was designed to move stock prices higher and lower in order to verify the behavior of trading algorithms in a controlled environment.

This is insane. Make one wonder, what is or isn't actually being deployed in prod in 2022.

[1] https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stoc...

coder running down corridor to the trading room, bumping past people and sending sheaves of papers flying

"Power Peg has triggered! Tell them Power Peg has triggered!"

So... you're saying the company activated a tool called "Power Peg"...

and.... they got f***ed?

Well, what did they expect?

Yep. They got... power pegged.

Back in the day $440M loss due to coding error was a landmark warning case. How could this happen??

In 2021 alone something like $10B was lost due to bugs in defi land.

Something about the worst possible thing could happen tends to happen eventually and it gets worse every passing year.

Those $440M were lost by rich people who had invested in a hedge fund, not poor people who bought crypto lottery tickets in the hopes of getting rich quick

Also a fuck load of money had been printed in the mean time. https://fred.stlouisfed.org/series/M2SL

Was just about to comment along these lines. If I read about this a few years ago I would be shocked. Now after seeing so many flubs in the crypto space, my reaction is just 'meh'

I actually always think of the Knight case and similar ones when people see a DeFi organization have an issue and extrapolate that to an issue with the entire DeFi concept.

Its so obvious that those people have no clue whats going on in the markets they respect. Truth be told, many of them dont like markets at all. So its just a lack of exposure and compounded ignorance.

Many traditional finance issues are fixable though, there are many more errors which don’t become big stories because they are reasonably reversed as only minor inconveniences.

like how Credit Suisse is going to reverse their Bill Hwang losses? I guess in this conversation we can't distinguish from irreversible asset value and liquidity issue to misdirected transactions that inherit partial reversibility.

similarly, maybe you/they just don't see the headlines of thwarted attacks in DeFi that work specifically due to design considerations.

I'll take the permission to fail. The rapid iteration creates some really fascinating systems in very short time periods, for me. One project implodes, 100 (or 1000) more harden, bigger money comes in creating more assurances for users like easier recovery and compensation paths, all while continuing to rapidly iterate.

Where is this innovation happening? We’ve had crypto for 13 years now, and just yesterday we had people flat out losing 10k into the ether because their transaction “ran out of gas” without completing. Isn’t that plenty of time to develop systems that avoid basic issues like suddenly burning all your money?

It doesn’t really seem to be innovation, just rediscovering 100 year old scams and failures that have to be readdressed.

So, what discussion are you willing to have?

For your strawman argument, all of their client side wallets told them “its not worth it bro”, the people did it anyway. Thats a consumer issue not a technology issue.

I'm generally skeptical of ethereum but i don't think this claim is quite right. First, they didn't lose 10k "into the ether" it just went to miners as a fee for processing their transaction. Additionally, the fee was paid for priority. There is a limited amount of block space so people can pay extra to make sure they're included. I know it's cliche but this is literally a feature not a bug

Fees for what, when your transaction did not go through and you did not get the NFT you were paying for?

Knight Capital Says Trading Glitch Cost It $440 Million - https://news.ycombinator.com/item?id=4329101 - Aug 2012 (90 comments)

Original Post-Mortem: https://www.sec.gov/litigation/admin/2013/34-70694.pdf

My small footnote to this story: we had the look on the "blind bid" to purchase the portfolio of erroneous trades on the program trading desk I worked on. Ultimately we didn't bid (thank you risk department) and it traded away to Goldman as the article correctly reports. Would've been a great trade though. I estimated it netted Goldman $2m+

Important to add that this occurred in August 2012 and not in 2019 as the title implies.

I had interviewed there in my garden year (2010-2011) and was ultimately not considered for a role as a high-frequency quant.

What is a garden year?

It's paid time off from a financial entity (hedge fund, prop shop, etc.) where you are paid not to compete in their space.

In my case, I had a year to sit-out as a HFT quant from 2010-2011.

Note that it sounds great - you're getting paid not to work - but nobody likes to make you an offer, as it's non-binding and your perceived/real value is declining as the time goes by.

Hey on that topic of a non compete, I've recently been offered a different internal role that would force me to sign a non compete. What was your experience like interviewing/searching for new roles? I would also like to hear what your general opinion on what it make it worth it to sign a non compete, although it sounds like you a generally negative view of them from this comment.

Peanuts, just a regular day in DeFi.

https://rekt.news

What's amazing to me is that there are many software engineers here who can recognize how errors like this can so easily creep into software (some would say they're inevitable in any sufficiently complex software) but they still somehow think immutable smart contracts on blockchains are somehow still the future.

Crazy.

They know but they don’t care. Crypto is get rich quick

Hindsight is a helluva drug. The SEC report cannot be viewed as a “postmortem.”

https://www.kitchensoap.com/2013/10/29/counterfactuals-knigh...

Interesting. Five years prior, this story was posted on this blog: https://dougseven.com/2014/04/17/knightmare-a-devops-caution...

Really good write-up. Perhaps there's a dawning realization that the model of 'move fast and break things' is fatally flawed?

> "Knight’s IT project managers and CIO should have pushed back on the hyper-aggressive delivery schedule... Thirty days to implement, test, and deploy major changes to an algorithmic trading system that is used to make markets daily worth billions of dollars is impulsive, naive, and reckless."

The fact that since 2008, the portion of all stock trades in the U.S. taking place away from public markets has risen from 15 percent to more than 40 percent is also kind of astonishing. It's long past time to re-erect the walls between commercial and investment banking.

Here’s a 225 million dollar oopsie from 2005 https://www.foxnews.com/story/typing-error-causes-225m-loss-...

Today there was a 300B oopsie in Europe caused by a Citibank "glitch" bloomberg.com/news/articles/2022-05-02/citi-s-london-trading-desk-behind-rare-european-flash-crash

It only was a sudden drop in share prices, which quickly rebounded. The amount of money that actually changed hands due to that mistake will be tiny in comparison.

Here’s the same oops in 2001

https://www.wsj.com/articles/SB1007117680496415760

> However, a dark cloud remains: market data suggests that 70 percent of U.S. equity trading is now executed by high-frequency trading firms

Is it just me, or was this one of the scarier statements in the article? High frequency trading does not seem to add anything, it's not a market instrument to balance anything. It's more like gambling with the odds slightly in your favor.

Oh course, I'm a laymen when it comes to high frequency trading and might totally wrong.

Market makers provide cheap liquidity. Or in other words, compare the total transaction costs (bid-ask spread + commission) you would to pay today to trade 100 shares of stock today vs 30 years ago. The difference is staggering.

My experience in quant finance is that strong engineering teams are viewed as cost centers and are not well compensated the way they are at “real” tech companies, especially when the firm isn’t doing well. Management foolishly tends to only value seeking alpha, rather than reliability of existing alpha.

Unsurprisingly poor management tends not to realize when their software is built on a house of cards.

Definitely saw bugs cause large losses (8 digit numbers).

Feel like this way more of the case in the past than it is now. Engineers at top prop shops today are compensated more much generously than at all but the most senior engineers at a FAANG. For example, word is that new grad hires at Jane Street are getting first year guarantees at around the 450k range, which is much higher than an entry level engineer at a FAANG would make.

Culturally, there's some truth in what you say though I think: alpha generators are probably more respected and are pulling more TC in good years.

My brother worked in a medium-sized fintech not long ago. This example is shown in their onboarding process. I guess the same goes for other companies out there.

Wow, I just started a new position at a tech company in the NYC financial sector, and this was part of today’s training!

Fascinating read. I love code horror stories such as this.

There are several more of these of approximately the same order of magnitude but those firms are able to keep it successfully out of the media.

Who gets to deposit the bribes?

Excellent report of a fault case study.

Yes, check lists help, not just in medicine, but also in aviation. But automating things is of course the best option.

(2019)

Updated the title

at the end .. its just money going from one account to another right? Its not like some physical thing that has perished and cant be brought back. Why is it difficult to reverse the transactions?

Because those transactions cause other transactions, which cause others, and so on and so forth. You'd have to reset the market for the day.

Imagine how pissed you'd be if you made money off Knight's mistake and it all just disappeared the next day.

Except, well, cancelling transactions obviously does happen, sometimes: https://www.reuters.com/business/lme-suspends-nickel-trading...

Knight was probably too messy to rollback cleanly, but that just means it's a matter of cost/complexity/politics... if you're a big enough player, then the exchange will do you favors, like in the LME case.

Free markets, lol

This is much easier to do in a centralized futures market. I can't imagine a rollback in stocks being easy or possible.

From the OP

  Rules were established after the “flash crash” of May 2010 to govern when trades should be canceled. Knight’s buying binge did not drive up the price of the purchased stocks by more than 30 percent, the cancellation threshold, except for six stocks. Those transactions were reversed. In the other cases, the trades stood.

> The LME announced that all trades will be voided from midnight until 8:15 a.m. on Tuesday when trading stopped and added that it was considering a closure of several days.

> "People will be asking if this really a functioning market... This is meant to be a market of last resort and people can't get inventories to deliver against positions," said Colin Hamilton, managing director of commodities research at BMO Capital Markets.

There’s gonna be a pretty high threshold for this sort of thing. Higher than “one company fucked up and wants a do-over”.

Exactly this. If you're a market maker, likely your trades impact your own trades too. As you accumulate a position, your average price is going up with it. Trades should not just roll back because one large hedge fund screwed up. Imagine being a retail trader with that expectation. Would be nice!

> Why is it difficult to reverse the transactions?

Why should the transactions be reversed?

If things had gone according to plan, Knight would have made several million dollars that day, some likely because of a mistake by someone else or an unavoidable circumstance, just like it did on other days.

Those other people weren't made whole, so why should Knight be any different?

Because the people those transactions were with didn’t want them reversed, and there was no rule or law that required it.

Didnt the exchanges cancel the trades during the flash crash in 2010. I am sure the buyers of those transactions which were able to buy shares of good companies for pennies would not want to reverse those transactions.

Knight was not a big player so the exchanges had no reason to keep them happy.

sorta like a precursor to a smart contract hack

I read this

>Under stock exchange rules, Knight would have been required to pay for those shares three days later. However, there was no way it could pay, since the trades were unintentional and had no source of funds behind them. The only alternatives were to try to have the trades canceled, or to sell the newly acquired shares the same day.

And then I understand why /r/WallStreeBets and /r/Antiwork is gaining traction.

All it takes is a bit of organisation and the adoption of Govt tactics and practices which is ultimately violence and then just maybe you might see a Govt that works for the people and not the criminals, but I cant picture Bernie Sanders wielding a pitchfork!

Still I see Musk was market making with his tweet. I dont think you can be any more blatant! LOL https://twitter.com/elonmusk/status/1520650036865949696?cxt=...

No lessons have been learned. There is no catharsis. This blog post has meant nothing.

Reply on news.ycombinator.com