Hacker News Reader

I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.

As somebody who worked with them as a client, I can confirm this. There is currently a spec-level bug with their core Pillar engine and it was essentially bounced between several different teams and ultimately ignored as nobody's problem.

Unlike all of the "modern tech company" problems which are never ignored and only solved when someone's problem goes viral on social media.

They're a big company, some groups are better than others, some customers get more attention than others.

So basically like any other medium to large company? This doesn't sound unique in the slightest.

I would think that the company being a securities exchange would factor into the analysis. Don't you?

How does them being a securities exchange in any way affect the analysis of their software engineering practices? They're not some special snowflake, they can suffer the same software engineering and business process issues as other companies.

>> They're not some special snowflake

But they are. The consequence of a one-day or one-hour shutdown on their system is exponentially worse than most any other. I would expect them to have more rigorous systems, including more rigorous attention to development. Comparing the NYSE to any other business is like calling Fort Knox just like any other bank vault.

I disagreed with you until: "...like calling Fort Knox just like any other bank vault."

Interesting point that teeters on false equivalence. I think AWS or Azure might make for a better analogy. Your point identifies the inherent risk of actually operating a platform business. A bank vault is (mostly) synonymous with Cloud, in this context. If a vault is robbed or a cloud goes offline, losses extend beyond the business which inherently compounds the severity of downtime.

Linear loss vs. parabolic loss.

But if a cloud goes offline, there is damage to the economy linear to the length and breadth of the outage. Sure, there are losses to businesses serviced by the cloud's users, but they'll bounce back, even if a day-long outage was so severe as to temporarily ground flights and halt supply chains.

If a stock exchange executes trades at incorrect prices, even for a short amount of time, all of a sudden you're in a kind of non-linear sigmoid regime, where investor confidence can suddenly tip into panic selling and recessions can be triggered. Thankfully, that didn't happen here, but it could have. If you're going to give a company that power, you should better hope that they're held to higher standards than most dysfunctional tech organizations!

"If a stock exchange executes trades at incorrect prices, even for a short amount of time, all of a sudden you're in a kind of non-linear sigmoid regime, where investor confidence can suddenly tip into panic selling and recessions can be triggered."

This is false equivalence and slippery slope.

No they aren't.

There's far more critical snowflakes out there... FAA Airspace management, a medical radiation device, avionics in an aircraft, and facebook.

No company or organization is immune to bad business practices.

Them being a securities exchange does not somehow provide immunity from developing rigorous systems which have oversights, or make bureaucracy magically go away.

Likewise, the impact of an outage being more extreme does not mean the people there are infallible. Things slip through. Especially random customer requests being bounced around from team to team, the thing in question.

Then somehow we are in agreement. It is my impression that you were saying precisely the opposite, that why should we focus on NYSE's bad practices over others'? Well the answer to that question is, this is a thread about NYSE under a post about NYSE. And besides, we should be, if anything, more scrutinizing of their practices because of the outsize effect that disruptions in their services would have on the global economy.

> like calling Fort Knox just like any other bank vault.

Main difference being that most bank vaults aren't actually empty. ;)

Having been in the industry for a couple decades, and having worked at both, they're not all that different. Some groups are going to be better than others in the same company. Some companies are floating on venture money today, and might disappear tomorrow. Most technologies constantly cycle. Our experiences working at the same company were different.

Blame cultures and process cultures are both problems in different ways. Blame cultures don't care about individual accountability, only that someone suffers. Process cultures only care that no one suffers, not that individuals are accountable. Both have some misguided notion that something other than personal accountability can lead to good results. Misattributed blame and suffering does not deter poor performance or mistakes. Not even correctly aimed punishments are very good at that. Accountability isn't about punishment, it is about limiting power to the level of responsibility demonstrated. Rules and procedures don't prevent poor performance, they can in fact entrench and guard it, and they only mildly impact mistakes. Best practice can mitigate mistakes to the same extent or better (due to easier adaptability), but people keep trying to turn them into rules, and that has to be fought. If you followed all the rules but didn't get the job done, you still shouldn't be handed the same task again, but not out of blame.

I used to work for a small startup, and postmortems were truly no blame - engineers would talk about exactly what happened and wouldn't hesitate to put the blame on their mistakes.

But as the company grew, the postmortems became more about blame since now you're not blaming an engineer, but an entire team so singling them out isn't personal. The postmortems were no longer a single engineer describing what happened in his code, but were team leads talking on behalf of teams. They were all about shifting blame from your own team and talking about why a service from another team led to the problem, even if your team could have (and should have) been able to work around it without melting down.

I'm no longer at the company, but Postmortems are much more useful when they really are no-blame because you can get to the real root of the problem, but I don't know if that's possible in a large company.

As organizations become larger they become more political. It's unavoidable.

This happens within big organizations that are large enough where they start having that internal small company feel within units. I would say a good program, which could be small or a chunk in a massive org, does a blameless post mortem.

A few years back, a task to modify an index was given to a scrum team. The lead was away and the senior people could not be bothered. The junior developer stack overflowed an answer, asked for review, tested the script and let it rip. She missed that the change deleted everything if you noticed. Every environment, every data center wiped out. 10B records in each prod instance. Lessons were learned and processes fixed. She was not fired, but rather became one of the people safeguarding the keys to our prod kingdom as we fixed out broken process. I stole her away as my first report when I switched groups.

Suddenly I don't feel so bad about deleting an entire PVCS repository (happily answering 'yes' to all the 'are you sure?' questions) at 4:30PM on a Friday.

I worked for NYSE’s parent, ICE, and I have to agree with this. While there were many things I didn’t like about working there, the tech and the management weren’t involved in those things. A similar problem to this happened while I was working there, but it was on Endex and not NYSE. Management spoke with the responsible party, but he wasn’t fired and no punitive actions were taken against him. The blame game also wasn’t played. The team just decided to provide more eyes on the process, change the interface of the tools a bit, and move on. The company itself did face hefty fines for the screw up tho. Ultimately, the issues at ICE/NYSE are due to a highly bureaucratic structure and to onerous regulations forcing parts of that structure to exist. Given those two problems, I think ICE. does extremely well.

I've thought about getting into this... The stuff they work on is so incredible to me.

Here's a quote from their Pillar product page:

> Up to a 95% Reduction in Latency: The roundtrip latency on NYSE Pillar order entry sessions via Pillar matching engines has been reduced from ~592μs to ~32μs for FIX and from ~96μs to ~26μs for Binary, getting client orders into the market much faster. With a 92% improvement in the 99th percentile latency results, clients can also have more confidence in improved performance consistency regardless of market conditions.

Reading stuff like this makes my current work feel stupid by comparison.

> Reading stuff like this makes my current work feel stupid by comparison.

It makes our economic system seem stupid. Jesus, we're not calculating astrophysics or quantum mechanics. A made-up system should not require or depend upon this kind of speed or precision. Maybe we should chill.

Reminds me of those pro StarCraft players who keep unnecessarily clicking the mouse to keep their APM (actions per minute) stat high.

Absolutely agree. but "liquid markets are important" or something. rolls eyes

This is just another step in the endless journey of widening the gap between your average Joe and someone with access to high level financial services.

This is what enables products like robinhood to exist - platforms explicitly targeting the average joe investor.

wonder how they measure this and is this smart engineering from the exchange or just new fast network gear.

Just an educated guess (I'm in the same industry, have worked on some networking related stuff) But I think it is probably mostly network hardware and architecture. You can only improve so much from the code, the networking is where all the latency comes from.

Just naming a 'staffer' though seems to already be a way to apportion blame to a segment of the employees, insulating management from what was done. Named or not doesn't really matter, clearly blame is being assigned.

I think those are Bloomberg's words, or their paraphrasing of the grapevine. The high level people that I knew there weren't petty, and everyone was of the opinion that it didn't matter who clicked the button: we were all in the same boat.

Yea, it sounds like an issue with process and automation. It shouldn't have been possible for the staffer to make a mistake that would cause this.

Precisely. It is never just one error. At a minimum two and if you really stare at this sort of thing long enough it isn't rare at all to discover a whole chain of them. The only difference with all the times that it went right is that this time everything was aligned 'just so'.

Curious why the link to ICE.COM?

ICE owns NYSE and several other exchanges. ICE stands for Intercontinental Exchange. ICE is the IT administrator for these exchanges. I helped sell some router management software to them a while ago. ICE is fairly new, NYSE used to be independent. That changed sometime after 2008.

do they still do UDP packet loss replay over email?

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

You're not making it easy to get any answers, if you cut out 2 of the main reasons for people in general to change jobs.

I consider these things table stakes when choosing a job.

I'm not recruiting for them, just sharing my experience. I included their careers link for people who might be interested because I know they're always looking for good engineers.

There’s also geography. You go far enough East and you have mostly public sector or defense jobs. A little smattering of insurance data processing. And then fintech.

illinois.edu has one of the top rated CS programs, but once you graduate there are not a lot of options but to move to one of the coasts, or move back/to Chicago and try your luck there. Second City has a good deal of fintech.

Wouldn't working with systems that keep the largest stock exchange for the largest economy in the world running where a simple mistake can cause "mayhem" when the market opens be considered more "tangible" than working in AI or on self-driving cars? It just doesn't have as much street cred as working on those particular projects in the tech community.

Not necessarily. If you're the type that's into finance, then sure, that might get you out of bed in the morning. I'm not into finance and kind stand the culture that surround finance. Yes, it's big and touches every single one of us, but doesn't mean I want to embrace it and go to work in it every day.

If I can take that same skill set and apply it to something with a much better culture surrounding it that affects people in a positive way, then I would definitely choose that over finance any day of the week and twice on Sunday.

At the end of the day, if the NYSE did not exist, the world would continue to turn. It's just not that big of a deal to a heck of a lot of people.

>if the NYSE did not exist, the world would continue to turn

This is startlingly ignorant of the complex machine that is the modern economic system. If something like the NYSE was to shut down today it would be pandemonium.

There is a difference between 'I don't understand how something works' and 'I don't understand how something works, so it is worthless'. The former is healthy and the first step to understanding, the latter is ignorant, and the first step to getting more ignorant.

Relax, Ayn Rand cum True Believer complex.

The current state of business within the current iteration of how people interact with one another isn’t some necessity.

Yes, the world may fall apart for a relatively brief moment in the grand scheme of things — but then life will go on.

The first step to understanding this is to drop the superiority complex.

Very little is actually needed to keep the world turnin.

Exactly. Our current implementations of resource rationing isn't some fundamental of reality, or even needed by human society just 1-2 centuries ago.

I actually think both you and the guy you are arguing with are half correct.

The real answer here, in my opinion, is that yes there would be pandemonium, and then yes, the world would go on without it, but then something else just like it will pop up. And that is because a liquid market for financial assets (whether that is securities, options on securities, futures, etc) will always be a massive benefit to the ability of businesses to conduct business, and the ability of individuals to preserve and increase wealth.

I was remarking on the key word "tangible", not trying to express an opinion one way or the other on financial institutions. Accidentally forgetting to do something and ending up in the news because you caused havoc when the markets opened the next morning is more "tangible" (able to touch things directly) than working on AI or self-driving cars, at least currently. Certainly working in either of those fields might provide more benefits down the line.

All of my “culture” experience working in finance were uniformly better than pure tech.

The movie portrayals don’t match my experiences at all and I saw a lot more bad behavior in the tech companies I worked for.

Heck I saw more people working for the intellectual challenge of it in trading than I did in SV style tech firms where money drove nearly every decision.

It’s really hard for me to buy that SV style tech companies are a better place to work when for the last 2 decades the business models that have been front and center are panopticon style tracking to sell ads and legal arbitrage.

Oh, don't get me wrong. I pretty much abhor SV/VC culture too. It's why I don't have one inkling of a notion to work on either coast for the "big" corps.

It's not an either or, I can hate both ;-) I'm a big boy and get to make up my own mind on the matter.

Will the world stop turning if people stopped working on self driving cars or AI?

i'm guessing you're trying to make a point here, but care to elaborate on what it is? i think you well know the answer to the question

>If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars

Maybe they aren't as smart as they think they are? Or they find that there are interesting problems to solve in fintech? Problems they can tackle and see resolved in a realistic time frame vs 'tangible' (?) self-driving cars or chat bots.

I know AI encompasses a far larger range of things but right now, what problems is it solving? Artists, writers, and others can do that work. What do self-driving cars resolve beyond continuing the dominance of car culture in a world that could have better public transit and safer infrastructure?

You answered your own question. The only motivation is greed.

Taking a well paying job is greedy? What planet are you living on?

Serious Answer :

There are problems in Fintech that are absolutely worth solving for altruistic reasons. One that I think is very important and might even need to incorporate AI is this :

Larger financial institutions have access WAY more and WAY higher quality data surrounding stocks and options. For example, publicly available SEC filings contain extremely useful information about companies. Professional traders have access to services which provide this data accurately in programatic form (like an API). Us normal people have only the SEC filings themselves, which are enormous documents. It would be impossible to read them fast enough to ever catch up on all of them in the last say year. There are free APIs, but they are absolute dogshit and provide incomplete and inaccurate information.

If someone could democratize this and provide this info for free or cheap to the public, it would be an enormous benefit to the general public.