NYSE/NYSE MKT has temporarily suspended trading in all symbols | Hacker News Reader

NYSE/NYSE MKT has temporarily suspended trading in all symbols

nyse.com

326 pointsmmastrac136 comments

This isn't much of an issue for traders That's a nice benefit of having 40+ trading venues in the US. They had problems at the open as well with connectivity on one of their gateways.

Issues like this crop up all the time and most of the time they are resolved before the open. The good news is that they aren't reporting any lost trades or trade busts yet so this isn't as bad as the BATS open at BATS:)

Having said that, they've announced that they will cancel all open orders, that is a huge deal, I can't remember the last time they did that.

To put that into perspective, cancelling all open orders would be the Silicon Valley equivalent of Ebay loosing all bids on their current auctions.

NYSE has always been considered second rate in their IT compared to the NASDAQ and BATS and this won't do much for their reputation.

EDIT meant BATS not FB, thanks!

One other point to keep in mind when throwing around hacking conspiracies. The exchanges aren't running on public networks. You can't DDOS them or hack directly into the matching engines. Though I'm sure you can break in via some other NYSE owned network and make your way to the matching engine somehow.

To hook into the exchange you either go through a blessed intermediary like GS or you plug directly in via colocation. You just can't keep pinging the NYSE on port 80 to bring it down.

right. one trading head just noted that it's a "non-issue." the timing is interesting, though. it's midday so there's lighter volume. big problem is going to be with index calculation. other exchanges are functioning just fine and other venues are seeing plenty of action and stability.

Nothing to see here, move along.

Are there serious trading shops that don't route orders dynamically to multiple venues?

Yes in the sense that about 90% of hedge funds just use a SOR(Smart Order Router) from their executing broker. So from the funds perspective they are just sending an order to GS that should execute on the "US Market" and they expect GS to send it to the appropriate venue.

Most funds just trade through an intermediary like GS and all intermediaries have SOR's that route to each venue in the US as per RegNMS

The remaining 10% are your market makers and HFT's and for them this is a bit of a headache but it happens and they are well equipped to deal with this. No fund that I know of is a market maker for just the NYSE, but if there is then I guess they just go home early?

Yes, some strategies are highly exchange-specific.

I'm really happy to see that they're canceling all open orders. It's much better than letting them stick around and then execute with unexpected behavior. It looks like NYSE learned from what happened related the FB opening (just as you alluded to), and what happened with Knight around the same time. Playing it safe can be a smart move, even in the stock market.

My immediate reaction is that their system shouldn't require canceling all open orders. But I admit that I don't know much about matching engines.

If they roll out a new version of the module that creates orders, I would think it makes sense to include the version of this order module in the order data structure. This way they can delete only the bad orders, created by the faulty module version, instead of all open orders. But it's also possible that I'm way in over my head here :).

Actually, the FB IPO open fiasco was on Nasdaq not BATS. Nasdaq, has had way more serious issues in the last 2 or 3 years that I can recall compared to NYSE.

[EDIT] The BATS issue you may have been referring to is BATS IPO when they tried to list on their own exchange. The technology failed miserably that day, so much so that they decided to pull the IPO altogether. One of the biggest fails I can think of.

Yep, I was wrong, you are right! I meant BATS IPO on the BATS.

RE hacking, this is pretty much a case of https://xkcd.com/932/.

Great info, thanks!

(Very minor, but "ping" usually implies ICMP Echo Requests, and ICMP doesn't have ports)

UDP and TCP include echo ports (port 7 just like ICMP) and many tools that can run in non-privileged mode actually use the UDP echo instead of ICMP.

It's very common to "ping" a service by sending a valid request and seeing if you get a valid, expected, non-error response.

A tweet from last night: "Wonder if tomorrow is going to be bad for Wall Street.... we can only hope."

(source: https://twitter.com/YourAnonNews/status/618626955433349120)

Good call on their part. Coincidence?

That tweet is probably about the market situation in China.

I get the impression they'd heard on the wire that something might go down.

Given the blame coming out of China, it wouldn't surprise me if they tried to crash the market on opening with a flurry of sells. Looks like NYSE got a whiff of the pending trades and the NSA stepped in. All trades dropped. Crisis averted. For now...

It should be noted that this concerns only the regular NYSE and NYSE MKT (formerly known as AMEX). Their ECN NYSE Arca is working properly. So, this is really not that big of a deal for the market.

This doesn't mean all trading is suspended, it just means you can't trade specifically on NYSE or NYSE MKT (formerly Amex). You can still trade NYSE listed stocks at all the other exchanges (BATS, NASDAQ, NYSE Arca, etc). Obviously a black eye for NYSE but it's not as big a deal as the media are making out.

CNBC and Bloomberg have been repeatedly saying this

all open orders cancelled.

It's still pretty bad. Imagine being the IT dept at the NYSE right now. The trading is likely being rerouted to other exchanges. The Wall Street Journal website is also down.

Pretty bad timing, markets are already rattled because of China:

http://www.bloomberg.com/news/articles/2015-07-08/u-s-index-...

Regardless of the technical cause have a feeling this will make things worse and people more nervous.

The technical glitch is probably related to China news somehow, but it is rather meaningless for trading in general. NYSE has pretty antiquated technology compared to other exchanges.

> The technical glitch is probably related to China news somehow

If that were true, you'd expect to see volume spikes on multiple exchanges, which so far I haven't seen any evidence of.

Maybe their network administrator is stuck on the ground in a UAL plane.

Well, at least he/she can read the Wall Street Jo...oops.

UAL went down. WSJ down. NYSE halted. coincident?

edit: based on everything that i've heard/read, the incidents are most likely not related. sorry for asking the question.

Does this have anything to do with the Chinese stock market rout and the rumors circulating that it's synthesized to a certain degree?

interesting question and i'm not familiar with the rumors. is it that the shanghai index crash is synthesized? not being completely fluent in the financial sector, how would this be achieved?

Financial markets can be manipulated provided you have the right keys. At this point in time, this is just a wild conspiracy theory and it's explained as retaliation for all the cyber attacks allegedly carried out by the Chinese govt or people affiliated with it.

There's plenty of reason to believe China was in a bubble long before these last couple of weeks, and if you don't like somebody and want to do them the maximum damage, you want to help inflate the bubble, not pop it. (Well, at least, if you have a rational view of how markets work.)

Or retaliation for 2007-08 financial crisis. If I remember correctly, some Chinese quant gave the disastrous formula that caused it.

I doubt it is synthesized, they managed to make 52% of all listed stocks enter automatic freezing at the same time...

Most likely, yes.

The Journal's issue appears to be only with their front page. The rest of the site is working correctly.

mad rush to publish crushed the front page? wsj.com/asia 504s as well.

The TV is stating that US government sources do not see a connection.

Edit: Now the Secretary of Homeland Security is saying that there is no evidence of United or the NYSE being due to malice.

Edit (about an hour later): Now the White House Press Secretary is reiterating the point about no indication of malicious actors.

ok. i'm not near a tv so i can't tell. thanks for the info.

Now 2 hours later Tom Farley, President of the NYSE, is stating that it is an internal technical problem.

As the saying goes, never attribute to malice what can be explained by incompetence.

In the case of United Airlines, be prepared to adjust your threshold of expected incompetence dramatically upward; since their merger with Continental multiple years ago, system-wide outages have been an every-couple-of-months thing.

HN was down earlier today too, or at least timing out on my end. never saw that before.

that happens to me quite a bit. nginx gateway errors are pretty common on HN in my experience

Cloudflare Chicago?

That would be deadpan sarcasm there, without the <sarcasm>tags</sarcasm>

Well maybe sarcasm is not _exactly_ the right sentiment but you get my drift.

Ha, you have been lucky then.

Maybe there's a stock market hacker flying around on planes...kinda like in the movie Hackers. Of course, in that movie they had no trouble cuffing him without grounding the plane. :)

ZeroHedge was down for a bit as well. I thought maybe it was an AWS (or other cloud provider) issue causing all the problems, until I read about the NYSE issue.

No need to apologize, my Bayesian inferences are going wild too.

they're highly likely related, your intuition is telling you that for a reason.

I have a possible diagnosis- Technical debt has reached critical mass!

It looks like Anonymous tweeted something about this yesterday...

https://twitter.com/YourAnonNews/status/618626955433349120

We all knew today would be "bad for wall street". The Chinese stock market is crumbling (due to economic, not technical reasons) and Greece is in the fiscal equivalent of the crazy guy barricaded in his house with this gun collection.

From the NY Times:

"A trader on the floor of the exchange in lower Manhattan, who spoke on the condition of anonymity, said that after the suspension began, traders were told that the problem was related to updated software that was rolled out before markets opened on Wednesday.

According to the trader, the exchange said that the new software caused problems soon after trading began on Wednesday and the exchange decided to shut down trading all together to fix the problem.

A representative for the exchange did not respond to a request for comment on the trader’s account."

http://www.nytimes.com/2015/07/09/business/dealbook/new-york...

three cheers for SDLC!!!

Seems like they were having issues earlier today, since before market open. Looks like a network issue based on the previous reports. Those were marked as resolved; unsure if the current issue is related to the previous but it seems likely.

Other exchanges and trading venues appear to operating normally.

Well, one thing this has done is bring the paranoid out of the woodwork. I've rarely seen such a display of tinfoil hattery on HN as I have done in the last few days.

this is nothing compared to the comment thread on zerohedge. they're connecting jade helm, china, greece, and this into something pretty extraordinary.

Yeah, but that is Zero Hedge's audience though: the upper echelon, rarified air, pure-bred market conspiracy theorist. What surprised me is seeing it start to creep in HN in such a volume.

funny description. i wasn't familiar with them until today but it's fascinating.

I've read ZH for six years, it used to be much better until the tinfoil hat, buy gold now brigade took over.

And occasionally it still can be - the odd nugget of outside analysis and opinion from anyone but "Tyler," mainly. Don't get me wrong, I enjoy Zero Hedge these days too, but mainly for entertainment value than anything actually insightful.

It's just another advertising campaign for Mr. Robot.

Good one! BTW, I like that show. Technology wise, sort of believable, and entertaining.

Apparently its an staypuft attack https://twitter.com/staypuft/status/618808119695544321

More info on them/this?

BTW, is NYSE still run by a Tandem?

Edit: I hope so! Great platform.

It is now owned by Intercontinental Exchange[0]

[0]http://www.nytimes.com/2015/07/09/business/dealbook/new-york...

They're called HP NonStop now. Great platform. Yet, even the best platforms can't stop stupid code from ruining NYSE's day. ;)

Very interesting, obviously matters very little to retail traders as most retail brokerages either sell order-flow to marketmakers (e.g., Ameritrade) or have their own smart-router that looks at the liquidity of all exchanges/ECN and decides how to route their customers' orders (Interactive Brokers).

Quick question for trading peeps out there, is there a reason why one would want to direct orders directly to NYSE? Is it because it's still the place to trade bulk orders? (vs. say BATS or ARCA).

Systems have bugs all the time. Unfortunately, it was bad timing that all of these systems were affected by different issues.

You're better off using a jump to conclusions board rather than speculating on the correlation of these events.

This is a 'networking issue' not yet attributed to an adversarial compromise.

There is also an outage at United Airlines and the Wall Street Journal - none so far attributed to an attack.

Time to make the market makers fully electronic. No point in having to physically go to them. Why is NYSE so backwards compared with NASDAQ?

Possibly related: wsj.com is also down.

Only the front page is generating an error for some reason. Deeper links like http://www.wsj.com/news/economy still work fine.

probably it was a huge volume of trades at the open due to chinese market halt

Good thing I still have all the MREs I bought for Y2K

Combining this with the United airlines "glitch" -- it is definitely suspicious. What are the odds of two high profile failures happening at the same time?

Sounds like a similar problem to the birthday paradox - only instead of looking at the likelihood of any two people in room having the same birthday, we are asking what is the likelihood of two major companies having the same major, visible failure day. There is a nice explanation and calculator here:

http://betterexplained.com/articles/understanding-the-birthd...

The parameters that you provide would be the number of companies that would be newsworthy and the number of possible "dates". You could probably back out the latter from an expectation on reliability (five nines, as an example) or think of it as the time between major, visible failures (e.g. enter 3650 if you think of it as a once in ten years kind of failure).

thanks. that's an excellent read and adds tremendous balance.

If there's smoke there's fire. In my view, Occam's razor might indicate that coincidence is unlikely, and that the two major incidents in US on the same day China's market crash suggest it's a state sponsored cyber attack.

China's market has been crashing for a few days now. How does a cyber attack on the NYSE help China? All trades will go through just fine on NASDAQ and BATS.

maybe china doesn't have a sophisticated cyber-offense strategy so they are hitting whatever few targets they can.

Interesting hypothesis

http://www.zerohedge.com/news/2015-07-08/what-first-world-cy...

China's cyber-offense is generally considered to be top notch.

> In my view, Occam's razor might indicate that coincidence is unlikely, and that the two major incidents in US on the same day China's market crash suggest it's a state sponsored cyber attack.

Occam's razor calls for an explanation that does not require the unneeded introduction of a new variable. If you need a state sponsored cyber attack as the alternative explanation, the coincidence theory should be preferred.

http://map.norsecorp.com/

Cyberattacks originating from China is not a new variable

> Cyberattacks originating from China is not a new variable

Sure, but its mere existence in other contexts does not automatically make it the cause of the NYSE etc. glitches. The assumption of causal relationship without any proof is what violates Occam's razor.

(And that map, while pretty, does not proof state sponsored cyber attacks. If I rented myself a $5 VPN account to Russia and then nmap whitehouse.gov, I would appear in that map as an attacker from Russia.)

Occam's razor would actually suggest that coincidence is much more likely explanation than a malicious hacking by a (especially particular) foreign state actor. You have to look at relative probabilities.

You state this in response to a comment linking to a good description of why, precisely, this is broken thinking. Did you even read it?

Also, you've got Occam's razor around backwards, always a good way to cut yourself in public (Occam's razor resists adding more complexity to the solution without evidence)

Also I've been having real trouble getting HBO Now to stream consistently. That's triply suspicious! Or a total coincidence.

your GoT frame-by-frame analysis will certainly be delayed

> What are the odds of two high profile failures happening at the same time?

Asymptotically close to 100%, given a little time.

And combined with the United Airlines ground stop happening now.. Something is certainly happening.

That has already passed http://www.latimes.com/business/la-fi-united-flights-grounde...

That's a pretty big leap to make. You can't take disparate events, equate them and purport "something" is happening. That just doesn't make any sense.

"NBC News quoted two unnamed U.S. officials as saying that there is no indication that the market shutdown was related to the computer problems that temporarily grounded all United Airlines flights Wednesday morning."

http://www.washingtonpost.com/business/economy/nyse-trading-...

SHARES crashes every few months like this, that's not news.

Consider this: what are the odds that a coincidence won't happen?

Someone bought the "hackers" narrative.

wsj goes down, nyse goes down due to technical glitch. seems like a crazy coincidence or some very large state who have a track record of infiltrating and disrupting America electronically at a time when the leaders of that state is feeling the heat and need to stop speculators.

so you're saying that china has initiated some sort of half-cocked single-exchange attack? all other exchanges are operating, and trading is maintaining volume, and asset prices are stable. not sure i'm seeing the reasoning for such an attack.

Its a distraction. We are here talking about the NYSE not the colossal Chinese market collapse.

This is what happens when you pay ENGINEERS less than 100k in NYC! This is what happens when you hire a bunch of H1Bs! This is what happens when you don't invest in tech infrastructure!

Why aren't Vladimir Tenev and Baiju Bhatt being acquihired? Why isn't David Byttow making 1 million plus as a full stack engineer for the NYSE? It's because companies, even those heavily dependent on their systems, see tech workers as a cost instead of THE business.

This will more than likely never change. Underpaid engineers vastly outperform their salaries. But when things like this happen I can't help but feel a little glee.

What are you trying to say? That somehow American born engineers are better than those from the rest of the world?

The attitude of a small subset of people in this country is a constant reminder of why I don't feel at home here.

Reply on news.ycombinator.com