Finance sites erroneously show Amazon, Apple, other stocks crashing | Hacker News Reader

Finance sites erroneously show Amazon, Apple, other stocks crashing

marketwatch.com

169 pointsdoener91 comments

This story isn't emphasizing an important point: the stock price of many tech companies - amazon, google, Microsoft - were pegged to exactly $123.46 for multiple hours.

This is likely 123.456 rounded to 2 decimals.

To me this smells like a software testing bug or hack, rather than "stocks crashing"

Good eye. That was probably the case.

Turns out they are scraping price data, the bots must have failed!

I saw it pegged at 123.47

rounding

Rounding from what? 123.456 to 123.47?

maybe 2 passes? 123.45678... -> 123.4568 (4 decimal places is common in finance) then to 2 places for the retail muppets 123.46

Stocks have two prices. I guess this would be reflected in test data.

Bid/ask?

The NYSE did exactly the same f*ckup in 2014. And as I would expect will happen this time, the NYSE did not care because it is a bank holiday in the US, but it is not in the rest of the world, and I saw many people complaining on forums that their brokers automatically closed their positions or made big margin calls.

So this is a pretty bad mistake.

There's no due diligence expected on the broker's part? Or was the accurate data simply not available to them by any means?

Well, the official price normally comes directly from the exchange. Brokers protect themselves against their exposure to their clients by reacting to the markets live. The exchange should be the authoritative source for prices.

As some have suggested, Nasdaq is confirming that this was test data that got propagated to third party aggregators.

https://mobile.twitter.com/Nasdaq/status/882054148413325313

http://www.nasdaqtrader.com/TraderNews.aspx?id=utp2017-05

Using production systems to run tests... What could go wrong?

No test like production.

It's actually considered best practice in various categories of technology to run tests on the production systems.

https://en.wikipedia.org/wiki/Built-in_self-test

When done intentionally.

I've seen lots of tests 'escape' their environment for a bunch of reasons (tests got sent to production with a deploy, human was on the wrong machine and ran a manual test command etc etc).

It's almost impossible not to screw up at some point, I try to focus on minimising the small mistakes (airline pilots have commented that it's not the first problem that kills you it's the second one).

That's why artificially differentiated steps are critical. If the "deploy to test" procedure looks exactly like the "deploy to prod" procedure, except with different servers, then you're asking for human error.

Ensure the test deploy procedure will always fail if run against prod targets when half-asleep. Whether it's something like requiring a different config setting or a reload or an account switch. Anything that breaks the process-as-usual.

I agree that there are a class of problems that you simply will not find on canary environments, but most of the places I have been will run their smoke suite independent of the actual service. A problem of this type, e.g. core to your SLA/SLO, shouldn't be tested on the live production deployment. An alternative would be creating a dummy ticker and playing fake data, but even that is dangerous on a live exchange.

One nice thing about SEC-regulated markets - unlike the various cryptocurrency markets - is that it's possible for trades to be rolled back if they are "clearly erroneous". That isn't perfect - if the data is only a little off, it might not be possible to get a trade rolled back - but it tends to limit how much havoc is called.

https://www.sec.gov/news/press/2009/2009-215.htm

(My understanding is that trades can also be rolled back in the case of security breaches, at least at the exchange and broker levels.)

Does that SEC possibility apply to Google Finance stock data? It's not like they're a proper broker or official provider if data (i.e. most of the affected financial sites have ample "price inaccuacy" disclosures).

I don't think the OP was trying to say it applies here, just that IF there had been some kind of hack or security problem, then the SEC has some tools for dealing with it.

In this case, if it was only the data going to out to third party stock tickers, then it probably doesn't apply. Services like Google and Yahoo finance make it really clear on their sites that the data isn't necessarily accurate, can be delayed, and isn't for trading purposes, so nobody should have been making trading decisions based solely on this data anyway.

The flip side of this is that no providers of pricing data are expected to stand behind their prices. It's basically arbitrary and at the whim of some people who very easily may not care about your situation.

Why can't majority participants in a crypto currency agree to roll back?

The majority of participants do not in a crypto-currency.

Only the majority of the hashing power.

If a majority of participants rejected blocks that were on a chain that did not rollback the erroneous transaction, then the hash power would be forced to follow.

I believe the term for who decides is the "economic majority".

Hashing power can not roll back transactions. Hashing power alone can not change rules embedded in the software.

Proof of work is used to order transactions in the global ledger to avoid the double spending problem.

They did in Ethereum after the DAO got hacked.

Anyone could roll forward all the transactions again, so you need to do more than just roll back. You need to declare the transactions invalid somehow.

And unless every participant is in on it, you will end up with two cryptocurrencies afterwards. Holdings won't match on both chains. Find an exchange that is not running the same chain as you and spend a transaction that is valid on his chain but not the other, and by transferring out other cryptos you could empty their wallet.

Cryptocurrency trading usually happens on an exchange and the exchange almost certainly won't make any on chain transactions every single time a trade occurs, they will just swap the balances around inside their own internal systems. The on chain transactions will only happen when a user withdraws their currancy from the exchange. So trades certainly can be very easily rolled back by exchanges up until the point that one party withdraws from the exchange.

And during major instability and crashes it's not unusual to hear that the exchanges disabled withdrawals temporarily. I assume this is at leaat partly until they have chance to asses if any action is necessary.

It's not a property of a regulated market, it's a property of a specific centralized system. Nothing prevents SEC from "regulating" a system where such rollback is impossible, and nothing prevents centralized cryptocurrency exchanges from rolling back transactions that haven't made it into blockchain. There can even be non-immutable blockchains, which rollback mistakes, such as Ethereum.

Not second order effects. Like if I have a stock affected in my portfolio, my broker may close a position to reduce my leverage or execute a stop loss order. They can't sell that stock because the US is closed, so they will sell the other positions outside of the US, which are unaffected by this bug.

These transactions have been induced by an error but are not clearly erroneous and will not be cancelled.

It happened in 2014 when the NYSE did a similar fuckup. Their reaction was "who cares, US markets are closed". There was a lot of livid comments from canadian investors on forums.

> The problem may have originated with data services providing Google and Yahoo their financial information

I'm sure Google and Yahoo pay a small fortune for this data so I'm wondering if they are aware of the situation and holding these data services companies accountable or not.I'm also sure these data companies sell their data beyond information sites to investment firms & others that make buy/sell decisions based on this data. Yahoo does disclose[1] that they get some data 'direct from exchange' though.

[1] https://help.yahoo.com/kb/finance-for-desktop/SLN2310.html?i...

I'm fairly certain they're aware of the problem :)

And yes, I bet this is going to lead to some interesting meetings between them and the data providers.

I've actually seen a number of times where Google Finance has a daily low that was never reached or a high that wasn't shown on the graph (sometimes many percentage points away). It's also notoriously bad when there's a gap up or down, and will sometimes cut out various parts of the graph.

But still, seems to be one of the better ones out there, as I still use it. Just don't treat it as authority.

Another interesting example of questionable data is the premarket CNN feed at (http://money.cnn.com/data/premarket/). I have sometimes seen a stock trading premarket down 99%, which almost always looks like an off by some factor of ten of the price.

Google Finance does not even have a working price plotter, that's pretty awful in a basic way.

and their financial news is flooded to uselessness with bot-written "stories" from sites like "The <placename> [register|tribune|bulletin|news|alert]". The service is pretty garbage now.

Of all of google's services Finance is the worst, it seems to be supported by cheap contractors in Elbonia. I mean Flash charts??? Sometimes stocks that were removed remain in the lists for months. The stock screener seems to be a student take home test where the student failed.

If I had to guess, it looks like google and yahoo track by time. Each nth second it plots a point, so it's likely to miss graphing the actual high or low of the day.

I thought this too, but looked at the daily high and daily low at Fidelity, and it matched the actual low on both the graphs (which were the same). I figure it's either some data blip, or maybe some kind of dark pool trading that was logged after hours.

This was a market half-day: exercising some code-paths that haven't been used since the day after thanksgiving '16. I do find it surprising that google and yahoo are both affected, as they used to have very different patterns of stock data errors (yahoo was much higher quality).

Sometimes I wonder if it would make sense to place long-term buy orders at a ridiculously low bids. Just in case.

On Bitcoin exchanges, sure. But here I think there's too much liquidity and automated trading.

I believe the recent Ethereum flash crash demonstrated that this strategy wouldn't work for cryptocurrency either. Ethereum price dropped to $0.10, but it rebounded in about 5 seconds because trading algorithms put in lots of bids at the really cheap price.

It did work, but not for many.

https://news.ycombinator.com/item?id=14611653

https://news.ycombinator.com/item?id=14690908

Too much liquidity?

Except these weren't real prices. Buy orders would not have been triggered.

It seems like the price was trying to be set to 123.45 -- there are some discrepancies on different sites; not really clear whether any trades took place in little or no volume after hours, or purely feed error.

As far as I can tell, it happened after the markets closed. Even so, my brokerage had the correct prices the entire time.

Recently when BTC flash crashed (or maybe it was Ethereum) those who had placed such orders made a ton of money. It's a safer bet with virtual currency because of its instability

It was isolated to a single exchange GDAX and happened with both Bitcoin and Ethereum at different times.

The equities markets have Limit Up/Limit Down bands (LULD bands) that prevent big swings in prices. Like the other commenter said, your strategy might work in cryptocurrency markets, but not in equities.

It works if your orders can be executed in after hours trading when liquidity is restricted.

Equity exchanges have cutoffs for limit prices. Even if you are able to do that you're setting aside capital that could have been used somewhere else.

Such trades would be reversed.

Unless you cashed out.

It takes 3 days for trades to settle, so you can't really cash out quickly. Google "T+3" for more details.

Not in cryptoland. The guy who bought ETH at $0.1 could have sent the ETH out of the exchange minutes later.

Isn't that sort of what happened in the "flash crash"? Or was it purely market-makers and algorithms working their way down to 0? Big regulated markets now have circuit breakers to prevent this, unfortunately.

But the idea is that you put a buy option at the low price.

Ie, you say "I would like to buy 1 million microsoft stock, at the price of 1 cent".

The flash crash was cause by stop-loss orders. IE, if the price gets low, then sell. Instead of price gets low, then buy.

I am not a trader but I believe stock markets take place on an order book with bids at low prices and asks at high prices, with the market clearing in the middle. The upshot of this is that your trade for a million shares will sit on the books like everyone else's, waiting for a seller. If somehow a seller entered a market order to sell 1 million shares for whatever price it can get, and somehow the computer matched it to your order, then that would actually be the last price traded, i.e. a crash.

tl;dr: your strategy is more likely to contribute to a stock crashing than you think.

That would mean locking in the amount you dedicated with very extremely low (lottery level) chance of any return. Most probably, you would not see that money ever again (provided you don't decide to remove your bid, obviously). Better give it to a charity :)

This could be the beginning of a thriller movie. A nefarious hacker group demonstrates a powerful zero-day for a dangerous client by creating national panic over stock prices.

This is silly. Blame the third-party providers on this one, for using exchange prices during a holiday when the exchanges are closed ...

"Will Briganti, a spokesman for Nasdaq, was unable to immediately comment on the situation." https://www.bloomberg.com/amp/news/articles/2017-07-04/nasda...

One of my grad school classmates spent some time in finance and always used to say something like "your best results are really just bad input data". This has proved a worthy maxim many times over.

What if a multi-government coalation broke out of the lobbyism hypnotization and would split up amazon. Could such a drop be realistic?

Black Monday - fake news edition.

And we trust that software can drive our cars for us?

Stock tracking websites don't need to be perfect, just need to be better than the humans they replaced. Faster, free, and less errors doesn't mean they must be error free.

Self driving cars just need to be better than humans, and given the number of teens and senior citizens on the road...

Statistically that's true but realistically I wouldn't want to use a self-driving car that's only a little better than the average driver. It's one thing to have a friend that's a bad driver and another thing to ride in their car.

But they are still on the road. They're still a hazard to everyone around them. Pedestrians, cyclists, and other drivers are all at risk.

Yes but they'll continue to be on the road until they can get in a self-driving car that they can trust.

Never mind trust: teens and seniors generally don't have the disposable income to spend on high end brand new cars (or expensive retrofits of sensor devices). The first buyers, apart from the wealthy, would be people and corporations that spend the most time on the road: generally tending towards being better than average drivers.

And even if self-driving cars were exactly as good as the drivers they replaced, half the purpose of buying them is to spend more time on the road...

Elon Musk said that he is aiming for his self-driving cars to be 10x safer than the average driver (without LIDAR btw).

This is true from a pure-risk perspective, but I suspect we will find that humans are far more forgiving of the errors of other humans than they are of the errors (or worse, harm-minimizing calculations) of machines.

As someone contemplating the purchase of one, a self-driving car needs to be better than me. That's a higher bar than better than average even if it's not objectively correct.

This is only true if you are on an otherwise empty road.

On the other hand, as a passenger on public transport, taxis and ride shares it only has to be better than (often over worked and over tired) average human drivers.

Do you ever use a taxi?

Because if you do, you are putting your life into the hands of someone who could be a much worse driver than you.

Instead of putting your life in the hands of a terrible taxi driver, would you put it in the hands of software that is on average better than a taxi driver? You are in a taxi, so you don't get to choose to drive it yourself.

Not gp, but I would personally expect a taxi driver to be better than me seeing the amount of time the diver spends on the road. The driver wouldn't be profitable otherwise.

That said, I also mirror gp in that any self-driving car must be at least better than myself for me to trust it. I imagine that's a sentiment shared by many people.

I agree this is the natural bar, but don't know how to reconcile it with the average person believing they are a better driver than average (https://en.wikipedia.org/wiki/Illusory_superiority)

What do you mean it wouldn't be profitable?

I am not paying the taxi driver money because they are a safer driver.

I am paying them for the convenience, and the ability to not take a car with me.

Also, taxi driver incentives are wrong. A taxi driver, even though they have lots of experience driving, have an incentive to drive very quickly and therefore dangerously.

We are all running software to drive our cars. Sometimes the software gets drunk, or has a stroke or a diabetic fit or just drops hot coffee all over the hardware.

Yes. If this is indicative of the failure rate of computers in general, then cars will be PHENOMENALLY safe. Humans are doing a garbage job, killing people left and right. So what's your solution that isn't human or computer?

I'd also hazard a guess that having humans update tens of thousands of stock prices every few seconds would result in more than one mistake in a decade.

Software ALREADY drives your car.

You press the gas pedal, and a computer reads that signal and puts gas into the engine.

We trust software to fly our planes and run our nuclear power stations... So what's the problem?

Those are simpler problems. There is more at stakes, but they don't have to contend with a kid running after a football.

Wow, that's crazy! What algorithm did this, or maybe it's some scheme manipulation for profit? You can never be sure when something like this happens, flash stock crashes have a history that's muddy at best. :/

It's just a data feed error. The markets have not actually moved.

your paranoia level is set a titch high - this is incompetence, not malice.

Reply on news.ycombinator.com