With 61 Seconds in a Minute, Markets Brace for Trouble
bloomberg.com
bloomberg.com
Real: true time, showing the 61st second
Comp: computer time, only counting 60 seconds
not smeared,
time counted at regular speed
...--->| |<---...
Real 54 55 56 57 58 59 60 00 01 02 03
Comp 54 55 56 57 58 59 00 01 02 03
|<- smearing ----------->|
time counting slowed down
by 20%I guess people don't want a database of leap seconds and a lookup each time time is rendered, but ultimately it seems like the best solution which is least likely to break things.
One drawback of your approach is that I can't just see if a time = 0 mod 60 to check if we are on a minute boundary.
Your way is what is used by the GPS network, IIRC, and it works for them, so there you go.
It appears to have this stuff down to the second. For example:
# Zone NAME GMTOFF RULES FORMAT [UNTIL]
Zone America/New_York -4:56:02 - LMT 1883 Nov 18 12:03:58
# Zone NAME GMTOFF RULES FORMAT [UNTIL]
Zone America/Chicago -5:50:36 - LMT 1883 Nov 18 12:09:24
The leap second caused our databases to all peg at 100% CPU/memory consumption..
Given the wide adoption and commercial nature of mysql, I can't begin to imagine how systems designed to be super precise will handle this..
I remember being utterly ridiculed that weekend, after swearing about fixing servers on Facebook.
It didn't take long for apologies to arrive on the following Monday, once the corporate sysadmins logged in and saw anything Java-based had also massively, massively failed.
It's the small (and petty!) victories.
What was the error in Java?
Are they made in a distributed fashion where it is assumed that two clocks can never be synchronized with full precision.
PS - I am a noob in markets and trading.
That said, I'd be surprised if keeping the clocks together is what is causing the worry. Much more troubling are third party bugs (kernel, jvm, db, etc) and incorrect assumptions in their code.
This same story can be equally applied to any industry honestly, but the markets make for better headlines.
GPS + NTP +- 12us
As a quick example, Javascript is not designed to handle leap seconds correctly, so I doubt we'll ever see the value 60 out of new Date().getSeconds(). Things will just be off by 1 second for a second.
Edit: Another example from the Windows Time service:
> The Windows Time service does not indicate the value of the Leap Indicator when the Windows Time service receives a packet that includes a leap second. [...] Therefore, after the leap second occurs, the NTP client that is running Windows Time service is one second faster than the actual time. This time difference is resolved at the next time synchronization.
It will just be a second off and resynchronize next time.
Also, not sure if any of the systems mentioned in the article run JS for critical server side code.
We've been here before, June 30th 2012 also contained a leap second (see Wikipedia). The article only forecasts doom without looking back at previous leap seconds and evaluating its effect.
Clearly you didn't read the article.
If you had you would have seen that they noted that last time it was on a weekend when all the markets were closed.
That isn't what you said in your post, now is it? You said "We've been here before", but as the article points out, we have not.
In any case,
Traffic lights and apache for the most part do not care what time it is - and certainly not to the second. Market orders are timed to the millisecond, they care very much.
Slewing vs stepping or any other de-synchronization could cause chaos with them, not so with most other fields. You might not care about HFT, but this applies to all orders, not just HFT. And like it or not the stock market plays a large role in everyday life.
"Being a second off and resynchronizing next time" could be a million dollar difference, there is no way that you can shrug that off.
As a developer for a financial firm, this leap second is an extremely big deal; the fact that this is the first time in history that financial markets (which are normally millisecond accurate) have had to deal with this phenomenon will absolutely result in something somewhere going wrong.
This article isn't exaggerated; it's there to highlight what's going on for the traders and brokers who aren't technologically advanced.
again, same question - who cares? apart from some already super rich HTF guys.
> "something somewhere going wrong" - um, this is practically constant situation for quite some time, in or out of markets. But don't let my relaxed monday mood disturb your doomsday freaking out :)
> This article isn't exaggerated; it's there to highlight what's going on for the traders and brokers who aren't technologically advanced.
traders and brokers who do actual trading and not HFT will not be affected directly by that (ie their books), it's more like a sidenote that "if you see some blip in feeds during that time, this might explain it"
there is no way HFT is less important than traffic lights
one keeps the economy of the world flowing, the other
stops people getting driving tickets.
This thread has reached sort of silly proportions. Of course electronic trading impacts most people's lives even if they don't want it to, but in reality the markets are largely self correcting at this level, so even if there is some oddity associated with the leap second it likely won't have long term (or maybe even observable) impact on the wider world.
I know "market liquidity" is an argument for allowing HFT, but haven't hard a good explanation about why the economy would suffer so much if the ultra high frequency trading just wouldn't exist.
Note: I have about as much knowledge about this as people who suggest how SpaceX should improve. Just sayin.
Of course computer systems everywhere will have issues with an extra seconds (deltas being negative that can't be etc) but most of those systems aren't trading systems. Why is it that "markets" are in the focus of this? At y2k we were worried about planes falling out of the sky. Surely there must be worse things than stock market computer systems being confused?
The short explanation is that the speed is a tool for managing risk (like most other market tools) and by removing it, you remove that ability. That risk must be priced into the market somewhere and that will be in the spread that everyone pays.
Will that be worse than what we currently have? Who knows, but what we have is working pretty well with regard to providing liquidity (it is cheaper and easier to get in more markets than ever before and the margins are as low as they have ever been on providing it), so why mess with something that has so few down sides?
> Surely there must be worse things than stock market computer systems being confused?
There are, but Bloomberg doesn't specialize in them, and they don't garner nearly the HN upvotes.
High enough that the firms are at least receiving a normal profit. And given the opportunity cost that all these smart people incur by working in this industry, this normal profit is rather big, and no doubt that the extraction of this normal profit has non-neglible deleterious effects on the economy.
As did the system of pit traders that it replaced. The margins and accompanied spreads are lower now than previously.
> no doubt that the extraction of this normal profit has non-neglible deleterious effects on the economy
As with any market activity, it needs to be compared with the benefits it provides.
When I look at high frequency trading, it says to me that the game is rigged. It's like going on Jeopardy up against a machine contestant that always buzzes in first.
How on earth can I possibly succeed as an individual investor in a trading environment where institutional investors are so privileged? How many other ways is the ostensibly neutral marketplace overseer profiting by offering those who pay-to-play opportunities to shave pennies, nickels, dimes, and dollars from me?
Economists have sort of argued that it was impossible to succeed in the way you're talking about since long before HFT came along. The classic explanation is presented in https://en.wikipedia.org/wiki/A_Random_Walk_Down_Wall_Street.
And when I hear this complaint, I don't understand the rationale. Why shouldn't participants that do something professionally, invest in infrastructure, invest in research or other kinds of IP have an advantage. If they didn't that wouldn't be a market, that would be a lottery.
Quite simply, the market dynamics that make it such that you might want to be involved in it as an individual are created by the sophisticated market participants engaging in individual zero sum trades that add up to a beneficial whole and they have been for hundreds of years.
Computers have made this process more efficient and you should be celebrating your ability to trade for cheaper than ever. That you don't is a sign that you don't understand how the markets work now nor how they have ever worked.
Edit: the more I thought about this, the more I decided it was overly harsh. I think well informed participants can disagree about the values of HFT and needn't celebrate it. I do think that the idea that markets don't (or shouldn't) reward more engaged participants is fundamentally broken.
We're talking on the order of zero liquidity for 30% of the days (weekends and market holidays) in a given year, and severely limited liquidity outside of market hours.
If liquidity was as important as we like to say, this just wouldn't be the case.
"If liquidity was really required at the millisecond level, market participants would not accept markets being down for holidays or weekends."
But that is not the point of the millisecond (or sub) resolution. The point is so that the people that provide the liquidity during operational hours can add as much information as they can into their pricing models, thereby requiring them to take on less risk with each bit of liquidity they provide. This less risk gets passed on as savings to the rest of the market participants in the form of smaller spreads. Any increase in that resolution increases the risk to the market makers, who will therefore need to increase the spread to compensate, making trading for everyone more expensive.
This would be true if the markets were only open 4 hours a week as well, though trends are towards more 24x5 (at least) trading, not less (and I think that is a good thing).
Soft-realtime systems in C++ instead of safer garbage collected runtimes.
Parsing market data on FPGAs or Cell processors instead of in code written with emphasis on safety.
Distributed trading systems that couldn't afford the latency budget to pass everything through centralized risk management, and therefore not taking into account the entire order book before authorizing a trade. Instead a series of compromises and less optimal failsafes.
It also just lead to more expensive practices--communication between threads using spinlocks instead of the normal waking a waiting thread through a system call; basically converting the spreads we could capture into data center waste heat in order to beat out the next guy.
Artificially slowing things down a bit would have just reduced costs and increased safety for everyone.
I'd also suggest that your experience (many of which I share) do not necessarily mean that speed is making things riskier, only that your centralized risk management was not as good at lowering the cost of risk as opposed to getting fast. As for the dc energy to spread arbitrage game, that is the result of far reaching policy decisions well outside of HFT and I largely agree that energy is too cheap.
I think that any artificial slowing of the market is likely to have really bad unintentional issues. I'd much rather we incentivize the market to fixate on something we care about (ie price) rather than try to subvert the laws of the market.
One approach that appeals to me is to remove the sub-penny rule that is artificially propping up spreads.
Why? It could severely hurt market liquidity (which everyone claims is so important) if there is a mistake that sparks a panic.
> was not as good at lowering the cost of risk as opposed to getting fast
Can you reword this a bit? I don't understand what you are saying.
> I think that any artificial slowing of the market is likely to have really bad unintentional issues.
Can you give some examples?
> One approach that appeals to me is to remove the sub-penny rule that is artificially propping up spreads.
This would be a decent thing to do, it would take some money out of market making, reducing the returns fueling the latency arms race a bit. Lower returns would also give less justification for some of the risks that are taken today.
A transaction tax would help in a similar way, but work more broadly against other high volume, low latency, low value traders. Index-to-underlyings arbitragers (maybe not the best example; sub-penny would decimate (hurr) them as well), etc.
Because we've seen what happens when a very major market maker blows their risk configs, it is a headlines day but the market largely recovers quickly. I'm more interested in systematic risk and I believe that any "fix" to the speed issue in HFT is likely either just going to shuffle advantage from one group to another or have unintended consequences.
> Can you reword this a bit? I don't understand what you are saying.
It was more risky to be slow than to use a central (and assumedly better) risk system. At least that was the implicit lesson learned from a market maker not using their own systems. I understand trading incentives enough to realize that might have been short term thinking trumping long term thinking, but I'd rather adjust the incentives there as well (long term bans from trading for risk limit violators maybe?)
Further, I want market makers to be able to be innovative with some of their risk controls, as I believe that it could lead to better risk controls more broadly.
> Can you give some examples?
The most common response that people trot out to "fix" HFT is to introduce batch auctions. I believe that this a) won't remove the incentives to play speed games and b) will make it harder for market makers to price spreads appropriately leading to them making them more expensive.
A transaction tax is similar. I view transaction taxes as pass throughs that will largely be paid by non-speculative investors. That is, we actually largely don't care (I don't think) about the number of transactions, so why would we use a tax to discourage them? Lets discourage what we actually want to discourage (I'm not sure what that is by the way).
> Index-to-underlyings arbitragers
I'm not so sure. In the short term they and market makers would get wrecked, but I believe quickly they would adjust (or go out of business) and the pricing models would be better for it, and spreads would go even lower.
Knight Capital, for example, was just at the threshold where they were effectively able to be bailed out. Nothing guaranteed it. Other large market makers are owned by banks and are essentially risking capital reserves. We haven't hit a scenario where a hit to one participant has had systemic effects, but that doesn't mean it isn't a risk.
> It was more risky to be slow than to use a central
From this and some of your other comments I see you are taking the stand that every market transaction is about pricing risk, and so talking about risk separately isn't ever a useful concept. Maybe you can say that under lots of abstraction, but it seems to make it unnecessarily difficult to communicate with people who speak of it with the normal meaning.
>innovative with some of their risk controls, as I believe that it could lead to better risk controls more broadly
Decentralizing risk controls in the way I mentioned is strictly worse in every metric but latency. Or at least there was no other explicit reason why we did it.
>The most common response that people trot out to "fix" HFT is to introduce batch auctions. I believe that this a) won't remove the incentives to play speed games and b) will make it harder for market makers to price spreads appropriately leading to them making them more expensive.
I'm not necessarily proposing batches, but a significant fraction of daily trading volume already goes through the opening and closing auctions every day. When stocks are halted and re-opened, they go through auctions as well in many exchanges, just opening right back up conceivably leads to worse pricing than the auction system with published imbalances.
My point about market hours, weekends, and holidays, is that all this stuff is based more on tradition and precedent than any kind of goals. Participants that trade through microsecond latency advantages are extracting a tax on everyone else. A transaction tax takes out a lot of these opportunities, and can be used for something more productive than the waste heat result of running a bunch of spin locks in a datacenter.
I completely disagree with this and have seen no proof of it one way or the other.
One data point that convinces me lacking any proof, is that spreads are tightest in the products with the highest competition for latency. It seems that speed is bringing the tax down not up.
Well, it is a Bloomberg article, so their subject matter might be skewed towards the market.
Sorry, but a minute delay is definitely not a solution to your idea of unnatural market manipulation. Artificial delays can be seen in the Chinese equities markets - trading halts given a +/- 10% intraday move, trading halts for up to 10 trading days before news announcements, you can only see quotes every half a second -- all measures to benefit retail traders over institutional traders, a paradigm the American markets inherently oppose. HFTs solve that problem by providing retail investors the same kind of access as institutional investors.
>I know "market liquidity" is an argument for allowing HFT, but haven't hard a good explanation about why the economy would suffer so much if the ultra high frequency trading just wouldn't exist.
HFT is a natural effect of electronic trading in contract to traditional forms of market making and liquidity providing. More often than not, thin spreads are often returned back to retail investors.
Other than some anecdotal evidence of "flash crashes" [1][2], how does HFT negatively impact the market?
https://en.wikipedia.org/wiki/2010_Flash_Crash
http://www.bloomberg.com/bw/articles/2012-08-02/knight-shows...
Pausing for a second, going back a second in time or changing how long a second is all are hacks that are bound to cause some edge case failure somewhere. I'd like to see a minute have a 60th second just as February has a 29th day.
The Earth's rotation is slowing, but it's not a steady process. For example, the Boxing Day earthquake in 2004 permanently decreased the length of the day by about 3 microseconds. Here is a graph of the changes over the past few decades:
https://upload.wikimedia.org/wikipedia/commons/5/5b/Deviatio...
Also, given all that variance and the fact that we only seem to be shifting around 1 minute per century at the current rate, we probably could get by having a leap minute every few decades when necessary.
This works fine for any system trying to interact with other computers and humans while keeping consensus on the time. Any system that is trying to determine earths yaw would no longer be able to rely on F(time), but that is only a minor inconvenience.
The complication is just that there are a whole lot of systems out there that assume that every minute has exactly 60 seconds with no exceptions, so you end up with crazy workarounds like smearing to hide that 61st second from such systems.
It's different because you don't have to keep track of what an authority is saying or deal with the actual time adjustment except once in a century. Once in a century is acceptable since we will only deviate by around a minute at our current rate.
The problem with leap seconds is that time is often represented internally as seconds from the Unix epoch, and this simple integer counter assumes leap seconds don't exist. In effect, there is no way to actually represent a leap second in many computer systems, so the question of what you do is basically a choice of the least painful lie. Handling leap seconds in the situations where people care about them requires a lot of complexity for an event that happens only once every few years.
There's a movement to abolish leap seconds, which I am very much in favor of.
It clearly isn't since the original post highlights three different ways people adjust the time making all these organizations lack consensus on the time.
Move to leap hours, and by the time it's needed, let the AIs sort it out.
Having a scheduled addition of the leap second only corrects an issue resulting from our timing being slightly off (by about 1 second every couple decades). Now, if the second had to be add in changing frequency... the changing in frequency would be a result of the earth's rotation changing... but even still that wouldn't be why the second itself is being added, is it?
In the meantime, the Paris-based International Earth Rotation and Reference Systems Service will keep track of the gradual slowing of the planet, caused in part by drag created by the Moon. Millions of years ago days were 22 hours long.
Consider the leap year for a second. We add it because our years have 365 days is slightly incorrect (by about .25 days a year). It isn't that the number of rotations per revolution is or isn't changing. That isn't relevant to our use of a 1 day correction every 4 years. If some force caused us to revolve around the sun slightly faster and it took 365 exact, then we could remove the leap year.
Removing the leap year would be a result of our revolution being slightly faster. But the leap year itself is not.
In short, the delta of the leap correction is a result of the change in the system being monitored, but the leap correction itself is because our system of measurement is off.
Seconds are not defined by the rotation of the earth, any more than they're defined by its revolution.
The problem is in some sense, the opposite... our timing is now so good that we can notice the natural variations in Earth's rotation to astonishing accuracy. Thus, if we want to keep the clock in sync with what the Earth is actually doing, we need to add the leap second.
(There's quite a bit of debate about that "if", and IMHO right now industry after industry seems to be experiencing major, expensive disruptions and repeatedly incurring risk, to save astronomers from having to consult a lookup table that they of course would build into computer programs and pretty much forget about ever after. But I digress.)
The Earth's rotation really does wobble by some amount due to natural shifts in density, temperature changes, and any number of other things. It's why we don't have a schedule of leap seconds for the future. If it were merely a matter of accurate timing we'd have a schedule, because our timing is accurate enough by orders of magnitude to make a schedule for a hypothetical perfectly-steady or perfectly-slowing Earth for millennia in advance. The problem is that we can't predict what the Earth is going to do.
And simply put, the leap seconds are added so that clock noon, solar noon don't drift too far apart.
before I did some research I thought this would be a good "anti-HFT abuse" idea but I still think it has considerable merits.
Much of the slowing is caused by tidal friction with the Moon (which recedes slightly each year), iirc eventually the Earth will become tidally locked with the Sun (same side facing all the time).
This gives rise to problems for computers since they are irregular and unpredictable, and can't be made part of algorithms. In extent, to properly calculate a precise time difference (actual elapsed time) over several years, you actually need to consult a table of added, historic leap seconds and take these into account too...
So the day is not 24 hours or 86400 seconds, but rather 86400.002 seconds on average. That's about 2 milliseconds or more of deviation per day and for a whole year that's about 0.7 or 0.9 secs worth of deviation. Yet we still pretend that the day has precisely 24 hours, hence the need for leap seconds.
Given that since 40 years ago since leap seconds were adopted about 25 leap seconds have been scheduled, that sounds about right.
Cumulative deviation since 1972 will be 26 seconds.
26 seconds over 43 years (1972-2015) is 0.60 seconds/year. If this matches the average offset from 1820...1972 (which would add up to about 92 seconds), it means that our average is wrong by ~0.6 seconds, but the time length doesn't gain additional seconds due to "spinning down" (caused by tidal interaction with the moon, or whatever other effects there might be).
That's actually also what the graph suggests: It's havnig a short-term daviation of +/- 1ms over the course of (judging by eye) weeks, and a long-term deviation of +/- 2ms over the course of decades. The average deviation of 0.6seconds/year (number of leap seconds inserted) translates to an average of 1.65ms/day from 1972-now.
In practice, it would take so long for the Earth to tidal-lock itself to the Moon, it's most likely some other major event will happen first (like the Sun expanding and swallowing the Earth).
When you know what the problems are likely to be, you can of course pick the approach that's least likely to cause you problems! But if you're going in blind, then this seems like it could be a safer way to do it.
(You might use some kind of a curve to do this, so it's smooth like. I imagine this is what the comment about weighting refers to.)
I understand that this isn't ideal for people dealing with actual dates and times, but the upside is that timespans expressed in seconds are now universal, so for example Unix timestamps actually have meaning that doesn't change according to earth's rotation.