Financial systems take a holiday
bitsaboutmoney.com
bitsaboutmoney.com
Technologists thought, how easy it is. A year has only a few hundred days, and there are only a few dozen possible rules for buses. Brute force is enough.
Nobody got the problem right. It gave me a great appreciation for how complex when technology interacts with culture and customs, long before my professional career.
Th +10sih comes if you work more than weekly hours, or if you have a day-based contract (i.e. you don't count hours).
Most day-based contracts are written saying "you owe x days to the company in a given year", so a holiday on a weekend becomes irrelevant.
Ah, but that's not so. For example, if New Year's Day falls on a Saturday, then the NYSE doesn't observe the following Monday as a holiday - happened in 2022, for example.
They also avoid being closed more than 3 consecutive days, which is why they are open for trading on Black Friday (but they do close early!).
When you guys quote salaries in monthly figures.. Is the hourly rate fluctuating month to month based on the number of weekends and holidays? Or it vis a vis some standard amount of work days?
Yearly wage / yearly hours works
February 28, 1923 (Wednesday)
The nation of Greece used the Julian calendar for the last time before adopting the Gregorian calendar, used by most of the world, the next day. In that the Julian calendar was 13 days behind the Gregorian, the day was noted as "February 15". The next day was March 1 rather than February 16.
Seems like that is a red herring since Easter is always on a Sunday, which, it seems to me, is all that could possibly matter given the description of the problem.
edit: or, because the buses would observe the Easter holiday the next day, quoting OP:
> including the observed holiday which is the weekday after the holiday if the original holiday falls on a weekend.
Then you have Ash wednesday and some other holidays around Lent (6 weeks before Easter)…
You’re still going to need that data entry guy. (Plus the two more that is the root problem)
if this data entry was so important, why is there only one guy responsible for it? where's the backup? where's the redundancy?
Sometimes, ad hoc things happen in our culture. Spurious bank holidays get added on a day’s notice in exceptional situations. These can be inserted into the cached DB. There is no calculation to back them.
This is a big one. And you don't get much notice, maybe a day or two. Typical ones in the US are death of a President or prominent Senator (I _think_ we had one for John McCain). Much easier if you can insert the holiday in a lookup table and have all your systems adapt.
Unexpected bank holidays (happened to me in 2002 during a financial crisis), official days of mourning, etc. Nothing to compute there: a government declares a holiday and you just enter it into the table.
Sure, you can compute Easter (you can just look up Knuth’s explanation of the algorithm for this…), but you can’t compute government decrees.
We do generate all the official calendars though, so we have print options to print various views of it(paydays, etc) and we also provide .ics feeds for Google/Apple Calendar, etc.
Like I mentioned in a diff. comment, we generate all the official calendars of the org from this data. We print them, we generate .ics, etc.
So when a mistake happens, every employee that cares about that date will find someone to complain about it to, and ultimately someone somewhere goes, oh bother, we forgot to update the Calendar, and they promptly fix it.
Plus we avoid all blame, as it's very clear that we don't invent the calendar data. For us the HR dept is in charge of maintaining the calendar.
> Nobody got the problem right. It gave me a great appreciation for how complex when technology interacts with culture and customs, long before my professional career.
For those of us who weren't there, could you let us share in that appreciation: why wasn't brute force enough?
Wouldn't it be better for users if our financial systems worked like the Google example? Why can't I as a regular retail investor trade a stock on Saturday? Why can't I move money from one bank account to another on Sunday? Imagine not being able to send E-mail on Easter because nobody at your ISP was there to push bits around or whatever these banks' excuse is?
Specifically, a lot of measures to prevent something going wrong, because the the damages are far greater when moving things of great value than an email.
You technically can trade stock on a Saturday, just not using all the aforementioned plumbing of a popular stock exchange and brokerage. And you can move money on the weekends, just maybe not with the help of banks because they consider it not worth their while to offer the service on weekends.
It seems silly that bank transactions, stock trades, purchases of insurance or anything like that requires human intervention. It's like an HTTP request requiring a human to put it through. Bonkers.
But overall, I would bet it’s just a low demand thing that isn’t worth it. Suppose you opened an exchange that was 24/7. Would the cost to trade on your exchange be sufficiently low to compete with the current exchanges with limited hours?
Technically, if you had a paper stock certificate, you could sell it or buy it, and if you had a bag of money, you could move it. But whether or not other people will help you is probably dependent on how much you can pay, which for 99.99% of people, is probably not worth it to just wait a couple days.
Think about real estate. It’s possible that you have a paper deed and draw up some documents and buy or sell it in 30 minutes, but the cost of something going wrong is so high that it makes sense to take a little time and use lawyers to double and triple check things.
Said bank employees would rather have the same number of days off we the rest do.
Never worked with financial transactions but I have a feeling what you said qualifies as 'famous last words'.
Take something really simple like a bank cash transfer. Do you really think that no one ever accidentally puts in a typo in an account number or an amount? If that happens then the transaction either fails or doesn't do as desired. Either way someone has to manually resolve it.
Take that and put it on steroids for securities transactions.
Surely you must appreciate how obnoxious it is when people assume that software development is just “translating the requirements into code”, right?
As a general guideline I have deleted the word “just” from my vocabulary when talking about anything engineering related. Doubly so when it’s a system used directly by humans. And double again when there are legacy systems involved that weren’t designed to have human judgement deleted from their operation.
Here’s a fun example of the complexity even when there are live humans in the loop. I’m Canadian. My client is Canadian. I sent him an invoice using QuickBooks with the option to pay using QuickBooks Payments. He pays the invoice sitting at the table with me and a transaction shows up in my books but no money in my bank account. That’s normal, it usually takes a few days to clear. A week passes and still no cash. I call QuickBooks and they provide me with an ACH tracing number; they tell me that they released the funds days ago. I call my own bank and provide them the tracing number, my bank tells me they have no record of that transaction ever coming to them. I call QuickBooks back and escalate. They insist that my bank screwed up and lost it and provide me with the same tracing number. At least after escalating I now have a direct line to a case manager at QB. Back and forth, back and forth, and eventually I get an email from a VP of Compliance at Chase Manhattan asking that I call them.
Now… I don’t do business with Chase. My client doesn’t do business with Chase. QuickBooks does business with Chase and Chase wants to talk to me about financing Iranian terrorists and FINCEN. Chase happily accepted the money from QB and gave them a receipt for it and then the money disappeared into a mysterious compliance department who, apparently, can’t legally tell me why I was flagged or what I can do to ensure this doesn’t happen again. But after a quick chat about my business and relationship with my client the money magically showed up in my bank account.
I knew someone who worked in KYC/AML at a well known 24/7 US crypto exchange.
Their whole team would regularly have to work weekends to keep up with the transaction requirements.
Then these transactions will be delayed anyway. You could still process 99.99% of the transactions on Sunday.
Visa, Mastercard, Paypal as operate 24/7. ATMs operate 24/7. Neo banks allow intra-bank transfers 24/7. And they deal with the fraud and AML stuff.
The banks can't afford for humans to be involved in too many transactions.
Markets have chosen to concentrate trading to specific hours so as to concentrate liquidity and to help out the professional traders. If everyone's trading 24/7 then it means you need to be prepared to trade 24/7. Having defined market hours means defined hours in which important things can happen. And since the people who work in the market can also define the hours, they've chosen to give themselves reasonable working hours.
In a world where the major markets are trading 24/7, traders need to be at their desk 24/7.
> Why can't I move money from one bank account to another on Sunday?
Instant money transfer means instantly draining someone's bank account with no recourse. Checks have plenty of opportunities to stop the money, so you can write a check on a Sunday as easily as a Wednesday. Wires are more irreversible and so you can only send those when the bank is open and there's someone to double check any suspicious behavior.
It's only the regular Joe's that are not allowed the priviledge to trade at all hours.
[0] https://seekingalpha.com/article/4453440-after-hours-trading
Changes to market structure should at least be analyzed thoroughly as we don't want any "parasitic" strategies to gain from changes.
We have instant 24/7 money transfer in EU/Eurozone using SEPA Instant Credit Transfer (well technically it has "up to 10 second" delay) and it hasn't really been an issue.
Hang Seng still even takes a lunch break.
This argument doesn't actually work; the period during which you can recover a check that shouldn't have been valid is (much) longer than the settlement period. This immediately shows that instant transfer does not mean instantly draining someone's account with no recourse. The recourse period isn't related to the settlement period.
Stock trading on exchanges traditionally has delayed settlement. You don’t need to have the money and stock at hand when you make the trade, but you’re expected to provide them on the settlement date. For a long time this was two days later, but is now being changed to only one day in the U.S.
Trading on Saturdays would either mean complications for settlement, or that settlement must happen on Saturday too. Surely not impossible, but these things have ripple effects that are not immediately obvious.
> “Why can't I move money from one bank account to another on Sunday?”
This exists in Europe with 24/7 SEPA instant credit across banks and Eurozone countries. It doesn’t cost anything extra.
But getting to that point needed a serious mandate from the European Union, years of planning, and cooperation by the various governments. The effort was worth it because it’s a far better solution for consumers than e.g. cryptocurrencies or the American proprietary payment solutions like Venmo.
I'd take my EU strong from an economic point of view and without "yay sunday SEPA transfer" any day over the EU we have today. The EU is a massive failure and it's going to fall into oblivion versus the US and China: especially now that the EU voted to kill its main economy, Germany, by destroying german automakers (with the mandatory shift to EVs).
The EU produces metric shittons of rules and legislation which brings absolutely nothing of value to its citizens.
If anything I don't think the EU is something to strive for: on the contrary, it should serve as an example of what not to do. Even if this means no "yay, instant sunday payments".
…but other than that, what has the EU ever done for us?
https://genius.com/Monty-python-what-have-the-romans-ever-do...
[1] GSM was created as a European standard by the EU
Absolutely, if you discount minor regional conflicts such as Word War II.
Even if you disregard that point, though, the overarching point of my comment still stands.
I'm starting to understand why the sentiments around EU here on HN is completely different from what I hear from anywhere else.
Ah, the famous European continent that includes only Western Europe. Approximately one million people died in the process of breaking up Yugoslavia.
Even if you disregard that point, though, the overarching point of my comment still stands.
The right wording might be "there were not yet a war between internationally recognized acting EU member states". That may still be an achievement, because you cannot say the same for a lot of other alliances, e. g. for NATO.
Now innovative companies like Tesla and BYD come and just eat their lunch.
Of course the engineers that work at German car manufacturers love combustion engines. That is why they chose to go work there in the first place. That is why they had (and have) such a disconnect from the consumer market. Cars are built by people that love cars, not by people that want to get you from A to B. They're people that love the solution, not the problem.
The inertia of big companies starts even in people who are not part of the company yet. It is the people who love finicking with the intricacies of combustion engines that end up at car manufacturers, not the other way around.
you are right thou, solutionists love problems and will even create new problems if their solutions no longer apply.
"Killed itself"? Do you know VW is the largest car company in the world?
German auto makers did that all by themselves. The first Tesla Roadster was released in 2011, it took a whole goddamn decade for the German industry to react.
And now the Chinese are churning out cars at a ridiculous speed.
> This exists in Europe with 24/7 SEPA instant credit across banks and Eurozone countries. It doesn’t cost anything extra.
> But getting to that point needed a serious mandate from the European Union, years of planning, and cooperation by the various governments.
Hm, it also exists in the USA with credit "within a few minutes" across banks (the one time I've used this, it resolved within a second), and doesn't cost anything extra. Getting to that point did not require a mandate from anyone, or government involvement of any kind.
It does seem to have been a four-year cooperative project between several banks, though it is not immediately clear to me that instant settlement was a founding goal of the project so much as something that got implemented after four years.
It’s not available with all banks. Both sender and receiver have to actively sign up to Zelle. And the transfer limits are quite low.
SEPA instant credit works with all existing bank accounts in the Eurozone. You send money to someone’s IBAN which they’ve already had for decades and are accustomed to using for bank transfers. If the receiving bank doesn’t yet support instant credit, it becomes a regular bank transfer that usually clears the next banking day. (It’s like batched email delivery back in the dial-up days. As a sender, you don’t need to care if their email system does immediate or batched delivery. And of course nowadays everyone has immediate.)
The transfer limit is 100k EUR, so the same system is useful to both businesses and consumers.
I think Zelle is limited at $5k last I tried? (When I lived in New York, I tried figuring out if I can pay rent without the absurdity of paper checks. But Zelle’s limit was so low that you can’t pay a Manhattan family apartment rent with it.)
Having worked in the industry for some years: because it's always been that way, and because the people who make financial systems are lazy and bad at their jobs and don't want to admit they were wrong.
It's happening, very slowly, just yesterday I was reading about "the first 24/7 stock exchange" making another filing or something. But the financial industry is conservative and most users are happy for it to be - sure they'll talk about how they want their bank to have a better online service, but they'll keep banking at the big four names they recognise rather than the upstarts that offer better service.
> Imagine not being able to send E-mail on Easter because nobody at your ISP was there to push bits around or whatever these banks' excuse is?
Here in Japan a lot of government systems work like that.
Still, it's an old, large and hugely complex interconnected system made up of literally thousands of banks employing millions of people and several regulatory bodies run by different federal and state administrations.
Such interconnected systems/networks are notoriously hard to evolve, especially given that, as you also noted, people are often extremely conservative when it comes to their life savings. I don't think it's fair to blame the people working in the industry for all of that or even call them lazy.
I worked with some incredibly motivated and talented people too - and saw many of them ground down as basic improvements took literally years to make it through in watered-down form, if they even saw the light of day at all. I was actually amazed by how few people in the financial industry are incompetent in the "couldn't write fizzbuzz" sense, but there seems to be a big-picture insularity where stuff that would be basic and common sense in every other industry (e.g. automated CI) is just not done.
Yes, I guess there are a few precious hours of our lives that aren't crushed under the perversity of the Protestant work cult, we should seek them out and grind them under the heel.
if you want to do it on NASDAQ/NYSE/... then those businesses have decided they only want to operate certain hours
> Imagine not being able to send E-mail on Easter because nobody at your ISP was there to push bits around
emails stuck in a transfer queue somewhere is not exactly a rare occurrence
What is the cost, in money time, attention, engineering effort of trying to build for five 9s of reliability?
That would have several benefits:
- Remove the perverse incentives encouraging the use of the stock market for gambling
- Free up intelligent people who are currently wasting their lives on said gambling to do things that actually benefit society
- Encourage creation of some other way for people to save for retirement that isn't a casino where their life savings could disappear at any moment
That saying there are some markets where auctions do take place, as well as RFQs, etc. They work well in illiquid markets.
If the markets are trading continuously, benefits of parsing new information go to those who parse it the fastest. If the markets trade in occasional auctions, everyone shares the burden (or gift) of new information equally, and no resources are wasted in, say, building a somewhat more straight glass fibre connection between Chicago and NY, then building a sequence of microwave towers, because light moves faster in air than in fibre.
Because stock exchanges don't run themselves. There's all kinds of services with humans in the loop, and without these, you'd have a market with much less guardrails than the current public stock exchanges.
You can make a case for such a market being good or bad for society, but it currently simply wouldn't be legal in most countries.
> Why can't I move money from one bank account to another on Sunday?
You can in many countries! In the EU, SEPA Instant is going to be mandatory soon. In the US, I can already do this between two of my bank accounts; most banks unfortunately don't participate yet.
The larger problem seems to be that the US simply does not have customer-initiated account transfers (to third parties) in the first place. In a way, the reason why you can't do bank transfers on the weekends is that you can't even do them on weekdays either.
Nor does Google, yet somehow they are able to handle running 24/7.
Google doesn’t get in as much trouble if there’s some downtime, especially for consumer facing products
Because for you it means spending few minutes to open your app and send and order, for other market participants it means organizing holiday shifts, night shifts, and working long hours. You just cannot keep the strategies running without a human eye, the lack of oversight could be too expensive. Exchanges and banks and big players can afford that (though still, you're making some people more miserable for unclear reason), but can the small prop shop with tens of people trading on multiple exchanges do the same?
There's a settlement concept called T+2 (soon to be T+1) which is Trade date plus 2 days; this is the day when the cash and shares must be exchanged.
On the Friday morning before last Veteran's Day, we get operations emails from one of the Banks we service that our Settlement dates for the previous day's trades are all incorrect and causing issues for them. I was very confused and very worried that I somehow forgot Veteran's Day.
But... then I realized that it was a Canadian Bank settling US Equities trades!
In their business practice they would settle the trades on a day all the US Banks would not. T+2 for the Canadian Bank was different than T+2 for US Banks. So we resubmitted all the trades with a settlement day of Veteran's Day.
As OP says, managing business calendars is hard for many reasons. I also got really jacked up by a Leap Second, a story for another day.
For the most part you can just buy a Bloomberg feed to get all these calendars, if you can afford it.
Even if you have all the calendars, it’s still non-trivial to apply the calendars correctly for different counterparties based on instrument and function.
And corporate actions are awful. The Bloomberg Data license data sometimes doesn’t match what the terminal has. Bloomberg said they updated it in the terminal but would not fix their bulk data sources since the errors were for actions that happened a while ago.
Your vendor says thing does not happen on holidays or weekends. There might be some other special day where things also do not happen. Keyword is “training day for staff”. Also end of year holidays. They will send you an email about it. Maybe 3. You read the email (important!), and put it in the vendor-specific list of holidays in your code (also important!)
And of course because you like your sanity you use whatever iCal date relativity respecting library to do logic like “the next Wednesday from today” or whatever.
Some things take 1 step. Some take 5. Being able to do something that takes 5 steps without writing a 3000 word screed about the world being broken is the sign of a successful person. Document it, do it, maybe improve it a bit, move on with life.
The point is you don't write the is_holiday function, you might write "is_jp_govt_holiday" or "is_mufj_bank_holiday" (though really it might be more "is_mufj_bank_transfer_csv_acceptance_day" or whatever), that might look at "is_jp_govt_holiday". You write functions to their contexts, in a specific way. This lets you write "hacks" that aren't hacks when you get the specific email. A bunch of tiny, specific functions calling other tiny, specific functions.
5-word functions might look gnarly but they will be specific, and you got the magic of autocomplete to save you some typing.
It feels to me like the article sets up the confusion it must resolve by starting from national holidays... I don't think any of my employers have had a holiday schedule that matched another. There's a lot of variation on what holidays companies do and do not recognize, often with compromises around giving longer runs off around Christmas. If you really need to know when another company is doing business I wouldn't have ever thought there was a solution other than "collect a list of holidays each of your business partners observes."
Hell, a lot of US states don't match the federal holidays.
If anything the banking industry is unusually consistent for practical reasons. But even then, I worked for a major US bank and discovered that our holidays didn't quite match another major US bank we did business with!
When you consume data and trade in global markets, figuring this out is non-trivial and usually process specific.
The next complication then is that these calendars can change. In Hong Kong, we have typhoons quite occasionally between around August and October, and if the observatory declares a T8 or above, several businesses close, including the stock exchange. It then depends on the exact timing of the raising and lowering of the signal whether there is the usual trading, or half day, or no trading, which implies no closing price, thus an additional holiday for the trading calendar.
To treat holidays properly, one must then take into account the day it was declared for some purposes.
- When Dubai was awarded the 2020 World Expo the following day was declared public holiday (yes, less than 24 hours notice)
- In the UAE, the death of a Sheik is often followed by 3 days of mourning where government offices are closed (this can include a lot of office workers one might not think of as "government", as the respective emirates runs a lot of companies)
- When Qatar scored a goal at the world cup, a public holiday was immediately announced for the following day.
- Even planned public holidays, such as Ramadan (not a public holiday as such, but work does grind to a near halt) and Eid are determined by sightings of the moon, and the start and length of these are often announced days in advance.
For Dubai at least, they have something like 90% foreign workforce, so it is easy to find people who do not observe Eid.
But in the context of the article, we are talking banks and offices and particularly KYC requirements. I’m a bit unclear how banks are affected by these “tomorrow is an advocate holiday!” announcements, particularly the banking software.
No techy discussion of holidays should be considered complete without a mention of Eid. When is Eid? When it’s decided Eid is!
This seems wrong? 5 9s is probably a reasonable benchmark or even unattained goal for Google Search, right?
Google search fails for me perhaps about that.
Now to get five nines for Google a bunch of parts below it have to be more redundant and reliable.
https://cloud.google.com/storage/docs/availability-durabilit...
Could be once a year lol
I don’t know if it’s possible to describe finance better in less words?
Anyway, if this quote piqued your interest, you should familiarize yourself with this author’s still incomplete work: https://open.substack.com/pub/concoda
Also: I guarantee you that there’s a team at every major U.S. bank with a crisis management plan ready to kick in when Jimmy Carter finally passes: U.S. markets are closed by custom on the day of a Presidential funeral.
There’s little worse for an ops team than a holiday that you can’t predict.
The Japanese emperor suddenly dying and thus the dating system spontaneously changing without warning?
One of the relatively few bits of ultimately consequential engineering I did in the last few years was assisting a colleague in grepping for comments we had both left identifying code to update in the event of a new era. (We are currently, of course, in Reiwa 6, and Heisei ended in what most HNers consider 2019.)
The absolute most nightmarish situation you could possibly imagine is the overnight batch cycle running late or even worse, having some portion of it not complete. I've been involved in the cleanup of those sorts of issues many times, they tend to be incredibly complicated chains of hundreds of programs from all different departments, and there is not a modern concept of a single cohesive "application" that runs. Its a very crazy world that even web developers will inevitably encounter a bit of if they ever take a job at a bank for instance.
this chap has clearly never worked for a bank
in the 2020s I had to write code to handle files in EBCDIC
I suspect that system producing it will outlive IP, TCP, HTTP and JavaScript
Also quite funny that black friday was translated as a sale on black things in at least one instance. When I grew up a decades ago in east/SE Asia, black friday was not really a thing, but I guess times change.
Asia has adopted a lot of the Western holidays, in the same way that Lunar New Year and Diwali and Eid have been adopted in the West.
It's one of the better aspects of globalization that the world's cultures can mix.
I will add another common source of problems in Italy: every single city or village has its saint, and if it is that saint day on the calendar that city or village shuts down.
But here lies an issue: if you are employed in a different city then you reside (e.g. you work for a company in Milan but you live in Rome), you get the worst combination. You don't work on Sant'Ambrogio's day (Milan's saint patron), but you get to work in your city's holiday, so when everyone is enjoying and having fun you're working.
And yes, this has technological implications of the sort that author describes.
https://en.wikipedia.org/wiki/Date_of_Easter#Gauss's_Easter_...
It's probably the most byzantine date equation I've ever had to deal with.
Technologists are often defeated by their assumptions of how easy it is to solve a problem, but it’s also how we get ourselves in to solving impossibly hard ones.
Of course the natural conclusion is we repeat the same architecture in the name of new ones.
Easy peasy, as long as N is currently >= 2. Changing from T + 1 to T + 0 is a big switch. T + 0 settlement isn't really effected by Wednesday being a holiday in that settling transactions isn't delayed, with the large caveat that entering transactions is.
Maybe processing is not so much dependent on how many workdays a week has?
They make money on the float when money doesn’t move. They make money by efficient compression of business with fewer workdays.
I think this is where he goes wrong. The fact that things stop working overnight means that there is some human intervention required some of the time such that the risk or cost is too great to run continuously.
Same thing happens with porting mobile numbers between carriers. Sure telecommunications systems are computer systems but some percentage of cases must require oversight or intervention enough that it’s not viable to run overnight otherwise they’d do it.
Why not just assume they are not available on any given day unless proven otherwise?
A positive list of what's actually confirmed is much better than trying to find every obscure holiday everywhere on Earth and then adding it to a negative list.
From working at places that do electronic trading, most have some sort of central db, as well as procedures that mostly work for those times when you realize it's a trading holiday or a special session Right Now.
What's worse is when timezones/DST changes at short notice, because that involves rolling tzdb, or shudder the JVM.
This is BS. 5 nines is great for pretty much any system especially if extrapolated over 5+ years. I stopped reading right after this statement as the rest is probably BS.