TSB Train Wreck: Massive Bank IT Failure Going into Fifth Day
nakedcapitalism.com
nakedcapitalism.com
- 'TSB gave me access to someone's £35,000' (bbc.co.uk)
- TSB Bank: Botched upgrade has left customers unable to access their accounts
- this
- Warning signs for TSB's IT meltdown were clear a year ago, according to insider
Here's some of the highlights
- Login screen regularly throws a message from the backend saying 'java.lang.NullPointerException' (so now I know what the backend written in)
- My balance was totalled incorrectly, which made me appear to be very wealthy.
- Payments had gone missing from my statement (how do I know they were missing, one of them was my pay)
- When I did login I tried to transfer some funds and was met with 'Payment successful' followed by a java.lang.IndexOutOfBoundsException. Had the payment worked? Who knows
- They have a fundamental scaling issue.
I've never seen anything like it. I do not get the impression this will be fixed quickly.
There is no way in hell I'd be attempting a transfer when the bank is in this state. I'd borrow cash from friends, take out a payday loan, sell the family jewels, anything else.
Even though TSB will pay you back with interest the fact that you've used the payday loan will sit on your credit file and damage your history in the eyes of some companies.
but I have used a payday loan in the past for an emergency when my wife needed to extend her job hunt by 3 weeks and we had credit that was about to go to collections and my guaranteed annual bonus was a month out.
There are some cases where a payday loan can be used effectively to PREVENT damage to your credit history.
And I've also been told (By financial industry credit checks) it being on a report is seen unfavourably.
Since ever internal pages like your accounts, transfers etc had suffix .jsp so that's not a secret.
It's interesting to see that credit cards work fine, but debit cards and online statuses don't. I'm also lucky to have emergency funds on Monzo and in cryptocurrencies.
Edit: I'm prone to idiocy.
I'm sure there are plenty of people waiting to do just that.
Yeah yeah "get a different job" or "you made the mess, you fix it" but massive failures like this aren't orchestrated by the front-line developers, they happen because people many steps removed from the work make bad executive decisions.
A deadline was probably set long ago, and no one adjusted it when unexpected problems and complexities arose.
I have limited experience with financial systems (zero directly working with banks), but I've noticed a trend where institutions are willing to take huge gambles with the integrity and security of their systems if the risk isn't that great.
For them, risk was losing money, and while you'd think that having severe downtime or awful security practices would result in losing money, I didn't meet a single manager that believed this. They were happy to cut corners wherever because it got the job done faster, and under budget, and that's what they were judged on.
A mate of mine worked for a large UK-based bank, and pretty much confirmed this for me when we asked about what it was like to build web services for a huge bank. From the outside, their devs were the cream of the crop. Oxbridge educated, some ex Google or Microsoft guys, and years of experience in building huge systems, but a management culture that saw them as code monkeys, and wanted work delivered that was an inch off of negligent if it meant it was done quick.
The risk to the managers was their job, but from an upper management perspective all that mattered was whether it would directly affect the bottom line in an obvious (non-tech) way. Bugs, security breaches, or poor availability didn't affect this, and were worthwhile risks.
I just left one of these orgs about a month ago where the program CTO of the bank (title inflation) wanted to hire me direct because they believe they need to have a core competency that isn't aligned with external political agendas, but largely the other senior managers just see having their own techs as a risk because they can't outsource the blame.
That and directly answering questions or questioning process upsets a lot of people because they have fiedoms. Like someone asks you for all your IP address you have on cloud and what server they're associated with. They have no ability to handle the dynamic nature of these changing but their job is to write them down so you need to fall inline sorta deal...
so while I can too place some blame on developers it only takes one rogue and a bad change management system to get code out that should have stayed home
And if this job was falling behind, they may have contracted additional developers who came in to do work in a hurry, which can lead to an even worse and misaligned codebase, etc..
Edit: Turns out I missed the most important factor. According to https://news.ycombinator.com/item?id=16939848, they had brought in IBM.
TSB just chose to screw their customers, and make them pay for the problems, rather than spend the money to pay for the problems themselves.
"Fast and Loose" may mean rushing changes through with unrealistic timelines, outsourcing the software development or/and infrastructure management to the cheapest location possible, sacking onshore experts to save money, etc. The UK Government is fully complicit, as their support for RBS (owned by the Government themselves) during their ruthless cost-cutting showed.
Banks are of systemic importance. People suffer if they can't make payments on time or can't access money that's needed for something essential. But the systemic importance is not reflected in the legislation: we have a situation where Banks outsource their IT to Indian service companies and do their upmost to move the "heavy lifting" of software development to the cheapest location they can find, while the social networks and search engines are recruiting the top talent.
The thing is they had the money to throw at the problem. Just keep on hiring young devs at a bit above market rates. Th name would look good on your CV.
Investment banking is actually a lot saner - Thatcher’s reforms in the 80s went a long way towards destroying the Old Boys Network in that industry niche.
Banks also have lots of money to buy political influence. Underregulating an industry with systemic importance invites disaster when component failure becomes cascading failure. But sane regulations, difficult to achieve for any industry, are especially challenging for banking, and then even more difficult to maintain.
I can't understand people with one account, one card, and 10 pounds in their pockets.
See also recent Swedish fears that their cashless society is very vulnerable to a digital attack.
As much as reasonably possible, I keep facilities like personal banking, personal credit cards (always at least two), any mortgage or other secured loan, and any business financial services with different organisations that aren't part of the same group. There's often small print in bank terms and conditions about being allowed to grab money from anything of yours they have access to if there is a problem with anything else, and frankly I don't trust them not to abuse that, particularly if they've already made a mistake that caused a serious problem in the first place. Also, the FSCS compensation limits tend to be per person per firm, so dividing assets can sometimes increase the amount covered if anything really bad happens to the financial firms.
And yes, I do also have some emergency cash hidden in safe places. I don't get crazy paranoid about it, but it's a substantial amount, certainly enough to fill my tank and buy essential provisions for a while. I would be concerned about becoming a truly cashless society as well, for this among other reasons.
User friendly error message of the year!
[https://twitter.com/thejackthomson_/status/98856435451268710...]
It doesn’t appear they were involved in the original migration.
As far as I know Monzo is using microservices (a bank is probably big enough for that to be the right choice) to try and limit the damage of anything insane like this. The way they handled Prepaid to Current Accounts was absolutely seamless https://monzo.com/blog/2018/04/05/how-monzo-to-monzo-payment...
Monzo/Mondo had issues on their own, and in fact they have very frequent issues with their payment processors.
The current account (now the only account as the prepaid beta programme has ended) runs on their internally developed payment processor, and has so far had minimal issues.
More information here: https://monzo.com/blog/2018/04/04/ending-prepaid/
I know someone who interviewed at Monzo and said the interview process and technical test (write a web crawler in Go) was generally a pretty good process. However they "noped!" out of there when the interviewer said that they don't do unit/automated testing, and it's the developer's responsibility to make sure their code works.
This was a while ago, so hopefully things have changed or hopefully the interviewer was not telling the truth (perhaps looking for a reaction?). However if it is the case then they're about as "modern" a bank as all the rest of these legacy banks, or at least will be in a few years time when their technical debt starts to catch up with them.
The same thing happened with Tangerine.ca when they launched their new website late last year. Tangerine is owned by one of Canada's "big 5" banks, Scotiabank.
For at least a week after the launch, users were experiencing login errors, timeouts, incorrect balances and features not working. Backend code exceptions would bubble up to the surface. It was ugly.
But even with the backend issues ironed out weeks later, users continue to lament about the horrible new UI, which over 6 months later, is still a UX nightmare on desktop web browsers.
Reading the 400+ Facebook comments on their "launch day post (they since removed the post from the timeline) is quite an eye opener.
Apparently they did minimal testing and disregarded all user feedback during the beta phase.
The comments are still accessible from the post photo here: https://www.facebook.com/TangerineBank/photos/pb.46798899664...
EDIT: Nope, I'm wrong. That would involve sending sensitive data over unencrypted channels, which is a no-go.
Personally I wouldn't care and I'd want that, but that's irrelevant.
Why do you think that? My inbox is plenty of (unsigned) PDFs coming from mx<n°>.<mybank>.eu ...
Headers show they're relayed from <redacted>.internal.<mybank>.eu (private IP included), that spf authentication is ok, but dkim is not even provided.
The GP was talking about encryption: can anyone else involved with any system your email passed through read your intimate financial details?
TLS means in transit interception is not easy and SPF that the address is not spoofed, but without DKIM I'm not able to verify that the content is not forged. As such, for example, my email provider is able to read those emails, and probably to modify them without me noticing.
If something doesn't grant confidentiality, data integrity and authentication, it's not encryption. I don't think TLS alone is enough to qualify an email as an "encrypted channel".
In short, banks sending out PDF statements by email would be dangerous, as sillysaurus3 originally implied.
I'm not trying to argue that it's ok, I'm saying that in my experience bank employees use email for confidential communications.
It’s lower effort for me to take an envelope and put it in a “2018 statements” folder than to go download statements.
It’s also easier to throw out that folder in 2020 than to sort out digital documents.
More seriously, I would definitely move my money somewhere else if I had an account there. Not because I would worry that it will disappear (government insurance & all that), but just because this looks like the kind of problem which will take a long time to fix, and which could strike back.
Furthermore, there's a lot of process for switching banks that can be done by the customer through the bank they're switching to. While your savings may be stuck in TSC for some days, you don't need to go to TSC at all if you'd choose to switch your wages and recurring bill payments to somewhere else, you just go to the new bank.
I've switched banks over far far less. I don't know how TSB has any customers left when this is done.
Like what? You can't take cash out from ATMs or from bank offices, you can't login so you can't make a transfer, some people can't use debit cards... so yeah, run.
Are they using IBM technologies for their stack?