Deutsche Bank's “dysfunctional” IT division (2018)
efinancialcareers.co.uk
efinancialcareers.co.uk
I can't choose a password for my account; instead they assigned me a 6-digit PIN (Not kidding. A PIN).
When I registered my account at a local branch I gave them my dutch phone number. Their website's transaction page wouldn't load due to a JavaScript parsing error over the dutch phone number format. I called customer support and they basically reacted as if water was burning. They would promise to get back to me. 3 weeks later I still couldn't access my online bank account's transaction page. I ended up eventually manually editing the HTML in inspector in order to be able to navigate to the page where I could update my phone number to a German one and I never heard back from support.
I tried opening a depot account for trading stocks. It just says "Internal server error. your account id is incompatible. Please open an additional account with Deutsche Bank". I then tried this; had to go to KYC all over again; but I can't because I can't register two accounts on the 2FA app that is mandatory to have on my phone. Customer support adviced to get a second phone (??).
and I'm just a normal customer. It's impossible to bank with this bank. Every time I have to interact with them it feels like i'm in some weird fever dream.
How they describe it: https://www.bundesbank.de/en/service/reporting-systems/exter...
Any single transaction exceeding 12500€ entering (or leaving) Germany has to be manually reported here. So for example if you're living in Germany and working remotely as a contractor for a US based company you'll probably have to report your income every month.
There are many exceptions that don't need manual reporting though, like if you're buying physical goods from overseas. I guess those are already automatically tracked through other ways.
For fun, I just tried to login at ~9:30 PM local time on a Sunday, and it said:
"Access Denied"
"You cannot access the resource you have requested at this time."
"The resource you have requested is protected by a policy restricting access to specific time periods. Your request has occured outside of those permitted time periods."
My favourite thing on that website is the script block inside a noscript tag that they actually expect to run. Things like that tend to suggest the developers have absolutely no idea how to code for a web browser.
I mean what else should they use?
works okay but the credit card integration is a complete sh*t show
Reality is much mundane and boring: the person writing the web app has no vested interest in the app actually providing a good service for customers. His or her boss doesn’t care either.
It’s simple disinterest: the self-interests of the coders are too far removed from the interests of the customers.
If you’re a solo dev working on a startup and every lost customer is money gone from your pocket, you will pay attention to every little detail.
If you’re a cog in a giant soulless machine, you do the bare minimum and go home.
To give a few examples:
- Running all services under a single Linux account, then requiring vendors to support password encryption in the startup scripts (everyone can access our production passwords!), then proceeding to store the production password in plain text in GIT and Confluence where everyone could read it.
- Everyone (including contractors) had access to the production environment through the testing environments, including unrestricted access to the production database (without requiring as much as a password). This was not deemed nearly as important as the passwords in the point above.
- I received an official warning for disclosing to AWS/GCloud that we did not have CI/CD setup but would be interested in getting one up and running at some point. This was considered to be "sensitive" information.
- Ansible deployments taking 12 hours (on a 6 server environment) because as it turns out, some random "security" settings slowed down SSH, incompetent people not really understanding what they we're doing did the rest.
- Ansible being blocked because someone in the security department read that it only required HTTPs access, turned out that they were reading the documentation for the WebUI.
- Requiring manual deployment of PR's to test environments and manual testing, the "evidence" + logs of doing this was to be documented in a Word document and attached to the PR.
- Automation and DevOps was banned because "The law does not allow us to do this"
- No rsync due to security reasons, great fun when you had to sync 200k small files :)
- PRs were kept around for months, so "senior" developers could estimate the risk. This included fixes that were hotfixed in production so that it was guaranteed to break again at the next 12 hour deployment (thus giving the same developers even more reasons to be even more "cautious" with all future PRs)
- Three different database schema's, three different database technologies, microservices, a React frontend, 50 outsourced developers for an internal CRUD application with 2000 entities, to be used by 10-20 users
- All .exes not on a whitelist were banned, so in order to run your application you just had to rename it to word.exe :)
- At one point external developers were hired, we requested a desktop for them that would allow them to "write" code. They got a locked down VM with notepad installed and nothing else
Both founders are German speaking. Good guys.
Although that was a small firm not a giant
The firm in my post above is very much a Swiss firm though, you've probably heard of them, they're not a bank.
Lack of talent is not a national issue in Germany. German unis and schools produces great devs, along with the slef-taught "hackers" and tinkerers which are quite numerous there. The whole "talent shortage" in Germany is propaganda from shitty companies engaging in wage suppression, and actually a shortage of pay that keeps decent devs away from such companies.
It's more the lack of good pay/equity to motivate people, but the main issues are the immovable and outdated bureaucracy without any wish to address it, and the heavy top - down approach where top management is clueless and dysfunctional, scaring away the top talent over time and ensuring that only the equally clueless yes-men stay, and the result is what wee can see.
These large German "ships" are only good at bureaucracy, maintaining the jobs and status quo for those at the top, not innovation, efficiency(despite the stereotype) or customer satisfaction.
My favorite about German companies is project management done in Excel, and bosses who don't want to use Jira and demand their seniors send them emails explaining the current work status. You can't make this up.
Just kidding, nightmarish as it is, beats emails and excel for sure.
Classic strategy :D, those aren't seniors: they're Gods.
I want that on a shirt lol
- technologies had to be evaluated to be allowed on the network. This process took years. Git (as a replacement for subversion) had been in evaluation for over 14 years at the time.
- network authentication mechanisms made building an http/s Webservice internally impossible without the security teams approval which takes months.
- ironically the 'rules' around 'security' were regularly broken and not audited when it was convenient for someone with the clout to make something happen.
From a domain like db-service.com, only text “there is a problem with your account, please call xxx”, with xxx a number that was nowhere to be found on their website.
Turned out to be a legit inquiry from some audit department.
When I asked support why I have to pay for my depot (even though its free if I have a checking account) they said they signed me up under two separate customer IDs, but they cannot simply consolidate them. I have to open a new depot and transfer my stocks.
This is a property of large, bloated legacy companies in any field. IT is still seen as a cost center and a secondary concern rather than the enabler of their business. As a result, pay/resources and "political capital" (for the lack of a better word) are allocated accordingly.
IT folks there aren't given the pay nor recognition they deserve, so no good talent joins or stays for long enough. Junior talent that joins ends up just learning from the mess and has no chance of actually becoming "good", so the problem continues.
Furthermore, the messy and unefficient IT systems benefit many people there, from lower-level menial positions whose jobs would be obsoleted by good IT to managerial positions who have a large list of reports to manage which gives them prestige and justifies their salary. Third-party suppliers also benefit as a bad IT system requires constant attention while a good system would require less attention (and a competent in-house team can attend do it, requiring no third-party involvement). Bad IT can also serve as cover - problems can be blamed on it instead of incompetence.
Fixing it incrementally from inside is politically impossible as people who rely on the status-quo will fight you on every step of the way. The only potential way is the organizational equivalent of a "full rewrite" - set up a subsidiary, give it unlimited money and task it with building a competing product. Operate it like a startup with the appropriate culture (especially regarding tech). Once the product is competitive, migrate customers onto it over time. This should be feasible at least for retail banking as UK fintech startups proved it's not actually impossible to create a bank from scratch. Rinse and repeat for every vertical of the business.
My learning here: new is not new if the culture stays the same. Also: never underestimate the power of old. People always talk about the new, shiny stuff. But old was there first. And is much more resilient than it seems.
* paying contractor rates to thousands of people
There is also a large cultural component. Having lived both in The Netherlands and Germany, dealing with banks or government institutions is a stark difference. The large Dutch banks (eg. ING) are efficient, you can arrange everything online and they have modern apps. With German banks for every small administrative change we had to go to a physical office and the apps/website looked as if they were literally from 1999 (and weird security practices like using a pin to log in). Similar with government institutions, in NL almost everything can be arranged online with a single authentication method (everything from requesting a parking permit to filing for taxes). In Germany it's a lot of literal paperwork, going to offices, waiting in line, and no integration between administration of different public institutions.
My wife (who is German) says that one factor is German history (nazis and the stasi), which makes people distrust any kind of central (digital) administration. So all databases are uncoupled from eachother, people prefer cash, etc. Another thing that is really stifling in Germany is the hierarchy, there is so much formalistic bullshit around company/government/... hierarchies. I experienced this a lot when working with the university administration (I was a researcher/lecturer at a German university). No one dares to make choices that can have some impact, afraid that higher-ups, the court, or whatever will punish them for making the wrong choice. So they hide behind rules and necessary decisions are not made/postponed. Almost no one stands up to their leadership, so even they have wrong ideas, there is rarely any counterbalance.
True.
> Another thing that is really stifling in Germany is the hierarchy, there is so much formalistic bullshit around company/government/... hierarchies. I experienced this a lot when working with the university administration (I was a researcher/lecturer at a German university). No one dares to make choices that can have some impact, afraid that higher-ups, the court, or whatever will punish them for making the wrong choice. So they hide behind rules and necessary decisions are not made/postponed. Almost no one stands up to their leadership, so even they have wrong ideas, there is rarely any counterbalance.
This depends a lot on the company, and is something that you, in my experience, mostly experience at big companies, government-run companies and government agencies (this includes university administrations, even though at many faculties, things are completely fine (even many professors prefer to keep away from the university administration as far as it is possible in their job)). So, if you really don't like this, attempt to look for a smaller company as your employer and/or find a job in the private sector.
I can hazard a guess why this hasn't happened in Germany but I don't want to make assumptions based on stereotypes, lol.
Culturally though, older Germans mistrust then and want to use the older, established banks with physical branches.
In the case of N26, the exploits that CCC found were disclosed and fixed before the presentation, this was also a while ago now, 2016-17 maybe? I'm not sure what the situation is now, though.
Every management team I have ever worked for in banking - I have worked 4 stints in 4 different banks as a contractor - has been so old school that its been impossible to apply any up to date constructs from the professional software world.
I once had a manager who had been with the bank for 40 years. He simply drawed 3 boxes on a board, drawed a few lines between them and asked "how long will this take ?"
Hmm...what is in the first box?
Old school, grey and stale. That's banking for you in Europe at least.
Recently there was an issue where a field overflowed which broke literally everything. Then they added a zero to the start as part of the specification which broke it even more because half the companies on the planet were treating it as a string and not a number.
If there's anything I've ever learned about working in the financial sector it is that the only thing that is protecting you is the legal framework around it, not the banks or financial companies which are universally a fucking shit show.
So, Deutsche Bank is still crap at IT.
[1] https://www.anwalt.de/rechtstipps/erneute-probleme-bei-n26-b... [2] https://www.businessinsider.de/wirtschaft/einem-n26-kunden-w... [3] https://www.businessinsider.de/gruenderszene/business/n26-er...
I tried Finom, but the experience was abysmal. They behave like some sort of shady porn subscription, not a place I would want to entrust my money with.
I’m not sure about giving money to these Neobanks. They’re not really cheaper, and have so many downsides.
It has gotten that bad that the regulator is starting to get interested [1, German]. Luckily I was not hit by the more serious problems of missed payments. Really "funny" to see that nothing has changed since 2018, I would have thought this is a current article if it weren't for the year in the title...
[1] https://www.tagesschau.de/wirtschaft/verbraucher/umstellung-...
P.S. for the Germans looking for a privacy.com alternative (not available in Germany): Revolut took less than 30min to setup from scratch to first CC payment, not affiliated or anything, just really happy with a financial service not stuck in the past
Carron Energy 2004, Cardiff
Better Payment 2014, Berlin
United Financial Group 1981, Garden City
Quantiguous 2014, Mumbai
Given all of these needing to integrate together to some degree, and the bureaucracy of banks in general, add that to their resistence to change / improve something related to the tech if it still works, none of this surprises me.
You see these kind of problems in all conglomerates which have to absorb other companies into their ecosystem.
Edit :
Slightly unrelated but I have just seen that she has passed away last year from cancer.
https://www.fnlondon.com/articles/former-deutsche-bank-chief...
Edit 2 : my mistake of details
Their IT was as diverse as it gets, and then they were told to move into one building and "make it work".
It's as messy and as broken as you imagine it is.
Yes because one division is sending the data in XML and a 3 letter county code and values in thousands of monetary unit and another system does things in a completely different way
It's not necessarily ungrateful. It can even be interesting, but it all depends how everyone approaches it and the most important thing ever is: realistic timeliness and outcomes.
Understanding semantics is required, not just form.
These banking systems are all massively complex one off systems, often with very different sets of functions or even definitions of the problem, and the implementation details are different in almost every way possible. They’re the exact opposite of the sorts of problems modern trained AI systems are good at dealing with.
Changing data types in an interface is a boring job (didnt Office Space joke at that?) but doing this via some broken AI will probably introduce even more spaghetti.
Reality is that those systems should be killed one by one and finally when there is one system, it should be upgraded to something better. What is very difficult.
> Reality is that those systems should be killed one by one and finally when there is one system
It take so long to replace each of the systems, by the time you get to the "one true solution" 40 years have passed and you are back where you started.
You’ve misread the article, she was diagnosed with cancer on 2003 aged 35, and died this year aged 55.
DB happens to have both a broken engineering culture and (at least historically) an even more broken culture of mis-selling and fraud, but there's nothing to say that one always engenders the other.
The view on technology and infrastructure is very primitive... as you can also read from just the header "IT division". I initially thought that they were competent but greedy turns out the banks are really just greedy, naive and sometimes... plain stupid.
I don't know if it's unique to Europe, but I am never working for a non tech-first company again.
> I don't know if it's unique to Europe, but I am never working for a non tech-first company again.
For what it's worth, Goldman Sachs is reasonably competent in tech, and used to be even more so (at least in relative terms).
Bloomberg is about half-way between the banks and the more competent tech companies.
(I worked at both, and a few other financial companies and also at Google, Facebook etc.)
In Europe, tech is seen as "just IT" especially in non tech corporations. It's much less important than in the US.
It's a pretty minor thing but personally I think just the fact that I rarely see the distinction between "software engineering/development" and "IT" in Europe is pretty telling.
The salaries are also another indicator.
It's so bad that they're losing serious business on this despite being competitive on fees and service.
Bloomberg is great if you're big and bring a lot of money to the table. For any small/mid-size firm (think less than 5M ARR on Bloomberg licenses) the answer for even basic requests is often "we'll file an enhancement request" only for things to disappear and never get adjusted. To give an example, it's still not possible to get an hourly export of the trade history in CSV or XML on a SFTP server. There's workarounds but it's definitely not half-way competent.
I last worked there in 2018. So my experience is a bit out-of-date.
> Bloomberg is great if you're big and bring a lot of money to the table.
My experience is from working inside of Bloomberg and trying to get stuff done. I don't know how it looks like from the outside.
Though I can tell you that eg Google was more competent on the inside when I was working there (in 2014 to 2016); but from the outside they are probably not easier to work with than Bloomberg?
This is why fintech (although a very overused word by now) is such a breath of fresh air in the banking industry, and desperately needed.
> "Hammonds was innovative, but she couldn't do what she set out to do," says one DB insider. "So we're just sitting here, waiting to see what happens next."
Yikes! Sounds like an impossible uphill task.
Is there any examples of such turnarounds in the recent years?
If you ever visit Germany, prepare to not be impressed by their Internet access (but possibly impressed by everything else)
Apart from energy independence and defense.
A culture in which software people are considered to be really low status is pretty much the global default and is still common today, even in the US. I worked for American finance types before. They couldn't understand why programmers earned so much and why their software team couldn't be completely outsourced to India. They took perverse pride in having absolutely no idea what their own software stack actually did. Last time I checked they had got rid of the only people who were any good and then outsourced the rest (to India, of course), and apparently lost the ability to ship new versions of their software in the process. Even in the US many investment banks have totally dysfunctional IT, with the possible exception of Goldman which is famous for actually being good at it (although from reading the comments in this thread maybe that has changed?).
Why is there such a thing as a "tech" firm when all firms use tech? It's because tech firm is really meaning a firm created and run by programmers, as that's the only environment in which they can get respect and a productive setup. If it weren't the case companies like Amazon could never have existed because they'd have just been crushed by other better established retailers doing the internet well.
Investment banking organizational structures are set up such that bureaucracy encourages 'silos' where teams can deliver quicker if they don't depend on other teams, and performance related pay dominates the managers (managing directors) thinking.
Having worked at Megabank, it ran at least one system on SunOS (not Solaris), well into the late 2010s. It had Windows XP, long after end-of-life.
Nothing about that story is peculiar to DB.
I'll raise you a company-wide ticketing system written in COBOL, running on an IBM mainframe.
Once they are done with all this standardization, they will realize they just have new roadblocks, and will spend a half decade trying to fix those too.
One of my favorite stories to retell is my wife had a job there also and she had a medical issue from a car wreck. Her boss said the bank couldn’t function if she was doing physical therapy two hours a week, basically fired her, all the paperwork was drawn up, sent to a manager in London to sign. The manager in London was “too important” to sign things so he just leaves it for someone else to rubber stamp in his name the next day. Meanwhile my wife goes across the street (literally), gets a doctor to sign her disability paperwork, walks it into HR - and instead of doing physical therapy two hours a week she was on paid medical leave for almost a year. Her termination paperwork did get signed eventually but because she was on a protected leave they had to throw it out.
Just FYI.
I suspect the problem in DB is not it's aging, disjointed IT infrastructure, but the culture.
It paid well but fuck that noise.
I interviewed at DB around 2018, and from that experience (what's the worst thing about working here, I asked my interviewer. The politics, she replied) plus that of all my friends who spent time there, this article rings pretty true.
But I'd love to get some more up to date info, so would be happy if you could share.
Kannon isn't even mentioned in the older article.
Note that I am not claiming that Deutsche's IT department is no longer dysfunctional! Just nothing in TFA provides convincing evidence about the current state of affairs.
Though in that case half the mess were from mergers leaving the bank with multiple instances of similar yet incompatible IT systems. Another source was the wave of regulation and reporting requirements crashing onto finance after the subprime crisis.
It turns out that they were mixing up that ticket with another one I bought, and I only managed to find that out after almost 1 month of trial and error talking with their staff, which pretty much resembled a chat-bot experience.
We put ourselves in a sort of quicksand. Technology's promises are tempting, but there are hidden costs. We didn't know that and now we're finding out what happens when complexity is not tamed. Lessons are being hopefully learned, but I'm afraid that what we currently see is just the beginning of the nightmare
Did NOT leave me with a good impression of German IT.
They are horrible places if you take your craft seriously.
I had no idea 45 different operating systems still existed that were functional
They’re talking about the “banking” operating systems.
Also note that Deutsche Bank (like many global financial institutions) is not really a single company, but more like 10 different companies all operating under one umbrella.
It’s all very well pointing at a company like Goldman and saying “look, they do IT well” when they don’t have the same range of commercial and consumer businesses as part of their company. If you’re just an investment bank, life is a bit simpler.
On a more general note I feel like in our industry there’s a tendency towards “perfection worship” where people think there’s some kind of immortal “perfect solution” that a company needs and anything less than that is terrible and must be chastised.
This is egoistical and forgets that IT and computing exists to enable companies and not the other way round. The world does not do computing for computing’s sake. Company change can be almost constant, changing systems to keep up is hard and expensive, being able to live with imperfection is critical to commercial success.
A standard part of an acquisition is integrating and migrating IT over a defined time period, so you don't keep accruing and perpetuating technical debt. Who were the CIOs at DB?
You probably don't want 10 different desktop hardware providers and IT support systems, but it might be a bad idea to try and unify the trading systems for slightly different looking business units unless there's some competitive advantage to be had by doing it.
There wasn't a CIO at Deutsche when I was there, it all fell under the remit of the COO who was this guy: https://en.wikipedia.org/wiki/Hermann-Josef_Lamberti