Hacker News Reader

Comment by mihaifm | Hacker News Reader

This is mostly a PR move. I currently work at DB, I can say that they make it look nice from the outside, but when you get here what you find is a large mess. It's a big difference between a bank and a company that actually ships software. Most of the work here is done in interfacing systems together, and moving data from one place to another. There is no focus on quality, just patching things together and making them work. This is no surprise when a lot of work is done by contractors hired for 6-12 months. Software and tools are old , but I guess this is a know fact. There is a lot of duplication, there are usually 3-4 systems that can do the same thing, when a high level executive decides that then need to create a new one from scratch. Sure there are exceptions, there are people who work on trading algorithms, but these are rare occasions.

These are some of my impressions after some time spent at DB, sorry if it sounds too negativistic.

View on HN

This is not only a DB specific, a whole banking industry works like that. Systems that have <10 years are considered new ones. To keep things working, just produce 5x more services created solely to move data from one place to another. No focus on programming discipline or sense of purpose and constant fear to change anything, because that could break things and we have not enough tests or monitoring infrastructure to even detect that. But in finances, IT is not on a focus. After all, we're just a "cost center" ;)

This is my exact experience while integrating with ERP systems in at least four global firms. Except I'd change the phrase "not enough tests or monitoring" to "no tests or monitoring except the ones I wrote".

It's a fact that you can't ever have enough tests, because, for one, the tests are not tested.

That's funny in practice. But does not have to be a problem in principle.

There are ways to test your tests (in a way that does not lead to an infinite regress), and alternatively you can also prove your tests correct; or just use plain old code review.

Of course the tests are tested – by the code they're covering, which is also covered by other tests.

I heard some banks still did crazy things like calculating mortgage payments with spreadsheets.

Spreadsheets are heavily used in all financial sectors. Personally I've connected many trader developed pricing spreadsheets to etrading systems at investments banks. Last year I was contracting for a large UK mortgage lender who used spreadsheets for all their product dev & spec. Spreadsheets enable bankers to move quicker than bespoke or vendor supplied solutions, hence their enduring popularity.

Any complicated spreadsheet might be a bug ridden mess, but at least it's one the finance guys understand and can augment themselves.

Alternatively they could wait for in-house IT to take a few years and millions of pounds (add going running over schedule and budget, too) to produce a bug ridden mess that misses.

>but at least it's one the finance guys understand and can augment themselves.

And more importantly it enables the "money people" and the "1s and 0s people" to work together throughout the software life-cycle. When the people who input the data understand what's going in and the people who read the output know what should come out the black box in the middle is a lot easier to create and maintain.

Programmers tend to look at shitty programs and imagine better ways that it could have been done... In practice the competitor to a shitty program is not a better program, but no program at all. Even shitty software can still make the world a better place, and the preponderance of half-assed bug-ridden brain-dead garbage software should be seen as a good thing.

thank you. I have the feeling that many participants in this overall chauvinistic discussion here see themselves as artists that always produce perfect outcomes, ignoring the history and productivity of the software actually in place.

And on this positive note, you might want to have a look at efforts to make spreadsheets better: https://www.microsoft.com/en-us/research/publication/improvi...

Hell, we even look at good programs and imagine we could have done them better.

Trading desks and other tech savy groups overuse spreadsheets so that can minimize their interactions with a bank's underwhelming IT department.

Of course, the above is a generalization, but I stand by the assertion most tech savy bank workers try to interface with Bank IT as little as possible.

I once did some consulting for a very prominent bond fund manager. His main tool was an excel spreadsheet which was the most godawful mess you can imagine. Dozens of separate pages with tens to hundreds of thousands of cells each. I ended up writing some VB code to dump all the formulas into a text file where I could parse them and do some automated analysis (using Common Lisp). It turned out that the spreadsheet was in fact as much of a train wreck as it appeared to be. Most of it turned out to be dead code, i.e. >50% (I don't remember the exact number) of the cells did not figure into the final result.

But the worst part was that there turned out to be a single input that completely dominated the final result, and that input was a "gut feel" that the fund manager had about which way the market was going to move. So after all that calculation, the upshot was that the fund investment decisions were being made based on this individual's intuitions, and the entire spreadsheet was just window dressing. Ironically, the audience for the window dressing was the fund manager and his team because the spread sheet was considered proprietary, a closely guarded secret. That's the reason I'm not revealing the name of the manager. I'm probably still bound by the terms of the NDA.

That was the moment that I realized that much, if not most, of Wall Street is a colossal scam.

Not so much a scam per se, but a colossal example of correlation does not equal causation. At least for the old world traders. Newer hedge funds are definitely churning out returns that indicate something proprietary is going on. But anyone bragging about a gain of +2% over the market is kidding them selves.

I can underwrite this. I am currently working for a(nother) bank. And I want to add that a lot of the legacy systems, procedures and thought processes stem from the misinterpretation of regulations.

When a bank gets a new regulation on their table, they tend to create a department that compartmentalizes the subject within the organisation. That's how we get the exact opposite of what the Agile mindset is. And that's also how we get organisations where nobody wants to work in, except if the pay or day rates are high enough to excuse any suffering. And that's how bad, bloated software is created.

This is not banking specific, the whole SW industry works like that. I have worked over 20 years in a couple of companies, where SW was an essential part of our products. Yet the quality has never been good, it's a bloat of solving similar problems again and again. Architecture is a mess, even if there are n architects who can present nice diagrams. (I have seen n=1 and n>30, but the fundamental problem is the same)

To extend my thoughts a bit.

Yes, I believe that some sectors might be worse than others. There are metrics and tools provided by the Software Engineering discipline to measure quality of architecture or code. However, they have remained mostly academic. There is no standard metric, which is widely accepted to be useful. And even if some rare(?) company might systematically measure their code base, they don't put the results in the their home page and not even in the contract with their customer.

As long as that does not change, it's just more or less educated feeling where the situation might be better or worse than elsewhere.

Working for multiple banks in a another country, i can confirm that this might as well have been written by someone here. It's the same everywhere.

I heard Goldman Sachs is a bit more on the ball.

Day to day traditional banking operations will be slow to innovate. Partially due to the regulatory burden as you describe. But also the perception there is little financial upside for the investment.

Outside of that however change is happening. And you will see new products coming out of "Innovations" groups operating entrepreneurially. My colleagues at WF and JPMC are busy experimenting with distributed ledger tech, mobile payments, global micro credit, and other "fintech." And the rollout of Zelle, instant peer-to-peer cash transfers, happened in weeks not months. Literally I was invited to participate in a beta program around the beginning of summer and a few days later billboards advertising the service started popping up around town ;)

It's worth stressing your point on perceived limitations of financial upside. I work for what is effectively a fintech business within a large financial firm and this is a constant hindrance to innovation because investment tends toward short term gains. I think it really comes down to how low the bar is set for software used by financial institutions with respect to originality and pushing boundaries. There's simply not much pressure to do extraordinary things because clients are accustomed to such laborious, outdated systems and processes. I hear a lot of fluffy talk about leveraging modern technologies and being pioneers but when it comes to building out teams and buying into exciting projects, leadership can't seem to find their wallets unless a client has specifically asked for something.

You mentioned that your colleagues are working on things that break this mold - I won't try to speak for their companies, but in my experience those sorts of experimental projects (e.g. distributed ledger tech) tend to be wildly underfunded and consequently don't deliver much beyond an overblown, hype-building press release.

It's probably worth noting that Zelle was mostly a rebranding of an existing service: clearXchange. Or, rather, a branding, as prior to the Zelle name, no one advertised the underlying clearXchange network.

I strongly agree with your general point. Just like any extremely large business, there's small sects pushing forward in various ways, while the vast majority toils away on SSDD stuff.

Sounds just like the investment banks I worked for. The software is just accretion of patches and fixes.

Betas and hacks that make it into production because no-one is allowed to re-engineer things correctly.

Imagine in 2017 still having to run IE6 because there's a web-ui written in a COBOL system back in early 2000 that everyone is afraid to touch. And supporting Solaris 2.6 for similar reasons. You don't have to imagine.

i appreciate the thoughtfulness of this thread very much... obviously, defintely hit a nerve.

and I agree with everything that's been said. i've been talking with a couple different banks (across the US) and they all are sharing the same concerns, whether their systems are VB6 or newer.

i'm with a group of machine learning and software maintenance folks who are trying to build tools to make refactoring/ migration/ speedups easier. if anyone is curious, we'd love to get feedback on the tools, see if they might help in similar situations, or could be improved.

>but when you get here what you find is a large mess.

Had a friend that worked at BBVA Compass. Same deal: huge mess, overworked employees, and awful experience setting up and getting into a workflow.

Somebody told me that at Lloyds Bank, the core standing order system is still written in pounds, shillings and pence. There's just a translation routine sitting on top of it.

It seems more like a move to get people to standardise on the DB code as a means to send trading instructions via Symphony,

OT: Imagine you hire somebody and this person publicly posts that some of your company's work "is a large mess" after you worked months to improve the situation. Even if DB hasn't deserved any credits for this "PR move", is this fair?

Why is a PR move worse than an employee who publicly badmouths his employer while being on the payroll? And why is this tolerated (upvoted) when it's a large corporation?

Edit: thanks for downvotes (this was expected)

It's perfectly fair to point out how things actually look in practice. It's upvoted because it presents another (probably more realistic) picture instead of the marketing spiel that DB wants you to believe.

Why do you not think that employees should be allowed to discuss how their workplace looks in reality?

Reply on news.ycombinator.com