Building a Modern Bank Backend
monzo.com
monzo.com
To me creating a back end banking system using a load of software that's very new and still very much developing is a risky move. Once they're in production bank back ends tend to be around for a LONG time (in my experience in UK FS there were a load of old back-end systems that were maintained out of necessity as they were too expensive to re-write and the product they supported was still in use)
To me banking back-end systems should be boring from a tech perspective. Use well known components where the limitations and downsides are well known, and also if it were me I'd use components that I could get a support contract for. You don't want to be relying on open source support at 3am on a Sunday when you hit some bug or other and the FCA is breathing down your neck to get your transactions processed in a timely manner.
* Or whatever the UK equivalent is.
Strange that because banks have been using relational database technologies for that task for years.
Any internet company wouldn't even make it past day one if they need to sink a few hundred thousand just to get started.
Commercial support: send email. You get an automated message back with a ticket number. Around 6:30 AM you get a call from Delhi that satisfies the 4 hour response SLA, and the support analyst says that they will look into it. At 11 AM you get a request for a dump and certain logs, some of which you had already sent them...
Open source support: after getting zero google results, you look through the source code while running a debugger. It takes you two hours to find the place where it makes an assumption that a signed variable will never be negative, an hour to test your fix, and five minutes to send a patch to the mailing list. It is coming up on 6:30 AM, and you conference in the two people who need to approve emergency changes in order to get that approval. (This is a bank. You can't unilaterally make a production code change.) Everything is deployed before 8 AM.
From friends' experiences, Microsoft and IBM support can be very very good for the right price.
That said, I know at least some financial institutions use their fair share of open source tools. They're often best of breed.
Normally it goes for India for a few hours where they rattle off something that is in the knowledge base. Then when that doesn't work it goes to the next tier which is some support person in the US who does exactly the same thing. Then finally you end up with a support engineer who is finally technical and will usually do a screen share to walk through a solution. But if it's a bug you will have to wait for an actual product engineer who will finally start looking through code. I've seen this process take days.
And yet people are acting as though the big RDBMS vendors are somehow a good fit ? Crazy.
3 AM on a Sunday, you discover a critical bug in a library provided by a third party.
Commercial support: You call them. It takes an entire day of awkward back and forth emails and phone calls, but you eventually get an engineer to send you a a patched library for testing. It works.
Later, when auditors ask you about vulnerabilities, you can ensure them that you track $VENDORS supported long-term releases, and your contract has a security and support SLA.
Open source support: you find nothing on Google or Stack Overflow. You haven't looked at this source since your initial code review (you're a bank, you review all third-party code in use.) It's 3am, you're tired, and none of this is making sense. You've posted in the projects IRC channel but nobody's listening, and there's no good responses to your posts to the mailing list. You have to pull other team members out of bed, and together you get a reasonable test and patch implemented, which for some reason the maintainers aren't accepting quickly (if ever) and now you're running a local fork of your library. Hope your internal security auditing practices are solid, because there's no guarantee you didn't accidentally create a vulnerability during your pre-morning hackathon.
---
I don't love commercial software typically, but the support and auditing benefits are frequently the overriding factor when choosing for a regulated industry.
I'd be genuinely interested to hear, as it's something I see bandied around a lot but extremely rarely followed through on as code reviewing a modern stack is a huge proposition, and manual code review is very expensive/time consuming.
Your open source case is also perhaps a touch optimistic, it implies you have technologists capable of debugging and patching every single part of your tech. stack.
As companies grow and need more software and also as the tech ages that gets less and less likely to be the case in my experience.
Starting with a 'team of 3' and building a fairly fancy system with a lot of open-source 3rd party solutions ... making all kinds of assumptions about cost, scale etc. - this is worrying.
In finance and banking, I think that 'simplicity' 'reliability' etc. are the key issues, and it probably should feel boring. I can't imagine that the 'cost of servers' is going to be the key issue, especially since these things are pretty cheap these days.
When you look at how much a startup invests in various things ... I'm not sure that server cost is an issue for anything but entities like Netflix etc. wherein you have serious scale.
Messaging startups like Kik have 200M users and I don't think they are that fancy, granted, it's not messaging.
I hope I will never have to rely on a bank that uses this architecture. It seems they are vastly underestimating the complexity of distributed systems and the range of their failure modes. It's all cool if you're building a web site, less so if it's about your money and being unable to pay at a gas station.
It's just that rather than have a single monolithic app in front of it they have broken it up into a suite of smaller, independent services. And many of those services will have their own database which contains a transformed, optimised slice of the total EDW data.
Everything around that (trading models, development, research systems, reporting) can be microservices, but it will all rely on the core system.
You seem to be implying that you know something that the very talented developers I worked with don't. Be curious to know what that is.
How is that true? These companies have shown that you can scale well if you don't have monolithic approaches, but that doesn't mean that you cannot scale if you have one. Just because someone is successful by using approach B doesn't make approach A unsuitable.
Quite the opposite. Some banks handle more than a billion transactions per day on monolithic infrastructure (think of big investment banks with a sizeable amount of trades per day). It's very costly, but it does work reliably.
EDIT: One thing I don't get: Most banks use a central database somewhere because you need to have accounting values that you can rely on (e.g. how much money you have on your account). How do they do that? They must use some synchronization with locking that works across all services which rely on them, but how will that scale well when they get dozens of transactions per second? How do they guarantee the integrity of their data?
Most interesting was "...Deploying Kubernetes in a highly available configuration on AWS is not for the faint of heart and requires you to get familiar with its internals, but we are very pleased with the results..."
Wonder how much pain and suffering there was getting this to where they wanted it.
The only part I was concerned about was 150 microservices so early in the game, moving to message queuing while still very small, and the RPC/locking stuff. Smells like over-engineering. But hey, I'm not there. There's a ton of stuff that goes into making decisions like that. Thanks for the story!
You see that as a good thing?
It's a difficult situation. I don't want to spend all day reading an article defending CoreOS. But if you throw that buzzword out, I'm assuming you've had some good discussions about your OS selection and could provide that depth if needed. Same for the others. They're not right or wrong choices, but I'm assuming they're there not because the team wanted to be trendy, but because there was real architectural need.
If we're discussing architecture, by necessity we're going to have to skim over a lot of stuff. I was okay that given the team's desire to prevent a single point of failure that the pieces listed worked towards that goal.
It's very easy to put too much stuff into a system. It's also very easy just to do trendy stuff without thinking through the consequences. I'm giving them the benefit of the doubt until shown otherwise -- and at the end of my comment I was clear about what some of my concerns were.
I would argue that as long as you don't actually lose any customer's money, its OK to screw up once in a while. My e-bank site its down once a week for deploys or so, my bank's mobile app didn't to certificate pinning, and they are still operating.
Actually I believe CAP isn't actually that relevant, because most money is transferred and there are double ledgers for that, which for sure isn't fully consistent.
As for the paid software and support, its a crap reason. The support guy isn't omnipotent, at best he's seen the issue you're having and will help you faster. If you have paid software and you have downtime because of them, good luck trying to get money from them.
And there's nothing wrong with microservices, as long as you don't need to call a bunch of them to update my balance. I don't care if I try to change my address and the service is unavailable, or if I don't receive a weekly notification. Its much better than having e-bank site downtime because they want to deploy a new version of the webapp.
As long as you maintain account balance integrity, I'm sure you guys will do just fine. All the best
Unfortunately banks in the UK are regulated by the FCA and they are more than a little picky about that kind of thing. As an example RBS were fined £56 million for an outage a couple of years ago (https://en.wikipedia.org/wiki/2012_RBS_Group_computer_system...)
I would have thought you could start with a bog standard CRUD setup. How many customers are they serving at the moment? How many queries is one customer going to make each day?
What part of a bank are they actually trying to make? Just current accounts? Or multi asset market making? The requirements are quite different.
Also, on the business side, why does the UK market need another bank? It's a totally commodity service. Why would I want to use this bank? Do they offer something better?
-edit-
Oh and I forgot to add, they offer a complete API, so you or third party developers can build on top of your financial data. No other bank offers this today.
this part ain't completely right - all current banking backends on the market offer instant messaging integration, otherwise they wouldn't be bought. it's not straightforward API calls directly in the backend (which some would prefer to not expose directly anyway). a bit of glue code is required but this is how in the moment these systems evolve to conform to ever changing regulations for example. otherwise banks would have to shut down
Best and easiest security to achieve and maintain is no public access at all (something like physically separated networks vs authenticated ones)
The alternative to open APIs is that third parties ask customers for online banking passwords. This practice is significantly less secure than, say, OAuth2 or similar and is generally against a bank's TOS, which makes the customer liable.
My bank: I look on their website which says I can phone up and do it over the phone. I phone up, go through the automated options and then I'm told that actually I have to go into a branch with ID and proof of address because apparently I'm not registered for 'telephone banking' (why?!).
The nearest branch to me is open 9-4.30pm on weekdays and 9-1pm on Saturdays. I generally work at these times so I have to take time out of work to go into a branch, queue for 10 minutes and eventually get it changed.
Monzo: I open a live chat within their app, answer a couple of security questions and the address is changed right away. It took about 5 minutes at most and I could do it while working on other things.
My bank also charge a small fortune for foreign transactions and contactless won't work abroad. Monzo is free and contactless has so far worked fine in 3 different countries.
The UK banking industry is massively behind the times with regards to customer service, facilities and technology. The potential for Monzo is huge if they can pull off what they're trying to achieve.
I can also imagine that changing the customers' address without proof of address could violate KYC/AML rules.
So while I agree that banking in the UK can be a pain (although less than in other countries), I think it's mostly due to legal requirements, not bad intention.
With interest rates that low, retail banking is all about cost cutting. They could save a lot more money if you'd never have to visit a branch, but verifying an identity via chat is not easy.
Obviously banks have a lot of regulation they have to comply with but I think a lot of it is simply a case of "that's how it's always been" and a lack of competition due to the high barriers to entry.
It is also the only service with a mobile app that doesn't look and feel like a complete shit show - usually because they are web apps being wrapped in a tiny bit of native chrome.
Is it commonplace for a bank to be a British Ltd, and are they part of a separate banking emergency fund to accommodate for that because they have a banking license?
Also where do I have to look to find a real world address for Focus FS Ltd (company behind Mondo)? Looked around the internet after finding nothing on monzo.com. I know startups do not list contact info these days, but whois info is also opaque. I'm asking this because they advertize themselves as a modern bank, but I cannot find a real world address after 5 minutes of searching.
basically all companies should file with companies house...
Bit more about Mondo/Monzo on wiki https://en.wikipedia.org/wiki/Monzo_(bank) (they recently change the bank name)
Initially they only offered prepaid cards while they waited for their banking licence. They're not covered by the Financial Services Compensation Scheme (FSCS) at this point but will be when they start to offer current accounts.
They've recently got their (restricted) banking licence so they'll be part of the FSCS fairly soon. Part of their restrictions is that they can only hold up to £50,000 ($65,000) in customer deposits, the FSCS covers up the 75k in savings so customers money is protected. (http://uk.businessinsider.com/uk-digital-bank-mondo-lays-out...)
I think what they are doing is bold (especially in terms of security, I think their head of security role is still open?) but ultimately the right thing to do for a truly disruptive banking startup.
I don't think it's possible/easy to bring the features people want from modern banking software front end (free text search through transactions etc), with a legacy banking backend.
I know since right now I am customizing one of those for my employer.
Except Google does have a "single monolithic codebase", so actually, the jury's still out on that one. Personally I wouldn't bet the farm either way at this point.
Kubernetes is FROM Google. Along with some of the most advanced distributed systems implementations in the world.
And yet, both Facebook and Google have large, monolithic codebases.
Building the best X is about using the best tools to do what X needs. Not everything craves massive scale and a TON runs better on a well provisioned RDB!
Want to revolutionize banking? Fine, but don't forget the fucking CAP theorem when you're storing my cash.
I agree that using microservices and kubernetes is not what allows us to provide a good user experience. We could provide the same user experience if we had built everything on rails and postgres. In fact, that would have been much easier.
The reason we are investing so heavily in a rock solid platform is that we want to still be able to offer the best possible user experience in 10 years time. This is what we mean when we say we want the platform to be "extensible".
Many large banks IT systems are not extensible, in the sense that it is very expensive to make changes. Take, for example, the ability to freeze a card in the app at the tap of a button. A friend at RBS told me that they considered this feature many times, but it took a long time to work out which 20 IT systems would need changing and some of them had been under change freeze for a few years. So eventually the idea was discarded as too expensive.
The freeze card feature would be easy to implement for a startup, regardless of their stack. The key is to still be able to implement such a feature easily in 10 years time :)
I'd also wonder if a post marked "Technology" with a title that includes "Backend" really needs to pare down the jargon and focus on the user experience as opposed to say, this one https://monzo.com/blog/2016/08/08/updated-design/ , which has minimal jargon. Also when talking about architectural decisions across an entire platform, you're going to have to mention a lot of different technologies.
Finally, to quote you: "an article like this is always going to get a response like mine" == "it's not actually my fault, the post made me do it" - it's not an acceptable excuse.
EDIT: I'm guessing you work there. You disagree with me; that is fine (many do). But... How quick did this fall off the front page?! From that alone, I'd claim my negativity is more representative of the readers that post was trying to target. I'm sure your company is trying to do something awesome, but you're not going to win hearts and minds with tech fodder like that.
I'm interested that you say you were looking for "rock solid" when you're using what is a pretty new stack. CoreOS and Kubernetes are very new products, innovative, exciting yes, seems early days to describe them as "rock solid".
Out of curiousity did you consider using a more conservative stack (e.g. ASP.NET ?)
And I'd suggest you're wrong. Neither complexity or age of a system is usually the issue. In fact, I'd say technology is almost never the real issue, but rather organizational red tape. Those owning the various systems often operate as more or less autonomous entities, largely with their own change policies and process, and I'd be willing to bet this will be a bigger hurdle than anything technical. Also, there will be a fair bit of politics involved.
Perhaps a small outfit as monzo can move fast and be nimble now, but if you reach the point where you have multiple system owners you will inevitably run into these problems, and they have almost nothing to do with technology. FUD, often fueled by compliance requirements (whether actual legal compliance or perceived such) also plays a large part.
For me personally, I sometimes wish the red tape would just go away, and at other times I'm glad it's there to save us from some of the madness I've seen people try to get deployed.
Source: I have worked with and in top tier financial institutions for about half a decade at this point, including RBS.
Organisational red-tape is obviously part of the problem but that's part of what I meant by complexity. Change in large FS companies comes slowly because of their size and complexity both organisationally and technologically.
Once you get a complex network setup and a load of systems that aren't designed to work together (which tends to happen over time as technologies shift) implementing new layers across the top of them takes time.
One of the problems with changing existing systems is that impact to production is something which has to be avoided, so changes to those systems have to be planned carefully. Now it's arguable that some organisations take that too far, but it's not fair (I think) to suggest that all of the change controls in place in those companies are "FUD".
My experience is that whatever technical issues may arise really aren't that big of a deal, provided you do the proper research to realize what a change to the system actually means. Significant effort, perhaps, but not particularly difficult from a purely technical standpoint. But it usually means coordinating with different parts of the organization, which can be excruciating, and more often than not has little to do with technological aspects and more with territorial politics and process. That's not necessarily a bad thing, like I said I've seen it stop some really horrid stuff make it into production, but it can also introduce so much friction that you give up before even having started.
I don't rightly know where the balance lies. For a small outfit like monzo it makes sense to be nimble, but for a huge organizations like RBS, UBS, JPM, HSBC etc., there are so many things at play that being conservative isn't necessarily a bad thing. Plenty more degrees in this hell than people generally believe there is, that's for sure.
We need a protocol for rpc, http+json was the slowest one --> clear winner.
Everytime we need a new function, we can make it a service and make it distributable on our polyglot (because why not!) docker infrastructure behind http load balancers + json data-type and make it infinitely scalable!
For example, we chose HTTP as our RPC protocol because it is widely supported in many languages, and has mature support in Finagle. HTTP/2 fixes most of the performance issues with HTTP, and we have options (eg. to use mux) if we still find there are performance problems – although at the moment, we are quite happy with performance.
JSON can be faster than binary in many cases: https://github.com/eishay/jvm-serializers
And your sentence sounds like the ideal scenario, no ?