How we secure Monzo's banking platform
monzo.com
monzo.com
A couple of other areas that aren't mentioned, although perhaps they're still doing them are around container breakout risks.
There's no mention of what (if any)hardening is being done on the container runtime, either restrictive seccomp, Apparmor/SELinux policies or using something like gVisor/Firecracker. With this year's number of container breakout CVEs, seems like an important area.
A related one is whether container aware runtime security is being used to detect where an attacker might have got access to a single container and be trying to breakout to either the underlying platform or to other containers in the cluster.
Your suggestion being? Putting a sandbox inside a sandbox? How many layers deep should this be, before being considered "secure"?
If you run a fully virtualised instance you get your own kernel and aren’t relying on process isolation.
Would you be happy if your cloud provider was running your containers on the same virtual I stance as someone else’s? Most people wouldn’t be.
There should be proper hardware-level isolation here, depending on the scenario. Most cloud companies can't afford that though, because they're not rolling out their own hardware.
Most people absolutely would not.
But that's exactly how VPS hosting works today - you don't get your own private blade unless you're ready to pay premium prices and have the competence needed to run them yourself. The technicalities of how private resources in a VPS are isolated from each other will differ, but the concept remains the same nonetheless.
People bite the bullet, only to be subject to things like rowhammer [1], or other container escape scenarios [2].
The top comment in this thread reflects the proper way of dealing with this: containers or sandboxes are may not be treated as a secure boundary.
[1] https://www.usenix.org/conference/usenixsecurity16/technical...
[2] https://www.intezer.com/blog/research/how-we-escaped-docker-...
No, VPS is isolation by virtualisation, not containerisation.
The clue is in the V in the name.
This is just not true, or at least it's extremely disingenuous.
Container isolation relies on the Linux kernel. Other than seccomp-denied syscalls (which aren't a thing in k8s by default) any program in the container has full access to the kernel. The Linux kernel has massive attack surface, especially to root users.
VM isolation like Firecracker is much safer. The attack surface is considerably lower. For one thing, you can isolate the process in the guest just as well as you could outside, further limiting attack surface. But more importantly, an attacker either has to attack:
1. Firecracker
2. KVM
Both are very small codebases.
Firecracker is:
1. Written in Rust.
2. Sandboxed aggressively.
KVM has basically never had a public guest to host breakout. You can read about one here, https://googleprojectzero.blogspot.com/2021/06/an-epyc-escap...
So, to recap, we have "security boundary relies on a fully exposed Linux kernel" and "security boundary relies on hardened, tiny, security-driven programs".
It is not even close.
> There should be proper hardware-level isolation here, depending on the scenario. Most cloud companies can't afford that though, because they're not rolling out their own hardware.
Hence hardware building hypervisor support in.
The challenge with Linux containers as used by Docker/Containerd/CRI-O et al, is that containers run against a shared Linux kernel. The Linux kernel has a very large attack surface, so it's easier for attackers to find some way to bypass the restrictions it tries to enforce. If you look at this year there have been several Local Privilege Escalation issues in the Linux Kernel, some of which have allowed for container breakout.
If you compare this to a hardened hypervisor (e.g. Firecracker) there is a much smaller attack surface visible from inside the container. It obviously could have a breakout vuln. but there is a lower chance of that occurring.
Gvisor and firecracker are the most popular sandboxes for containerized workloads.
How good of a boundary it is may be another story. There's some seccomp filters going on and namespacing is pretty sweet too.
But an attacker can escape by exploiting the kernel, which I think most security people would consider to be not particularly high effort.
So, suitable for internal services that you generally trust, not suitable for hostile code or highly exposed services. In an ideal world maybe we'd all use Firecracker but it's not nearly as easy to do that vs just putting something in a container.
For the most part this is troublesome when parts of the kernel or host userspace code are not fully aware of the different forms of namespacing (there are still portions that just check for an effective UID of 0, without checking whether they're in a namespace for example). These are the components where a lot of container breakouts happen and is largely mitigated by having internal processes in the container not running as root in the namespace. Dropping privileges to a different user still trace's it origin back to the root user on the host, so in some cases being partially aware of namespaces in a section of the kernel or host user code actively hurts the security by tracing the user back to root and using those privileges again. SELinux really tightens the potential to pull these shenanigans, but most production k8s clusters at least that I've seen are built on Ubuntu where those protections aren't available. In this case the security layer is once again SELinux not the namespacing.
As long as the container runtime is performing the various namespace isolation primitives starting from the root user these container bypasses are going to be a risk. There are 'rootless' versions of containers which can only use the privileges available to lower (presumably heavily restricted) user but those aren't widely used. Once again this is relying on the security protections of the host user authorization, not on the namespaces.
The networking analogy is NAT. People treat it like a security layer as it kind-of-sort-of looks like an ingress firewall since you can't directly address devices inside a NAT, but its not and can be pierced pretty easily. NAT is not a firewall. Namespaces are not a security layer.
That's not true
Having a capability inside a user namespace permits a process to
perform operations (that require privilege) only on resources
governed by that namespace. In other words, having a capability
in a user namespace permits a process to perform privileged
operations on resources that are governed by (nonuser) namespaces
owned by (associated with) the user namespace (see the next
subsection).
On the other hand, there are many privileged operations that
affect resources that are not associated with any namespace type,
for example, changing the system (i.e., calendar) time (governed
by CAP_SYS_TIME), loading a kernel module (governed by
CAP_SYS_MODULE), and creating a device (governed by CAP_MKNOD).
Only a process with privileges in the initial user namespace can
perform such operations.
Yes, like I said:
> But an attacker can escape by exploiting the kernel, which I think most security people would consider to be not particularly high effort.
> Dropping privileges to a different user still trace's it origin back to the root user on the host
It does not. Only if the process creating the container is root, which with unprivileged user namespaces is not (necessarily) the case.
> The NS_GET_OWNER_UID ioctl(2) operation can be used to discover the user ID of the owner of the namespace; see ioctl_ns(2).
"root" isn't the point anyways, it's about checking capabilities. The problem is that the Linux kernel has historically not cared about root -> kernel privesc, and containers expose more attack surface because of that. But an attacker outside of a container can still just enter a namespace (user namespaces are unprivileged) and perform the same exact privesc, so containers aren't making anything worse.
> As long as the container runtime is performing the various namespace isolation primitives starting from the root user these container bypasses are going to be a risk. There are 'rootless' versions of containers which can only use the privileges available to lower (presumably heavily restricted) user but those aren't widely used.
That's not how namespaces work. Even with 'rootless' containers your guest has CAP_SYS_ADMIN. The only difference is that the daemon that starts the container isn't privileged because user namespaces are increasingly becoming unprivileged. Rootless changes nothing, except that attacks against the daemon itself won't be an insta-privesc to root on the host, they'll only be a privesc to the user running the daemon on the host.
Anyway, let's step back.
What is a security boundary? I would say it is a mechanism by which an attacker is restricted where the attacker must exploit a vulnerability in order to get around that restriction. By that measure, containers are a boundary. Is exploitation difficult? Not necessarily, like I said, the Linux kernel has loads of attack surface. But it meets a reasonable criteria for a boundary.
As an example, chroot on its own is not a boundary because attackers can just call chroot again - this requires no vulnerability, it will never be patched, and you need another layer to prevent that. Containers have nothing like that, there is no "just let me out" syscall, you require another vulnerability.
You can read more about user namespaces here:
https://www.man7.org/linux/man-pages/man7/user_namespaces.7....
https://www.datadoghq.com/blog/engineering/dirty-pipe-contai...
Worth noting that even basic hardening in docker will prevent a lot of them. I say "in docker" because K8s disables seccomp, which matters a lot since `unshare` is denied by docker's seccomp and is very useful for attackers in a container. If you use Docker the main thing to do is just not run as root.
If you do that much, and it's not hard at all, you are in a much better place than a default k8s pod.
Now all we need is to somehow capture some of this "best practice" and make it normal practice, enabled by default and documented well so that organisations don't set stuff up and then disable all the controls because it is too hard to understand.
AWS/Azure/GCP/... for example have published IP-ranges of services. If a zero-day for any of those services is released a hacker can already narrow down the attack-range and gain a lot of time.
This is insane. What am I missing here that an organization is bragging about having 2000 moving parts?
And is 2000 moving parts too many? How many moving parts do you need to run a bank? I can imagine they're having to comply with ~2000 legislation clauses, for example. Isn't that just the complexity of their domain?
Quantity of moving parts is a nearly worthless metric on its own without describing the scale and complexity of those moving parts.
The domain is obviously very complex as its not just a ledger, but risk, loans, overdrafts, savings, etc as well as premium offerings, interacting with different payments networks like FasterPayments and Mastercard. Then there are the budgeting features, payments between friends, support system (Which is probably quite complex in its self), card issuing, internal dashboards and metric services, etc, etc, etc. I can easily see how this gets to 2000 services and I didn't even begin to think about the business accounts, US accounts, etc.
At least, they've only added 500 microservices in 2 years (not even 1 per day, how sad!)
I can imagine 2000 microservices being rather low for a bank.
I do believe AWS is likely far more secure than any DIY computing environment but even so, should banking be on cloud infrastructure? I'm not saying I think this is a bad idea but it came to mind when I read this.
Also, is it really a good idea for a bank to be talking openly about its security strategy? Isn't an important part of security not to let on anything that might be used against you? For example if determined hackers know your systems then they can keep an eye out long term for vulnerabilities in those technologies and be ready to strike. Does this sort of thing matter or not?
As to your second point, security through obscurity is generally believed to not be worthwhile.
What if AWS gets cracked/hacked/compromised?
I know it's not happened yet, but it's not impossible.
* For individual banks and their customers, is it more likely that an AWS-wide exploit will compromise an AWS-hosted bank, or is it more likely that a self-hosting-specific exploit will compromise a self-hosted bank?
* For society, is it better that security efforts are concentrated in on centralised providers like AWS, or is it better that security efforts are distributed, on individual hosting entities?
In reality it's always about tradeoffs: who to delegate to and who to trust.
The difference is that cloud relies on public services, which once compromised (e.g. via social engineering), allow for lateral attacks resulting in much bigger impact (e.g. Lapsus$) across the complete customer base. This makes social engineering much more attractive in cost vs impact. The resulting monoculture in not only the software, but the infrastructure and configuation also increase the impact on technical attacks on specific exploits.
What are the public services that AWS relies on, and how are they different from a bank's server farm, or a bank renting out space in a datacenter?
The same, really, applies to all other concerns.
All of these services are hosted by AWS in a multi-tenant fashion, sharing not only the code, but infrastructure and configuration patterns.
Is not advertising your security architecture "security through obscurity"?
Security only through obscurity - sure.
But obscurity as an additional layer, as part of a defence in depth strategy, still has some value.
It’s rare for any large org to publicly discuss any details of its security design, let alone a bank. Monzo must be supremely confident in their system to go public with this information, or judge that the marketing/recruitment benefit outweighs any potential risk.
On the second point, I'd say it depends on the level of granularity and detail. Here they're describing general mechanisms and they're not saying that this is all they do, so I think it's a good thing.
In general relying on obscurity for your security is a bad idea, as attackers will often find a way to get that information. That said I wouldn't give attackers a complete schematic of my env. and every protection, no sense in making things easy for them :)
For legacy banks, it will be much harder to move a mainframe based core banking system to the cloud, or migrate from a mainframe system to a new cloud system.
Banking has a fairly predictable usage pattern, but there will be black swan financial events that cause 100-1000x load. On top of that, how else could you serve customers around the globe with reasonable latencies?
These are genuine questions. I’ll admit I’m an engineer whose entire career has been during the cloud era. I don’t see how cloud’s advantages of scaling and worldwide “edge” locations can be replicated by the average bank’s tech team.
I can only speak to the banks I've have the opportunity to work with who stayed on prem or built their own internal cloud infra; what you call waste, they consider a premium/cost of business for security and resilience. I'm sure a lot of folks would've said the same thing about JIT supply chains (that had been squeezed to be as efficient as possible) until they unraveled.
> I don’t see how cloud’s advantages of scaling and worldwide “edge” locations can be replicated by the average bank’s tech team.
As always, "what are your requirements and what are you optimizing for?" Most folks don't need web scale nor edge locations [1] [2], they'll get by just fine with a CDN and some API endpoints [3].
[1] https://news.ycombinator.com/item?id=19576092
[2] https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
> Most folks don't need web scale nor edge locations [1] [2], they'll get by just fine with a CDN and some API endpoints [3]
Isn’t using a CDN using cloud?
I'd be interested to hear how this Monzo bank addresses the problem of someone walking in the home of one of their programmers and lifting access keys to AWS whilst that person is at the supermarket, and leaving with no-one the wiser.
Or installing a keylogger USB device onto their keyboard cable.
If you have AWS keys on staff laptops at home, you've already failed.
We don't allow any code at all on local machines.
But other than that, enforcing session auto-locking should work fairly well. Of course, if this is combined with some kind of agent that checks whether you're doing something that the employer defeated with a mouse jigger, all bets are off...
They can also enforce using MFA for AWS (and probably for GCP and Azure, too, but I don't use those) and not use plain access keys.
They do remote desktop connection into their PC at the office. The work laptops given to them can't connect to anything but the PC in the office(so, it's merely a thin client). A fingerprint reader device connected through the USB port and a physical device for generating single use codes are used in addition to username/password.
At the start of the pandemic they used their own laptops to VPN into the work network and then connect to the remote desktop but not too much later they switched to complete solution.
I know this because I helped with troubleshooting connection issues. Does look quite secure to me, the only difference to an office environment security seems to be the possibility of an intruder to make an employee do something at a gunpoint.
The lag from remote access (Citrix) drove me nuts!
* Work laptops are all using trusted computing, tamper detection and remote attestation which, while imperfect, does provide some verification that the hardware isn't being tampered with.
* Additionally it means if you try to access any service from a non-work laptop (or a work laptop failing remote attestation), it doesn't let you in. Even if you have all the credentials.
* Passwordless authentication means capturing PIN codes with a keylogger is of very limited value unless you also steal the laptop. Even then, an additional factor is required such as mobile push or biometric.
* No developer should have access to any AWS keys that would grant access to production data. But in any case, we use AWS SSO which only returns temporary AWS keys.
* There are lots of systems that monitor for anomalous activity. For example if a user account suddenly starts hitting lots of access denied errors or accessing things they don't normally access, that's a hint they've been compromised.
https://www.vice.com/en/article/bvg7n3/monzo-freezing-closin...
https://www.reddit.com/r/UKPersonalFinance/comments/db9j5z/m...
https://www.theguardian.com/money/2020/jun/06/monzo-customer...
I'd be far more interested in a blog post explaining that than how great their infrastructure is.
One gets the feeling they're hiding behind the "we can't tell you why" excuse to escape accountability for a broken system that's literally ruining lives.
Statistically it doesn't seem likely that Monzo and its ilk would be so overrepresented merely because their customers are more fazed than others about having their money taken away. Still, if the FCA have published meaningful statistics I could be proved wrong.
https://www.which.co.uk/news/2021/09/why-banks-are-freezing-...
This in turn means that they will have their risk assessments inverted from the usual high-street banks: optimise signup/account creation flow, and deal with AML requirements in a slightly delayed fashion. Making it really easy and smooth to open a current account brings in a surprising fraction of the crowd who would be rejected or otherwise earmarked by high-street banks.
Being digital upstarts, these modern banks also don't have the fraud and risk departments their established competition has. In order to not get hammered by the FCA, they almost certainly veer on the blunt instrument side when dealing with suspicious activity. And law of large numbers guarantees that there will be a significant number of false positives.
This, I should add, at a time when there was a high incidence of fraud with people trying to open business accounts for dormant and non-existent companies simply so they could claim Govt backed loans where it was clear that very few checks were actually being carried out.
How do you know the scale and how are you not sure that it's a vocal minority? After all, Monzo does target more tech-savvy users that might be more likely to voice their frustration online than with other high-street banks.
> One gets the feeling they're hiding behind the "we can't tell you why" excuse to escape accountability for a broken system that's literally ruining lives.
Because maybe they are being forced to hide behind that? UK banking regulations are notoriously strict when it comes to what banks are allowed to share with their customers.
Read the articles and look for similar experiences for other banks. There's definitely a lot more noise about Monzo doing it. Maybe (as another comment suggested) there's a selection bias at work but I assure you it's a real thing, and appallingly handled by Monzo when it happens.
> Because maybe they are being forced to hide behind that?
That misinterprets what I said. If there are rules that say they don't have to be accountable for a misfiring fraud prevention system that hurts real people, where is the incentive to fix it?
If that's the case, it's fairly obvious why there'd be a lot more noise - a lot larger percentage of people in that age group doing crypto, and a larger percentage are "very online" and likely to complain about it there.
I don't know anything about Monzo specifically, but perhaps anti-money-laundering laws might be the reason. Monzo is presumably obligated to take steps against money laundering, under UK law, where it's essentially an offence to say Our system has flagged you as a potential money-launderer. [0] Freezing your account without comment may be their safest course of action.
edit I see this explanation was mentioned in the reddit thread.
[0] https://www.lawsociety.org.uk/en/topics/anti-money-launderin...
Source: used to work on Monzo's financial crime team.
And it’s not just a criminal act for the organisation, but the individual customer service agent can be held personally criminally liable for disclosing, even accidentally.
So yeah, they have a really strong incentive not to tell you anything.
A frozen account could be due to any number of things. The customer service person may or may not have access to the reasons. In any case, it might be money laundering (from the bank’s perspective), therefore they can’t tell you anything.
Of course this is ridiculous. Due process should exist, even when a private organization “accuses” someone of illegal activity.
This is why you don’t build your own solutions to well solved problems.
Other banks run their AML systems in batch at the end of the day, and will only freeze an account after the money has left.
So yeah, you’ll see a lot more people complaining about Monzo, because their money will be frozen while they’re trying to exfiltrate it. That doesn’t mean Monzo has an unusually high false positive rate, it just means that people are used to being able to get away with fraudulent behaviour, and have their accounts frozen empty. Rather than being stopped mid-act and having their accounts frozen while they still contain illegitimate funds.
There should be a penalty for this. When "innocent" users are denied access to their money, those actions are much more felt by the user than the bank. That user's money is a rounding error to a bank, but to the user it is everything.
An incorrectly frozen account should come with some sort of "oops we're sorry" type of something that a bank can understand: monetary reward for the user.
I would rather criminals get away with a laundering transaction than me not being able to buy/pay for something in my day to day because of some dumbass AI algo.
There are, the FCA and Financial Ombudsman hold banks to an extremely high standard and hand out fines and compensation on a regular basis. Denying access to funds is a serious issue which banks and regulators take very seriously. However there’s an acknowledgment that AML systems have false positives.
> I would rather criminals get away with a laundering transaction
Easy to say when it’s not your money that’s been stolen. Most money being laundered through retail banks like Monzo isn’t drug money, or high stakes bank heists. It’s the millions stolen every year from normal people subjected to high pressure, complex, coercive and extremely effective social engineering scams.
We’re talking about peoples house deposits (because their solicitors emails got hacked), their life savings, their retirement funds.
Don’t assume you’re immune to these scams, or that you’re paying a cost for it. The scams trick even the most savvy individuals. Costs are borne by everyone with a bank account because victims are reimbursed, and I can assure you, no bank is sacrificing their profit margin to do that.
False positives are one thing, but it shouldn't take a year to resolve?
And yeah as others have said that’s not a Monzo thing not telling you, no UK bank would tell you why they think you’re laundering money
That's what's so kafkaesque (and frankly rather violating) about the experience: you literally haven't the faintest idea what you've done wrong and the bank refuses to tell you.
[1] https://www.thebureauinvestigates.com/stories/2021-07-28/mon... / https://www.forbes.com/sites/afontevecchia/2012/07/16/hsbc-h...
Perhaps the law is too strict due to the impact it can have on individuals.
My employer paid me a couple of grand in expenses, which I quickly moved to another account. That triggered an account freeze.
I asked the customer service rep for "why" - couldn't tell me. They then asked me for payslips to prove my story. I told them feck off because they could see the monthly entries marked "Salary" from the same source.
I escalated the matter and it took several hours before my account was unfrozen.
Lesson: only keep a minimum amount of money in UK bank accounts.
I've also had great experiences with Monzo, so will also back them up.
As others have pointed out already, AML/KYC laws are strict. They are strict in general for financial services, but for banks, because of their privileged position in the financial system, its even stricter.
But there is a second aspect which is that challenger banks such as Monzo take an even more cookie-cutter approach. If you don't fit their definition of what a "client" is then you will be in for a hard time. Normal banks do this too (to a degree) but challenger banks are much more hard-core about it because if you fall outside the cookie-cutter then you mess up their fragile business model.
Case in point, I know of a well-known, well-established, UK VoIP operator. They moved their business over to one of these challenger banks (might have been Monzo !) because the challenger bank provided APIs to enable integration to their internal systems, which is something that the old-school high-street bank did not offer - and the banking fees were lower too, always a bonus !
TL;DR: $challenger_bank had a definition of a client that did not include provision of VoIP services. So after about a year as a client, said VoIP provider found their account frozen (in this instance they were explicitly told, it wasn't a silent freeze). VoIP provider attempted to constructively engage with $challenger_bank but it was like talking to a brick wall "computer says no".
(N.B. I have oversimplified the story a bit, so please don't nitpick !)
Nope, not them. I'm surprised they haven't been frozen given Monzo's stated policy[1]:
"Certain industries have higher risks, where we need to put extra checks and controls in place. We’re currently focusing on industries that don’t need these. In future, we might offer accounts to some of these industries. But we appreciate this is disappointing for some businesses at the moment."
Followed by a long list that includes "technology equipment, like lasers or telecommunications"
(For those wondering, telecoms wasn't there at the time the other guys had an account, and they were not selling equipment anyway)
Don't touch them with a ten foot pole.
(I don't work for Monzo, but I've been a happy customer since day 1)
You are simply a false positive of this ineffective policy that has a success rate of 0.2% according to recent United Stations study: https://www.effectiveaml.org/un-slams-aml-success-rate/
Globally, it has almost zero impact on crime for an infinite cost.
Switched to Starling instead and created one instantly.