How we protect our most sensitive secrets from the most determined attackers
monzo.com
monzo.com
Are we talking the real most determined attackers like the US or China who can easily deploy $10B (i.e. a team of ~3,000 people fulltime for 3 years) for a strategic advantage?
Or $1B (i.e. ~300 people for 3 years) which is within reach of every government, international criminal organization, and thousands of multinationals?
Or $100M (i.e. ~30 people for 3 years) which is now just a line item for those organizations?
Or $10M (i.e. ~3 people for 3 years) which is 10x better than the "gold standard" used by every other bank in the world, but still profitable for small criminal outfits doing ransomware attacks?
Are there any legally binding marketing statements made by any associated security executive on where they sit in these 4 orders of magnitudes of "most determined attackers"?
It is frankly absurd that we continue to allow statements like "secure", "reasonably secure", "most determined", or whatever without expecting some degree of quantification within 4(!) orders of magnitude. If you can not quantify or demarcate an accurate lower bound within 4 orders of magnitude you probably have no idea what you are doing.
They’re still a target for someone wanting to cause disruption or engage in hybrid warfare - imagine a simultaneous hack of all the neobanks, possibly followed by a few of the traditional banks a week later for maximum impact?
They’re still a target for someone wanting to pivot into other organisations they might be connected to, such as card networks or regulators.
The small and new nature of the Monzo makes it an attractive target.
An attacker may well use slightly smaller, slightly less established business to other, more established businesses, as a pivot point. Since this is a 'bank' it will have some connection with other institutions. It does not need to be SWIFT or similar to be exploited.
> ... don't think state-sponsored attackers are a major problem. Those rarely need the money ...
I disagree with this too. State-sponsored APTs may very well take advantage of financial gains, both to enrich the attacking country, and throw off investigations. (e.g. DPRK, Iran)
Just my opinion.
Where can I apply?
You're probably in charge of a government-backed digital currency and out to make alternatives look unreliable.
Industrial espionage. Blackmail. Unmasking informants or undercover law enforcement.
It's kind of a disaster that they even collect this stuff.
https://www.iana.org/dnssec/archive
https://www.cloudflare.com/dns/dnssec/root-signing-ceremony/
I'm not blaming Moz, CACert were breaking new ground in trying to build a transparent process for a member-run CA. I believe they failed to find a suitable auditor that didn't cost a mint, and could work within the CACert transparency constraints.
Then LetsEncrypt came along... and ate their breakfast.
Source: formerly at one of the largest banks.
I'd skip to Chapter 2: Risk Management and Chapter 3: Fixed Facility SCIF Construction
Hopefully by physically removing where possible? Also consider microphone and speakers if you’ve not already..
I mean it's seven, it's literally written in the attached PDF describing the ceremony procedure.
for all you know, I rolled two dice to decide how many keyholders I'd put in each part (which I would totally never do, would I?)
"Our entire system is air-gapped, which means it is physically isolated from the outside world and has no way to connect to the internet."
Point being that yes, you need to ensure that your root certs are super secure and that you have a full documented, videotaped chain-of-custody. But then at some point you need to use that root cert to sign some other cert that DOES live in your infrastructure and is used to sign requests. How do you control your systems at the point where you need to use your cert (or a cert chain) to sign requests that by their nature must be connected to the Internet?
There's definitely a much bigger risk with an intermediate on the internet, however we at least have the mitigation of being able to revoke it if it goes wonky, the main priorities I see in this space are: (1) minimising the chance of a compromise as much as possible (very clear and well defined communication channels so that as few things are open to even being poked at for vulnz over the network as possible) and (2) being ready to react if there is a compromise (revoke the cert and recover)
Fortunately, because the usage of our CAs is very tight to Monzo and our partners, we can reach out and explicitly ban a certificate from our machines (and tell partners to do the same) without too much trouble (as compared to Public CA's who have no chance of being able to do this) in addition to following normal PKI revoking procedures.
You can use Validation Authorities to avoid needing the full on root that can issue new certs just to revoke an existing one. https://en.wikipedia.org/wiki/Validation_authority
If you feel differently, by all means go do business with a bank that doesn't waste all this effort.
Only above the non-trivial limit guaranteed for UK regulated banks.
The problem with hacking a bank is all they can really do is try to screw up the data or lock it down. Even people who work at the bank can't figure out how to move money around, much less where to wire it to to profit anonymously.
Root access to servers, to secret keys or password is not beneficial. There are many systems validating each other. Even if you got access to one system, even if you control it for some time, you cannot really permanently transfer any significant amount of money anywhere. Fraud detection and AML systems will trigger an alert. Behavioral analysis systems, which are must have in industry, will trigger an alert, to say nothing about many other systems I really do not want to discuss in public.
The inclusion of NSA stuff got my eyes rolling. If you’re worried about NSA implants, you probably should be thinking about the provenance of the HSM, smart cards, etc.
No, it is not. That is the point. There are many levels of protection, many checks and validations, and even if you pass all of them still a lot of things can be reversed pretty easily. Banks are not cryptocurrency clowns with private key obsession.
it's someone who found the video of the DNSSEC key signing ceremony and decided to implement it at their company
I think this is an unfortunate framing. The property of public key cryptography is that unlike secrets such as passwords only one party has the private key. This instantly eliminates an important class of potential problems.
And unlike with something sophisticated like an asymmetric PAKE, I think it is relatively practical to explain this benefit to end users, Monzo's counter-parties can't lose a private key they don't have and can't produce signatures from Monzo themselves. If there are signatures that shouldn't exist Monzo knows the problem necessarily must be with their systems.
British Banks rely far too much on trust. A symptom of this is that periodically a big merchant (e.g. a supermarket) will accidentally run a transaction feed twice (e.g. all card transactions at Tesco on Thursday happen twice). The bank could insist (as it does for its own personal account holders) that these transactions have unique IDs authenticated by the card, which would thus mean the duplicates are rejected, but it trusts these huge merchant customers, so all the transactions they say occurred are just assumed to be fine, and the result is that the banks eat a bunch of customer anger and costs. It was remarkable to me that this would happen in 1995. It's outrageous that it still happens today, but it does.
It's an interesting contrast with how carefully root CA keys are handled in this kind of setup, compared to things like Kubernetes clusters, where you'll typically get 3+ root CA keys in the clear on the API server disks, and I've seen them committed to configmaps (e.g. in RKEs default setup), and even checked into GH repo's...
- Since all the components we purchase are kept air-gapped, you'd need to already be on a machine in the air-gapped system (which isn't assembled and powered on except when we need to deal with key material) to exploit a vulnerability
- We're keeping things as minimal as possible in what we trust from coming out a store, for example if we purchase a Laptop we gut it of most of its components before it goes near our key material, Laptop's dont need batteries (or even CMOS batteries) to run basic live systems so they're going out.
- Compromising one part of the system won't let you sneak private key materials out on its own by-design, the part we've tried to make the hardest is exfiltration, the only material that leaves the air-gapped system leaves either as QR codes on a screen or on CD-R drives that we keep for years in a safe in their own tamper evident bags. This is all part of an effort to try and make sneaking private material out (even if you had full control of the system) as difficult as possible to do without being detected.
There's also projects to guess keystrokes based on sound which don't even require you to be on the host device https://news.ycombinator.com/item?id=29266783
Totally understandable if you can't/don't want to answer obviously.
I also suggest using at least dvdisaster (I suggest it's RS03 codec in augumented-image mode) or equivalent (if you come across a proper competitor to dvdisaster RS03, please let me know).
What that hardware will be is still an open question, but probably BD-R HTL.
Also worth noting (I think it's in the script too from memory) we make at least 3 identical copies of each CD in case of CD failure.
presumably this is kept in a locked cabinet, in a basement locked room, behind a sign "BEWARE OF THE LEOPARD"
Admit it...y'all just want a good reason to go to MicroCenter, right?
https://www.amazon.com.au/first-100000-Prime-Numbers/dp/B089...
It's a symmetric algorithm, so this doesn't solve any problems Monzo actually has, whereas the elliptic curve public key crypto they use does
It's known to be cryptographically weak. If you write small messages, by hand, and your recipients are, even if they have more technology, reluctant to try to decrypt large volumes of messages too great in number to all be from you, you're safe. But if you used this industrially it would get broken at scale.
As a comparison, RC4 has a biased keystream just like Solitaire, and it has been successfully attacked in practice since unlike Solitaire it was actually in use on the public Internet. The attack consumes a very large number of messages, transmitted over HTTPS in the course of hours and so would not be at all practical for a system used with a few hand-ciphered messages.
First draft (very rough) was 22nd October
Second draft (toned around a bit) was 5th November
Security review + a proper edit happened while I was on Holiday on the week of 8th-12th November (lots of comments left for me), basically nothing got removed, I think one image and about 2-4 lines of text from memory
I came back to a bunch of comments (nearly all on the Monzo Tone of Voice https://monzo.com/tone-of-voice/), went through those, proof-read and then got a green light from:
- Engineering
- Marketing + Press
- Some other folks in Security (because of the content)
And then we moved this over from Google Docs where I was drafting it into our Blog system (Contentful), had a quick skim to make sure it read properly and then published :)
e.g.
A decision has been made to close your account …by monkeys
vs
We decided to close your account ... by monkeys
HSMs should only sign messages if a minimum number of authorized users request the message to be signed (by its hash), and downstream systems should only trust messages signed by a quorum of HSMs (at least 2).
For banking as an example, each bank should require some minimum number of distinct authorized signatures on each transfer request from another bank.
Individuals can have a personal public/private key, but HSM actions should also require multi-signature requests from both the user key (stored on FIDO2 or smart card) and a device signature from a list of devices the user is authorized to use, which reduces the risk of rogue devices tricking users into signing requests.
Go back to using bog-standard laptops everywhere because no single device is in the critical path of security; if a single HSM loses its keys then have it generate new ones and add the new keys to the set of N possible keys allowed for root signatures and remove the old one. Only if too many HSMs need to have their keys restored would administrative users have to resort to SSSS to recover all HSM keypairs at once, but the procedure would be done one HSM at a time with potentially different sets of users responsible for recovering each HSM. There never needs to be a single set of people that can restore the entire PKI; instead have K groups (for K KSMs) of N people, of whom M of each group are required to restore an individual HSM, where only L HSMs are required to operate the PKI. There can even be some intersection between the groups of users per HSM so long as the intersection between any two HSMs < M.
I wish it was available as a reproducible build on F-Droid. That's how critical apps ought to be distributed.
This is a common trope in security-oriented businesses: more security is a good thing, but the consumer often isn't afforded the same attention.
If you're talking about security, surely a closed-source app is just obscurity, there should be (sounds like there are) better precautions in place that it doesn't add anything.
If it's ripping off features.. meh, is it really development time in frontend logic/UI that keeps Monzo ahead of any competitors its ahead of? (I'm saying that vaguely and unassertively because users seem to get very tribal and argumentative when it comes to 'challenger banks'!)
If it's wanting to hide the API for whatever other 'it's proprietary' reason, well Monzo has a ('beta' and 'developer only' for years) public API anyway. And 'open' banking of course, so you can use TrueLayer or whatever anyway.
Who could use the source code? Their competitors? Maybe, but there isn't that much competition in that space, and actually developing the app is certainly not the most difficult part. Auditing it probably is, and using a competitor's open source app as a base is probably a no-go for a lot of established players. For what purpose would they copy a functioning app?
One thing I can see as an argument against would be raising the bar for would-be copycats and phishing attempts. Not that it raises it too much either.
Or is it to limit documentation of their API? I know security trough obscurity is thought as a good additional measure in traditional banking, but that doesn't seem to be the case here. And reverse-engineering an app wouldn't be that complicated.
I would actually love an open source, cross-bank app. Not sure why it couldn't happen. I'm not sure either what the value proposition is for banks to try and limit that kind of development. Control the user's interactions with the bank as much as possible?
It's like these banking apps requiring "safetynet" or similar: they trust google or the manufacturer more than myself, while I am the actual client!
Can't really comment on why other banks don't talk about this so you'll have to draw your own conclusions on that :P
Why go after Monzo when you can go after XYZ root cert trusted by every device out there?
If you could actually do anything based on the knowledge here, you would go after a browser root cert because your hack would be exponentially more effective.
edit since I can't reply down level further: Yes I would be very careful about security related marketing and really consider if its necessary at all.
You can always be hacked in some other way, so I guess we should never write anything about security ever?
>edit since I can't reply down level further: Yes I would be very careful about security related marketing and really consider if its necessary at all.
Interestingly, the security people at Google, Cloudfare, Microsoft, and just about every other major tech company (and security company, certificate authority, etc.) agree that openly talking about security best practices is.. well.. A best practice. And that keeping security practices secret (obscure, you could say) benefits no one.
Not sure why you have to shoe-horn marketing in every comment, literally anything a company posts is arguably considered marketing, what's your point?
I published this because I want to be open about how we do these things, because I think we safely can be and it shows that we care and don't just pay lip-service to security
If you'd like to show us up for our security though, please do peek at our HackerOne program. I'm a program admin on it and I'd love to read more interesting reports https://hackerone.com/monzo <3 (I know it doesn't have a paid bounty yet, I'm advocating for it)
>banks don't use to describe openly the technologies or methodologies they use for good reasons
I don't think it's "for good reasons". I think it's for "don't want to be found to still be using XP" reasons.
most of the stuff is security using multple layers - which can be a pain to modify
Any technical debt is evaluated as a risk and signed off by the business
the higher the risk the more hoops you need to jump through for security sign off (to the point where they will not sign it off) and higher the risk, the higher up the food chain of managment that is needed to sign it off
obv if the execs are not bothered about security then anything can happen
but if something goes wrong a big finger will be pointing at that person who signed it off - which seems in it self a massive deterrent
Yes. Generally speaking, an open and auditable system is more robust and secure than a closed and non-auditable system.
There are obviously limits (e.g. publishing what specific VLAN numbering scheme you use is obviously not helpful to anyone and just provides information that wasn't known).
But yes, it is best practice to publish and accept feedback on generalized procedures.
You should sign up for the MDSP dev-security-policy mailing list and see how the (open) conversations have continued to improve security for all.
I fail to see any positives in openly publishing anything, unless you provide an extememly very detailed view - I see only negatives
It is a shame the article doesn't bother to cite nor link to its source.
Of course by now you can just buy similar devices on the free market, what used to be expensive high-tech in 2008 is now possible with commodity hardware.
In this case, the author posted here on Hacker News, which provides a wonderful opportunity to accept feedback. And it looks like the author added some text giving more context explaining where it came from:
"COTTONMOUTH-1, a USB cable manufactured by the NSA that looks like a normal USB cable, but has a hidden implant that allows an attacker to inspect data on the cable wirelessly, modify data as it travels over the cable and install malicious software on connected computers - US National Security Agency, Advanced Network Technology (ANT) Division"
It is nice to see authors being responsive to comments here.
But that isn't the point. You can cite just about anything -- ranging from phone conversations to in-person interviews, from cookbooks to journal articles. Yes, even documents from WikiLeaks or other data dumps. It isn't hard.
Citing your sources shows that you are careful and you respect your readers. Make it easier for them follow the trails where they lead.
It also saves time, energy, and electrons. Write out the reference once, help N people later.
The flow of the text is broken by the Cottonmouth cable and caption. I'm not suggesting removing the image; it is interesting and serves as a good example.[1] I'm suggesting another round of editing to improve the reading flow.
Additionally, since so many people scan text rapidly, some visual changes are needed too. The caption looks too much like a follow-up paragraph. To fix, consider:
A. Include the caption in the bounding box used for the image (which has white space on both the left and the right).
B. use a different font, font-size, or style for the caption
C. using a different background color for the (image + caption)
D. separate the (image + caption) from the surrounding text with vertical lines
I'm inclined to think that in this case, A with B would be sufficient and would work well. You could also pair that with C and/or D, being mindful that the latter two could be overdone.
[1] Why stop at one example? I think a carousel of examples would be even more interesting!
I wonder if Monzo has done a similar deep dive into how they structure their architecture/security practices to protect against the threat of this information+encryption keys being stolen.
Any Devops on HN to tell us about the last time they implemented a change on a Monzo API ?
These sorts of precautions are pretty common at large companies, there's some good coverage of them generally here: https://en.wikipedia.org/wiki/Offline_root_certificate_autho...
The people who implemented this process are the same type of "IT/devs" or "Devops" you mention. There isn't a whole lot of throwing things over the wall to Ops, or for that matter putting up arduous processes against "normal" engineers.
404 is it a private repo?
We're going to remove that link
Exceptions are evaluated, documented and acted upon from the moment they are discovered. The Ceremony Administrator decides what to do about them after consulting the Internal Witness (but the CA gets to decide), at the end of every ceremony (with exceptions or not) we ask every participant to sign a sheet saying they're happy we effectively haven't breached security (last page of here: https://monzo.com/static/docs/redacted-root-certificate-auth... ), if they don't sign, that would kick off a large discussion about whether we're happy to continue using these roots or replace them.
From memory we've had about 2 exceptions which have been resolved, both very minor (as ever, when you get into the room stuff comes up that you couldn't have planned for, if I recall one was something silly like a typo in the command documented on the script, so we just ran the command without the typo, etc).
I mean when administrator denies interruption. Rotation of administrators and signatures is nice, but there's still one person that makes the most authoritative administrator. The bug to address here is conformity, see https://en.wikipedia.org/wiki/Asch_conformity_experiments
It's entirely up to the people authoring the website.
That is exactly why many do it. The chance of getting fined is low, practically non-existent in most cases, but making the user experience difficult both increases the chance of people just saying “fuck it” and agreeing¹. Those high up the data collection chain also hope that annoying enough people with the bad UX will make them campaign to have the regulations rescinded² in their territory.
---
[1] losing people like me who say “fuck it” and move on elsewhere, is seen as a small price to pay
[2] even though the regulation isn't at fault, the deliberately bad UX is
Jaywalking is illegal in many places too. It's an enforcement problem, not a legislative one.
https://news.ycombinator.com/item?id=29266521
And why it's just a laptop here:
https://news.ycombinator.com/item?id=29267685
To be super clear, the actual private keys only ever exist on HSMs
This sentence made me believe in the incompetence of the authors.
Sorry for not being up to your standards I guess :(
I am an incompetent welder, for example.
My point is that if you are removing the hard disk to prevent state storage, that is ineffective, and if you think that it is effective (which you stated), you are objectively factually incorrect.
TLDR: it's not bulletproof but increases the effort required from an attacker significantly.
There are better ways of doing this. There are embedded systems that can run general purpose OSes that cannot store permanent state, such as rpi <=3 that are much better suited to such things.
We also don't run that many complex commands on the actual OS, it's basically one of three things 99% of the time:
- Mount this CD containing some CSRs or etc
- Ask the HSM to please sign this
- Burn a CD with these files and then report the hash / PGP word list of the content on the CD
You can tell if you're using a part of the OS that hasn't been used in a session yet because you can hear the CD drive with the OS CD in it physically spin up :)