obeattie

Best thing about incidents like these: post-mortems for systems of this scale are absolutely fascinating. Hopefully they publish one.

The only reason to consider Cloudflare, for us, is DDOS mitigation.

Commenting is painful.

At Monzo where I work, we use Dropbox Paper quite extensively for internal RFC's in the engineering team. We'd probably prefer to use a Google product since the rest of the company uses Drive so heavily, but Docs is simply atrocious when you need to insert some code.

I imagine what is considered an "enticing" cover for a book aimed at scientists and engineers has changed a bit between 1997 (when this was published) and now.

Monzo | Full time | London/remote | Visas sponsored | https://monzo.com/careers/

At Monzo we aim to build the best current account in the world. We are always keen to hear from capable, creative engineers who want to help us accomplish that goal; specifically backend engineers and Android engineers, and we also offer paid engineering internships.

Our engineering team have a variety of different backgrounds: we have several non-graduates; only a couple of us studied Computer Science; one of the team has a degree in Marketing; some of us have worked in huge companies; some have only ever worked in startups; others are former consultants. As long as you enjoy learning new things, we’d love to talk to you.

If you’re unsure about applying or have any questions about the role or team, please don’t hesitate to email me (oliver@monzo.com) directly :) We're very open about what we do in general, so our blog is a good place to learn more about what we do: https://monzo.com/blog/

We're _very_ directly involved with regulation, and with regulators :-) It's a very important part of our business, so we put a lot of effort into ensuring that we please regulators _and_ that we run our business in the way we want.

We don't store things that are actually secret, as k8s secrets ;-) Hashicorp Vault is quite good at this.

To be clear, we (Monzo) are not writing banking software in PHP, or using MySQL.

Because of its disk layout, Cassandra is truly excellent at time-series, append-heavy data. In our setup, the data is partitioned by time bucket and n number of labels (one of which is the request ID).

We may unify the two at some point, but there's no immediate need to do so. While the write use-case is quite similar across both, the read use-case is quite different: slog requires reasonably low latency reads soon after the data is written, data can age out after 2-30 days depending on severity, and sometimes dropping events is acceptable. It would be acceptable for reads from the "archival" system to take minutes or even hours, the data should be kept forever (or for a long time), and dropping events is never acceptable.

This is great feedback – thank you.

A lot of the feedback from my blog post a month or so ago was that the "why" was not clear enough, so I guess I focussed more on this. Now I've (hopefully!) covered that better, I agree that more depth on these topics would be helpful.

>what really limits most financial institutions from embracing a lot more modern tech is their core systems of record AND the acceptance of said systems by their governing agencies.

I just want to say that I think there is a huge amount of FUD about how you can and cannot build your technology as a regulated entity – and in particular as a bank. In reality, close to 100% of requirements from a regulator will tell you _what_ you must build, not _how_ you must build it. Even then, especially in terms of resilience and security, they are almost always a subset of our own requirements.

What can be more of a challenge is convincing an auditor that what you have done is acceptable, since it can be so different from what they may have seen before. Again, I don't think this is a reason to compromise. We see technology as a major competitive advantage, so it is worth the effort to find open-minded auditors, and spend time to explain and demonstrate how (and why) our software meets the requirements.

I don't think there's any way we could build a secure, resilient bank with the kind of product experience we want, AND do it on the budget of a startup if we approach technology through the same lens as existing banks.

Good question. We have a couple of approaches to this:

* Every request that comes into our system is assigned a unique ID, which is propagated on every downstream call and returned in a response header. When logs are emitted during request processing, they are tagged with this ID. A system we've built in-house indexes these log events against their trace ID in Cassandra (on a separate cluster). This lets us take a failing, slow, or otherwise interesting request and look up all the things that happened to it during processing. Events in this system are TTL'd according to their severity – so an event at critical or error severity is kept longer than one at debug severity. * stdout/stderr from all our containers is forwarded to journald on each host. Logstash then pushes all these logs to Elastic (and also to permanent cold storage). This is useful to look at the "big picture" and means we can analyse all the logs in aggregate and makes it very obvious when something is _very_ wrong and causing a lot of requests to fail, but is less useful than slog for pinpointing a specific issue.

It's worth also noting (since I find often a whole load of things can be mixed into logging) that we do not drive our monitoring off these logs, at least at the moment. We have separate systems for that.

Author of the talk here. Happy to answer any questions anyone has. This post also contains more info on how we build our systems: https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac...

And as an aside, I'll also be giving a longer talk at Kubecon going into more detail on some of this stuff :-)

Almost all of them interact with other services to do something or other.

I'm the post's author. The key point I was trying to make was not that monolithic systems cannot handle the throughput, but that they become more and more difficult to extend and maintain as they grow. None of the companies mentioned started out with a microservices architecture – they moved to them (at great expense) when they found development difficult to scale.

As others have mentioned, you can find it on Companies House, or on the the FCA (Financial Conduct Authority) register: https://register.fca.org.uk/ShPo_FirmDetailsPage?id=001b0000...

Author here. We are in beta and have about 40,000 cards issued to customers in the UK, with a waiting list of about 200,000. I think we have good reasons for making the architectural decisions we have made, some of which are laid out in the post, but perhaps I did not explain them in enough detail.

For example, we chose HTTP as our RPC protocol because it is widely supported in many languages, and has mature support in Finagle. HTTP/2 fixes most of the performance issues with HTTP, and we have options (eg. to use mux) if we still find there are performance problems – although at the moment, we are quite happy with performance.

Potentially; we don't have many remote employees yet (but we'd love to in the future) so we're still finding our way with it.

I'd say it's unlikely, but anything is possible.

Absolutely. We don't have an ad up for internships specifically (we probably should!), but you can fill out the backend application and mention you're looking for an internship :)

Monzo | Backend Engineer | Full time | London or Remote | https://monzo.com/

At Monzo we aim to build the best current account in the world. We are always keen to hear from capable, creative engineers who want to help us accomplish that goal.

We’re still a relatively small company with only 10 people in the backend engineering team and 50 people in total. This means you will need to wear many hats in the beginning but it also means there are many interesting challenges ahead that you can specialise in if you like. Broadly, the work we’re doing covers these areas:

* Distributed platform: We aim to be the first bank without a single point of failure

* Banking and payments: We write code that moves money

* Product engineering: We help our customers by hiding the boring parts of money management

* Financial crime: We need to stay ahead of criminals trying to steal data and money

* Internal tooling: We automate everything so the rest of the team can work smarter

Our backend architecture is composed of hundreds of distributed Go microservices running in the cloud. They are managed by Kubernetes, store data in Cassandra, do RPC with linkerd/Finagle, and use Kafka for asynchronous queueing.

Lots more detail is on our website : https://boards.greenhouse.io/mondo/jobs/244898

On Mondo, there simply are no passwords at all. Instead, when the user wishes to log into our first-party apps, we send a login link to their registered email.

We'll almost certainly add additional required factors to this process (eg. biometrics), as we see the user logging into the Mondo app on a new device as one of the most critical from a security perspective.

Why? In order to have a Mondo account, the user must have our app installed.

In the proposed implementation above, the _only_ piece of information that a user enters inside the web view is a username. The user must then use the native Mondo app on a previously-authenticated device to complete the OAuth flow. The Mondo app could also require biometric (ie. Touch ID) authentication.

While a malicious application can inject JavaScript to intercept the username, this alone is useless to an attacker.

The user may be on a desktop, or a different device. We want to support such flows too.

I agree in the "native app on same device" situation, we can bypass all this by bouncing straight to our app though.

The flow we (Mondo) are considering for this:

1. MyLittleApp opens web view to log into Mondo

2. User enters something to identify themselves into the web view (eg. email address, phone number)

3. We dispatch a notification to the user's registered device (ie. the Mondo app where the user is logged in – this may be the same device or a different device)

4. User opens the Mondo app and accepts/rejects the authorisation request

5. User returns to MyLittleApp, OAuth flow completes

In this flow, the user is not exposing their login credentials to the app… at worst, the app could extract their email/phone number. It also introduces another factor into the auth flow: the user's registered device.

Agree completely, this is a really well-written and informative article. Even more impressive is when you scroll to the bottom and notice this was written by a "17 year old student from Brighton who programs for a hobby."

The outbound dispatcher here seems like a prime candidate to write in a different language (Go for instance). Presumably it just makes HTTP requests so can be very simple -- building the request payload for instance could still happen in Python. And as you use Celery you already have a message queue over which it could communicate with the rest of your application.

All I can see is a striking resemblance to https://en.m.wikipedia.org/wiki/Biological_hazard