Monzo: All payments are failing temporarily | Hacker News Reader

Monzo: All payments are failing temporarily

mondo.statuspage.io

42 pointscharlieegan354 comments

Goes to show, you can shake-up a sector with your modern infrastructure all you want, if you still have to plug into it somewhere...

To be completely fair to Monzo though, my limited understanding is that this reliance on a 'supplier' for payments will go away when the beta ends and Monzo's a full-fledged bank/current account, rather than just a prepaid card.

EDIT: ^ That was correct, that is the intention: [0]

> When we launch the current account later this year, it will not use a third-party card processor and instead transactions will be processed entirely on our own technology.

[0] - https://monzo.com/blog/2017/03/05/outage/

They're currently dependent on Wirecard for the underlying processing until they get their full banking license. At which point they can ditch that dependency - that's expected to be in ~May this year.

Edit: My mistake, its GPS doing the processing -- who are currently experiencing issues.

Wirecard are the issuing bank but not the card processor.

the processor is the dubai based GPS https://twitter.com/GPS_Processor

So it is the Wirecard infra failing then? And it depends if they can ditch that dependency; they will have to work with GPS or whoever themselves then. That is quite a lot of work. There are companies providing an intermediary to lessen that burden. Ideally you would want to cut it out indeed.

I believe it is specifically GPS, however I believe they are also doing that directly for their current account launch - they're putting a lot of emphasis on doing everything in-house

> I believe they are also doing that directly for their current account launch

They are, I just updated my top-level comment - they confirm that in the blog post the CTO linked in another comment here: https://monzo.com/blog/2017/03/05/outage/

My crystal bubble says it's gonna be a fun time when they'll roll the current accounts and be the payment processor.

It seems that currently they are (mostly) stateless and build around statelessness. i.e. get clients orders and pass it back to the payment providers.

Once they have current accounts, they'll have to hold serious funds and will be held to a much higher standard, which is a whole level of new difficulty to achieve. (unless they find a 3rd party service to outsource all of that :D).

Source: Switching from running market-makers to financial exchanges. Same stuff behind the hood, one is 10 times harder than the other one.

But then we can just have Monzo's own infrastructure fail or so. In saying this, I'm a happy Monzo user for almost a year and use it regularly, but it really dawned on me today that new technology brings new ways for things to fail, and when it's to do with my bank, it worries me a bit.

The nature of the way Monzo operates makes this much less likely to occur.

And if it does happen, the nature of how they operate means it should be significantly faster to resolve. Particularly because there is no reliance on a third party

Pedantically, they'd still be reliant on MasterCard's systems, and their connectivity to them, but if MasterCard went down Monzo wouldn't be the only one with problems.

Care to explain why? What is this "nature of monzo operation" that solves errors quicker?

If anything, a small startups who has to redo everything from scratch with limited resources seems like there is huge room for errors, and not many people to support and fix them 24/7.

They are a tech company first, financial one second. They made a decision very early on to build everything from scratch in the "right" way, pretty much no matter how long it would take or how much effort would be involved.

Their systems are incredibly stable. In fact, I don't think I can recall a single bit of downtime which was actually down to their system since November 2015 when I joined.

In the case which happened here, some kind of unexpected migration gone wrong - Monzo's systems are highly decoupled and separated. So any given database issue will only affect that given service. As each service is so simple and contains relatively little code, working towards a resolution can be achieved significantly faster. The smaller scope of responsibility also means that a mistake is less likely to happen in the first place.

> Their systems are incredibly stable. In fact, I don't think I can recall a single bit of downtime ---which was actually down to their system--- since November 2015 when I joined.

Downtime is downtime.

Ignoring and shifting the blame to third parties will not fly with customers who are unable to buy their groceries. Just saying ;)

That is exactly why they're bringing it in-house so the blame would be on them and they're in control.

I work at Monzo. We've posted an update on the situation on our blog: https://monzo.com/blog/2017/03/05/outage/

Thanks for posting this. I appreciate the transparency.

This caused an embarrassing moment for us this morning when buying groceries with a queue waiting behind. Since it was only two items, the clerk was kind enough to let me pay later.

Would have really appreciated a push notification as soon as the issue occurred -- they sent one a few hours too late.

If you're running a critical service, please make sure you have an appropriate and working notification strategy for major failures.

CTO at Monzo here. We are really sorry about that.

One of the things we have learned from this incident is to proactively reach out to all customers as early as possible, even if it means alarming a large number of people.

We did tweet about the incident and updated our statuspage the moment we discovered it, but we didn't proactively message all customers until a few hours later.

Monzo was your only payment instrument? If so, they weren't the only fail today.

No it wasn't. I couldn't remember the pin from my other card off the top of my head because I've been using Monzo for many months. I've learned from my mistake, I hope Monzo has too.

one trick for remembering pins is to decide a number that acts like an encryption key for all your cards. Then calculate the PIN by adding the key to four digits on the card. Then you only have to remember the key.

Is there anything really wrong with sharing a PIN for all your cards? What sort of attack does this make easier? Maybe if someone sees you entering a PIN for one of your cards and then steals your wallet?

Ah. I hate it when that happens. Hopefully Monzo will keep up their end of the deal in the future!

No reason to think that now that there is overwhelming evidence of the opposite.

> No reason to think [Monzo systems will be available in the future] now that there is overwhelming evidence of the opposite.

What would that be then? This outage is not Monzo's, but the processor's, affecting any card issuer that users it - a function Monzo has stated [0] it will do for itself without a third-party when it offers full current accounts later this year.

Calling this Monzo's failing is like complaining that Trello went down earlier in the week (due to the S3 outage). It's perhaps even less fair, since I imagine using multiple AWS regions is easier and more common practice than using multiple card payments processors. (Though I am just speculating on the latter point.)

[0] - https://monzo.com/blog/2017/03/05/outage/

So you are saying that my credit cards and current accounts should be held to the same standards as an average todo-list app?

No offense but I am gonna stay with my current bank ;)

> I am gonna stay with my current bank

When Monzo is actually capable of replacing your present bank, i.e. it offers current accounts, this issue won't exist.

> current accounts should be held to the same standards as an average todo-list app?

If your present bank used the same card payments processor, it would be experiencing the same outage. That's my point, not that it doesn't require more stringent certification than Trello.

> So you are saying that my credit cards and

There's no credit cards and the company's stated no plans to ever offer any, preferring to instead ask What problem makes you seek a credit card, and rethink what a modern solution looks like.

> No offense ... ;)

Why would I take offence? I'm just a user, no skin off my back if you're not ;)

I went out this morning with only my Monzo card because I've gone and lost the card for my actual bank account. Glad the first thing I did was get cash for the bus otherwise I'd have had real problems, as it was I just had to cut things short and come home. I had a very upset three year old as I discovered this when I went to pay for a Happy Meal, but otherwise no harm done (and he got one for dinner instead!)

Almost had the same problem a few minutes ago. Waiting in the queue at Boots and planning to pay with my Monzo card when I decide to check HN while waiting. The fact that this item was front page enabled me to get out of the queue and find my wife who had a different card and avoid a problem...

(Oh and quick note to Jonas, the iOS app did not push any notifications whatsoever...)

The notifications were sent via Intercom. Do you always get a notification from customer support when they reply? It's the same system, so if not there is likely some issue which needs to be resolved.

Can't recall ever getting a customer support notification, but do get notifications on spend and other expected events.

I got a push notification about the issue, but didn't read it. On paying for a coffee this afternoon either my payment worked or the barrista didn't read the decline message - a little lesson for both of us to read and not assume :)

Also from the status page:

    If you have incurred FX fees because you were unable to use your Monzo card, we will reimburse you for those fees. Please get in touch once this issue has been resolved.

Just makes you love them that little bit more!

Was it contactless? I've found staff really aren't used to online contactless transactions which can decline, they just take the "card read successfully" beep to mean transaction approved.

I've (accidentally) had a free dinner once when I used a locked prepaid card and neither me or the staff behind the bar noticed that it declined.

It was. Happened to me in a bar once too, but I got the decline notification so went back (since it was my local haha!)

I've heard that first paragraph -- not from Monzo, I'm not their customer -- so many times that it really doesn't mean shit anymore, even if 100% sincere.

It's right up there with all the companies that stress how "your security is our highest priority", right after they get compromised.

Hmm... Revolut had a similar problem today (https://twitter.com/RevolutApp/status/838392571827208193). Wonder if they are related, do they use the same processor?

I think they must be using the same processor (or some other mutual bottleneck upstream) as Revolut is still experiencing "payment declined" issues as of some minutes ago.

The recent post on payment processing is helpful, if you haven't read it, to help you know what is failing (the processor), and where they lie in the chain of entities responsible for payment processing.

https://news.ycombinator.com/item?id=13791913

I don't know much about card processors but could you have two card processors and fall over to one if one failed ?

Not an ideal situation for a 'bank'

Other banks relying on this third party will also be down. Actually happens all the time. Banks are very good these days at not losing your money but nothing else is guaranteed to work.

Monzo is actually slightly better in telling people. I even got a push notification earlier saying the service was down, which a normal bank won't (or can't) do.

Couldn't agree more. Plus, soon they'll have built their own infrastructure and this kind of thing will be less likely and at the very least easier to deal with.

Banks can do it, they have your phone number to message you.

They could do it, the problem is would they? The most I've seen with major outages at major banks is they end up on BBC News which is easily ignorable

They don't have a status page, though.

Doubt the folks outside of HN is much interested in reading websites.

They are interested in the modern (phone) app-y nature of it all though. So much so that Monzo has stated repeatedly they won't offer a web app at all - the one major thing I dislike about it.

None of the actual banks I use are having any issues. One of the most important things a bank can do is give you the ability to make payments. In this case, the fact that a 3rd party going down will stop this core functionality is horrifying.

"None of the banks I use are having any issues" - they all use third parties and they all have issues sometimes. Fact.

yup its Wirecard

http://uk.businessinsider.com/british-fintech-wirecard-monzo...

the processor is actually GPS (https://twitter.com/GPS_Processor)

Osper, Curve, Monzo and Revolut affected

which processor do they all use?

Reply on news.ycombinator.com