546 karma · joined January 30, 2013
Both occasions the pieces weren't structurally important, and were small decorative elements.
It cost £20 and I was dubious about it, but it cleans better than I've ever managed by myself with a spray. Lasted about 1.5y before it stopped being effective (just moves grease around rather than picking it up) and I went and bought another one.
Once you're storing a few million objects at standard durability it's par for the course.
This sounds like a bug we're aware of where an account that goes public -> private we'll reliably purge their tweets from the public index, but if an account goes private -> public sometimes we'll not re-populate the main index correctly.
> we had one person's account whose search results still showed searchbanned tweets
This part doesn't match what I'm describing, but could be explained by the logged in account having access to private tweets in search results that logged out / other accounts do not.
These ads are deliberately designed to prey on the desperate, unfortunate or the technically illiterate.
Some of them probably know exactly what they're doing and I have no sympathy, but some of them probably fall into the same category as people falling for Authorised Push Payment fraud.
It's just more expensive and another thing to maintain, and still doesn't account for _all_ failure modes (what if you sync really frequently and a bad change was made deleting all accounts?)
In my case I had a country name accidentally put in a region field, but it didn't look out of place because field labels are missing when filled in. Frustratingly, your contact card with that address will show a map preview that when tapped opens maps with a pin in the right place, so there's clearly some logic to "fix" incorrect addresses somewhere.
There were two - I pointed out both and they didn't seem super happy about it. To this day I'm still not sure if one of them was an unintentional error.
Complete outages are rare, and well-publicised, but things go wrong a lot more[1] than you might think without any communications to customers that anything is wrong, sometimes outright denying[2] that there's a problem.
https://www.fca.org.uk/news/news-stories/continuous-payment-...
> After buying a £2.70 gel screen protector on eBay, Lisa Neilson found her left thumbprint, which was not registered, could unlock the phone.
This suggests that an attack of "put a malicious screen protector on phone to unlock" is possible. I'm curious whether there was any re-training after applying the protector.
What worries me is the next unknown unknown, which is why we are insourcing. One thing I think Monzo can be particularly proud of is our incident response, and debugging of our own systems at speed.
That's why this article focuses more on what actions we are taking, and not what actions they are taking.
Just yesterday a major high street bank stopped sending payments for an hour, and was telling customers on Twitter that there were no problems.
Hell, the central system (what I called the Hub in this article) had a 12 hour split brain meltdown last July which had banks emailing each other spreadsheets back and forth for two weeks afterwards.
TL;DR- Largely Go microservices running on k8s, with http-based RPC calls for synchronous communication, and kafka for asynchronous communication.
As for sending and receiving of this kind of payment message, they are largely async but it does depend on the payment system we're talking about. When we build our own FPS gateway we're going to have to have something to manage "sessions" (TCP connections) which will block waiting for a response to an individual payment messages. Right now our communication with our third party Gateway is via a queue.
[1]: https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac...
The recipient bank will receive the money into their settlement account at that time. If the sending bank doesn't debit their customer then both sending and receiving customers will have the money in their accounts, but the sending bank will be out of pocket.
As another poster mentioned we already have a status page where we post about incidents as they happen (though obviously not in quite as much detail as here). Personally I think our main blog is a reasonable place to have this ️.
2. Multiple redundant payment processors would be great, but ultimately infeasible. As a settling FPS participant we have to have a single Bank of England settlement account, tied 1:1 to a "bank code". Multiple sort codes map to a single bank code, and migrating sort codes between bank codes is non-trivial.
It'd be great if we could migrate sort codes easily between redundant connections, but as we build our own Gateway we'll have complete control over how our failover mechanisms work. Here's to much greater uptime in the future!
3. As another commenter mentioned - yes! We're just doing staff testing for now, but we've got a waiting list up. It'll be a prepaid product issued by another bank before we get a US banking license, just like we were in the UK a couple of years ago.
What I meant here is they could tell that the corruption was being introduced by some component in their infrastructure, and they were only observing it for messages passing through one of their two active-active sites.