No excuses – we let you down
blog.revolut.com
blog.revolut.com
and
>"This server slowdown has never happened before"
Why does this transparency not include meaningful details of what actually went wrong? A "database malfunction" and a "server slowdown" are technical gibberish. This is not dissimilar to the kind of crap a traditional bank says when there are problems. It reminds of hearing "The system is down" from a customer service representative.
This apology is on the company's blog alongside a blog post titled "How to extract analytics data from your iOS application", complete with Swift code snippets[1]. So clearly they aren't worried about content being too technical.
We are talking about a bank and your money, why would it not be OK to err on the side of being too technical? Or at the very least provide a link for the more technically inclined.
[1] https://blog.revolut.com/how-to-extract-analytics-data-from-...
Because the people who can provide that are busy. This is customer communication, not post mortem from engineering. They had engineers spending whole day with extra hours on Friday to resolve the issue, then there was the weekend. I expect they'll take a day or more to dissect it internally, plan follow up, start working on it, etc. Publishing the plan can come later. (If they plan to do that)
What would you prefer them to do first: work on the follow up plan, or spend time writing a blog post about the follow up plan?
This blog post was published a whole day after the outage. Also I never questioned anything regarding the lack of follow up plan. I was questioning why technical detail wasn't more forthcoming.
>"This is customer communication, not post mortem from engineering ..."
The customer comms and service status were posted on their Twitter account on the 29th.
I still appreciate the intention to strive for transparency, but you need to do it right. If I ever do it again, I'll need to make sure that I know for sure how many people it's affecting, and how much.
What kind of a company are you building?
What examples can you provide of "being transparent and sincere in trying to do better" back-firing? Did massive clickbait articles get reported, blowing up the issue?
Then I figure out that I can't fix everyone's problems by fixing bugs only, get emotional, write a weird and over-emotional post, and just leave lots of customers behind at the end of the process.
It's still a small-scale issue if you ask me, as it is mostly personal, but it's an issue for sure.
> If you were forced to use an alternative payment method and therefore incurred fees, we will fully reimburse you for this amount.
does make me waver, they're obviously trying to do the right thing.
I'm still confused about whether they provide the same guarantees other organizations do under the FSCS. I wouldn't feel comfortable without this. Responses by their team to questions about this suggest they protect your deposits under some other regulation... but since it's different I don't understand it yet and trust it less.
I was stranded in Australia with my card at the checkout of a supermarket and my card stopped working with no explanation. An hour later, their Twitter feed told us that there was issues.
Another incident where I got cash out of an ATM. The ATM didn't spit out my cash but still debited the money. Revolut said it would take 90 days to get my money back. This was £500. At Christmas.
Unfortunately as a result of this, they lost my custom for life.
The biggest bank, Steward Bank, owned by the much adored Strive Masiyiwa and his Econet Wireless, is offline more than half of the time. They drop transactions and mobile money balances disappear without explanation or apology. Been happening for years now.
"At around 07:00 BST on Friday morning, our transaction database began to malfunction. Naturally, we followed procedure and switched to a backup server. Unfortunately, the backup server began to drastically slow down and was struggling to process live transactions."
That is very vague and sounds like a euphemism for "we don't know who changed the config file but it broke production and so we had somebody spin up a new instance on their laptop".
I any case I was in a foreign country when my card (and those of my travelling companions) stopped working without warning or explanation. I don't want him to be sorry, I want a rationale for why I shouldn't switch to an equivalent competitor like Monzo (on the assumption they have better reliability).
It's not a post from engineering. It's an announcement to customers. Most will neither understand nor care about the details of that infrastructure failing. What you want is an engineering post mortem which is a completely different kind of post. Don't get me wrong, I want them to publish one and would love to read the details. But their engineering team has (hopefully) much more important things to do at the moment.
What behaviour do you expect the app to have and why?
For all the funding they have, I'd assume they could fix these, but no, they don't.
It's the first line of text after the header.
edit: After watching the video it seems it's an international online bank that allows you to have and exchange among a bunch of currencies and pay with card at the cheapest rate possible.
It doesn't interest me because I don't plan on going out of the EU, where I can pay with just the phone in the vast majority of stores.
I always guessed it was something like, "Many WordPress and other blog-theme designers default to thinking of the blog is the product or main website itself, so their themes link the logo to the blog itself (and perhaps(?) there's no easy setting in the GUI to change this)?"
I see this default behavior on my own personal blog[1], which uses a fairly popular free theme, but in that case it's correct because my blog is really its own standalone thing.
[1]: Hehe, did you think I was going to try to plug my personal blog here?
The easiest way to run a database is to set it up as a SPoF.
You want to be serious and set everything up with redundancy, but then in case of problems it is just too easy to mess something up by doing the wrong thing when switching back or promoting a stale replica or something.
I don't even know if there are proper hosted replicated SQL-ACID databases-as-a-service available one would consider using for financial services and similar use cases.
There seems to be merit in paying DBAs proper money to handle these things.
ING uses Cassandra, which has no single point of failure, has been around for almost a decade, and would have made gitlab’s mistake a nonissue (and almost certainly would have made this one a nonissue as well - single instance failures don’t matter and linear scalability is a thing)
Here’s one: https://academy.datastax.com/resources/ing-groep-nv-exploiti...
And another https://m.youtube.com/watch?v=-sD3x8-tuDU
(And bad as it is, that's already better than British retail banks, though.)
Well, there's a weird turn of phrase. In what way is learning from the experience regrettable?
How do they make money?
Especially the "magical" debit card.
1. there's no monitoring in place
2. fail over is manual
3. there are no standby servers
4. nothing auto scales
5. backups are not tested
6. they lack sufficient manpower
7. there are no clusters
8. the service is one error away from an outage
It sounds like they completely ignored the operations side of running a service, and finally defaulted on technical debt. Yikes!I hope the best for them, but they have to start taking operations seriously.
Most people I've seen running companies are either clueless in general in IT matters (business or engineering people), or, "older" gents who grew up long before IT security and robust backups were "the norm" and stressed in every class and conference. If there's no direct money to be made, it doesn't matter.
Only in the last 3 years or so have I seen ANYONE of my clients care about potential hackers and breaches. And one of them only cared... because they were hacked by the Chinese.
Some of your points are almost dangerous. Not automatically failing over something handling large number of financial transactions may be a good decision, unless you can guarantee consistency between environments. Same goes for autoscaling databases (if you can even autoscale them in that environment - I'd expect them to be write-heavy and transactional, which is a good reason for scaling up, not out (or sharding or other things, not the point))
>No excuses — we let you down
>An apology from the Founder & CEO of Revolut
>The last 24 hours have been painful to say the least. Painful for our engineers who have been working around the clock to resolve the issue, but even more painful for our customers who have been unable use their accounts.
and
>You will get no excuses from me.
and
>We have let you down and I take full responsibility for what happened.
for what is essentially a very minor brief issue. He says:
>I would like to reassure you that all personal data and customer funds were completely secure at all times.
So I would like your help interpreting why he uses such completely inappropriate and over-the-top apology language for what amounts to brief [Edit: 1 day of] downtime.
What is your take? Why is it written like this?
Edit: I guess some Californians got upset and decided to downvote me? It’s not negative in any way, just a different style of expression. You certainly will never hear a Japanese or Romanian person exclaim “this is so amazing!” on a regular basis.
Also, I'm unsure of your exposure to Japanese, but superlatives are used frequently in daily conversation. "Sugoi"/"Suge" ("great/amazing!") or " kawaii" ("cute!") are a couple examples. There's some nuance and loss in translation, but the general usage and frequency is similar to American English from my experience with my Japanese in-laws, friends, and employees.
I do agree that Californians and our sun-fried brains seem more likely to take it to an extreme.
This reply contained all three of those. Kudos to the team for owning up to the problem and assuring their users that they take their money seriously.
However, the reverse is not true; an insufficiently-apologetic email would only serve to enrage customers who were already upset. These users would then be exponentially more likely to sever their relationships with the company.
Although I agree that this approach is a little off-putting, I think it may only read that way to someone not in that second group, which is really who the message is targeting.
That's my take, at least.
Revolut positions itself as a full service bank, including account number and sort code - they will have users that understandably use their product as their primary account. There will be regular users that will be unable to pay for lunch, and business users that will be unable to pay employees and suppliers (don't forget that it was last Friday of the month, a common payday in the UK). To let their customers go without access to their funds for an entire working day is not a 'brief downtime', it's completely catastrophic.
I'm also not surprised that user's did not hear back from support. I was formally a Revolut premium user that used the 'premium support', and they frequently had support response times >4 hours (one time over >6 hours). It's no surprise that when something serious happens their support infrastructure completely collapsed.
I want to give them the benefit of the doubt over their downtime, but I struggle to understand how it took them an entire working day to realise that an application server was playing up, and why a single server was able to take down up to 75% of their transaction processing?
Lastly, if there is anyone out there considering Revolut, please be aware that despite their advertising that they do not add FX fees they actually do add fees on the weekend: https://community.revolut.com/t/weekend-exchange-rate-surcha...
The title "there is NO excuse for 1 day of downtime" just seems so superlative to me.
It just seems to me that this is just an inconvenience. You yourself use the word "frequently" to describe the frustration you felt. a singe occurrence once doesn't seem like a catastrophe to me, though another respondent told me that it was.
For me, I don't consider it catastrophic if I can't access banking for 1 day. At any rate, thanks for the responses. It's clear some people consider this catastrophic.