Square is down

Probably some slight details of what is actually wrong, and where engineers are at in the process of fixing the issue. I don't think customers are looking for AWS-level incident details or anything, but people would like to know what is happening at a very high level, rather than completely generic updates for 16 hours.

like how other status pages do it

That makes sense, thanks for the perspective!

Your answer to "why does the technology stack of a critical multi-billion dollar business not have a failover ability" is "because it's hard"?

I worked at Square from 2014 to 2016 and this wasn't my experience.

In fact, Square invested a lot in infrastructure that was pretty impressive compared to other companies of a similar size. We're talking 800-1200 people and they had a single common custom deployment system that had been built before Docker was a thing, and later replaced that with a new in house system (p2) based on containers while I was there. They had a common custom CI system that all services used, and that actually worked quite well (much nicer than Jenkins which was common at the time). And Square really only had 3 languages that were acceptable to use for services (Java, Ruby, and Golang; node.js being phased out). I remember interviewing people at other larger companies and finding they didn't have things we had at Square in 2014, like a common way of configuring and registering services, or an easy way to see what's running on a particular box, or service-to-service ACLs, or a tool to coordinate online schema changes. Square even built their own S2S RPC protocol based on protobufs, and went on to participate in the design of gRPC.

Maybe it's changed since then, I don't know. But the culture was pretty solid when I was there. And the plateng team ran multiple DC failover tests of our service while I worked there.

From my understanding, yes, but they have rock solid uptime.

There's usually at least two single points of failure, the merchant processor (Square in this case, but other retailers use different processors) and the card issuer (Visa, MC, etc) responsible for approving/declining a transaction.

> How does VISA credit care actually work? (genuine question)

Full answer at the bottom. I hope this answers your question.

> Can a cybro-terrorist shut down credit card by DDoS VISA's server or something?

No. There's some nuance though[2]. You can't shutdown VISA globally, but you can probably target a set of customers for a particular bank, or a set of customers in a particular suburb.

In a single card transaction there are multiple parties (none of whom trust any other party): the acquirer, the issuer, the merchant bank, the cardholder, the cardholder bank, the PoS and the transaction switch. The transaction flows through some or all of these parties.

1. The acquirer[1], the issuer, the merchant and the merchant bank will all configure the terminal for when to go online and when to approve offline (using the card information and the cardholder information). Whose settings take preference depends.

This means, in effect, that small transactions (like your lunch) are approved offline and so even when the issuer service is offline for 16 hours, most people won't notice.

2. When the transaction is forced online it isn't going to VISA anyway. It's going to a switch (probably hosted by the merchant's bank, or a party that the bank hired to switch transactions).

3. The transaction is then switched to a bank, and the bank approves or declines the transaction. It may be the merchant's bank that approves it and reconciles with the cardholder's bank at midnight. It may (depending on the risk evaluation of that transaction) go straight to the cardholder's bank.

4. The bank verifies the transaction against a secure security service (Frequently called the HSM - Hardware Security Module) which is provided by VISA (or if provided by someone else, has to be certified by VISA/MC/Europay). The HSM decrypts the encrypted 38-byte payload from the card and verifies if the cardholder and card are genuine.

5. The bank hosting the HSM can then decide to either approve or decline the transaction. If the transaction is approved then the bank will reconcile with VISA at some later point (generally once a day at midnight, or similar).

6. In very rare cases, the bank might decide to ask VISA to approve the transaction. I've never heard of this happening (because I never worked on the backend side of things past the bank). If this happens, it's a relatively new thing (in the last 20 years or so).

So, the answer is "no", you cannot shutdown VISA cards by shutting down VISA's entire network. If VISA goes down the vast majority of online transactions will still work, and every single offline transaction will still work.

You have to DDoS every single switch at every single bank to make online-only VISA transactions fail.

Bear in mind that a single bank hosts multiple switches to handle the load, as well as spreading the transaction load across multiple third-party switches too.

The entire global card network is handling billions of transactions per second, and is set up so that a peak of double the current load doesn't break it.

Even if you managed to globally pwn every single desktop computer currently powered on with internet access, there may still not be enough resources to overload the global network to such an extent that you knock out card transactions globally.

The best you can do is probably hit a single bank or a single group of banks (which still wouldn't make a difference to most merchants as most have more than one bank anyway), or take down all internet in a single area (knock out fibre, copper and cell towers all at the same time).

The system is built for resilience and poor networking infrastructure, by adding in risk of fraud. People tend to forget that, for 4 decades, credit card transactions where a 24x7 0% downtime service. Well before 99.999% uptime on HA cloud providers were a thing, transactions were working just fine, just with an added cost to cover the fraud and only localised downtime (a single bank, a single suburb going down, not the entire network).

This is also why crypto-currencies bring no value to the transaction/payments space in terms of resiliency.

[1] Acquirer includes recent things like those USB/BL card-reader dongles you buy for android phones that are EMV certified with the app they come with.

[2] I may be misunderstanding, or even misremembering some of the details, and some of the details may have changed in the years since I first worked on this stuff.

I'm not the person you're responding to, but I built POS on Square. Or more accurately, Square is the merchant processor that's integrated into a very large piece of operations software I wrote/manage for a regional retail chain. The simplest explanation is that the company was already using Square when the software was commissioned, and didn't see an advantage in changing processors. They already had terminals everywhere provided by Square. The software gives customers the ability to put their cards on file (stored by Square, when they make purchases through the online store) and gives the retail outlets the ability to charge those cards on file. Meaning most of the transactions are done through the Square API now, not the terminals. The terminals serve as a backup so they can take payment in person if our servers go down.

In any case, I built all the software to talk to an in-house, isolated Payments API that I wrote, so that every part of our code that processes payments sends requests to our API, which handles all the back-and-forth with Square and distills the results. If it were necessary, I'd just change that API to talk to some other processor. But that wouldn't be a one-day job. As it is, it would take days or weeks to even arrange a banking relationship between the company and a different processor. I've argued for a backup to be built to a second processor, but the company hasn't wanted to deal with the cost or the hassle. They would also then need to have two separate terminals at each checkout register, and we would need to store customer cards on file with two separate processors.

Even so, perhaps this week's events will change their mind.

But I'm curious what you think would be a "more robust" option, for API access + physical terminal, since aren't all merchant processors prone to the same potential issues?

Tips calc you almost have to wait till last minute - unless your running payroll offset in the past. That's not a you problem, the system should be available.

And, for Square, at least split reporting from processing. Why have two critical systems down at once?

I heard from a friend in a wine shop that even offline sales weren’t processing.