Square says it has resolved day-long outage | Hacker News Reader

Square says it has resolved day-long outage

techcrunch.com

37 pointsrossdavidh22 comments

This seems like it went under the radar a bit.

I cannot comprehend the impact if someone like VISA went down like this for the same amount of time. I feel like you might be getting what you are paying for with those merchant fees. They seem like extortion until you get a chance to take a tour through the fully-staffed command center and get to see just how much pain & suffering that real, live humans go through 24/7/365 to ensure the payment network is stable.

During my time working for Pulse (now part of Discover), I was expected to respond to a network outage within 60 seconds. Like, on the phone having a conversation with the bank's IT team. Anything going beyond 5 minutes would be escalated to a multi-party bridge call that doesn't end until everything is green again. The notion that the entire payment network could be down for more than a few hours is unimaginable at the scale of an organization like Visa, Amex or Discover.

Square is built on relatively modern (eg this century) tech stack. Modern tech stack are inherently more fragile than the real legacy stuff. The Mainframe tech that powers visanet has enormous practical limitations. But holy crap is it stable. Something about the dozens of abstraction layers that hadn't been invented yet when the original ISO payment standards and processes were first invented. The very close to metal and low surface area for breakage. And the benefit of decades of experience in these legacy stacks to shake out bugs. All of Visa's global commerce literally runs through just one of seven (mostly one of three) physical computers. Crazy stuff. But probably not sustainable many more decades. Eventually core networks will have to migrate to modern hardware, OS and SW stacks. That migration could create a lot of benefits, but I would not expect that maintaining the currently observed ~5-7 nines reliability to be one them.

I wonder how much of the stability is the extreme reluctance to do any upgrades or touch anything related to that mainframe code. In my experience with larger institutions using mainframes, its largely "code freeze, do not mess with it".

I wonder if this is a consequence of the stability, or a cause of it? Modern tech stacks are always upgrading components underneath to remain modern and frequency of changes cause instability no matter how hard you try. Not really any meaningful tooling to get around that yet

> I cannot comprehend the impact if someone like VISA went down like this for the same amount of time.

Last year one of Canada’s biggest ISPs went down for about a day. Knocked out its entire home/biz fixed internet, mobile network, private links, everything.

A bad update on Friday morning that seemed to catch them off-guard when it failed, so response was delayed as their processes were poor for a potentially (and ultimately) enterprise-breaking change.

Their CTO was on vacation and unaware for a while because they were on a dead company phone and figured it was a roaming issue.

Because the towers were up (but islanded), if you had a Rogers phone and tried to call 9-1-1, it would just fail instead of failing over to other networks because the phone saw the tower.

Many biz couldn’t do electronic payments, and any cloud POS knocked out so sometimes cash didn’t work either.

This also knocked out the Interac/Debit network nationwide because they depended on this ISP, even if you used another provider yourself.

https://www.cbc.ca/amp/1.6515869

https://www.itworldcanada.com/article/interac-outage-exacerb...

Our regulator let them redact tons of info in the investigation, which is hilarious because the Canadian or US Transportation Safety Board or US Chemical Safety Board would make the investigation totally open and say “fuck you” to your corporate “secrets” (that everyone should avoid, not steal).

We don’t treat telecom infra like we do other physical infra. And our regulator is toothless.

Didn't Rogers have two outages last year? I only really remember the one you mentioned, but I seem to recall cell service being down another day as well. My spouse and I use different cell providers (Bell and Rogers) to avoid a single point of failure.

2022: https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...

2021 (but was generally just mobile/voice, not fixed line): https://www.cbc.ca/news/business/rogers-outage-1.5992954

The status updates on this incident were very frustrating: https://www.issquareup.com/incidents/06zffytdqjdz

There is a global outage, stopping hundreds of millions of dollars of credit card transactions from processing, and all the detail they share is:

- "We are continuing to work on a fix for a disruption"

- "At this time, we do not have a solution for the disruption, though we have all the right people working to get it resolved"

- "Very sorry for the inconvenience today."

- "Thank you for your patience!"

I thought they scrubbed their hourly non-updates, but they’s just buried a level, e.g.:

https://ca.issquareup.com/incidents/wyb09mlsvvpq

I have never understood this line of thinking. Why do you, as the customer, care about the details? It's not like the issue will be fixed any faster if they posted updates about redeploying a service, waiting for CI to pass on a commit, someone investigating a bad database query, or toggling feature flags.

I don’t necessarily care about the technical details as a customer, but I do care about things like “we are prioritising restoring our PoS payment processing first, then online payments second”

After 8 hours, telling me about my patience is a slap in the face

> Why do you, as the customer, care about the details?

I might be a gigantic customer. I might be so big that I get to have a little bit of influence over certain strategic decisions you make in your product stack.

When I worked in the industry, there were big customers that would frequently tour the facilities to get a sense for capabilities and compliance. Part of the tour involves a private datacenter in central US that houses the mainframes and strictly designed around those assets. I was allowed to visit the facility and developed a deep sense of respect for how much capital & energy goes into making this stuff the bedrock.

If you are interested in doing 1bn+ USD business through a payment network, you are going to want hands-on assurances that things are going to support your needs.

So, even the barest of detailed reporting from Square would go a long way in helping to establish this sense of "bedrock", imo. Let some bigger customers push back on technology decisions that resulted in this outcome and insist upon more resilient choices.

The worst part for small businesses is that Square's offline payments mode (where you can accept credit cards without live authorization, at risk of later declines when they post) has been unreliable historically. With some Square outages, customers have been logged out, and then unable to enter offline mode to accept credit cards. Not enough attention has been paid to this critical feature unfortunately, it does not seem to activate properly in many situations. I hope it worked OK for businesses yesterday and cards were able to process today.

Anecdata - a local farmers market weren only taking cash payments yesterday AFAIK - it sounded like a big deal

Good reason to always carry a bit of cash with you.

Yeah yeah, you’re losing $5 per $100 a year, but situations like this won’t affect you.

If visa or Mastercard go down, I could say ATMs being emptied out.

Latch and Luxer One are two other apps that log you out if any HTTP error code returns - but instead of locking you out of your payments service, they lock you out of your apartment building. Apparently the lazy way to code “access terminated” these days is “if http.ok is false”.

This is a much larger problem. Fail open and you potentially let someone who has been evicted with a relatively unsophisticated method. Fail closed and you lock someone out.

Now, if you are going to say you are solving the problem, it means you have to think deeply here. However, all of the existing solutions have had unpleasant tradeoffs in degraded cases.

Trusting the client to be permitted to reauth is a guaranteed failure scenario, because then they can just put their phone in airplane mode and disable background updates.

Issuing time-restricted certificates to the client that expire after an interval is the only way to be certain, other than the property manager simply walking to the lock in question and using the lock app on it to force out your client.

Interpreting “No route to host, as various mobile networks do when your only cellular connection is Emergency Only, as “Purge the current session” is not beneficial to anyone’s security, and results in angry tenants trapped without cellular service in your parking garage, that then call emergency services via the elevator to get out. (This is not a theoretical example.)

I assume the core problem is that they only refresh their session once a month when the previous session expires, and if that refresh fails, you’re locked out — because they didn’t code it to update the session expiration each time it successfully phones home your lock’s status when used, and so when the monthly interval is up, it logs out because it incompetently manages the OAuth session.

My understanding is that it did not.

I was at a business yesterday that was affected by this. Their POS restarted several times and then the rep tried to have me pay through the phone app instead. Eventually the payment started processing an hour later for me.

"One X user responded by saying that their small business missed out on more than $2,000 worth of sales due to the outage, during what is the biggest month of the year for the company"

Ouch...that's sad.

And they say cause of outage not yet known...unknown, or not yet reported/shared with the public?

A friend that works there said it was DNS related.

I can second this.

Reply on news.ycombinator.com