Hacker News Reader

Comment by chatmasta | Hacker News Reader

The Co-Op (grocery store chain) was hacked around the same time in likely the same incident. It took three weeks for them to get food back on the shelves at my local store. I don’t understand how that’s even possible… what happened to all the meat and vegetables in the supply chain? They just stopped flowing? They rotted? Why couldn’t they use pen and paper? It’s unbelievable to me that a business would go three weeks without stocking inventory.

View on HN

You can say this because ignorant, stock inventory is really hard especially huge warehouse where many items come and go 24/7

they can "move" it of course but who can guarantee how many amount goes from where and who ????

paper and pen where there are thousand items in single rack is nightmare, I can tell you that

well, apparently co-op couldn’t answer those questions with their computers because they got locked out of them…

Don't call someone ignorant for asking a question. He said "I don't understand how". If you know the answer, answer. Don't call him ignorant.

Yeah, I can do better in that part

at first I thought he underestimate this part of industry, I initially thought this because its common on HN mock tech company

I chatted to a staff member on the checkout of my local coop supermarket

She said that every shelf item is ordered on a JIT basis as the store stock levels require them - there are no standing orders to a store

Based on that, I presume they didn’t really know what any store would need

Even when they were struggling my local store still had a decent stock of lots of stuff - just some shelves were empty

You could (and people did) run this in the pre-internet days with basically just phone calls and a desk to receive them. The problem is that by now this represents an incredible increase in manpower required overnight.

I remember when I was a teenager working the register at a local store. The power went out one day, and we processed credit cards with a device that imprinted the embossed card number onto a paper for later reconciliation.

That wouldn’t work today for a number of reasons but it was cool to see that kind of backup plan in place.

I’ve seen cc impression machines within the past 5 years in small town america

In the UK the credit / debit cards I've had issued in the last few years have been flat, with details just printed, so that level of manual processing is presumably defunct here.

Don't forget chip & PIN is state of the art novel tech in the US. (From memory I think it was required here in the UK from Valentine's day^ in something like 2005.)

(^I remember the day better than the year because the ad campaign was something like 'I <3 PIN'.)

that is mostly because major US retailers sued Visa/Mastercard to make it not enforceable via lower interchange fees, since then they would have to change tens of thousands of point-of-sale systems at each one

And you need a process to follow. You can't just have nearly 4000 supermarkets ringing up HQ at random and reading out lists of 1000 items each. Then what? Back when a supermarket chain did operate like that, the processes like "fill in form ABC in triplicate, forward two to department DEF for batching and then the forward one to department GHI for supplier orders and they produce forms XYZ to send to department JKL for turning into orders for dispatch from warehouses". And so on and so on. You can't just magic up that entire infrastructure and knowledge even if you could get the warm bodies to implement it. Everyone who remembers how to operate a system like that is retired or has forgotten the details, all the forms were destroyed years ago and even the buildings with the phones and vacuum tubes and mail rooms don't exist.

Of course you could stand up a whole new system like that eventually, but you could also use the time to fix the computers and get back to business probably sooner.

But I imagine during those 3 weeks, there were a lot of phone calls, ad-hoc processes being invented and general chaos to get some minimal level of service limping along.

> Everyone who remembers how to operate a system like that is retired or has forgotten the details

Anyone who’s experienced the sudden emergence of middle management might feel otherwise :) please don’t teach those people the meaning of “triplicate,” they might try to apply it to next quarter’s Jira workflows…

One day you'll find a sheet of carbon paper in the office laserjet and you'll know it's starting.

I wonder if we could negotiate a return to typewriters and paper if it means individual offices and a tea trolley?

I agree, although it seems like a failure of imagination that this is so difficult. The staff will have a good understanding of what usually happens and what needs to happen. What they are lacking is some really basic things that are the natural monopoly of "the system".

Perhaps we need fallback systems that can rebuild some of that utility from scratch...

* A communication channel of last resort that can be bootstrapped. Like an emergency RCS messaging number that everyone is given or even a print/mailing service.

* A way to authenticate people getting in touch using photo ID, archived employee data or some kind of web of trust.

* A way to send messages to everyone using a he RCS system.

* A way to commission printing, delivery and collection of printed forms.

* A bot that can guide people to enter data into a particular schema.

* An append only data store that records messages. A filtering and export layer on top of that.

* A way to give people access to an office suite outside of the normal MS/Google subscription.

* A reliable third party wifi/cell service that is detached from your infrastructure.

* A pool of admin people who can run OCR, do data entry.

Basically you onboard people onto an emergency system. And have some basic resources that let people communicate and start spreadsheets.

part of the problem with emergency systems is that whatever emergency system is going to take you from zero to over capacity on whatever system it is, particularly if you are requiring communication from suddenly over-burdened human staff working frantically, and these processes may break down because of that.

In my case all the perishable shelves were empty - no fruit, no vegetables, no meat, no dairy. I checked every few days for multiple weeks and it wasn’t until three weeks after the incident I was able to buy chicken again.

It’s possible they were ordering some default level of stock and I just didn’t go at the right time to see it, but it sure looked like they were missing the inventory… when I first asked the lady “is the food missing because of the bank holiday?” and she said “no because of the cyber attack” I thought she was joking! It reminded me of the March 2020 shelves.

You forget we have entered the “Who the fuck cares?” era. When no one in the chain is incentivized to care, things just fall apart.

Interestingly Co-Op is so-called because it’s a cooperative business, which vaguely means it’s owned by its employees, and technically means it’s a “Registered Society” [0].

If you check CompaniesHouse [1], which normally has all financial documents for UK corporations, it points you to a separate “Public Register” for the Co-Op [2].

So, your comment has more basis in reality than simply being snark… the fact that “nobody is incentivized to care” is actually by design. That has some positive benefits but in this case we’re seeing how it breaks down for the same reasons nobody in a crowd calls an ambulance for someone hurt… it’s the bystander effect applied to corporate governance with diluted accountability.

[0] https://www.gov.uk/hmrc-internal-manuals/company-taxation-ma...

[1] https://find-and-update.company-information.service.gov.uk/c...

[2] https://mutuals.fca.org.uk/Search/Society/7240

I guess this is more snark, but honestly I am genuinely shocked when people care about anything anymore. Sad times.

There is a serious crisis of competence and caring all throughout society and it is indeed frightening. It’s this nagging worry that never goes away, while little cracks keep appearing in the mechanisms we usually take for granted…

I’m not following your logic. The co-op is designed for everyone to care _more_ because they are part-owners and because the organisation is set up for a larger good than simple profit-making.

In practice the distinction has long been lost both for employees and members (customers), but the intent of the organisational structure was not for nobody to care; quite the opposite

But there are millions of part-owners. Every “member” of co-op (i.e. a customer in the same membership program that just lost all their data to this hack) is an owner of it. Maybe the employees get more “shares” but it’s not at all significant.

And at the executive governance level, there are a few dozen directors.

There is a CEO who makes £750k a year, so it has elements of traditional governance. I’m not saying the structure is entirely to blame for the slow reaction to the hack, or that there is zero accountability, but it’s certainly interesting to see the lack of urgency to restore business continuity.

My family used to own a local market, and as my dad said when I told him this story, “my father would have been on the farm killing the chickens himself if that’s what he had to do to ensure he had inventory to sell his customers.”

You simply won’t get that level of accountability in an organization with thousands of stakeholders. And a traditional for-profit corporation will have the same problems, but it will also have a stock price that starts tanking after half a quarter of empty shelves. The co-op is missing that sort of accountability mechanism.

Responsibility diluted to the point of no actual responsibility?

Exactly, the bystander effect. But it’s not strictly due to the large size. Other big companies get hacked too. But if they have a stock price then there’s an obvious metric to indicate when the CEO needs to be fired. It’s the dilution of responsibility combined with a lack of measurable accountability that causes the dysfunction.

The problem is that cutting IT and similar functions to the bone is really good for CEOs. It juices the profits in the short/mid term, the stock price goes up because investors just see line go up, money goes in, and the CEO gets plaudits. There's only one figure of merit: stock price. What you measure is what you get.

It's only much later that the wheels fall off and it all goes to hell. The hack isn't a result of the CEOs actions this quarter, it's years and years of cumulative stock price optimisation for which the CEO was rewarded.

And you can't even blame all the investors because many will be diluted and mixed though funds and pensions. Is Muriel to blame because her private pension, which everyone told her is good and responsible financial planning, invested in Co-Operative Group on the back of strong growth and "business optimisation intiatives"? Is she supposed to call up Legal and General and say "look I know 2% of my pension is invested in Co-Op Group Ltd and it's doing well, and yes I'm with you guys because you have good returns, but I'm concerned their supermarket division is outsourcing their IT too much, could you please reduce my returns for the next few years and invest in companies that make less money by doing the IT more correctly?"

The incentives are fucked from end to end.

When everything is done by computers, no human really knows what needs to be done even for a simple thing as buying vegetables.

Buying and distributing vegetables for stores is not remotely a simple thing. It includes statistical analysis with estimates of demand for every store, seasonal scheduling, weather awareness, complicated national and/or international logistics, plus accounting and payments.

Some or all of those may be broken during a cyberattack.

That’s a good point but perhaps you underestimate the ingenuity borne from constraints.

If you’ve got trucks arriving with meat that’s going to expire in a week, and all your stores have empty shelves, surely there is a system to get that meat into customer mouths before it expires. It could be as simple as asking each store, when they call (which they surely will), how much meat they ordered last week, and sending them the same this week. You could build out more complicated distribution mechanisms, but it should be enough to keep your goods from perishing until you manage to repair your digital crutch.

The suppliers will know and be able to predict what a large customer like M&S is likely to order. They will probably be preparing items before they are even ordered. And surely their must be some kind of understanding of what a typical store will receive.

Reply on news.ycombinator.com