London Drugs closes stores until further notice due to cyberattack | Hacker News Reader

London Drugs closes stores until further notice due to cyberattack

cbc.ca

81 pointsnvy76 comments

Context for those outside Canada, London Drugs is more of a department store than a drug store / pharmacy. They tend to have lots of general goods, computing and camera and electronics departments, and so on. I used to sell top-end Nikon SLRs there, of all things… along with the shittiest Star Trek inspired telephones Curtis ever made.

Wild. These types of attacks would be mild inconveniences if they had procedures in place for continuity of business.

I am thinking:

- Manual record keeping (ie, physical ledger)

- Procedures to call insurance companies and verify coverage

- Procedures to ask local doctors to phone in (or hand write) prescription requests instead of using e-prescription system

- Credit and debit card information captured at point of sale with offline device. Processed outside of impacted systems or when systems recover

How 'Procedures to call insurance companies and verify coverage' prevents store shutdown? In my worldview insurance is 'sometimes we pay if ensured event occurs'. Lack or presence of the insurance changes nothing about the attack.

They're a pharmacy, they get directly reimbursed by customer's drug insurance plans and only charge customers for the uninsured amount. They need to submit prescriptions to insurance, and find out how much insurance is covering, before filling prescriptions.

They are completely down because of the breach. Insurance in any context is irrelevant.

You are mistakening medical insurance with computer/cyber insurance.

The post above you is talking about having manual communication with the provincial and federal medical insurance (Canada has national insurance) to confirm customers have the appropriate insurance ready so they can dispense pills, medication, and medical equipment, which is life critical equipment and supplies for many of their customers, without access to computers (which is doable if they had the procedures ready)

It’s to do with business continuity of the pharmacy’s work, nothing to do with breach.

You are assuming about the wrong kind of insurance…

Health insurance (not life, auto, home, commercial) often requires pre-authorization. At least in the USA. Some health plans like “hmos” are very stingy with many rules that prevent payment. It’s a fucking mess.

It might be different for Canada though, so this continuity step could be omitted.

In Canada you can buy your prescribed meds and then file an insurance form. I did that myself more than once when 'computers were down' but cash registers were not.

Yes, let's all do business like it's 1980, just in case the internet goes down.

All of this is possible, it takes a lot more time. That's why they are moved to "emergency only" business.

yeah, it's much better to do biz like it's 2024 BC

This is when cash comes in handy, but we're also transitioning into a cashless society so that option will slowly be gone.

Do stores really need to be connected to the internet all the time?

For pharmacies? Yeah - stock, ordering, prescription validation, (country dependent) drug seekers checks, vaccination records, ...

It's not just payments.

> This is when cash comes in handy, but we're also transitioning into a cashless society so that option will slowly be gone.

No, because giving money is only a part of the transaction.

> Do stores really need to be connected to the internet all the time?

Yes, they need to record transactions (in some cases live for tax purposes), update inventory, and in the case of a pharmacy also check medical files (if such a feature exists in the country in question), verify insurance information, check usage details on the specific drug, etc etc.

Some of those could be batched offline and verified when the connection is back up, but others can't.

Considering that stores existed long before the Internet, perhaps all this is just a convenience that has become a liability.

So how did this work before the Internet was commonplace?

> Some of those could be batched offline and verified when the connection is back up, but others can't.

That feels like someone decided that implementing a resilient business continuity plan wasn't worth it (which it may as well be, the impact is great but the likelihood low), e.g. manually making phone calls to verify the needed information, having backup paper copies of documents and so on.

exactly, some transactions can be streamlined with digitization but most rely on humans giving a fuck.

Before internet was commonplace you would have to go to a doctor and get a paper prescription which was sometimes done on a paper with watermarks and verification was that this piece of paper has a stamp or a seal on it and doctor's signature.

There were more forgeries with paper prescription than there is with online system.

You can still ask and get those AFAIK.

Are there provinces with electronic prescriptions? I always get a piece of paper (in Alberta).

I've had cases where the pharmacy (indeed London Drugs) phoned the doctor to ask them to fax a renewal.

In Ontario my doctor sends prescriptions "electronically". In practice it's not clear whether this is like an email, or whether someone behind the scenes prints out the prescription and faxes it. Apparently the local clinic has a team that is solely responsible for faxing things on behalf of the doctors.

Here in europe it is a government database where all the clinics and pharmacies are connected to, so doctor essentially creates a record in the database.

When you visit pharmacy they ask for id and enter your id number system shows them all your active prescriptions and past ones as well, which sometimes helps when your prescription is not renewed for some reason they can give you a week supply while you sort it out.

That depends on the country. In France it varies by doctor, some will use Doctolib (a great third party private company that does appointment scheduling, video consultations and digital prescriptions) which allows you to have a digital prescription that you click on a button in the app/website to share with a specific pharmacy, and when you get there they just get your social security card and... then print out your prescription, and scan and print on it how and when was it fulfilled. Others just give you an old fashioned hand written note, or print an A4 sheet of paper.

Alberta is pretty close to being electronic.

That makes sense given the shift to electronic records that has happened in the last few years, but obviously not a factorwith this incident.

Pre–internet, depending on the size of the store and retail chain, your cash registers might tie to a local small system which in turn talked to a central mainframe (IBM, Dec, Unisys, etc) using APPC over SNA, or TCP/IP, or Decnet, or whatever Unisys used. Leased lines were not particularly fast, but did not need to be, you’re talking less than a megabyte of data at most per day.

Lost of labor hours are the real answer.

Basically what would happen was that someone would write it down on a paper form, mail that form off to the typing pool at the corporate headquarter where some clerk would then type it into the central system with a delay of 5-10 days from point of sale to recording into the inventory management system being totally normal, leading to a lot of overstocking and waste.

So to run offline would mean getting a hold of a lot of people that aren't there anymore in addition to reintroducing all of the risk(and fraud opportunities) that running without real time access to centralized data.

Often local servers with daily batch uploading (at the time over phone lines) to larger corporate servers (often time shares)

>That feels like someone decided that implementing a resilient business continuity plan wasn't worth it (which it may as well be, the impact is great but the likelihood low), e.g. manually making phone calls to verify the needed information, having backup paper copies of documents and so on.

When you wake up one morning and you're under a cyber attack and no longer have network access, how easy do you think it is to just start manually phoning in prescription information?

This may come as a shock to you, but the internet has greatly increased the efficiency of business over passing paper around and making phone calls. You can't just flick a switch and go back to the old system.

> Some of those could be batched offline and verified when the connection is back up, but others can't.

One of the things that can be batched offline of course is card payments, which sort of negates the OP's point.

Credit cards can still use imprinter (from what I'm able to google but I haven't seen one in years), but debit transactions such as Interac need a connection.

Offline transactions are quite normal for many reasons

https://www.businessexpert.co.uk/payment-processing/offline-...

> Do stores really need to be connected to the internet all the time?

Some of them do.

A few weeks ago, I was at a Roam Burger outlet in San Francisco whose Toast point of sale system was down due to some server-side problem. They couldn't sell me a burger. Not even for cash. I had a nice chat with the store manager, who didn't have anything else to do. Then I left and ate elsewhere.

Toast docs: "If the restaurant cannot communicate with the Toast cloud, the devices cannot communicate with each other."[1] They have a lot of outages, according to third party monitoring.[2] Their own status page doesn't show those outages.[3] But their outage history does.[4] They're "transitioning" to a system where one of the local devices can be a host for the others when not connected to the "cloud".

[1] https://doc.toasttab.com/doc/platformguide/platformOfflineMo...

[2] https://isdown.app/integrations/toast

[3] https://status-dev.toasttab.com/

[4] https://status.toasttab.com/history

Back in the day when this happened employees would simply record the transactions in a physical ledger (my fancy way of saying pen and paper) and enter them later. Why is this not possible anymore? Do you need internet access to unlock your burger ingredients and turn on the grill? I would not be surprised if the answer is unironically yes.

I feel the underlying issue with a lot of these things is that no one seems to trust anyone, so nothing can be done "outside the system". As you say, the solution for these kind of outages are easy: just write some stuff down on paper and enter it later. But good heavens, we can't let people just enter data! Every possible avenue of abuse or mistakes must be covered.

Second problem is ill-designed systems which don't take exceptions in to account. Sometimes because of the preceding reason, sometimes just "oops, we didn't think of that".

The logistics fallout is a nightmare.

Let's say you're running a 10000 store burger shop. There is an outage and all of them are offline now.

There's the sheer hassle of recording everything and everything needs to be recorded correctly for compliance. Not only does it need to be recorded, but now it needs to be manually inputted back in correctly as well.

Let's say you could do that. More & more stores are getting rid of their fronting staff for the Kiosk systems. The store won't even have the capacity to keep up.

Now you've got boatloads of cash sitting in these stores that far exceed what normally would be there. Target for robbery.

If you pencil all the orders how will the fulfillment systems know when to ship you replacements and of what? Now reconciliation needs to happen across all of them to make sure they're properly stocked.

> Let's say you're running a 10000 store burger shop. There is an outage and all of them are offline now.

That happened to McDonalds on March 23, 2024.[1] Outlets in UK, Australia, Japan, Thailand were down for hours. Burger sales stopped at most locations.

No backup plan. Unlike Waffle House.[2]

This is a serious issue for disaster preparedness. The Waffle House CEO tries to get other key businesses to prep more. He says that if you can keep the Waffle House, the Walgreens, and the WalMart open after a disaster, the community comes back fast.

[1] https://www.bbc.com/news/business-68573106

[2] https://www.hrdive.com/news/waffle-house-serves-up-emergency...

Do they even have enough pen and paper at shops? And somebody that can use them in the right way to keep track of the transactions and make sense of them again later on.

> Yes, they need to record transactions (in some cases live for tax purposes)...

I'm pretty sure there was a time all these were handled without internet connection. it's just as society we decided that resilient fallback methods are undesirabled because of [insert of favourite regulatory rule]

Or that tax avoidance is harder if POS have to directly report information live to the tax authority, and reducing tax avoidance is a good thing.

"who will think of the taxes!"

Yes, but they don't need to be able to communicate with the entire internet. A few routing and firewall rules would go a long way for most stores.

In some places like Turkey, shops traditionally keep a "debt book" where customers will accumulate debt for their purchases and pay once they have the money. With the prevalent use of credit cards the tradition is much less widespread today of course but it is still used by people who become unbanked for one reason or another(i.e. people who are persecuted, people who went bankrupt and want to keep their transactions out of books).

So, no, loss of connection or system break down won't necessarily mean that the trade stops. People have many ways of issuing IOUs and they can go creative.

>This is when cash comes in handy, but we're also transitioning into a cashless society so that option will slowly be gone.

Pharmacy IT infrastructure is a little more involved than the POS system at the checkout.

I dropped by an LD shop tonight hoping to purchase one item. The doors were locked, with signage that the stores were closed (I assumed they would have reopened by now). They had some staff on hand, mostly to pass along info to would-be customers and let pharmacy-goers in. The guy at the door said I can make an online order and pick it up. So I did so. The website said "available in under 30 minutes!", which was great! I wasn't getting the "item ready for pickup" email though, and eventually I went back and was told "we literally can't access the systems to handle the order", the ppl at the store are totally locked out of their online systems (I guess). Pretty dire. They will apparently start reopening stores in a "rotating" fashion, which is pretty surprising (especially for a company of this size). The guy had no clue when I'll be able to pick up my order.

> Retail and pharmacy chain London Drugs [...] has shuttered its stores across Western Canada until further notice.

> The chain says pharmacists would still support customers with urgent pharmacy needs [...]

Er... which one is it now? If they support customers with urgent pharmacy needs, the stores can't be completely shuttered?

Also, there is no information about any of this (including which stores are open and under what conditions) on their website - but maybe they also no longer control that website?

"support" doesn't necessarily mean they sell anything. I don't know about the system in Canada, but around here the pharmacy can hold your prescription repeats for example or other important information that you'd have to get before you can pick up your drugs from a different location. Or you may have local vaccination scheduled. Or other things that pharmacies do.

At the location I visited yesterday, somebody let me in because I wanted to go to the pharmacy, but they turned other people away. They gave me a few days' supply without charging me.

I have a choice of three very-local pharmacies. I stuck with one, and now they know me (10+yrs) they'll give me a prescription, and get the note from the doc (electronically) shortly, after holidays etc. I'm just honest with the doc - and he's 400m from the pharm, so no shenanigans there.

> Er... which one is it now? If they support customers with urgent pharmacy needs, the stores can't be completely shuttered?

Customers can still go to another pharmacy.

The last I remember LD was a Toshiba/IBM OS 4690 shop for their point of sale system. I don't think there would be compromise there, it's robust (enough) that I haven't seen (many) exploits on that platform.

The rest of the company network though (other servers, endpoints) could well be screwed as well as anything trying to hook to it.

Reading between the lines it seems like it's some kind of ransomware.

The attack was recognised on Sunday afternoon, so they chose to attack during the end of the weekend.

If all stores are incapable to service and apparently no customer data has been compromised, it could be ransomware.

So, let's say I'm a diabetic who needs his insulin and was planning on picking it up today. What then?

Attacks on Western digital infrastructure will continue until the "technologically talented" in countries like Russia and China (among other places) start seeing it as a way to invite imminent danger into their lives.

>So, let's say I'm a diabetic who needs his insulin and was planning on picking it up today. What then?

You go to your pharmacy and they'll give it to you and make a note for repayment. Or you go to one of the other 500 pharmacies in the Province.

Which threat actor is involved?

Its not really a surprise that many commenters here seem to defend the relatively recent digital dependence. Kind of understandable, because most commenters here are working in tech, and obviously dont want to see problems like this. But this is asking the wolf how the sheep are supposed to be guarded... When I go to my pharmacy, the only reason why I need electricity there is to be able to pay with my watch. But that is a convenience. I always have cash in my pocket. Digital prescriptions are a very recent thing here where I live, basically since COVID. Before that, it was a simple piece of paper, and there was no validation that would require internet connectivity. But commenters here still seem to frame it as if the world would go down in flames if certain digital services were no longer available. Thats wrong, bordering on deliberately maniuplative. If a cyberattack can lead to a closed drug store, the problem is our reliance on digital, not the fact that the cyberattack happened.

Computers make it possible to have the convenience of massive retail establishments with large inventories and a reasonable hope of finding what we want when we want it.

And moving production off to foreign countries helps reduce the costs, but creates a ton of availability problems in times of crisis.

IOW, we have to weigh the advantages vs. the disadvantages. Its not enough to say "Computers make big things easier". Besides, there is probably a reason why antitrust laws exist. Getting too big apparently has more disadvantages then just monopolies.

Entirely agreed but they still must query the national database of filled scripts (to prevent dr shopping e.g. opiate abuse) and that requires at least a phone line.

Which is only an argument against VoIP, nothing else :-) Phones exist since a rather long time...

Is there such a thing in Canada?

It’s provincial and most provinces have it with varying degrees of data collection and accessibility but data isn’t shared across provinces.

One of the top comments in this thread is questioning whether a store needs to be connected all the time, and society's move to cashless. I also scrolled through the rest and couldn't find any that seemed to defend this "digital dependence"

Perhaps you could link to some?

> When I go to my pharmacy, the only reason why I need electricity there is to be able to pay with my watch

No it's not. Things like fire safety systems, security systems etc all need electricity.

And you can pay with your watch on a battery powered offline device anyway.

The need is for everything else in the hyper-efficent supplychain.

Canada cybersecurity is bad. Recently the Toronto Public Library got hacked.

And our government is worried about online harms haha

You're using 2 examples of hacks to reach a conclusion about the state of cyber security across an entire country?

There is more https://www.cbc.ca/amp/1.7099290

https://nationalpost.com/news/canadas-cybersecurity-under-si...

And our government is worried about calling people names

I do have the feeling that Canada in general is a little bit complacent - it's been real cozy for a long time. But do I have anything to back this up? Not really other than working at different places and knowing that general feeling of - it's happening somewhere else.

Reply on news.ycombinator.com