Ocado's burning robotic warehouse | Hacker News Reader

Ocado's burning robotic warehouse

bbc.co.uk

73 pointszeristor75 comments

I found the fact that they're using modified 4G over unlicensed spectrum to control the robots pretty interesting. Latency, scalability and consistency seem to be the key drivers. There's more on that at:

https://www.mobileeurope.co.uk/press-wire/ocado-uses-unlicen...

https://www.cambridgewireless.co.uk/media/uploads/files/UK5G...

https://www.silicon.co.uk/networks/voip/ocado-4g-network-unl...

Any clue what radio IC they used? I didn't see the hardware mentioned in those links. Can you tune a vanilla LTE baseboard to unlicensed bands? Or did they use a software-defined transmitter with a custom LTE stack?

I believe most LTE chipsets can be tuned into any band you like, as long as you design your own RF front end (antenna and circuitry to/from the chip) and persuade the chipset manufacturer to give you a firmware update in such a way that they aren't liable for you distributing illegal devices. It's probable quite difficult to find manufacturer who is prepared to work with you if you're only going to buy 50,000 modules. I guess being Cambridge Consultants gives you contacts.

Given that Cambridge Consultants spawned Cambridge Silicon Radio you're probably right.

From the slides linked above they are not using LTE, but something loosely based on it.

So they may be reusing some PHY parts like the OFDM but the rest seems custom.

So basically if this is widely adopted 5ghz WiFi is screwed?

> But let's not forget there's a good reason why data centres full of computer servers are often built inside icy mountains or under oceans. Electronics get hot.

Gell-Mann amnesia effect applies here. This is pretty bad reporting.

Thanks

https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

You mean to tell me my ap-south-1 instances are not actually inside Mt. Everest?!

Nope, they're in an orbiting datacenter. You can't beat the dead cold vacuum of space.

PS: You can, the vacuum of space is actually terrible for cooling, I was making a joke.

Your objection is probably to the usage of the word "often" which is... debatable. It's probably also unrealistic. Otherwise yes, nobody ever built a DC underwater [0] or in the icy mountains [1] or underground [2].

[0] https://news.microsoft.com/features/under-the-sea-microsoft-...

[1] https://greenmountain.no/

[2] https://royal.pingdom.com/the-worlds-most-super-designed-dat...

Reporter was claiming that warehouse was freezing cold to keep electronics of robots cool. It's total nonsense. These robots probably emit less heat than regular forklifts.

The only reason for the freezing temperatures is to keep foodstuff fresh.

Also datacenters underground is only a marketing trick (we keep you servers in a bunker). If anything it only complicates getting rid of extra heat (eg. problems with London underground [1] )

[1] https://www.independent.co.uk/travel/news-and-advice/london-...

Given that the article also has no information about the cause of the fire ("The firm has not yet commented on the probable cause of the blaze and the investigation could go on for quite some time") it's also fairly easy to read the earlier "electronics get hot" as trying to point towards a suspected cause.

As you say, it's total nonsense - even if the cause does turn out to be an electrical fire, it's unlikely to be due to insufficient cooling of the whole warehouse.

I don't she meant to suggest that improper cooling led to the warehouse burning down.

She implied that DC are built underwater/in mountains (true) often (false) because electronics get hot (true...ish). And then drew the conclusion that this is one of the reasons the warehouse is cold (false) on top of the obvious one:

> The temperature was to protect the cold food, I was told.

I appreciate that "robotic warehouse" makes for a good headline but I just wanted to point out that only the part the retrieves the goods from the shelves is robotic. Humans are still packing and dispatching the orders.

It's amazing how many people in this thread are looking at the headline and jumping to the conclusion that there were no humans present at all.

> Humans are still packing and dispatching the orders.

Perhaps not for long, according to the article they're now training robot arms to perform the packing tasks.

That is a much harder task than just picking up boxes that are all the same size. It's essentially still a research task that many universities are working on.

I'd say we're at least a decade away from a robot that can pick up irregular items like a person can.

Is it that hard when you know, what are you picking? I mean, each box contains only one type of objects. Weight, shape and package type is known.

There's a finite number of SKU's it'll need to be trained to pick up and pack. It's not like it's having to learn to pick up and pack unknown things, it just needs to know how to pick and pack whatever Ocado carries in stock.

I reckon they'll have this mostly working in under a year.

Hopefully there were no humans present when the fire went up!

That's hilarious that a journalist thinks that datacenters are "often built inside icy mountains or under oceans".

It'd be funny if it wasn't entirely normal for the BBC's technology output.

BBCs reporting is nothing short of dire on everything at the moment. Random people on twitter are better journalists.

Heard an "Interview" with the head of Instagram on BBC radio yesterday. Mostly consisted of trying to extract clickbait 'headline' splashes, asking him if he was going to resign (what would that accomplish?), accusing them of incompetence because they hadn't solved the 'simple' matter of age verification, and then brought up legislation that "could involve jail terms for executives. Are you prepared to go to jail for instagram??". Absolutely no grasp or insight of any complexity or issues that might actually be relevant.

This was also a 'BBC Technology reporter' IIRC.

The BBC do seem to have realised this, considering a lot of their "stories" recently are literally just cobbled-together lists of tweets from these random people.

IIRC BBC Online was deliberately gutted by the top brass to keep the (Conservative) government happy because it was perceived as competing too effectively with commercial news providers.

So basically not lining up with the Daily Express?

Not so much about political positioning, but simply that the quality of the straight news reporting was high & there were no ads, so viewers were flocking to the BBC website in preference to other commercial news providers. This was deemed unacceptable by the Conservative government during a crucial time in the negotiation of the renewal of the licence fee, so the website was quietly gutted to appease them.

That’s the story I’ve been told at least.

If that is the case then it’s a sad outcome and it shows political overreach.

Broadly speaking everywhere the BBC did well the Tories took as reason to see them as impacting on free competition and commercial interests. There have been periodic Tory calls to privatise them for decades.

BBC online has been significantly limited twice, and BBC news more generally once (One relatively recent Tory change required the BBC to take on the costs of World Service that had always been Foreign Office until that point. Whilst editorially independent, the FO could ensure certain regions received output).

It is quite sad, as there were many areas they were doing well in.

BBC's tech stuff is possible poor, from editorial to accuracy and their reporters (for example Rory Cellan-Jones) have no background experience in the fields they are covering.

It is an interesting question: how do you plan for the unforeseen?

We as humans are very good at coming up with solutions to ambiguous/messy real world scenarios... robots less so. By definition, the current set of robots are programmed for specific situations.

As we automate more and more, I suspect we will have more and more “one off” catastrophic events. Yes, once we experience them, we can “test” against them. However it almost seems to argue for one amount of human in the loop no matter what, if only to give an occasional “that seems weird” check.

Am I missing something? Yes we can make an argument for machine learning to learn “nominal” and report off nominal... but that only works if you gave the right sensors being inputted in the first place.

Fire in buildings and warehouses is not an unforeseeable risk. It is a major, well-known risk.

Correct, this particular case is obvious (and they should plan for it). However what I am trying to dig into is the “less obvious”. An example, snow showing up in an area that hasn’t seen snow in the last 100 years. Or perhaps someone “crashing a car” into your factory. My point, and my questions has to do with the following at the core: “How do we replicate/approximate the ‘general’ knowledge that a human will inherently have?”

Yep exactly.

I agree - humans are able to collate generally disparate, individually insignificant events to identify issues and unexpected events, but the entire idea behind ML is that AI is trained to react appropriately to the specific range of possibilities it is provided with. While ML allows for significantly better generalization within the trained problem set, it still flounders anywhere beyond predicted/tested parameters.

Hopefully, developing gaming AIs such as AlphaGo and AlphaStar will help us with this, but for now I remain dubious - AlphaStar, at least, has trouble reacting at all to the unexpected.

I agree current automation can't handle unforeseen events, but I think it can flag them as they start.

Where I work we use a hybrid system that must be pretty common — instrument, establish patterns, watch for anomalies (automatically) and alert humans when they happen.

We clearly just need fire smelling and fire fighting robots. And robots to repair and maintain said fire fighting robots.

So better smoke detectors and better sprinkler systems, in other words.

No, that's old tech. We need sprinkler robots!

we need sprinklers that leverage machine learning to adaptively choose different fire suppressants!

Simply use function as a catastrophe detector and create a generic quarantine and run plan?

So their award winning sprinkler system clearly didn’t work.

Will be interesting to read the investigation report & see what actually went wrong.

A sprinkler system... for a giant lithium batteries fire? If it was an actual water sprinkler system that may have been an issue.

One hopes that they had thought of that, but who knows?

Looking at the warehouse videos, it’s clear that under the robots is a vast sea of individual towers of plastic boxes, all packed in tightly. I suspect that once that plastic was burning merrily, it would be very hard to stop & the heat would start cooking off the lithium batteries in the robots above. Hard to see how a sprinkler system can prevent fire from taking hold deep inside those towers of boxes, if that’s what happened.

A fire detection camera system could have picked out the heat plume in a short amount of time, and parked the robots as far away as possible from it. The article even comments on how cold the warehouse was, so any heat from fire in 'the hive' should have been easily detectable.

My bets are that it started somewhere totally different and unrelated to the robot system. Will be interesting to see why their system failed.

Could the robots bring buckets of water to dump on the fire? :)

I guess there's a lot of isocyanate foam insulation, once that gets going no sprinkler is going to help.

Isn’t this what the learning curve is about?

Although we’ve heard of hover boards, and the rare steals bursting into flames it is odd that battery robots on rails weren’t thought to be a fire risk.

That added to the fact that it wouldn’t be simple or safe to get to a fire in the middle of the grid.

For some reason I really felt for the engineers hearing about this. They've presumably lived and dreamed this for years. A bit like watching say a model railway go up in smoke: sure, no-one is dying, but what a waste of best-efforts.

I wonder how much building fires are a source of pollution in urban areas.

Whilst commuting into London Liverpool St I’d quite often see huge plumes of thick black smoke from fires in tyre dumps, which must be a huge source of pollution even with all the diesel lorries and cabs.

As an Ocado smartpass customer who is expecting an order today, this appears to just be manifesting itself as a few items out of stock.

Slightly disappointed though if I’m honest. This was the future. And now it’s going to be looked down on by every semi critical person as a complete failure even though the actual picking process was working very well.

For those who don’t know, Ocado was the only supplier in the UK that pretty much have a 1:1 correlation with what you order and what you get and haven’t been sitting on a shelf for days. All the other supermarkets are picked from local stores by hand and are quite frankly terrible with 2-15 items (from experience) missing on every order and some things very near expiry dates.

Sainsbury’s for example substituted me some bread which was out of stock with a fresh loaf expiring on the day it was delivered. The delivery turned up at 22:00...

This isn't strictly true. I know in South London that Tesco orders are not "picked from local stores by hand" - they're packed in and dispatched from the same enormous warehouses and depots that supply the stores, and there's some degree of automation involved (although not to the same level of Ocado.)

They only do that where there wasn’t enough room to build the home shopping distribution bays on the existing stores.

They're just trying to replicate the real UK supermarket experience, with the convenience of home delivery.

It's a feature, not a bug.

Its not that night and day.

Ocado will for example substitute some things that I don't really need, and aren't really substitutable. but if they don't have the nappies you ordered? No substitute, nadda. And it isn't like you can wait a week for nappies if you're running out.

Sainsburys I've always found ok, even for our Christmas shop, there weren't any substitutes.

Where Ocado is better is search. If I search for 'juice' in Sainsburys I very quickly get 'quinoa with juice of jojoba'. Ocado gives me actual fruit juices.

In case you don't realize: If they substitute something and you don't want it, you can just ask the delivery person to take that item back. I've not done that often, but when I have they've never had an issue doing it.

(the real big thing for me is that I have a weekly shop set up, and if I forget to look on their site, they'll send me something based on a tolerably ok prediction of what I usually buy; it far from perfect, but good enough that I occasionally doesn't bother log in on purpose, because I know it will be close enough to what I need to be ok)

You can reject things you ordered too, like the £40 of toilet roll added to reserve the delivery...... So I've heard of course.

This does work. I know someone who has done it a couple of times. I suspect that there’s an alarm somewhere that picks out regular abusers though.

I ask for no substitutions on Ocado. I haven’t had any missing items on my receipt for months. They seem to be pushing to do explicit stock allocation and actually remove the products from the store front now if they are out of stock. It’s a better experience if you get to choose up front that wing it on a substitute.

As for substitutes I’ve had some crazy ones from Tesco.

I've been shopping with them for nearly 8 years, haven't had that many issues.

I heard this warehouse only accounted for a small percentage of their capacity but I guess it's a big blow for customers who were primarily served from that location

On the flipside, I rarely have items missing from online orders. That's from a few years with Sainsburies and I've used Tesco for about two years now.

Waitrose do pretty well where I am (and Ocado dont cover my area). Although they dont have an app so you have to use the website.

They do have a reasonably nice app (again) now.

https://itunes.apple.com/gb/app/waitrose-partners/id13460641...

https://play.google.com/store/apps/details?id=com.waitrose.g...

Looking at other articles, this warehouse accounted for 10% of their capacity. It won't affect them that much, hopefully.

I have some criticism for Ocado. They were a freaking successful business in a no brainer industry... and then they decided to make robots out of a sudden.

Not only their risky robot venture cost them a lot, but they ended up burning more by operating them than regular warehouses.

Why?

Next time their plans should include some firefighting robots.

The article says they had a world class sprinkler system.

https://ocadoengineering.com/ocado-engineering-awarded-prest...

Was it arson?

I don't really care about this story. I don't feel compelled to care as it is not like other warehouse fires, e.g. the Asos one where you wondered about people and their jobs. When it is just robots and when you know the customers will just place their orders with one of the other click and pick outfits, e.g. Waitrose, Sainsbury's or whatever there isn't that requirement to give a damn.

In retail there are thieves who don't care about stealing from big companies like how they might care about stealing from small businesses as there is an assumption everything is insured. I am sure Ocado have insurance and I don't care too much about the shareholders of the insurance companies as I am sure they have their losses 'insured'.

File under 'expensive mistake' rather than 'human tragedy'.

The warehouse was not fully automated:

The robots, complete with green flashing lights, are collecting crates of food that sit in stacks beneath the grid and delivering them to chutes, where human workers beneath take the number of items they need to complete an order and then the crates are lifted back by the machines and returned to their spot.

https://www.bbc.co.uk/news/technology-43968495

There are some holes in your argument. Most glaringly, this fire presents a risk to everyone who has to fight it, and anyone else in the area.

If, and I know in this case it wasn't, your no-people warehouse was far from homes and was on fire, wouldn't one option be to just let it burn itself out? No risk fighting that fire...

Then I’d agree, but when you think of how far burning cinders can be carried on the wind, it would have to be very very far away. There would also need to be some legal framework to allow firefighters to say, “No thanks, we’re not risking our lives for your property, and we are 100% sure there isn’t a kid or homeless guy who snuck in for the night.”

http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...

Not really. It is not like they are heroically going in there to save cabbages and Pot Noodles. "Just think of those poor nectarines...!" doesn't cross my mind. On a scale of 1 to Grenfell it scores zero. Let the venture capitalists money burn...

In the UK we do have a huge amount of respect for those in the emergency services but it doesn't have the patriotic edge that Americans have in the post 9/11 world. Our firefighters are down to earth heroes yes, but not Hollywood glamorised patriot heroes. My grand dad was a firefighter so I am naturally inclined to think highly of everyone in the fire service, perhaps more so than most people.

Feelings are not strictly rational, my original comment was an observation that the incident lacked the feeling of awful that goes with a Grenfell or empathy that goes with things like the Asos fire where people were having their lives in jeopardy.

Very long comment for something you don't care about!

2 things that I find interesting are.

A. Humans are often seen as the weak link, and cause of accidents. That obviously didn't happen here.

B. As the article points out, humans would probably have smelt the smoke, and maybe the fire wouldn't have been so damaging, a human work force has useful soft benefits.

So no it isn't a human tragedy, but its a useful case study.

Reply on news.ycombinator.com