Toyota to restart Japan production on Wednesday after system failure | Hacker News Reader

Toyota to restart Japan production on Wednesday after system failure

reuters.com

74 pointsKnuthIsGod66 comments

My Toyota was the only car I ever bought that had exactly zero defects and worked perfectly in every capacity asked of it.

They are the Debian Linux of vehicles — older libraries but stable. No cutting edge package managers to crash your workday.

Therefore, for them to shut the line down means they are working through a serious issue, but as long as they keep that commitment to “quality, reliability and durability” and the reputation they’ve earned because of it, I’m sure the reason is worth it.

That said, the new Land Cruiser has cup holders for a third row seat in the US (a 250 globally, not sure why they decided to do a Lc brand on a 250 just for the US markets), but it… doesn’t have a third row to use them. It’s baffling and very not Toyota. So maybe they are slipping?

Two years ago, all Toyota plants were stopped due to a cyber attack. Toyota itself wasn't the target, but one of its suppliers got hit.

It was a belt manufacturer in Ukraine, whose cyber incident also affected VW, CARIAD, Audi, EvoBus, BMW and others.

Can't name more for obvious reasons, but more information can be googled easily.

NotPetya and 2014 changed how I see the world now. Everything is hackable, given the right chain of trust and dependency.

[1] https://www.reuters.com/business/autos-transportation/toyota...

A manufacturer in Ukraine? I wonder which hackers from which country would have incentives to do that in 2014.

I know, right? Rumors say it must have been a nation with incentives to conquer.

Jokes aside, I think that cyber warfare is the worst case scenario in terms of military efficiency, as it causes collateral civilian damage across the globe and its effects cannot be contained inside any borders.

And Toyota is a prime example of that. They ain't got nothing to do with the war, and were affected by it a bunch of times already.

You think the effects would've been more localized if the manufacturer were bombed instead? Collateral damage affected civilians because the target was civilian, not because of the method of attack.

On the contrary, look at Stuxnet: Infected well upwards of 200,000 computers, but only had any real adverse affect on 1: a Siemens Industrial controller controlling Iran's nuclear centrifuges. That's the kind of precision strike that would historically required covert operatives at immense risk.

Maersk was hit much harder than Toyota, and arguably with much larger global consequences.

> as it causes collateral civilian damage across the globe and its effects cannot be contained inside any borders.

Again it does sound like a modus operandi of a particular country... so many coincidences

Isn't CARIAD the software wing of VW? Why do they need seat belt?

There's a thing called "corporate VPN" and the way it works is that due to IP rights they share people working on projects, but each of them has a bunch of laptops for external corporations.

If VW and CARIAD work on a project, the guy at VW has a laptop from CARIAD inside the VW network; and vice versa.

Most of them even don't have Citrix VMs because they are hopelessly too slow to work with.

>Therefore, for them to shut the line down means they are working through a serious issue

This is quite the conclusion to draw based on your experience your Toyota vehicle. The product and supply chain teams won't necessarily have the same goals.

Given the reliability, why didn't you buy from them again? (I'm assuming you did not due to use of past tense.)

Great question. Honest answer: I bought an LC 200 and it was more car than I was using or really needed so I sold it. I fucking loved it.

Amazing vehicle. I’ve owned many amazing cars and it was an incredible vehicle. Top of my list. But I put 50k on it in 8 years and I just didn’t need it, so we decided to give it up for someone else to use and enjoy.

I’m driving my old 2007 Chevy now. Gets the job done. I don’t love it as much, but as I’ve matured I’ve really had less and less desire to own things. I just don’t care about owning such a nice vehicle. It's a wonderful experience, but I'd rather spend my money investing, spending time with my friends and family or helping somebody else out.

I don't know how to put it any other way.

I loved it, but I kind of fell out of love with the whole system of accumulation and ownership in general. I don’t want to collect things. I don’t find much joy in pointless wealth or fancy cars I guess.

I’d buy another one for sure if I needed it, but truthfully I’d rather ride my bike than have another ride. I don’t think I’m going to regret driving an old car on my death bed, even though I could afford otherwise.

It’s also deeply interesting how much people ascribe to the trappings of ownership and the appearance of wealth. I guess I lost my desire to have a nice car? Like who fucking cares? I know that’s weird. But none of that shit made me actually happy.

Travel, gardening, quiet love, children’s smiles and stuff like that is far more valuable than another car will ever be to me and I don’t want to waste my life paying for fancy shit that doesn’t make me truly happy.

Sorry if that’s too much of an answer but it’s heartfelt.

You have found happiness.

But don’t forget about being a complete idiot sometimes: if you avoid big trucks, fast cars, and cute girls - you’ll get boring to the people you love; because to them your dumb moments made you cool

Much love man, keep it steady

>if you avoid big trucks, fast cars, and cute girl

How do you sell this to your wife?

Lol. You two made my day.

My reason is super simple, no CarPlay at the time. Been a Toyota driver my whole life before the last car. There was a zero percent change that the wife and I would live another decade with some BS automaker ui. Toyota has seen the light and my mothers recent Camry purchase has CarPlay.

Speaking for the Australian market....

I had two Toyota Celia's and they were fantastically reliable cars. I would love to own another. But I look through the Toyota yard and everything there is a truck, or at least an SUV. So I have to look elsewhere.

From the top of my head, non-SUV/trucks:

Toyota GR Supra

Toyota GR86

Toyota GR Yaris

Toyota Yaris

Toyota GR Corolla

Toyota Corolla

Toyota Camry

Toyota Crown

Toyota Prius

Toyota Mirai

Toyota Aygo

+ Lexus models

There are more non-SUV/trucks than there are SUVs and trucks from Toyota. Seems strange you couldn't find any of these if you actually wanted one.

In the US, you can only buy new cars at a dealership. Not sure about other countries but here car sales have been declining or at least not rising compared to truck and SUV sales.

Btw, I believe the current generation supra is a bmw. They might call it a Toyota but it is still a German car.

every other market in the world the LC is 3 row seating. It isn't considered luxurious because the space in the trunk is more wanted. Look at LC in Philippines or APAC.

Absolutely. You know what you’re talking about.

So why didn’t they delete the body panel for the US? It’s a minor part.

You want Toyota to spend money revising a part...

To offer fewer cup holders to Americans?

Touché my friend. Accurate. But Touché.

Toyotas are too expensive today. For total cost of ownership, it's probably reached parity with used BMWs, it's just that with the Toyota you frontload the maintenance costs in the purchase price. The BMW will be much nicer to drive, but it'll have $1,000 in surprise maintenance (on top of oil changes) every year while being much cheaper to acquire.

When I was looking to buy a car, that was not what I saw. What you said may be the case with other brands such as Hyundai or Chevrolet, but for BMWs you are always paying more - frontloading and recurrent maintenance.

Except in the case of the i3 which has, by far, the highest reliability and lowest maintenance costs of any BMW ever produced. No wonder they discontinued them!

And also, every now and then, the BMW's engine or uber-complicated automatic transmission will have major issues, and you're out of $5k-$10k.

With regards to the cupholder, it's possible that Toyota is re-using parts from other vehicles which do have a third row, to bring down costs on the assembly line. Pure speculation on my part.

Yep, this is the correct answer. The new Land Cruiser is built on the same platform as the Lexus GX, which has the 3rd row.

Also, the 3rd row is being occupied by the hybrid battery system. In other countries where they offer a non-hybrid Land Cruiser, they have a 3rd row.

I have a 3rd row, but rarely use it. The cup holders are fantastic for easily accessible knock-knits, light flashlight, dog leashes, and infant stroller toys.

One could only speculate on the cause here without more info. I am reminded of reading The Toyota Way a while back, which does talk about how they would shut down lines for any defect. A coworker of mine was pretty adamant that this was A Good Idea(TM), but I thought about how much of IT best practices are about uptime and doing (often principled!) workarounds when you have operational issues, and moving forward in a degraded state.

I wonder if there are IT operations that do try to aim for the ~no defect approach like this.

A lot of the Toyota methodologies ended up in various agile software practices. Not so much operations of a live system, but development practices. For example, all your teams push their changes to an integration branch which is automatically deployed to staging systems for testing and automatically deployed to production. And when a failure is found, that pipeline gets shut down until the problem is solved. And all eyes are looking at that problem, rather than tapping away in their own world ignoring the 'somebody else's problem'. Nobody is pushing new code unrelated to fixing the problem, making the testing and deployment of an update slower. And in theory, improves your uptime because the quality of code ending up on production is better, and when that fails, fixes will end up on production faster.

That's interesting, I had never considered "main branch is red" as a similar fail state to the Toyota case, but I totally agree with that comparison!

>I am reminded of reading The Toyota Way a while back, which does talk about how they would shut down lines for any defect.

Like a crack in the frame?[1] Shots fired!

[1] https://news.ycombinator.com/item?id=37292321

In IT we (wrongly) use the word "production" to refer to the systems in operation serving customers: ie. the car that has left the factory. I don't know much about cars, but Toyota has a reputation for high reliability there.

In manufacturing, production lines instead refer to a previous step in the lifecycle, still in the factory. That's where you can pull the andon cord.

I’m sure finance folks are quick to hit the big red stop button if it’s causing invalid transactions to clear.

But generally the impact to the end user doesn’t really matter. There’s no reason to shut it all down if 1% of your request to a web server gets a 500. It might even make it harder to understand.

There’s no reason to shut it all down if 1% of your request to a web server gets a 500.

This is more like finding a bug in production and having all the devs stop working (including merging more changes) until it's fixed. Customers can still use the end product, you just stop making new things until you've solved the problem. There's some undeniable logic to doing that if you value bug free over fast.

Instead you should do the opposite: aim to get the process completed even in the eye of failures. Because failures at all levels, including hardware are a fact of life in computing, more so in distributed computing and almost every problem is now a distributed problem.

I would like to pose the claim that distributed computing is probably not more complicated than the distributed production of automobiles.

Maybe The Toyota Way is actually bunk from the get go. But if you buy into the idea that it does work, then why would it not be applicable in software development? This is the core thing that interests me.

Toyota sounds like the kind of place that still runs Solaris on SPARC and has no immediate plans to change.

From my conversations with some Toyota software engineers, they have a respectably sophisticated modern software/tech stack.

There's a ton of C++ since they have to interface with hardware. For tooling and internal applications it's a lot of Rails/Elixir/Javascript. Web APIs are backed by Elixir/Go. Data processing in Scala/Spark. Infrastructure in AWS, orchestrated with k8s. Truly agile prototyping then polish, significant automated testing.

Honestly, it makes "big tech" look like dinosaurs with their megasized-Java projects and 3:1 PM:engineer ratio waterfall planning once a year releases.

What they're missing is the operational experience with operating the software at scale.

You jest but back in the day (90s) one of my friends said they still ran Smalltalk VMs in production. That was 25 years ago but still possible…

Always headlines when this happens, but shutting down production like this is what Toyota believes is part of their magic sauce. They lose productivity when a problem shuts them down, but most of the time when things are working they gain due to lack of waste. But despite being a common part of curriculum around the world, it isn't that common to see, I think because intuitively it seems wrong somehow.

The Andon Cord: https://en.m.wikipedia.org/wiki/Andon_(manufacturing)

My best CEO borrowed that idea for our company and anyone could stop anything (or at least get serious consideration) by "pulling the andon cord" - if something was broken or being mismanaged, employees were empowered and encouraged and congratulated to fix it. Best engineering org I've been in.

Was there a mechanism in place to prevent people from abusing this process? (I’m assuming this is what you’re referring to when you said “at least serious consideration”)

I ask because based on personal experience and from reading comments on HN, most engineering orgs tend to be ego-heavy.

Were there times someone called something an Andon Cord situation and it turned out not to be? Of course! And that is fine. Were there "serial false andon cord pullers"? Not that I'm aware of - and if there were, the solution would have been to have an adult conversation with a focus on getting alignment. Pulling an Andon Cord was not for things you merely disagreed with, for that we relied on "disagree and commit" whereby you could bring data up later show it was a bad decision, but for now, work on making it work until you have that data.

> Was there a mechanism in place to prevent people from abusing this process?

Yes, it's called firing.

No one would pull the Andon cord if they thought they might get fired for it.

I think the flip side of this is that the workers are diverted from their typically assigned tasks and all work together to resolve the problem and they collectively take responsibility for enacting any lessons learned … it’s an approach that’s quite at odds with the traditional Taylorism approach of everybody just doing their own job. It might also be perceived as slightly Marxist by some while ironically being viewed with suspicion by unions … it also recognises that management don’t have all the answers. You would need an organisation with a lot of trust to enact something like this I think. Trust is in short supply in global business so you have to weigh the cost benefits of a global workforce vs cost benefits of organisational cohesion.

Toyota has major supply chain problems. I wanted to buy a Corolla recently, but the wait time was 8+ months, so I ended up buying a Honda Civic instead, which also had a wait time, but it was much more tolerable at 1 month.

The wait for a Toyota, if one wants any say in the options on the car, is quoted at six or eight months where I live. And as my friends have experienced, there's no penalty if the dealership takes longer than the quoted time to deliver a vehicle.

It's weird.

"Toyota's share price was up 0.12% at 2,439 yen in afternoon trade after spending much of the morning in negative territory."

It's nearing end of month in a low-liquidity market(summer, pre-labor day weekend). Next month is historically bearish(probably because the pro's return from vacation and start correcting all the summertime madness).

Version of news at the Japan Times: (no account/paywall)

https://news.ycombinator.com/item?id=37303807

Oh how I do hate the word "glitch". It tells you nothing and makes the whole thing sound like a happy little accident. A glitch to me is "Hey every time we order 200 of these fuses, we also get five lug nuts". That's a glitch. It's weird, but production moves forward. This is an incident.

No, it either a software bug, misconfiguration, network issue, hardware failure in Toyotas part ordering system or something to that nature.

Glitch just sits on two sides of the extreme. It's either too early in the investigation to call it anything else or it's after the investigation and there's not enough knowledge of the stack consisting of people, software, hardware (electrical, mechanical), and physics to figure it out.

Reminds me of my old college lecturer.

> There is no such thing as a bug or a glitch. Computers will only do exactly what someone tells them to do.

> You either have a mistake, or a lack of understanding.

Unless of course a random solar flare comes and flips a bit. ;)

Or any other error source like that. Thermal noise, background radioactivity, etc. As chip density increases the chance of bit flips also increases, and ECC RAM becomes more important.

Or something more simple like a browser update.

I’ve heard this thinking described as a fallacy somewhere - in economics I think - nobody has total knowledge of any complex system. So it is not valid to say any kind of complex computer system only does what manifests in its commands is the result of the interactions of many people, analogous to thermal noise jiggling a grain of sand in suspension.

Parkinsons law of triviality https://en.wikipedia.org/wiki/Law_of_triviality

I think you are referring to “bikeshedding” which isn’t quite what I had in mind.

I think what I’m describing is more closely related to https://en.m.wikipedia.org/wiki/Nirvana_fallacy

See also “Complete Information” https://en.m.wikipedia.org/wiki/Complete_information

The idea that all agents in a system should be aware of the consequences of all actions is itself fallacious in this thinking.

As such, the “glitch” can legitimately arise from the interaction of agents who do not have complete information.

I was once able to make a very solid case that a crash was due to a bit flip turning a two-byte cpu instruction into a one-byte “inc r8” and a garbage second instruction

I don't think bug means the computer is at fault.

Reply on news.ycombinator.com