Analysts argue that over-automation is to blame for Tesla problems | Hacker News Reader

Analysts argue that over-automation is to blame for Tesla problems

businessinsider.com

116 pointskristianp165 comments

Original and expanded story here:

http://www.businessinsider.com/tesla-robots-are-killing-it-2...

Can we please replace the link with this article?

It answers why the analyst think Tesla's way of doing things is an issue:

> the Japanese, try to limit automation because it "is expensive and is statistically inversely correlated to quality." Their approach is to get the process right first, then bring in the robots — the opposite of Musk's.

And, a hypothetical but expansive example on why Tesla's automation strategy might not achieve the desired cost savings.

Yes, sorry I should have used the original link.

the Japanese, try to limit automation because it "is expensive and is statistically inversely correlated to quality." Their approach is to get the process right first, then bring in the robots — the opposite of Musk's.

Of course, they use Japanese workers, which have a completely different work ethic than American workers.

I say that having worked in an American assembly line (not automotive though). I haven't worked in Japan myself, but the attention to detail of Japanese workers in even the most menial of jobs is amazing.

From what I’ve seen, Musk has actually been flying Japanese engineers into the Gigafactory (no joke!) They literally have their own buses going back and forth to the airport.

It's no secret that the gigafactory is a partnership with Panasonic.

In fact, my understanding is that Panasonic own and operate the cell production lines at the Gigafactory (which Tesla then take and assemble into packs in another part of the factory). Presumably these people will be Panasonic employees.

> Of course, they use Japanese workers, which have a completely different work ethic than American workers.

The Japanese manufacture most of their American market cars in America, using American labor.

Take your biases somewhere else, this isn't the place for them.

The big differences are union versus non-union and quality control, not necessarily the nationality of the workers.

So why did Johnny555 specifically bring up a purported relationship between nationality and work ethics?

Because he's prone to stereotyping based on nationality?

Then why are you chiding kchoudhu who's simply pointing that out?

I wasn't chiding kchoudhu, I agreed with him.

Given that e.g. Toyota's American workers are unionized, that doesn't seem to be the difference either.

You aggressively refuting a point the OP didn't make.

Cultural differences were an explicit problem when Toyota started to bring their manufacturing to America. This itself does not take any stand on work ethic, etc. Just that the Toyota system was a well designed totality that depended on the japanese culture as a necessary primer to get everything else working. Culture differences are a real thing, and human processes are non-trivial to transport directly for this reason. Training is of course the solution, but this needs to be explicitly taken into account - not just presume that everyone will behave automatically the same if they are given job titles and a factory floor where to operate.

Just looking at cars - the assembly quality of Japanese cars(Toyotas especially) lags far behind even French cars, and that's saying something. They are extremely robust mechanically(engine/transmission/suspension/drivetrain) but in a cold climate a Toyota will rust through 10x quicker than a Peugeot will, with Japanese cars using thin, minimal welds and little to no rust protection, plus in terms of panel gaps they are again far behind European cars. There certainly is attention to detail, just not in every aspect of car building.

I would not hold up French cars as the paragon of assembly quality. And I'm saying that as a huge Citroen fan (I've owned many different models, 2CV, DS, ID, CX, XM, C5, C8).

German cars are substantially better in quality, and finding high mileage German or Japanese cars is easy, finding French or Italian high mileage cars is much more rare.

> I would not hold up French cars as the paragon of assembly quality.

Isn't that the point of GP's comment? They're not saying french cars are well assembled, they're saying japanese cars are even worse:

> the assembly quality of Japanese cars lags far behind even French cars, and that's saying something

(emphasis mine)

And then goes on to say this is mostly limited to rust, which French cars used to do pretty badly but using better quality steel, paints and in some cases galvanization cured that problem.

French cars haven't rusted in decades. But that doesn't mean their assembly quality is better than the Japanese.

And I'm not convinced that Japanese cars are so bad in the rust department either.

The last batch of really badly rusting cars - ironically - were Mercedeses from the early 00's. Mercedes totally missed the boat on the water based paint front and got it dramatically wrong leading to a ton of re-work and expensive warranty issues.

I think that that was his point...

> the assembly quality of Japanese cars(Toyotas especially) lags far behind even French cars [let alone German]

I have to disagree, even though my knowledge is anecdotal.

All our Citroen cars NEVER showed any sign of rust. Our BX was 15 years old, when the hydro-pneumatic system fell apart. Citroen are known for excellent attention to corrosion protection on essential parts. I know a BX nearby from that era (must now be ~30 years old) that seems to be flawless.

Our VW Golf IV we currently own has had almost no exposure to the winter in his 17 years, apart from the last 2-3 years. He began rusting badly last winter, this will unfortunately end very soon.

High mileage German cars are available (I can only speak for middle Europe), because they have extraordinary value degradation (in terms of slow degradation), i.e. people still want them, because they are supposedly longer lasting, and supposedly better quality.

Maintenance statistics speaks otherwise. And god repairs on VWs are expensive. Just a new key fob is easily 5x as expensive as on a Citroen!

Edit: This blog-post [1] would indicate that my information is no longer valid. But then there is this post as well [2], which could be due to individual maintenance costs being higher for "luxury brands". Needs more research.

[1] http://www.jdpower.com/cars/articles/automotion-blog/5-car-q...

[2] https://priceonomics.com/the-lemon-index-which-cars-have-the...

BX was one of the first cars to have a lot of plastic in the body, you still see them occasionally.

Citroen CX and DS as well as 2CV were notorious for being rust buckets, the ones that are left have had tons of money poured into them. CX and injected DS were notorious for catching fire. Another major issue with the CX was euphemistically labeled 'gearbox separation' (I've actually had that happen to me on one of the two CX's that I owned, a 2400, super nice car but with the gearbox landing on the street it didn't get very far). The second one suffered a collapsed piston. All in all not the best score :), and that's before we get to rust issues. They were beautiful cars though, and super comfortable.

The more modern ones did not rust so much because Citroen got their rust problem under control somewhere in the mid 80's early 90's, hydraulics were a lot better too. The C5/C6 were probably the best cars Citroen ever made before they really became boring, and the DS is still a head turner after 60 years since it was first introduced, and it will always be my favorite car, even if I no longer drive them. On the plus side, my current car has the same suspension.

Parts for German cars are indeed ridiculously expensive but then again, they fail less frequently.

High mileage vehicles are the best way to check how reliable a car is, if it works out well the drivetrain and engine will outlast the chassis. I've had VW Golfs (older diesels) with 250K plus on them and a whole slew of Mercedeses, never a problem that caused the car to be stranded. I've never owned a BMW so I can't say anything about those but my French cars have been a long list of all kinds of interesting problems. I didn't mind, it taught me a lot about working on cars (and doing an engine rebuild on a DS is something special), but they never were very reliable. Oh, and I forgot to include one Citroen in my list, Xantia diesel which was fairly low mileage (180K) when it spectacularly blew up the head somewhere on the way back from Switzerland. That wasn't a fun trip at all.

Oh, you were talking about pre-80ies french cars...

Yeah, I remember my dad telling me those were known to rust almost immediately, and they apparently (to his perception, based on the cars we owned) not only got their act together, but excelled at it, somewhere end of the 80ies/beginning of the 90ies.

And that is true, the BX has considerable plastic parts (although only the front hood and the front mudguards, which is awesome if you ever bump something, nothing happens!), the non-visible chassis is still metal and very rust-safe.

On a side note: Our Renault 21 Nevada from the early 90ies was a different beast... I think almost everything on it had to be replaced randomly, and although it made it to 270k or more (I think), my dad was happy to leave it behind and he only kept it because it was a company car.

We used to joke they rusted in the leaflets.

Renault never really got their quality control in order, they are still hit-and-miss today. If you're lucky you will have a fantastic car if you don't it will nickle-and-dime you to death. I've had a couple of Espaces, I absolutely love the form factor but the reliability and general quality are horrible.

There were older Citroen cars with a lot of plastic/composite parts.

For example, the Mehari used a lot of plastic parts (ABS) and is from the late 60ies.

It didn't prevent rust completely however. For example, the Mehari was as prone as the 2CV to have a chassis breakup.

My father actually got one with a broken chassis (in fairness, more due to the fact it has been driven in very rough roads around South Europe and Asia Minor in the 70ies) and spent a summer disassembling it completly to replace the chassis (he still has a beautiful photo with all the parts laying around).

One thing to keep in mind is that the Mehari, as the 2CV, was designed to be a very simple and cheap car from the 60ies (the 2CV design even traces back to the 30ies). This actually somewhat wonderful as it's possible for a single person to service these cars alone with a few basic tools if he knows a thing or two.

Mehari and 2CV share a very large chunk of their design. And that was a very special car, I'd love to have one but they are hard to find in good condition.

It's funny. In my country Toyota is the most popular car and considered having the best quality. French cars are considered just the worst in any aspects, there's barely any (except Renault Logan and its derivatives, but it's more Romanian car than French). And German cars are considered terrible quality but having luxury features. But Lexus is just a dream of most Kazakhstani drivers.

there's also training to consider. japanese automotive worker have more training, less turnaround than other workers, a greater incentive to stick with a single company because of the "honor rules" that drive their planned careers and in general they are rewarded to optimized their workflow instead of berated with even more work.

also, local manufacturer where I live are famous for messing up hiring non qualified workers. Panda had a recall because seats weren't correctly screwed in, for example. most of the time when you dig the faults are in the manual part of the lines. And the Alfa I drive has only one misaligned component, and it's the manually placed windshield - the alignment mark are off more than an inch!

that's not to say that the problem is in the workers themselves, because their training and their efficiency is part of how the company configure the plant, but it's not just the work ethic of individual workers, more like the work culture of the whole company

I have visited a few factories in Japan and have seen manufacturing of components like heavy duty pumps and gas turbine blades.

You are absolutely right. The work ethic of Japanese workers is awesome.

I am an engineer and I am damn sure that the Japanese use humans to perfect a process and then replace them with a robot, with the same human manning the robot. Japanese workers and management always work together to perfect a process to such an extent that it becomes very easy to replicate it with a machine.

Its very inspiring to see the Japanese work.

> Japanese workers and management always work together to perfect a process to such an extent that it becomes very easy to replicate it with a machine.

It also means basically the workers behave more and more like a robot. I don't think it produces happy humans.

But getting a process as good as it can be should give people a feeling of pride and accomplishment. That should make them happy, no?

My guess is there is a balance somewhere.

But given Japan has such a high suicide rate and depression I would not bet on them for having found it yet.

Suicide rate and depression issues are, atleast in Japan, not usually related to work pressures, but due to social issues.

When you spent 8 hours a day, half of your conscious time, in a situation, I have a hard time it has no impact on depression and suicide. If you have a fullfilling activity for 8 hours, it's hard for me to picture the rest of the day canceling it to the point you want to kill yourself. I tend to think it's a general context, and it includes work.

But let's be honest, I haven't worked, let alone lived in Japan, so this is just uneducated gut feeling.

Ohh no, the Japanese work wayyyy more than 8 hours a day.

https://en.wikipedia.org/wiki/Salaryman

though things are changing nowadays.

Some people are happier with rote, meaningless work.

I have a relative who for 40 years has done the exact same job -- he drives a large dump truck from a stone pile in a rock quarry up to the crusher, dumps his load, and repeats, endlessly.

He's had many opportunities to transfer to other positions, but he enjoys his "meaningless" work.

He's retiring soon, which is good since the company has been testing an automated truck.

More and more like a robot, to some extent, but more and more skilled. This, I believe, instills a sense of pride and accomplishment. And when the same worker is rotated to another job, there again, he builds his skills. Of course, some may be unhappy, but I don't see any particularly negative aspect to being process oriented.

Moreover, even with the process automated, often, a finishing touch is given by the operator. And, often, Japanese manufacturing facilities, depend on human skill and ingenuity to extract maximum quality from the process, even if the process is heavily automated.

In addition to all the above, one unique aspect of the Japanese workforce and management, and I have seen this first hand, is that even the lowest level worker can freely give his inputs regarding a process and then the upper level workers, engineers and managers actually listen and discuss the inputs.

In the plants that I have been to, engineering, design and workers are all located on the factory shop floor and are expected to work together in unison.

Yet, given the choice, would you rather learn to turn a screw perfectly 10000 times or learn programming ? If so can I hire you to do just that ?

One seems way more satisfying than the other, not matter how good one can become at turning the screw.

To be fair, most processes are slightly more complicated than turning a screw. Most processes maybe monotonous but are sufficiently complicated to require concentrated effort to get right. For example a turning operation or a milling operation.

Sure. But I worked in a factory making screws for 2 years, that's why I'm taking about screws.

Even this simple product has numerous subtle steps in its production.

Yet on operator may very well do one to three different move, 10000 times a day. It's very tedious. Perfecting that is certainly satisfying, but it's hard for me to grasp how it can counteract the crushing experience of it all.

Japan has successfully exported its processes to non-Japanese countries, so "Americans can't build cars good" is not a good argument.

They were American processes before they became Japanese processes.

https://en.wikipedia.org/wiki/W._Edwards_Deming

This American Life did a couple of really good stories about NUMI, the ex-gm factory where Toyota attempted to export their process. Interestingly, this it's the same factory where Tesla builds it's cars today. Spoiler: many challenges to getting Americans to "build cars good" under GM management.

1. https://www.thisamericanlife.org/403/nummi 2. https://www.thisamericanlife.org/561/nummi-2015

This just seems a bit hand-wavy. What were the Japanese manufacturing, out of how many parts, what was the model lineup and were there any significant differences in the final product as far as customizations?

Tesla is solving a very narrow problem of a single model of the vehicle with limited parts inventory and somewhat reduced complexity (peculiar to EVs in general), offering very little customization (customers can choose the color and the interior, but that’s about it).

Thanks! We've updated the link from https://techcrunch.com/2018/03/28/tesla-is-overusing-automat....

Just a short time ago there were negative headlines about Tesla parts being built by hand [1].

https://www.google.com/amp/s/www.wsj.com/amp/articles/behind...

wallstreet analysts don't know jack shit and come out with statements whenever the wind blows their way from the woodwork. Wallstreet is short Tesla, bonds are trading new lows... Goodluck - so basically you are shorting the guy that is landing fucking rockets back down on this planet? great thesis. I will take the other side of that bet.

>I will take the other side of that bet

The stock ticker is TSLA, all you have to do is load up. No need for chest thumping.

I already bought in. "Overusing automation" is a new one.

Check the price now ;)

What does landing rockets have to do with building cars?

What does it have to do with automation...? Quite a lot.

Let's rephrase that: what does creating the world's finest space program in a mere decade and a half have to do with succeeding at profoundly difficult technological ventures where you have no prior experience? Quite a lot.

"He did this one hard thing, therefore he can do all hard things"?

Ah, so now SpaceX has been designated a "single achievement" (not a vast range of achievements over 16 years), and the singular "SpaceX" is apparently also the only thing Elon Musk has ever done. Is that what you're saying?

The problem comes when Tesla has spent immense amounts of capital on automation AND is hand assembling.

Disclaimer: I work for a Tesla competitor.

"production bottlenecks" == issues with the automation. Their solution is then more use of manual and quick fixes during QA.

that's the ironic outcome of their misguided attempt to just throw automation at every step in the process. They have ended up with the least automated car factory in the world.

Hmm, who do I trust, the guy who figured out how to land rockets onto a barge, or some “analyst” desk jockey who has been consistently wrong his entire career? Tough question. I think I’ll go with the rocket guy. He has a pretty good track record.

But SpaceX builds very complex machines at a low rate, whereas Tesla tries to build less complex machines at a very high rate. I'm not sure if knowledge is transferable.

This is an interesting juxtaposition, but I’m not (really) sure it’s true. Is a rocket, even one with an advanced control system such as those designed by SpaceX, really ‘simpler’ than an electric vehicle? We certainly think in those terms, because rockets are exotic to our experience whereas cars are experientially mundane for most of us, but I don’t think the facts of the matter are that clear cut.

Werner von Braun had to crash hundreds into London and Antwerp to learn how not to crash them at white sands and Cape Canaveral

And how does that compare to the development of early combustion or electric engines?

I'm not so sure about electric cars, but I would judge a rocket to be less complex than an ICE car. Rockets just have much more spectacular failure modes and are more expensive due to size.

There was a point in the early 2000s (and I don't know how long it persisted for) when ESA had a very hard time hiring rocket scientists because they were all being slurped up by car manufacturers to work on drive-by-wire systems.

They're certainly more complex as a result of higher quality demands. A car can be tested under real-world conditions and fixed if it shows any manufacturing defects. Any mistakes with a rocket are very expensive.

Testing is also very different. For a car, you can build a batch and test it extensively under all conditions, fixing errors as they come up. If you develop an assistance system, you can test this extensively before releasing it. Rockets are so expensive that SpaceX had only a few trials for their landing system. You need to make sure that both hardware and software work perfectly without ever testing them in production.

It's worth noting that one big advantage SpaceX has over other rocket makers is that SpaceX gets to inspect landed first stages. That makes their rocket development closer to airplane development than everyone else's rocket development, and no, it doesn't have to be perfect before it is tested. It just has to be good enough to land.

Does he really have a good track record? If I recall correctly the Falcon 9 has a terrible success rate for a modern rocket. Something like 94 percent.

How many years did SpaceX do over that modern rocket? How many years did NASA do to get to said modern rocket? I'd say he's on a pretty good roll.

SpaceX didn't start in a vacuum, or in the 40s.

And don’t forget in 1969 when they landed on the moon, they had to code all the timestamps in negative.

Neither did Ariane 5, Atlas V, Delta IV or Soyuz-FG. Of course SpaceX profits from the state of the art, but not developing the Xth iteration over a proven and battle-tested design with decades of flight-data is a disadvantage for SpaceX.

SpaceX benefits from being able to inspect landed boosters. This comparison is apples-to-oranges.

Falcon 9

51 launches

48 completely successful

49 partially successful

94% success rate (complete success)

96% success rate (partial success)

Ariane 5

97 launches

92 completely successful

95 partially successful

95% success rate (complete success)

98% success rate (partial success)

Atlas V

76 launches

75 completely successful

76 partially successful

99% success rate (complete success)

100% success rate (partial success)

Delta IV (including Heavy)

36 launches

35 completely successful

36 partially successful

97% success rate (complete success)

100% success rate (partial success)

Proton (since 2010 as an arbitrary cutoff)

65 launches

58 completely successful

59 partially successful

89% success rate (complete success)

91% success rate (partial success)

I don’t really think there is any way in which the adjective “terrible” would be an appropriate use of the word here. Also note the low numbers involved here, the different contexts and the plain arbitrariness of such comparisons. One failure more or less can make a world of difference here, despite maybe not being the most reliable indicator.

94% is about the long term ballpark figure for successful orbital launches but it is true that some newer rocket families may have ever so slightly higher reliabilities.

Now divide that failure rate from dollars invested in each project to truly see the differences.

(Adjusted for inflation etc and remove the arbitrary cutoff for Proton.)

Atlas, Ariane and proton have fifty+ years of investment tail behind them and a fifth generation in atlas is probably more than musk has had in spacex in generation terms no?

Does it include the rocket and satellite destroyed while refueling?

What about Soyuz? IIRC, they haven't lost a manned mission since 1971.

>What about Soyuz? IIRC, they haven't lost a manned mission since 1971.

Neither has SpaceX :)

Soyuz is somewhat more complicated since it's an entire family of rockets:

The Soyuz-FG (2001-today, currently does all manned flights) has a 63/63 success record.

The Soyuz-U2 (1982-1995) also has a 72/72 success record.

The Soyuz-U (1973-2017, also manned flight) has a 765/786 (97%) success rate, with no failure in their manned flights.

The Soyuz-2 (2004-today) has a 68/75 (91%) success rate.

Given the rate at which SpaceX has been improving their line-up you can't really do a 1:1 comparison between platforms that are essentially static and what SpaceX has been up to.

And over time their failure rate has been dropping steadily, since 1/1/2017 they've launched about half of all F9s and all have worked perfectly. It's gotten to the point where the press will crow 'failure' if they don't manage to land the first stage on the barge, which is apples-to-oranges with the rest of the industry, who are - rightly - getting seriously worried.

The report also seems to discuss the evils of excessive automation in general.

So yeah, this could apply to Tesla cars. Or rubber toys, which are also manufactured in a highly automated environment.

Not a comparison I would choose. He has a pretty good track record of producing a few dozen rockets a year. You could be generous and say a few hundred rocket engines a year. A comedian might say he has the same track record with cars - ba dum - tish!

Tesla's inability to quickly ramp up Model 3 production is pretty much one of those "fact" things at this point. The same goes for every model they've brought out, it's taken a long time to get decent production rates on any of them. So this isn't so much predicting problems he's going to have, but explaining the causes of all the problems they've already had and are having right now.

But that's the great trick he pulled....convincing everyone that not reaching 500,000 cars per year in 2 years is "failure"-- that Tesla is exhibiting an "inability."

But it is a failure. If you make predictions as a public company and then you can't live up to them you have a problem. That's not to say Musk has done some spectacular stuff but he's definitely not on target. Let's hope it all gets fixed before time runs out.

Investors are entitled to believe goals are achievable. If the goals are not achievable, they have to be corrected as quickly as possible or else legal consequences may stem. If you made an investment even with all the flowery legal contract language, how would you feel if the CEO said "it was always a moonshot" having previously said "it's achievable"

"He" figured this out, or some of the hundreds of engineers and scientists employed by him in an entirely separate enterprise figured it out?

They guy who figured out how to land a rocket on a barge is a spacex employee who is an expert in control theory, who was probably hired by another employee of Elon Musk. So I’m not really sure what you comparison means.

Or even more realistically, no "one guy figured it out". A team of people worked on it for years and eventually got it to work.

Musk created the company and instilled the work culture as well as the vision that made it happen - the hiring and all.

Can't help but think that if learning to get really good at robotics is a strategic goal, then that's something which might pay off later.

he's so obviously setting up for asteroid mining. autonomous spaceships, robotics and more autonomous vehicles, mining company etc.

when his 100% robotic and autonomous ships mine an asteroids and bring back huge payloads we'll still probably see articles like this.

> Hmm, who do I trust, the guy who figured out how to land rockets onto a barge, or some “analyst” ...

This kind of comment is a disservice to all the engineers who actually figured it out. Elon Musk is certainly a visionary and a financier, but is not a 1000x engineer/mathematician who did all the work himself

While it is a good idea to spread to credit for SpaceX's acheivements around, Musk himself has done more engineering of products at SpaceX and Tesla than any of his peers. His title as CEO of both companies is somewhat misleading as he has very little involvement in business operations of either company and spends most of his time engineering.

Source?

He is Chief Engineer at SpaceX, because they couldn't get anybody.

Go listen to Tom Mueller for example. Musk himself says at SpaceX 80% of the time he is working on engineering not traditional 'CEO' things..

The reality is that Elon is and was the Chief Engineer for the Falcon rocket family. He had the vision, the finance and he lead the engineering effort extremely directly and successfully. He overruled his top guys on multiple decisions and usually he turned out to be right.

Go listen to Tom Mueller on the design choices made for the Merlin example.

But of course nobody does everything himself, but for some reason I read this sort of comment made about Musk in every single thread but not with most other people.

Mr Musk has strategic vision, but not necessarily engineering vision. Which is fine, engineering is a huge field.

Bear in mind that the first time one of his staff proposed verticals landing rocket stages, he laughed.

He does have a degree in physics though. So he probably has a pretty good grasp on the concepts involved, as well as a finely tuned bullshit detector.

Who do I trust? A person I can identify by name and track a person down who has written an article. Or an anonymous account on HN named 'throwaway*'. Yea...you see?

Alternatively, do you trust the guy that is running a company that is going to be out of money in a few months?

Or to put it another way. Why is the guy running a car company busy putting rockets into space?

Appeal to authority is never a good argument for or against something.

Who do I trust, the guy who was told “It is impossible to reuse rockets” by all NASA engineers and built a company around it, or the analyst described as follow:

> Warburton, who spent his career before Wall Street at the International Motor Vehicle Program — a partly academic, partly commercial organization based at MIT — wrote that "automation in final assembly doesn't work."

It doesn’t even matter since it’s not public money. Which makes me wonder about the analyst’s intent: Tell people not to invest? Do people not know Tesla is super-risky? but if they succeed to obliterate this anchored rule that automation doesn’t work in the final assembly, isn’t that a superwin that stakeholders dream of?

Friends who invested in Tesla didn’t “invest” so much as “donated” money for what they think is a good cause. And perhaps because it’s fun to hear the NASA chief say “I had asked my engineers and they told me it wasn’t possible”.

I’d pay for that kind of fun.

I digged out a report from Bernstein (the same company that delivered the one reported in the article) from 2011:

"They have concluded that improvements to conventional engines will be key over the next 10 to 15 years and HEVs will become viable on a large scale by 2020. They believe near-term mass market adoption of electric vehicles is unlikely given the tough financial comparison with ever improving combustion engine vehicles. While premium-priced plug-ins may become viable earlier, by definition they will be niche products."

https://seekingalpha.com/article/310159-bernstein-and-ricard...

well maybe tesla really thinks the existing automation research is outdated and maybe they think it will massively pay off. but they surely must recognize that there are learning pains involved in this especially when it sounds like no other auto manufacturer is involved in this level of process automation, there's no one with relevant knowledge to simply poach, and therefore the cost of learning has to be borne completely by tesla... for what kind of improvement over tried-and-true existing processes in the short and medium term? for a new auto company that by many accounts is running out of money, this is a silly gamble to take. it might be a good bet 10 years down the road when Tesla is pumping out 1m autos/yr in line with BMW and MB but not right now when they are bleeding cash out of every orifice and failing to make deliveries.

sounds like one of those sad stories where the engineers fell in love with the stack and forgot about the product.

>sounds like one of those sad stories where the engineers fell in love with the stack and forgot about the product.

Sounds like Steve Jobs... in 1986.

https://www.youtube.com/watch?v=dSj6kvv7_Sg

There is running theme of Tesla "not knowing" specifics of mass producing cars. The head of prod for Tesla Peter Hochholdinger was at Audi for 20 years. He led the manufacturing programs of the Audi A4, A5 and Q5 vehicles.

Modern Audi production is an interesting example of horizontal integration which might not map cleanly to Tesla.

Chassis, drivetrains and minor parts-bin components are developed group-wide by VAG. Putting them together in the desired pattern and adding premium components and materials is what makes an Audi.

All Audi A, Q and TT models are examples of this. They are built in shared plants with other VAG models which use the same platform. Even the R8 has a shared platform.

In contrast AFAIK Tesla does everything bespoke without the advantages of volume and standardisation.

I would imaging Peter is aware of this difference and they are trying to mitigate the risks.

My previous comment on this subject (reproduced below):

https://news.ycombinator.com/item?id=16614247

----

Disclaimer up front: I work for a Tesla competitor.

Tesla is making several very interesting bets on the Model 3.

Bet 1: Model 3 mix - Tesla is betting that the ratio of expensive, high-content Model 3s to cheap, low-content Model 3s will be sufficient to make enough money to offset investments and pay back loans.

It will not be sufficient to make a small profit on each Model 3 sold given the debt load that Tesla has accrued, so they will want to sell a good ratio of expensive Model 3s.

If it takes to long to produce the cheaper Model 3s, Tesla will lose some potential customers. Many investors are looking at that "top line" right now - the number of customers and potential customers. If the top line moves too much, investors may get spooked.

Bet 2: Vertical Integration - Most automakers rely on a web of parts suppliers, who are under enormous pressure to reduce costs, but Tesla produces most of its components in-house. The contract with an external parts suppliers ensures that the supplier is responsible for any re-work or replacement of defective parts. This allows the automaker to concentrate on internal production issues.

Tesla's bet is that internal production of parts will lead to better and cheaper components. This has not worked for any other automaker.

Traditional OEMs shoot for a mix of components where the internally produced components are part of the company's core competence: Body Shells, Engines, Transmissions - and externally produced components may be generic - switches, latches, seats, frames, tires, wheels, etc.

If Tesla spends too much capital on component manufacturing, they will be inefficient and investors pressure them.

Additionally and probably more importantly, if Tesla is not able to spend the time and attention to iterate on cost and quality of these parts, it will also lose this bet.

Personal opinion: I think Tesla has learned the wrong lesson from previous dealings with suppliers. For instance, the original Roadster was designed with a two speed transmission. A supplier claimed they could make it, but it never really worked. Tesla learned the lesson that suppliers are stupid and suck at making new things - I think they should have learned that lesson that it is really really hard to make new things.

Bet 3: Automation - But first a detour - There are 3 main areas of auto assembly, and most manufacturers have already fully or almost fully automated 2 of them: Body Shop (welding and assembly of the body shell) and Paint Shop. The 3rd area is General Assembly.

General Assembly is the bloody, thorny, devilish poster child for multiple single points of failure. A high feature vehicle may have on the order of 1000 assembly stations (aka footprints) in General Assembly. The Model 3 is designed with much lower complexity in mind, and may only have 100 footprints.

If and when any of those 100 footprints has an equipment failure or parts issue, ALL 100 stop running in a short amount of time. Human assembly workers are rather resilient and can figure out a multitude of small issues on each and every operation. This may allow for a hypothetical variation of 5% in non-critical parts.

Automated assembly may only allow for a 1% variation.

Additionally, automated assembly only runs well when EVERYTHING is designed for automation. That is not impossible, but it is expensive and time consuming.

---

Most importantly, these three bets are linked:

If not enough base models are sold, the cost of design and equipment spending will be excessive on a per vehicle basis.

If internally produced parts are too far out of spec, you have strong negative impacts on automation.

If automation fails, you cannot produce enough vehicles at a low enough price to satisfy your low-end customers.

But, if Tesla wins all three bets, they win big time.

Comments like these are why I read Hacker News :)

Regarding these bets, I'm not sure on which side I'm on. You want them pushing the limits, Tesla itself was a long shot as a company. It's also meant to not just produce electric cars, but to also drive change in other car manufacturers so that they advance towards electric cars and maybe also self driving cars.

However, if they keep piling on the bets, at some point they will go under and I think we still need the company.

Your post makes me believe that they are pushing things a bit too much, like a startup. Yet they are at a later stage in the life of a business, considering their scale. It's like a bank using MongoDB 0.5 in production several years ago.

> It will not be sufficient to make a small profit on each Model 3 sold given the debt load that Tesla has accrued, so they will want to sell a good ratio of expensive Model 3s.

But automobiles are not their only revenue source. They also sell

* residential and commercial solar systems and roofing material

* residential and commercial battery packs

* batteries

* as of recently, electricity through Supercharger Network

Sure, but these are the bets for this revenue stream. There are other bets for those revenue streams.

Tesla's biggest Capital Expenditures have been in service of the Model 3, that's why we're talking about it.

I don’t know if they compartmentalize by revenue streams. If residential solar is the cash cow of the day, they will use the free cash to prop the Fremont factory.

The cash infusions go further - when SolarCity needed some expansion capital and sold bonds backed by residential customer solar leases, SpaceX (which at the time was flush with cash from a VC round) materialized as a major buyer http://blog.solarcity.com/spacex-invests-90-million-in-solar...

SolarCity was as far as I know a liability when it was merged with Tesla, not an asset. The only reason the merger happened is because Elon Musk had a big stake in both did not want to see SolarCity go under, it was 3 billion in debt at the time.

super comment. Thanks very much for a very informative PoV

I visited the Mazda plant in Hiroshima, and enjoyed watching robots working with people. (the window install unit is the one which I saw with a high degree of integration) It felt like the mix in a stable, well understood plant was pretty good. Welding by robots doesn't mix well with people side by side, but some welds were still being done by hand. I doubt they do this because they have choices: it would only make economic sense to use people to do some welds, if robots did them worse, given you have the robots there already with welding guns.

I think this kind of flexibility probably epitomises what car manufacturers have to learn from Taylorism onwards: it takes a long time to understand what mix of auto and manual construction works best. If Tesla is trying to short circuit this, thats high-risk: cutting time out of a time consuming learning exercise risks mis-learning, or not learning.

> Tesla's bet is that internal production of parts will lead to better and cheaper components. This has not worked for any other automaker.

It's working well for SpaceX who are doing just that while the incumbents in the rocket industry are not and the prices reflect how well it can work.

> Traditional OEMs shoot for a mix of components where the internally produced components are part of the company's core competence: Body Shells, Engines, Transmissions - and externally produced components may be generic - switches, latches, seats, frames, tires, wheels, etc.

Ummm, which automaker makes it's own transmissions anymore? Quite a few don't even make their own engines either (Dodge with Cummins engines, Jeep with VM engines, etc. etc.)

I don't think you can really compare a 10's each year assembly line Winther one that has to do 10's each hour.

One of the big stand-outs for me is that companies that are very successful in the long run change one major thing at a time. Changing one major thing is a risk, but one that can be dealt with and absorbed with a sufficiently strong cushion around the thing that is changing. If you change more than one thing at the time your cushion becomes very thin and any complex interplay between the various moving parts has the effect of exposing the world to the fact that you are changing many things at once and can not cope with the rate of change internally. Changing three major things at once is simply a gamble. It may pay off but there is a near certainty that you will damage the brand and there is a good chance you will mortally wound the company. I really hope that Musks bets will pay off but it is looking awful scary right now.

I heard the same thing in the 80-90s. Toyota has over automated and is spending more on robots, than it would have, had they used manual labour.

The funny thing about that is that GM (and others) assumed Toyota had more automation than GM in the 80s, but Toyota had LESS automation!

Toyota simply used their people more effectively.

In other words, they did have more automation!

Of managers and MBAs, not of line workers.

One of the things I remember from the plant tour at the facility in Toyota City was being impressed with the logistics they had to enable a heterogenous assembly line. One car rolled off every minute, but they were different models with different options. Most of the assembly steps were performed by humans, but all the parts needed for each step, which differed based on model/options, always arrived at the right station at the right time. There was a massive computer-controlled overhead distribution system that shuttled parts (including large/heavy parts like engines) around the factory that was damn impressive. With as many steps as were needed to assemble a car, that system was at least as impressive as the robotic welding arms and other fully-mechanized assembly steps.

Automation isn't necessarily just about replacing people. It can also be used to make the people you have a lot more productive.

I thought it was generally accepted that one of the problems with GM in the 1980s is that they spent too much on automation, whereas Toyota and other competitors spent more on just building better cars.

For instance: http://www.autonews.com/article/20080914/OEM02/309149956/a-s...

"GM number crunchers had pegged the investments in EDS, Hughes and Fanuc at $40 million in the long run. Instead, they cost $66 billion. In the end, GM's plants were no more efficient, and sales plummeted as money was spent on conglomeration rather than developing products Americans wanted to buy."

And now Toyota cars have a fantastic reputation for reliability. Surprise, surprise.

My 2p guess: maybe traditional automakers have a very good, experienced and cheap workforce where Tesla has many very smart and costly engineers. Focusing on what makes Tesla different instead of mimicking concurrence is risky, bold, but also very ambitious and promising (sorry for my bad english, I hope you see what I mean).

Your English is fine, except I think you meant 'competitors', not 'concurrence'

Yes. In portuguese, "[the] concurrence" is also a false-cognate for "[all the] competitors".

In my expierience with automation the last mile to full automation is biggest hurdles for little rewards. But when you finally reach 100% new possibilieties open that are just not available if your process is even in a very small part manual. I believe Tesla set their eyes on a bigger prize.

Automation pays off over time. I'm rooting for them.

Optimistically, those over-the-industry-ratio of line investments in automation will be profitable after more models are introduced for the mainstream audience.

Other car makers take no risk in the short term by not investing to automate more final assembly. But 2 times more is not like 10 times more. It is very probable that Tesla will get to have lower production cost & better capacity than other car makers in just a few more years.

Yeah I know they may need more cash very soon and are way under their targeted rate, but Musk still has lot of money in the bank I think?

I always think of Tesla of a company with a more long-term vision than other car makers. I think analysts sometimes just want a lower the price so they can get more of it.

I don't think Musk has that much money in the bank. I think most of his net worth is in his Tesla shares.

There is a reason most big car makers have not been selling electric cars yet, it is not very profitable, they believe that around 2020 batteries are becoming cheap enough to sell them. And if those companies can produce more and cheaper cars than Tesla they are going to have a problem.

There are only four or five significant old school car manufacturers and India and China bought some technology and are ramping up. The complex non linear investment decisions they have to make are just that: complex. If Tata or somebody succeeds in stealing mindshare and then market share leaving their run until after 2020 could turn out to be as suicidal as investing in cocaine to make the Delorean

Hopefully he can generalize their automation developments. Seems like this would have to be a bigger play then just the Model 3 at this point.

> But while all that exotic capital might allow Tesla to remove 5 workers, it will then need to hire a skilled engineer to manage, programme and maintain robots for $100 an hour (our estimate of a robotic engineers’ hourly rate).

These analysts, what are they analysts in?

More context:

"Let's say there are 10 hours of labour in final assembly (the part of the production line where parts, interiors and the powertrain are installed in a painted bodyshell). In a regular plant, final assembly typically has less than 5% of tasks automated. If Tesla attempts to automate 50% of these tasks, it could cut out 5 or so hours of labour. This might save $150 per car (assuming wage rates, all in, of $30 per worker, per hour).

"But while all that exotic capital might allow Tesla to remove 5 workers, it will then need to hire a skilled engineer to manage, programme and maintain robots for $100 an hour (our estimate of a robotic engineers' hourly rate).

"So the net labour saving may be only $50 per unit. Yet putting the automation into the plant seems to involve an apparent capital cost that's $4,000 higher per unit of capacity than for a normal plant. If the product is built for 7 years, that's over US$550 of additional depreciation per unit built. It's hard to see an economic case even if somehow the Fremont Model 3 line can be made to work. So why exactly has Tesla taken this route? It's unclear."

http://www.businessinsider.com/tesla-robots-are-killing-it-2...

The point is that that skilled engineer can manage and maintain robots across many factories. That's an aspect of Musk's "the machine that builds the machine" goal. It does have a high up front cost, but the long term advantage is real. Namely the ability to print factories, roughly speaking. Tesla is aiming for production volumes that are much higher than any current automaker.

I think he is preparing for automatad factories on Mars.

That skilled engineer is not something you easily clone and when you are trying to debug 250 stations at once you will need many of those engineers, not one that can maintain robots across many factories. That's for when things are working.

Meta-comment : I personally don't trust any information coming from businessinsider / wsj / theverge. Everytime I open one of their articles, I feel like it wasn't really worth reading. I only read the HN comments. At the end of the day, it's always tiring to analyze their articles to get the truth from the buzz and ove-generalization eg "the robot are killing Tesla". They are probably good starters for discussions on HN. By the way, the title on HN is way better than the actual title of the article.

It's funny how exactly everything Tesla does deserves a scandalous sounding headline..a couple of weeks ago they didn't have enough automation, now they're overusing it..

You can't make money on stocks that don't move. You can make money on stocks rising or falling.

This is a rule that can been extended and which in my opinion is the main objection to "AI will replace [insert any profession here]".

Even if you have sufficient data to train an algorithm, you still need AI experts who understand the domain (another big obstacle) and the cost of these AI experts creating and maintaining the algorithm needs to be less than the cost of the workers who's jobs you want to automate.

One thing this article fails to consider, is with more automation, there are fewer human limitations. For example, if a human is able to perhaps join a weld per minute, it's possible a machine could go 10x that speed.

Or make mistakes at 10x the speed.

If it were as simple as being able to perfectly replicate a skilled human and do everything 10x faster, then this should really be simple. The fact that automation is so problematic suggests to me that there are a lot more variables and problems.

I would argue that they are not simply repeating the same motions over and over again, but adjusting with real time feedback, making the speed and error rates dynamic rather than static numbers.

Great thing about Elon is that he believes in everything so much he is probably laughing reading those articles. I understand that myself, would lough too. There will be no future without 100% automation and for me this is the best bet you can make. We have so many areas of life that need to be automated and someone has to be first. Companies that have ancient roots are too unreformable and don't want to change what already works well for them, which is why BMW is lagging behind, despite the fact that it can produce millions of cars a year.

People will go crazy, say that he lost everything, he is stupid, etc... But he will not give a f%#k just to return with double power and hit like no other.

The devil is in the details. Automation is all about part variance, if they can't control that it will be a huge problem. Humans are flexible, robots much less so.

Didn't someone tel musk he cant reuse rockets and that was a silly and expensive idea. I wonder where those guys are hiding now?

One benefit of automation they also failed to consider is customization. All sorts of exotic designs, configurations, adjustments and test iterations could be made. Hopefully with a standard base so they can be repaired or swapped out later, but the less human training that's required to get an assembly line up the faster you can change and improve it

Another aspect is potentially repairs itself. If every part can be added by a machine, they could later be removed and swapped out as part of the maintenance cycle too

Those don’t make much sense.

Programming an automated line to build a car (or any other product) in a different way is a tremendous amount of work, since all of the motions have to be exactly right. Usually customization weighs heavily in the favor of humans, who can adapt much more easily to parts that are different.

Also, an automated production line isn’t going to be any use for repairing cars, all it can do is bolt them together.

These aren’t some kind of AI robots. They are mechatronics that are programmed to follow very specific toolpaths. Reprogramming them by even a few millimeters is a monumental task, and to test them, you have to try to build a car ...

Do you know this for a fact?

They already have advanced computer vision, for a new facility surely they'd be building this in at least partially

Is it just me, or is there suddenly a lot of media backlash against the wunderkind of yesteryear? Holmes, Zuckerberg, Musk — all sold impossible visions but they were just crazy enough to think they could pull them off because they were trying something new.

Was the whole innovation economy all snake oil? How many of the “Unicorns” will be left in the middle of the next decade?

Not sure if Holmes belongs to this list. Social network is a great thing and you can buy Musk’s car and see rockets launched and landed.

Holmes and Zuckerberg are not even in the same league as Musk. The first is a fraud, the second the sneaky CEO of one of the worst online entities (a perfect example of a fish rotting from the head) the third a visionary that tends to deliver long shots with alarming regularity. If Tesla gets its house in order it will survive and prosper, if it can't get its house in order Musk's personal brand will definitely be damaged, let's hope that SpaceX won't end up suffering from that.

With the cash crunch, this is going to be interesting to watch Tesla try to thread the eye of the needle.

I can easily see building out and proving the final assembly automation being related to Elon Musk's Mars colonization ambitions.

Sometimes optimization is not the optimal solution. They should have optimized their optimization methods first. Or should they?

Someone doesn't like your comment, but when I searched to find context for a quote from the article, I found this expanded paragraph from the Bernstein Research report:

"What is the inspiration behind Tesla's automation? Tesla has bought German robots and a German automation company (Grohmann). But the German OEMs – traditionally the most enthusiastic proponents of automation – have actually been rowing back on it in recent years. The best producers – still the Japanese - try to limit automation. It is expensive and is statistically inversely correlated to quality. One tenet of lean production is “stabilise the process, and only then automate”. If you automate first, you get automated errors. We believe Tesla may be learning this to its cost."

https://ftalphaville.ft.com/2018/03/28/1522248619000/No-one-...

The Grohmann acquisition was mostly botched, they got far less out of that than they would have if they had properly managed that.

https://www.theverge.com/2017/4/27/15457472/tesla-grohmann-e...

Couple that with a couple of Tesla bigshots walking when they heard Elons plans for fixing the Model 3 production schedule and you can see why things are not looking all that rosy right now.

https://www.google.com/search?q=tesla+execs+leaving

An example of: No pain, no gain.

LOL. Analysts are correct, but musk is more correct! (intuition!) Its like uno, automations not perfect yet, gaps in process, missing feedback signals and other signals on various fails. Solution! Densify all of the problems and hundred dollar hour guys in your super factory, raise TC, phase transition to factories making factories as a service, earth conquered, again. We are just getting to the full spectrum buildup, next.. red alert, its a tank rush!

Yea. I wouldn't be surprised if this was Tesla's PR itself. Make a bunch of people say you can't do something and then do it.

10x the PR and the you have successfully projected the illusion of capable of doing the impossible in front of the naive public...Worked well for Elon till now..

It didn’t work so well for Theranos.

Well, of course, you should have some sort of assurance that you ll be able to pull it. The PR tactics is just to give 10x PR when you actually do it...

And I think Theranos did much more than make people say what they are doing is not really feasible...

Theranos was a completely bullshit and fraudulent product

Imagine, LOL, this headline (now or >) "Slump in automation developments" almost as joked out as "Useful home robot, that while remaining totally and completely useful, is totally unable to kill the shit out of you by meerly having a series of unfortunate electric field states rise in its transistorized brain nugget" yeh laff

Reply on news.ycombinator.com