Comment by 01100011 | Hacker News Reader

Comment by 01100011 | Hacker News Reader

Parent Full thread

As far as I can tell, Tesla plays fast and loose, treating their product like a manufacturer of consumer electronics and not a manufacturer of a dangerous and durable good. That obviously allows them to out-compete other auto manufacturers who are more aware of things like product liability.

See the Toyota accelerator-gate lawsuits. From https://en.wikipedia.org/wiki/2009%E2%80%932011_Toyota_vehic... :

> However, on October 24, 2013, a jury ruled against Toyota and found that unintended acceleration could have been caused due to deficiencies in the drive-by-wire throttle system or Electronic Throttle Control System (ETCS). Michael Barr of the Barr Group testified[30] that NASA had not been able to complete its examination of Toyota's ETCS and that Toyota did not follow best practices for real time life critical software, and that a single bit flip which can be caused by cosmic rays could cause unintended acceleration. As well, the run-time stack of the real-time operating system was not large enough and that it was possible for the stack to grow large enough to overwrite data that could cause unintended acceleration.[31][32] As a result, Toyota has entered into settlement talks with its plaintiffs.

Not following software best practices(i.e. ISO 26262) can expose your customers to unnecessary risk and leave your company vulnerable to lawsuits. Tesla may learn a very expensive lesson one day, or they may get lucky. Time will tell.

---

Depending on the design of your system and software stack, switching SoCs can be anywhere from a massive effort to a simple recompilation. Even switching architectures could be trivial for some components(i.e. a UI written in HTML5 making REST API calls) or a nightmare for others(ECU, ABS or other real-time algorithm written to use bare metal modules on the SoC without an OS providing abstraction).

Toyota accelerator-gate is largely a farce. Almost all of the people involved with Toyota unintended acceleration were over the age of 65 and had the poorly designed loose floor mats in their cars.

Even if they had perfect readable John Carmack tier coding they still would have lost.

Sure but that's not related to the point I'm making. The point is that not following software design best practices can leave you open to liability, or at least put you in such a weak legal position that you are compelled to settle.

If/when Tesla gets hauled into court over its software flaws, any lack of adherence to best practices will make the company appear negligent.

This is a risk adverse culture then. Tesla is spending gobs of money specifically on building safer if not the safest cars out there.

How do you square that with them removing sensors from their cars and going all-in on doing self-driving exclusively using vision? If they were so passionate about spending money to make the safest cars, they'd be doing sensor fusion.

> If they were so passionate about spending money to make the safest cars, they'd be doing sensor fusion.

That's your opinion. They have shown data where the information from radar was of a lower quality than what their vision stack was providing (e.g. lack of vertical resolution). Watch their AI day webcast for examples. Their current vision-only stack has been validated with LIDAR ground-truth data.

Is anyone arguing for replacing vision with lidar and radar instead of combining it?

If the radar data conflicts with what vision is saying, which one do they trust? They have shown that their vision stack has surpassed what they can do with radar. So "fusing" that data in only makes it worse when it gives conflicting information.

Sensors are obviously important, but it doesn't really matter if two sensors conflict, the important thing is which one is consistent with their running model.

So far their biggest problem is that they allow flicker. That's why sometimes the software picks the wrong lines. (Of course this is a very hard problem. Our brain conveniently smooths over sensory changes for us, because that's how our everyday reality is. Things don't flicker in and out of existence, nor does a car suddenly appear as a different thing, then switches back.)

And if it turns out the sensor(s) failed, it has to be able to handle that too.

> building safer if not the safest cars out there

Unless the car loses power and catches fire. Then you can't open the fucking doors [0].

[0] https://www.washingtonpost.com/business/2019/10/23/man-died-...

Using 1 off cases based on unsubstantiated information in a lawsuit is not a persuasive argument for overall safety.

It's certainly a better argument than the GPs claim of "the safest cars out there" with 0 evidence given to substantiate that. I think it fair to refute such an unsubstantiated claim with a reported instance where safety was not maintained as can reasonably be expected.

Tesla cars have the highest safety ratings in the world and they have the the best active safety features and complete fleet compliance with their whole fleet for new standard of active safety features (literally the only company that does have that).

This is literally just the basic fact of the situation. If you want to disagree with literally every state safety agency in the world, then you are the one who needs to come up with real evidence.

It literally took me years to understand it - how can you accelerate by accident? Until someone mentioned that all of these cars were automatics.

But every automatic I've ever driven (not many - I prefer a clutch) moves forward unless you're pushing the break in. In the rest state, it's in motion.

The problem isn't Toyota, the problem is a broken system.

That's a mechanical side effect of the torque converter used in automatics, which even at engine idle with no acceleration input is able to transfer power to the drivetrain.

The Toyota case was an instance of the car's engine control software unexpectedly commanding acceleration not requested by the user.

In principle it could happen on a manual car as well, but most of Toyota's vehicles in the U.S. are automatics.

People panic and jam the wrong (or both) pedals. (There are quite a few people who use two feet to drive automatics, one foot for break, other for gas..)

> There are quite a few people who use two feet to drive automatics

They are very irritating to drive behind. Some seem to keep pressure on the brake pedal, and the brake lights stay on as they accelerate. When they slow it takes longer to realise they are braking as the lights have been solidly on.

Are you sure they're not just driving a manual transmission car? I can keep my foot lightly on the brake and apply the clutch to accelerate, without touching the accelerator pedal (diesel engine has a lot of low-end torque). Not that I do this in practice, except in drive-through lines just for the fun of it.

In first gear I can reach 8 km/h, but it's possible to get all the way to fifth gear while idling (takes a long time though).

I thought you would fail most driving tests doing that.

Perhaps, but left-foot braking is a useful technique, and common practice in some forms of racing.

In street driving, the main application is being ready to brake while driving normally. It is much faster to brake with the left foot hovering over the brake pedal than by moving the right foot from the throttle to the brake. Of course covering the brake pedal with the right foot precludes driving normally on a flat road or uphill.

This should of course be practiced in low-risk situations before being attempted in high-risk situations.

True but these are definitely not race drivers I'm seeing, jumpy older drivers mostly.

Not sure how is this possible. Automatic cars almost always require You to push brake constantly if you not driving. You can't turn car on without pressing brake, can't change to drive mode from parking state without pressing brake. There should be something very broken in Toyota case.

Automatic handbreak at a traffic light?

Easy to do. I stepped on the accelerator instead of the brake once. First impulse is to step harder, until you realize what’s going on. If your realization is slow, bad things can happen.

If bad things happen, the tendency is to blame someone else.

One solid step on the accelerator easily let's bad things happen below 1s.

I worked for ford many years ago and I remember a really funny conversation between young engineers and MBA types. The issue was the Lincoln Towncar was designed to have extremely low pedal pressure because that is what the old buyers liked. Combine old people with limited hearing, no pedal pressure, and a quiet cabin it was a perfect recipe for old people to shift and send the car through the garage wall. Lol. The classic issue of when customers want something stupid, do you build it?

The pedals are close together in my car, and a couple times I have hit the accelerator rather than the break. But I knew immediately what was wrong and adjusted.

Not entirely a farce.

The PCM (engine controller) that a university audit got access to had 12,000 global variables. Their conclusion was it was not possible to prove either way.

There was a definitely real issue with floor mats getting the pedal stuck. I don’t know enough background on it past the audit that was done.

I followed the accelerator-gate thing fairly closely, and I came out pretty convinced it was a computer failure. If it wasn't Toyota's floor mats were many times more likely to jam the accelerator than any other make or model.

There was a plausible root cause - a bit flip in the CPU. It was proved such flipping the right bit would cause uncontrolled acceleration that characterised each event.

I don't agree Toyota was guilty of shoddy work, or cost cutting. At worst, there were guilty of working some fairly shitty, hard to support code. (Someone below mentions 10,000 global variables.) If anything that shitty code just proves how dedicated Toyota was to testing something until until all the safety related bugs are gone, because despite some horrid kludges like replying on watch dogs resetting the thing when it got stuck and a reset so fast most people wouldn't notice it, despite the intense scrutiny of the code no one every found a safety issue with it. Toyota was well aware of a electrically noisy area like a engine bay could cause bit lips, and defended against it. Every variable was stored in two places in RAM, and on use they were always read and compared, and if they differed it was reset.

Sadly for Toyota they used a proprietary ECU and OS provided by NEC. NEC didn't let their customers look at their proprietary stuff. NEC didn't defend against bit flips. And it turned out a bit flop in a task ready bit could cause that task to never be scheduled again. If that task was supposed to turn off the cruise control acceleration, then bingo.

I have no idea what changes Toyota made in response to this mess, but if I was to hazard a guess it would be when it comes to software, they demand their supplies open their source to them.

From earlier discussions here (https://news.ycombinator.com/item?id=9643551 "10,000 global variables"), Toyota's conservative, arguably antiquated approach to software architecture wouldn't necessarily prove reliable as that software evolves.

Trying to assess whether thousands of global variables are still playing nice during a major rewrite to accommodate a new chip would be definitely be difficult and time consuming!

Not that a fast-and-loose approach is ideal, either.

Writing reliable software in the context of an organization is hard.

10K global variables sounds insane, until i realize "oh, huh, we have that too, we just call it _config_".

Almost every place we would have a constant (tuning parameters, etc) we instead have a configurable value with the likely default declared in code, but all overridable in config. Managing config can be a hassle, but the number of times we've merely had to tweak a value rather than roll a new build pays for itself every day.

We have thousands of such "global variables"... of course they are read-only, so aren't used to share state.

If Toyota is doing that: nbd. If they are using them to share state ... god have mercy on their souls.

I always interpreted the "10K global variables" to mean 10K directly accessible, heap-allocated symbols. It's a code reek. If you use a configuration object or other abstraction, the number of accessible variables may not change, but their scope and access method does.

It's a matter of interpretation I guess, but I would not classify a wrapper object (like .Net's ConfigurationManager singleton) as "10K global variables" even if the accompanying config contained 10K items, or if the ConfigurationManager backing store was preallocated in the data section of the binary.

>I always interpreted the "10K global variables" to mean 10K directly accessible, heap-allocated symbols. It's a code reek.

Which to me on an old _accelerator_ design is suspect because that's alot of heap for the micros that would have been used back in the day. This isn't a Linux system. It was at best a micro with like 8KB of RAM. (I'm not actually sure what it is but there's no way it was impressive).

If they used Simulink, the model output and input, aswell as state, are global variables when generated to C in some settings.

From the article referenced in that earliser discussion: https://www.safetyresearch.net/toyota-unintended-acceleratio...

> Other egregious deviations from standard practice were the number of global variables in the system. (A variable is a location in memory that has a number in it. A global variable is any piece of software anywhere in the system can get to that number and read it or write it.)

However, that was from the article author explaining the testimony, and not the testimony itself. It's not totally clear whether the variables were writable.

I would have guessed they were statically allocated rather than heap allocated, but what really matters is whether they were `const` — and they probably weren't `const` because otherwise this testimony wouldn't have rung true:

> "And in practice, five, ten, okay, fine. 10,000, no, we’re done. It is not safe, and I don’t need to see all 10,000 global variables to know that that is a problem," Koopman testified.

See https://www.youtube.com/watch?v=w5c5KzpamiM by someone who worked for a number of auto OEMs including Tesla where he explains how Tesla does agile hardware

I watched the video. Thanks for sharing. Powerful approach. But wondering how close it is from reality or not. Would be interested to know more.

I would suggest you watch the Munro Live videos where they disassemble and analyses cars. This company has been doing this for decades and recently started making some videos.

The changes in Tesla vehicles are absurdly fast. They disassembled original Model 3 and then China Model 3, and since then the refresh 3 and the Y.

Recently they showed the new Model 3 LFP (Lithium Iron Phosphate) and the electronics and how it was integrated is already different again.

How does that work out for maintainability? Is Tesla going to maintain parts inventory long enough to support owners for the lifetime of the vehicle(15-20 years?)?

> That obviously allows them to out-compete other auto manufacturers who are more aware of things like product liability

A lot of this era comes down to this. You have a legacy industry with tons of regulations, then a new guy comes up, steps on all the lines, and convinces people they're smarter for doing more for less. (up until regulations come back in the equation).

Its very easy to claim that everybody else is not following regulation and that the only reason you are losing is because you do. Its much harder to prove.

If anything, GM had more problems and failures of their EVs so far. So the claim that Tesla is ignoring liability and doing much higher risk development is literally just an accusation based on nothing.

In actual fact, GM is spending 10s of billions buying back every single Bolt and they have been forced to shut down the line. At the same time Tesla had no such issues.

So, how about these people actually prove or substantiate in some way that the new comer ignores regulation or quality. Specially when, they themselves have a worse record.

Oh in that case I was more thinking about Uber and the likes. In Tesla case, the only glaring problem is 'autopilot' the rest I have nothing to say.

Isn't the fact that Toyota lost this case damning to traditional automotive software development practices, rather than evidence of their importance?

The traditional players are already doing a bad job trying to write safety critical software the way things are now.

It is very troubling development when doing something so trivial even requires a computer.

That's a matter readily solved by a PID controller made of a few dozen 74XXX parts. Possibly made n-times redundant.

Even a more fancy LP solver probably wouldn't need a fully fledged computer.

It's really easy to over react because of the silicone shortage, but in many cars these chips are replacing physically manufactured parts or significantly increasing efficiency in a way that isn't possible with physical based components.

Electronic fuel injection is a _great_ example. Much of modern fuel efficiency stems from the fact that an engine can accurately control fuel in the cylinder based on a bunch of factors. Not only does this system require physically fewer materials, it uses less gas (in turn saving production effort).

I think you are misunderstanding me.

What I mean under a computer is a Turing complete machine. A circuit just doing some LP solving, or PID is not necessarily a computer.

By that explanation, I actually 100% confident I'm understanding you correctly.

Most of the functionality of these chips can be represented in some form of hardware. However, that hardware is often much more expensive, significantly less flexible, and likely significantly bulkier.

Very little applications strictly need a turing complete machine. However microcontrollers are ubiquitous because they are so flexible. One chip does the job of many, and without needing to be specialised, allowing all applications to take advantage of the economies of scale in semiconductor manufacturing. (Also as pointed out here you drastically underestimate the complexity of a modern ECU)

There’s nothing trivial about a modern internal combustion engine. The throttle control system is doing quite a bit more work than just opening and closing a physical throttle valve.

ECUs are simpler to build than the mechanical components they replaced and do things impossible to do mechanically (given the space and weight limits of a car).

Jadon Cammisa has a great series on this stuff. Here's just one episode on one topic, drive by wire throttle control: https://youtu.be/gKsCHx5NOMM

Most drive by wire systems I seen are nothing, but a simplest closed loop PIDs.

It's very close to what you are taught in the "How to write a PID controller" class without much extra creativity.

Have you looked at any vehicle from the past 3-5 years? I own one with drive by wire and you can feel and hear it reacting to variables when you push it to the limits (either in inclement terrain or on a closed course). The linked video has specific video examples from production vehicles as well.

Deciding how much fuel will be injected each cycle is not just a simple PID. The throttle body control may just be a basic servo (although it probably isn't) but the system controlling it is more complicated.

A separate circuit to map fuel/gas ratio for a given sensor input is also not a rocket science.

Simpler to build, and even simpler to overbill for them. Is insane to charge over 1000$ for some replacement logic board.

Let's start by assuming that the engineers are at least basically competent, shall we?

I mean, I could implement a PID controller without any 74xxx logic at all: a basic op-amp, a handful of resistors and capacitors and boom I'm done!

But there's a reason that engineering doesn't do that these days and it's not that engineers don't know how. My PID control built from an opamp? In 2021 that's almost always a stupid idea. We use computers because doing it digitally has many advantages. We don't use 74xx (or any other logic family) for this because microcontrollers are far more flexible, allow for behavior changes without reworking hardware, allow inventories to scale (the same component can be used in hundreds of products) and more reasons I won't detail.

The reality is that in 2021, often the simplest, most robust and cost-effective way to do something trivial is by using a computer to do it.

> The reality is that in 2021, often the simplest, most robust and cost-effective way to do something trivial is by using a computer to do it.

Very much not. Complicated MCUs are a very risky single vendor supply, and as we see now, people are now paying for an engineering overkill.

74xxx are available in every shape, and form, from dozens of suppliers.

Ah the classic HN "just use this things I just came up with".

https://xkcd.com/793/

I actually believe programmers/software engineers are the new physicists in this regard.

Ok, lets assume for a moment that throttle can be modelled by a PID plant (it can't) - is the circuit temperature stable? You don't want the characteristics changing as the engine. Now you need a compensator circuit. What about power supply stability and noise? Car power busses are hella noisy, due to alternator and spark coil. Now you need some serious conditioning and filtering. What about knocking? Modern ECUs adjust parameters on knock detection to reduce the damage from pre-ignition. Gotta put in a control system for that.

Repeat for manifold air temp, exhaust temp, emissions, rev limiter, etc., and as you mention, make that 3x for redundancy. Now you have hundreds to thousands of components, in a high noise, vibration, and temperature environment, and you have something which is about as efficient as an 80s car. And you can't as easily simulate it cause it's analog.

Or just use some digital chips and simulate all the tuning.

Okay, so you've designed it. Now, make sure it works with the dirty electrical signal of a typical car's power bus. Then make sure it fits in the space requirements of under the car's hood, or in the door, or any of the other tight places that these embedded systems go. Then make sure that it's verified to act over several years of vehicle life. Then, make sure your suppliers will guarantee you that they won't discontinue the part for at least 10-15 years because of legal requirements for spare parts. Just throwing discrete logic at the problem doesn't always help.

Reply on news.ycombinator.com