I'm waiting for a recall fix for the underpowered Sync 2.5 system to correct a backup camera problem. I'm not looking forward to worsening of all the current bugs with USB audio file playback that cause the UI to hang or not show a fully rendered display.
Maybe things just really suck for embedded in the states? But since my last year of university I’ve been inundated with recruiters for embedded positions, and I’ve never had a problem finding work. ~75th percentile in salary alone for software engineers in my area, ~55th-60th for Canada. I make more than every JS developer I know who graduated with me, except for the ones who moved to Seattle, Vancouver, or the Bay.
Now sure, if you’re looking for 500k+ jobs, embedded isn’t the area to be in, unfortunately. But I prefer low-stress, fun-environment embedded jobs, and don’t mind trading off salary for that. Different strokes.
At this point, when I wanted to get back into hardware, it made more financial sense to outfit my home office with all the measuring instruments, debuggers, tools and other equipment necessary for embedded work and do it as a hobby. If I had the space, I could even get full-size CNC machines and still come out ahead cash wise. It’s insane.
It’s no wonder they can’t find experienced embedded devs, when it makes no financial sense to stick with it over a decade.
But yea, a single class probably isn't sufficient and also I image a lot of embedded companies have a preference to hire someone already familiar with the chip they are targeting and the toolchain for the stack. I also see a lot of asking for experience with RTOS, which in my class, we didn't use an RTOS.
I did some initial requirements work on a system to monitor continuous-web papermaking machinery; the line had to be stopped, physically and completely, within 100ms if anything went wrong, because an uncontained web of paper can literally cut people in half. They wanted, in order to be able to hire, to use one of the embedded flavors of a well-known consumer-grade OS, and I had to prove to them that there was no way to make any of them safe, at any cost. And they knew their hardware, because they had built it themselves.
The absolute last resort is a watchdog timer that hits the reset button if N milliseconds go by without the software telling it it's okay. This is what you have to implement if you are dealing with buggy and undocumented hardware -- as, all too often, you are. Sometimes you can get some doco for $ and an NDA, but then in order to get the real doco it is much more $$$ and a much tighter NDA, and the existence of that option is not even divulged until after things have already gone very far south.
If it were only a matter of reading the top-level doco for this or that chip, there would be no issue.
If I were selling hardware I’d want it to be as open and well documented as possible. So that more people buy it and so that I get credit for all the great stuff people make with my products.
1) The more you open up your design and its behaviour, the more your competitors can learn about your product and how to possibly improve their own. Even stuff as basic as what specific features/capabilities a specific SKU at a specific price point has can be useful information.
2) The behaviour may be sufficiently undefined as to make documenting it impractical (or a bad look). Specs may also be padded (“up to 14 bits of SNR” may mean you’re getting 8 most of the time unless you’ve got a golden sample, and you’re not getting the distribution without paying big bucks and signing a big NDA). This ties in with 1) - if your competitors know your exact yields, they might be able to advertise being better/more reliable more truthfully, or even cheap out on their manufacturing a bit to drop their own yields down to match or just slightly beat yours.
3) The behaviour might be unknown. There’s obviously a crazy amount of validation testing that goes into high-end chips, but even the best test plan can miss things. This is especially true when you’re talking about high-speed stuff and anything involving power delivery/voltage fluctuations, or async/pipeline executions, or a million other things that can go wrong. Again ties into 1) - if your competitor knows that your chip might deadlock the radio with an obscure pattern of inputs and control signals, that could give them insight into how you’ve laid out your silicon and might give them optimization ideas.
4) If all the available info is given out freely, then potential customers can easily compare manufacturers and pick the best one. The manufacturers don’t want this, unless they’re the best, for obvious reasons. And, because everything’s locked down so tightly, no one knows if they’re the best until the chips are on the market and the volume contracts are already signed. And those contracts are hard to break, since the specs agreed upon are pretty vague due to 1-3.
5) The manufacturer knows their chips suck, but needs them moved anyways. This is rarely the case from most non-discount manufacturers, but it can happen. In this case, you don’t want to give away anything you don’t have to, because most info you give out is going to drive customers away to a better option. Good example in the consumer space is Intel refusing to publish acceptable voltage specs for their 12-14th gen Core chips, which resulted in motherboard manufacturers overvolting and killing high-end CPUs to try to meet the frequency specs Intel was advertising. If Intel was truthful in their voltage and frequency specs, there’d be a minuscule percentage of chips that could actually hit the advertised frequency at safe voltages, and 99% would have worse performance than expected, which would almost definitely result in lower sales.
6) The behaviour may be highly dependent on external factors. Basic example, a chip with external DRAM might have its execution pipeline stalled more or less frequently based of DRAM spec, or a wobbly voltage regulator might be known to cause lockups when certain executions are happening. Are you going to tell your customer those problems, or just say “we recommend high-speed DRAM and high-quality VRMs?” Especially if the other guy just says “we recommend high-speed DRAM and high-quality VRMs?”
The world would likely be a better place without such logic, but the incentive is there. Until someone comes and breaks the paradigm, I don’t see things changing.
This is an industry that is about as far from the light of science & enlightenment as it is possible to get, ensnared as deeply in the entangling anti-human anti-science Intellectual Property qualgmire-hell as can be got. Oh sure plenty of science goes it! It's fantastically interesting & technical! But aside from some Application Notes write-ups trying desperately to help move the practice along, move it out of jank, knowledge goes in, but it doesn't ever come out! There's such a lack of peershios with which to practice science, to report your findings to, to replicate works on.
The software world talks about its patterns and practices. The biggest industries on the planet are building software like wild AND are mad into open source. But... computer engineering is the shadowland, where no talk nor victories that happen there are allowed to be shared, where nothing escapes confinement. What a fucking plagued awful land of people unable to ever do the right thing, unable to bring their work out of the dark & into real civilization.
What does writing ABS module software look like? I'd actually love to know--it's not an area where you can "vibe code" your way to a 'working' product.
The process is so far removed from typical web and business slop that it's an entirely different world of its own.
To be fair, im still not sold that this is an advancement except maybe in simplifying the number of components. I'd prefer the car to work without "updates" and DLC. Why does my car need a firewall??
For what it's worth, I work in this industry and the general rule of thumb is that every increase in validation from QM (standard quality) up to the various levels of safety critical code has up to 10x the cost per line of code of the previous level.
Exactly that was done for decades.
Heck, manufacturers were issuing service bulletins to fix the fuel maps in their cars in the 1980s.
Now that all vehicles have entertainment systems connected to the internet, I guess it is tempting to use that to reprogram ECUs, I haven't been working in this area recently though.
The first use case of connecting entertainment systems to a vehicle bus that I can remember was to read some engine settings and turn up the volume on the radio at higher speeds.
Is anyone actually begging for this though? And why do you need a full bus? This feels like a luxury car problem that could be solved over I2C or something.
I’m reading this whole SDV thing, and outside of using less ECUs, it seems like an overengineered solution to what was hardly a problem. If we can update ECUs already with OBD-II, step 1 is just making a virtualized OBD-II port that the infotainment system can talk to. Everything else can then stay unchanged until later.
A "virtualized OBD-II" is really just a UDS server if I understand what you're trying to convey. UDS is a dumpster fire of a protocol that should be expunged from existence, but my personal feelings aside can be run anywhere you want. That exists. I'm not aware of many systems that directly connect the infotainment processors directly to critical CAN buses. Usually there's an intermediary component to isolate them.
Yes, but code that doesn't get written does not have bugs. And I don't want to control the rear window defroster, wipers, climate control, fog lights or whatever, on a touch screen menu buried 7 levels deep while going 130 km/h. It's bad enough that coffee makers, light bulbs and tooth brushes now have updatable firmware.
Why? If the rest of the car can function within design specifications for years, why can't the firmware?
I'm fine with updates to add compatibility with new protocols and such, but to me a bug implies there's a standing problem with the current system that's not due to some sort of wear/changing standard/component damage etc. While one can point to examples of cars with defective mechanical designs, I don't think anyone considers it impossible to create designs without such defects (where defects are defined wrt. specifications), why is this the view in software engineering?
But, do you have an example of a software project anywhere that's bug-free? I'd include the space shuttle code, but even that famously high quality development process produced a (low) number of bugs.
The reason for this is a physical limitation: the cars weren’t shipped with wideband O2 sensors, so there’s no way to measure the AFR when wide-open (since it’s targeting a significantly richer mixture, and narrowband O2 sensors can only signal whether a the combustion is stoichiometric, or rich or lean relative to stoichiometric, with no further info). The implantation is probably not a bug but rather a compromise; in an ideal world, the “most recent” BLM will hopefully be from an “almost wide open” part of the map, and the general rich/lean characteristics will be close enough. And, the fuel table in the factory tune is quite safely rich when wide-open, so even with a leaking injector causing the idle BLMs to be way off, the fuel being pulled when wide open will still be completely safe.
Aside from that, 128k of bug-free code.
What was wrong with ECU and ABS etc software prior to the OTA era that we're now apparently entering?
I've had plenty of cars--too many--and outside of a few warranty repairs involving re-flashing ECU/ABS(maybe), this was a very rare occurrence.
(Not counting deliberate tunes or re-flashes for modification purposes)
One, it's expensive. If your update takes half an hour to apply, under the old model someone's being paid half an hour to apply it. Either the manufacturer cuts the billable hours to the dealer and the dealer loses, or the manufacturer is paying that half hour out of increased prices to the consumer. With an OTA system there's usually no cost to anyone besides network traffic. This amounts to billions of dollars in savings for manufacturers.
Second, owners hate 1) paying for updates and 2) getting notifications about it in the mail. It generates bad press and bad experiences for the manufacturer.
Three, it makes the production line more efficient.
Four, the old systems sucked to maintain and for techs to use. They were also insecure and retrofitting security is impossible in a standards compliant way. The internet people have done a much better job with their standards.
Five, most owners are not like you and I. It's a feature for them that their car gets improvements and fixes automatically.
Six, you can be pretty certain what the rollout distribution is. Regulators don't like it when owners are driving around with years old recalls active because they forgot to schedule a dealer appointment. Manufacturers don't like keeping the inventory around.
Seven, "networked services" can piggyback on the same infrastructure and provide additional revenue streams. Certain corporate types think of this as one of the main benefits. Remember how manufacturers used to sell annual maps updates that no one bought? Some consumers also enjoy these sorts of networked services, which frankly I find a bit baffling.
Hmm, I disagree. Bug-free systems are expensive and hard, and get more expensive and harder as complexity increases, but you can absolutely produce a car that never needs updates. The vast majority of computer-controlled cars from the 80s to the early 2010s never needed updates, and the ones that did were performed at dealers (and were usually for non-critical things, because the critical things were simple).
GM had a good run in from the mid-90s to the mid-00s producing bug-free cars, even with some complexity. I don’t know of any software issues on any cars with LT1 or 3800 engines, nor with any of the tech in the Northstar Cadillacs. Displacement-on-demand could be considered a buggy implementation, but it was working as designed, and never got patched out, so I don’t think it counts.
That’s of course ignoring the decades of cars that had no computers at all. No software bugs being patched out with OTA updates in a carburetter (you have other problems obviously though, namely terrible fuel economy and emissions, and generally lower reliability).
If you make it a hard requirement for a car to be bug-free (maybe outlaw OTA updates and force physical recalls on any software problem?) I can guarantee manufacturers can make a bug-free car. It’ll just be way less complex and have way fewer flashy features, and will either cost more or have lower margins. It’s been done in the past, it can be done again.
There is a sweet spot for the level of computerization in cars. We had it somewhere around the year 2000, then waaaaay overshot, and haven’t corrected back.
Alot of embedded stuff is outsourced and doesn't want to waste the computing power for stuff like stack canaries. I recall the following from making a tool for a dlink? router?
//Reads a file name foo ReadFilePath() { // Get file name // TICKET 21321: Fixed crash by increasing buffer size char FilePath[100]; ReadFileName(&FilePath); }
It sticks out to me, since the crash was clearly from a buffer overflow, and they had this documented in the source code that increasing the buffer size fixes it. What they didn't realize was that the bug would still happen and you could get a buffer overflow from this and do whatever you wanted. This is the level of programmer you're dealing with who's writing embedded software in an overseas sweatshop. And the talent isn't even there domestically since they're severely underpaid compared to someone writing simple javascript.
The pipelines to create more such people are sorely lacking though
But if you take a couple C/assembly/systems electives, look for internships at hardware companies, build a couple of toy projects on the side, and graduate with even a modicum of embedded experience, there will be companies that will hire you, pretty much guaranteed. You won’t be making 250k out of the gate, but you should still be making a more-than-livable salary (and frequently in a lower cost of living area than, say, the Bay), and if you pick companies correctly, you can be working with and learning from some truly genius engineers.
The pipeline’s there, it’s just not attractive (read: $$$$$$$) enough to pull in most people in the industry.
can someone tell me if there are any course that taught this??