Comment by assttoasstmgr | Hacker News Reader

Comment by assttoasstmgr | Hacker News Reader

Parent Full thread

This. The OP wrote:

> They tried to lowball him badly. Like 1/2 the rate he asked for (contract) and probably 1/3rd or 1/4 what we made at Google

FAANG arrogance never ceases to amaze me. It's like these guys live in a different universe blissfully unaware of the world around them. Did it ever occur to him maybe Ford is paying normal average human salaries and not grossly inflated Google wages?

Remember Ford (and this analogy extends to most companies in traditional industries) still has to be able to sell cars to the average blue collar Joe at a price point he can afford, and still turn a profit. Outside SV most people don't drive $80k Teslas. Somebody's still gotta write software for the $16k econoboxes. I'm sorry but nobody deserves to get paid more than e.g. a specialist doctor to hack JavaScript code I don't care how smart you think you are, when there's a guy in India probably just or nearly as good at it who'll do the job for $40k/year. FAANG is swimming in money and can afford to piss away $$ in talent wars, artificially driving up salaries and the prices of everything else, which is why no average person can afford a house in SF anymore. Those that won the FAANG lottery and managed to cash in? God bless them. But don't let the reality distortion field make you think that's how the rest of the world works.

So for context, I'm in automotive on the FAANG side and work closely with people at traditional companies.

There's some misunderstandings here. For one thing, there are obviously skilled people working at every salary level. What you're missing is that this is largely irrelevant. Relative compensation (in software) is a shockingly good proxy for all the associated support and people investments to support them. Employees that cost less are less supported and thus less enabled to produce high quality work than their better-paid counterparts, as a very rough rule of thumb.

It's my experience that OEMs are significantly less invested in their developers and the quality of the software they produce than any FAANG. They don't consider software to be a core competence and they rarely invest in the technical expertise to understand quality or push it internally.

The scale of dev costs is also pretty minor for vehicles as a whole. Amortized labor costs are typically on the order of 10-20% per vehicle. Devs are a small portion (~100s-1000s) of the tens of thousands of engineers that work at major OEMs and their suppliers. An extreme 2-3x wage increase would end up as a few percent increase, but the truth is there are cheaper ways to do that than salary. "All" they have to do is care about the software components the way many do about the mechanical components. It's a cultural problem rather than a strictly monetary one.

>> They don't consider software to be a core competence

Because it isn't for OEMs, theirbequipment is their core competency and aoftware, especially the entertainment side of things for automotive OEMs, is just part of that.

Would you argue that software isn't a core competency for Apple or Samsung because they sell phones? Same situation.

Modern vehicles are just distributed computers with tires, and self driving vehicles are worse. Virtually every feature is either massively supported or entirely enabled by software.

> OEMs are significantly less invested in their developers and the quality of the software they produce than any FAANG.

We must be using different FAANG software because the ones I use on a daily basis are slow and buggy as all hell. The last few versions of macOS/iOS have been atrocious. Every version of CarPlay I've used has bugs. Amazon apps are full of glitches. Compare that to developing engine ECU software which I think is a far more difficult engineering task that the average overpaid UI JavaScript slinger could not handle. You can't just update ECU software every week because you sloppily overlooked something. Unless you're Tesla of course.

I don't deny that there are many outstanding, technically excellent engineers at FAANG that deserve every penny they make. But the sad reality is many of them are just TC-chasing posers who got lucky and have subsequently deluded themselves into believing that whatever narrow expertise they now possess is worth far more than it is on the open market.

I think you’re conflating the UI of Amazon with the more difficult problem of getting a large distributed system to work at scale.

Conversely, if your ECU has minor glitches, will you notice? Unless it results in catastrophic failure, you will not even know if happens to result in the occasional 0.1 mpg increase in consumption.

And I’m sure VW and the other companies put a lot of resources into software engineering the exhaust emissions defeat workaround.

Want to know something terrifying? ECU code is often nearly as sloppily written as that JavaScript. They follow MISRA and usually have a couple "QA" teams to catch the most egregious errors of course, but the overall quality is eerily similar to what you'd get if half the team came straight out of a time machine from 1995. I've had to give informational sessions on why boot systems should have redundancy and what undefined behavior is, for example.

And yet the ECU shipped with nearly every car for the past few decades almost never fails through millions of cycles.

I think all this "ECU code is bad" stuff is just people who don't like the style of the produced code. Are you sure those edge cases aren't accounted for in other ways? Maybe they decided that everything should be "best case" code, and any error just resets the ECU which boots up before the next spark event needs to be triggered.

I just cannot square how much people complain about the code in ECUs with how utterly reliable they are. Consider the group that investigated VW cars cheating on emissions. The system was complex and obfuscated, but also powerful, reliable, and configurable by the manufacturer.

A similar argument was made by Toyota about their systems during the lead-up to the unintended acceleration debacle. I recommend Koopman's talk [1] on the subject for details, but long story short the Toyota ETC was apparently utterly reliable and installed in some of the most popular vehicles on the market at the time. However, detailed investigation of the systems and code behind the few publicized incidents revealed basic system issues like the watchdog failed to detect task crashes, rampant memory safety issues, and even a failure to follow their own internal code guidelines. All of that was manifesting in people's vehicles, but it was only rarely turning lethal and even those cases went mostly unrecognized.

Things have improved significantly since the Toyota acceleration issue, in large part because all those details came out. Model-based design is now basically standard for most ECUs, which eliminates large amounts of buggy human-written code. Formal methods tools that don't entirely suck exist. However, most of the issues Koopman points out (especially memory safety) still exist in many places on modern vehicles and all of his points about the issues with relying on testing to surface quality problems remain relevant.

[1] https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_...

https://youtu.be/DKHa7rxkvK8

I think that's besides the point. If a qualified developer can get paid more elsewhere, they'll go elsewhere, and Ford will end up with marginally- or un-qualified developers. That's just how the job market works.

Only if money is the only thing you're optimizing for, which isn't necessarily the case for everyone.

I once went for a job because of the money alone - it was the worst experience of my life and made me realize that money is not worth optimizing for beyond a certain point of comfort.

Sure, but you have to look at total costs, span over lifetime. How long will it take you to pay for that house you probably live in? How much time do you waste commuting? How draining is your job? How much do you spent when going out on some show, restaurant etc? How close to true nature are you?

I've followed money in my job decisions in the past and it has worked out OK for me so far, but of course I have no clue how my life would look like with other choices. But living in cramped place like SF for double the salary (thats max) and double the costs... no thank you.

I agree mostly with your points, other than when you get into how much the cars cost.

The unit price of the cars isn't really relevant to the cost of developing it's software. A low cost model that moves sells a lot of units, can likely afford to spend more on software compared to a high cost model that moves only a few units.

That's not how unit costs and margins are calculated.

I don't think I described how unit costs and margins are calculated.

>> The unit price of the cars isn't really relevant to the cost of developing it's software. A low cost model that moves sells a lot of units, can likely afford to spend more on software compared to a high cost model that moves only a few units.

You didn't? Of course software costs are part of the unit costs for each car produced, just how those development costs are actually allocated is up to Finance (accounting and controlling) to decide. And of course those costs are relevant, as all the other costs of a car maker.

Putting your accusations of arrogance aside, the point is ultimately this: unlike the work done by a worker on an assembly line, software work scales out "infinitely." You may pay that team of developers a huge amount but the work they have done is reproduced at "zero" cost hundreds of thousand of times.

It makes sense to invest in quality there.

I have made the point that ad revenue distorts FAANG practices elsewhere. I agree on this. You'll note that myself and my friend had no intention of seeking a FAANG salary elsewhere, it's unrealistic. However. There are limits. And sane practices.

The low compensation band reflects a lack of recognition of a) the importance of software and b) the economics of it. It isn't just that they pay bad, it's that they produce bad software because they don't get it.

on the other side, I'm not beholden to work at less than what I could potentially make just so some blue collar worker can have a car.

Reply on news.ycombinator.com