Elon Musk: Tons of C++/C engineers needed
twitter.com
twitter.com
Edit: There are lots of junior-sounding devs on this thread. Don’t let me discourage you. Just be sure you’re joining a team with strong engineering leads that will be good mentors.
[0] https://www.tesla.com/careers/job/autopilot-systemssoftwaree...
If they give you like 1 short task that you're supposed to solve on a white board, well, that doesn't really map well with what you're actually supposed to do.
But if they sit you down for 8 hours with a reasonably interesting task (both optimization and architecture wise), that's a pretty decent filter. It just takes a lot of time and effort from both the company and the candidate so almost nobody does it.
It's not like the usual filters (engineering experience or education) are this amazing gold standard either. Just because someone wrote barely passable C++98 code for 15 years that doesn't make him a great programmer, and neither does having degree.
I have deep reservations about using members of the public (and others on the road) as guinea pigs. Before the Walter Huang fatality, maybe, but after that and how Tesla acted about what was clearly a software bug in software that wasn't fit to be on public roads ("beta" label or otherwise), I could never justify working for such a company. Imagine being a software engineer on AutoPilot at the time, and then having the company you work for excuse your bugs by blaming the victim for not circumventing AutoPilot controlled steering into a wall.
Software development ethics don't come up a lot. But I feel like AutoPilot is up there with medical software, aviation, and military, in that mistakes can easily cost lives. I want to be able to sleep at night.
It is a cool problem-space, but only academically.
Maybe the guys he publicly fired were idiots, but even then you can manage that differently. It was a to a time when he got a lot of crap from media, but that is not a good excuse. Seriously lost faith in the company for that.
If you are so cautious that you never ship any self-driving code, there are lives that may be lost due to accidents caused by human failure. These accidents might have been avoided if an AI was at the wheel.
I respect this type of work is not for all people. I applaud the brave companies plowing ahead.
So you need ridiculously huge samples to get statistically significant data on the safety of a single manufacturer's system. As in the order of magnitude of a hundred thousand miles per car for every one of the ~ 900k Teslas sold so far.
Thus you reach a paradox: with current trends in "normal" car safety improvement, you won't know for sure if a particular self-driving model is safer than non-self-driving models until they are close to end-of-life. And then it's a bit late to find out.
AP will never prevent all injuries / fatalities due to road conditions and other human drivers simply being imperfect. It has a good chance to prevent a lot of otherwise fatal crashes in the meantime however. One could say it would be unethical to not even consider the alternative here.
Just some food for thought. I'm a software engineer myself and Model 3 owner.
My Volt had the same exact hardware as AP1 (from Mobileye). GM decided not to play games with marketing, and had it only intervene in cases of immediate danger.
For example, you could try and get it to follow lanes, but it's guidance would be directly proportional to your steering input, and it would intentionally stop intervening the moment you were back in your lane, eventually disengaging if you weren't trying to correct at all.
-
At night in pouring rain coming around a tight highway turn, my car stopped for an overturned box truck covering 2 of 3 lanes and allowed me to go around it. It saved my life with very little fanfare.
AP didn't have to be called AP. AP didn't have to be something you can let take over (except jk don't let it take over or we'll victim blame you when it kills you).
AP could provide traffic-aware cruise control, could take over when you drift out of your lane, swerve away from imminent crashes, and you would still have been saved from that T-bone, and Walter Huang would still be alive today.
But then Tesla couldn't charge extra for Advanced AP, then extra on top of that for FSD.
Everyone who works in the safety field has to deal with this. How do you think the MCAS engineers feel?
But let's rewrite history, and say Boeing decided to scrap MCAS before it ever shipped on a 737 MAX. Now let's say a MAX crashes from stalling on takeoff - the exact scenario MCAS was meant to prevent. How would those engineers feel? How would we be looking at Boeing if we knew there was work done on a prevention device, and the higher-ups decided to scrap it before it was completed?
No one engineer or manager is responsible for a tragedy. Organizations, sure, but no organization can be perfect. You can only try to use whatever power you have to guide it towards better practices and regulations.
This sounds like another example where his hard core coding tests filters for code-grifters.
You know the type: People who game the system by studying L33T coding challenges, and can hack out some nonsense from copy/pasting stackoverflow. They get ‘something’ working, and ship it, and declare that it is done. But when you really investigate it, you find that it is a disaster waiting to happen.
And in general, they cannot engineer a formally-verified, fault-tolerant system to control complex systems, complex heavy machinery, and intricate business logic.
But, these are the engineers that will pass his hard core coding challenge. Welcome to the so-called world of Software “Engineering”.
Do any automobile manufacturers actually use formal verification? How do you even apply such techniques to things that leverage ML like Tesla’s autonomous driver?
It seems like ML is statistically safe for everyone else, until that moment when you become the “statistic”.
Put me in a real environment with access to documentation, actual team members, and time to learn and understand a system? That's where I actually shine.
Something I kinda like the idea of for a "code interview" would be a "reverse interview". E.g. give them some sample code (likely from production) and ask them to critique it and review it.
I'm no expert in SQL - but I can still spot an injection vector. If I need to use SQL regularly then I'll learn it as needed, etc.
You wonder why they're not using D or even Rust or Ada to get systems that are more provably safe?
Good engineers that already know C/C++ are 20x easier to find.
There is nothing taught in Computer Science at college that can't be learned online.
I appreciate that Elon is hiring for skill and knowledge and not a paper that says you graduated from some expensive school and therefore are automatically more qualified.
I've had interviews before where people have given me code on paper and asked me to critique it; also gave me a program where they'd introduced a bug, and asked me to find the bug.
It's a shame these things haven't caught on. Developing green field code is only a tiny fraction of what we do.
Or did you assume that "hardcore coding test" by default only means algorithms questions that you can memorize the answers to?
And are you really sure the people "copy/pasting stackoverflow" are the same ones who pass algorithms interviews? Sure doesn't match my experience.
In any case, you sure jumped to a lot of conclusions from a single tweet.
So even a competent programmer, who might have spent last decade writing GUI in Qt, could be practically worthless for real-time embedded C/C++ while passing any whiteboard tests.
It’s just less likely you’ll hear about it if it doesn’t involve a spectacular plane/car crash, missile launch, radiation release, etc.
Ad targeting comes to mind (e.g., accidentally pushing opioids and worse at mentally unstable people because the computer inferred they are likely to buy).
It sounded pretty straightforward. Coding test, talk to a bunch of people onsite, etc. Each of the people would write up a report and it would work its way up to Elon Musk and he would make the final decision as to whether or not to hire me.
In a small company of up to ~200 people, sure. A department that's maybe a layer or two beneath him, ok, maybe? I don't remember where this was exactly, but it was something involving web services or similar in C#, and I don't even think it was front-facing, but for internal use, so probably somewhere in the IT department.
For my personal tastes, when the CEO is making hiring decisions on every position, that's an element of micromanaging that I'm not comfortable with, so Tesla is pretty much off my list.
I'd love to work there, but I know I would not put in the effort to pass the coding tests.
I am aware of MISRA C for automotive but is something like that sufficient to guarantee against unintended behavior ?
If not, can someone comment on why this isn't the case today ?
I work on space craft flight software and exactly none of it is formally verified. As far as I know, it's not practical to do so. If we could do it (without ballooning cost and schedule 10x), we would.
A large portion of Microsoft's in house developed device drivers for windows are formally verified.
Boeing together with DARPA built an autonomous helicopter on top of the formally verified Sel4 microkernel https://ts.data61.csiro.au/projects/TS/SMACCM/
Another one:
Wireguard uses formally verified implementations of ed25519 https://github.com/project-everest/hacl-star
Honest question (I obviously don't have a ton of experience in this area), what does "model-checked in TLA+ though the applications themselves are not written in TLA+" mean (in terms of the guarantees you're getting)? If you're not verifying the actual code, then how do you prevent bugs creeping during whatever happens between verification and actually writing the application code?
The HACL* project looks interesting. I would definitely accept that as an affirmative answer to my original question.
Code review! So, SO much code review. Also testing and type systems and linters and code review.
(There's also some work on automatic synthesis, like PGo, but that's all very, very far from being production-ready.)
The main benefit of using TLA+ is that it lets you hammer out issues in the high-level design. Less "oops this has a buffer overrun" and more "oops if this service stalls out for too long our recovery mechanism will violate a core safety guarantee." Turns out that lots of the nastiest bugs are in the design, not the implementation, so using TLA+ saves a lot of time and money. But you still need to code review!
Yeah, but we already have processes to manage the latter. It's the (potential) buffer overruns that keep me up at night!
>Turns out that lots of the nastiest bugs are in the design, not the implementation
I'm not saying it never happens (does TLA plus make sure everyone's assuming SI units?), but that hasn't been my experience at all.
I'd _hope_ you'd have those processes, working on space flight and all :P. Good processes > good tools. It's easier to start using a good tool than overhaul your process. So for people who don't have that kind of discipline or heavyweight process, TLA+ gives a lot of benefits for really cheap. AWS, for example, found it really easy to slot into their workflow.
Re buffer overruns, I just remembered OpenComRTOS, which used TLA+ to verify low-level properties of their code.[1] They found it super helpful.
> I'm not saying it never happens (does TLA plus make sure everyone's assuming SI units?), but that hasn't been my experience at all.
'Fraid it probably can't help you with the SI units. I'm thinking about "design bugs" more like this:
> 1. Suppose we have a document with version 57
> 2. A bulk of delete and reindex will remove the index-v57, increase the version to 58 (for the delete operation), then put a new doc with version 59. While the engine places the index-59 into the version map, the safe-access flag is flipped over (due to a concurrent fresh), the engine won't put that index entry into the version map, but also leave the delete-58 tombstone in the version map. The delete-58 tombstone is stale because the latest version of that document is index-59.
> 3. Another bulk of delete and reindex will increase the version to 59 (for a delete) but won't remove docs from Lucene because of the existing (stale) delete-58 tombstone. The index operation will append document (version 60) to Lucene (instead of overwriting). At this point, we will have two documents with the same id.
Which was a bug TLA+ found in ElasticSearch.[2]
[1]: https://dl.acm.org/doi/10.5555/1779934.1779955 [2]: https://github.com/elastic/elasticsearch/issues/31976#issuec...
Never saw any crash happen there.
You should think of these systems like you think of garbage collection. They lower (sometimes eliminate) one particular kind of problem, and that's it. It does not prove the code works, doesn't OOM, doesn't segfault, doesn't dos itself, doesn't enter infinite loop, etc.
Logical problems (and problems in C extensions) are of course always possible, therefore you have Stateflow and other methods. Other formal methods cannot guarantee that neither.
In Ada you can do Design-by-contract but in the SPARK subset you can actually formally prove the implementation's correctness.
SPARK is used to my knowledge in rail signalling system, air traffic control systems, Class III medical devices, and now also NVDIA plans to use it for sections of their ADAS platform.
I think the cost aspect is overblown. The biggest issue is that anything related to Ada is not really that well known and often what is known is based on stereotypes from the 80's. Also for most people they would not see it as career growth move to switch to an Ada based technology stack.
I suspect there'd be a lot of stuff using either SPARK or Event-B, but Tokeneer[3] is the only thing I found with five minutes of Googling.
[1]: https://www.microsoft.com/en-us/research/publication/ironfle...
[2]: https://www.cs.purdue.edu/homes/pfonseca//papers/eurosys2017...
The problem in both these cases is that "formally verified" just means that you verify your specification doesn't violate anything you've said it shouldn't, but you might have written your specification incorrectly, or your specification might not match what you say it matches, which then makes the whole thing irrelevant.
That's not to say there's no value in formal verification, but I tend to see the value more in verifying some tricky protocol or very concurrent thing works properly (because these often have subtle bugs that are very hard to detect from inspection), rather than as a way to verify that your whole system works.
Plus for autopilot, how would you even specify that? The problem is too big and ill-defined to be specified in any way that would be amenable to verification. Of course, individual sub-problems could be verified but that's no guarantee that the system as a whole works.
And just to dive into this a little more, let's say the first example you gave fails, and we have one sensor saying something is 10m in front of us and the other saying there's nothing for 100m, what do we do in this situation? Well, it depends on the context, maybe the one sensor is picking up a bit of retread tire in the middle of the lane and we can just drive over it, or maybe the other sensor is malfunctioning and there's actually a bear in the road. I don't think there is an easy answer to what to do in general just given the sensor state.
I mean, they're out there: according to the SPARK Wikipedia page, it's been used in things like some Rolls-Royce jet engines, some military planes, some air traffic control applications, a cubesat project, etc. But there's orders of magnitude more engineers out there proficient in C and/or C++, plus even if you're willing to train, it's not that easy to find engineers willing to move into something that doesn't have broad industry support and might therefore limit their future career moves.
C and C++ are already both used for safety-critical systems. MISRA C is the standard in automotive, and there's civilian (DO-178C) and military standards for C++.
“ If you ever need a solid ideas man with no coding knowledge whatsoever, hit me up.”
To which Scott Lawson responds:
“ Ideas are cheap, execution is gold.”
Finally Wilson again:
“ Anyone can learn to code, you can't learn how to have good ideas.”
I assume Wilson (I’m using their names as presented) is kidding, in which case it’s very funny.
Looks like Tesla though are doing C with C++ features rather than C++. I'd also be interested in knowing what their "hardcore" questions are like...I heard that they ask a lot from your past experience.
Tesla's UI is in Qt 5 which is fairly C++
The time necessary for this means a huge opportunity cost for every applicant.
* whats wrong with rustc
* Intellij rust is a neat editor. You'll need clion for debugging though.
Exactly, there are none (yet). Unlike C++ which has Qt and WxWidgets which are all mature open-source GUI libraries and are used in production and seen in the wild.
That is one setback when trying to consider using Rust for cross-platform GUI development.
Rust for the Red Planet
Rust for the 2020s
Mars is covered in Rust
It just makes sense :-)
Recruiters shouldn’t group the together.
I have yet to meet a C programmer who can actually write C++ code. They just use the C++ compiler to write C compatible code.