Hacker News Reader

Comment by blantonl | Hacker News Reader

These folks that design and run these experiments must have nerves of steel.

You know that feeling when you have a file server that is half way around the world and all the sudden it's gone unresponsive? Then you have to start a recovery process with on-site people, etc? The dread? The uncertainty?

Now think of a principle scientist that quite possibly has devoted their entire life's academic work to one of these experiments, and something like this happens. From concept to funding to launch to all the billions of dollars to all the math and luck and other stuff that must go right. I don't know if I could do it.

And then it comes down to one damn tiny pin or something and all could be lost.

It would be like spending 20 years configuring a server just right in a specific manner, deploying it, and it's taken out for ever because of a typo in code somewhere - and it's all gone. Poof.

Nightmare fuel.

View on HN

Indeed. I'm a rover driver & arm operator on Curiosity and there have been a few sandy patches where things were a little ... touch and go. I often think: what if I made a bad call, and my drive ended the mission? Even with a mission that's been operating for more than 10 years, it would be awful: there are hundreds of people involved, including scientists who have been planning investigations for the rover to conduct along our journey over the next few years. I would feel terrible if it was my fault that we never got to their research site of interest.

But then I remember: I'm not an airline pilot. I'm not a surgeon. Yes, my decisions matter, but this is not the highest-stakes job out there, not by a long shot.

You’re driving a ‘car’ on Mars!

Does it actually include touch and go like driving with the driver as decision maker? I expected that every move would be scrutinized by a team of specialists?

In a nutshell, one rover planner (that's what drivers / arm operators are called) writes the sequence of commands, a second rover planner checks it and then walks through it line-by-line out loud with another ~2 rover planners watching. Then the second rover planner walks through the whole thing again with many other teams listening in and asking questions.

The first rover planner also consults with a surface properties scientist during development of the drive command sequence.

So, if things go wrong it's never really any single person's fault. But still, that first rover planner drafting the initial version of the command sequence bears a fair amount of responsibility.

Just wanted to add that if you're a US taxpayer, that's your car we're driving. We really do try to take good care of it :-)

Considering the relatively small size of the budget that might be the most carefully managed bit of taxpayer money in the entire country ;)

Just wanted to say that this is the kind comment chain I come to HN for, when a Mars rover driver comes in with a relevant and amazing anecdote. Absolutely love this. Thank you foobarbecue!

Aw, shucks. If you're interested in more info, there's a paper about how we work: https://doi.org/10.1109/AERO50100.2021.9438370 , the mission blog is https://mars.nasa.gov/msl/mission-updates/ and this reddit channel is great: https://www.reddit.com/r/curiosityrover/ .

this is fascinating.. thanks for the insight!

The rover has a bit of autonomy so it won't drive itself off an unseen cliff or whatever, right?

The autonomous driving capabilities on Curiosity (known as "guarded" and "autonav" modes) are quite limited and haven't been used recently. They don't work in rough terrain, and we've been in rough terrain for a while now. This means that we can only drive to the end of the 3D meshes generated from images from the previous parking spot. When the terrain is acceptable for autonav we can usually rely on orbital imagery and drive farther.

We do set limits on suspension angles and tilt of the rover. So we'll say "stop driving if you are tilted more than 20 degrees," "stop driving if your differential angle goes above 7 degrees," etc.

Thanks for explaining this. Super interesting stuff. It’s the type of comment I love to read here.

Everyone who ships hardware products has this problem. You don't want to recall thousands of items because you forgot to add a capacitor in your circuit.

Software engineers are just lucky ...

Many are. Some of us write very high-stakes software.

And there are a lot more parts to a software system.

I disagree, I do both, and a modern ECU hardware can be extremely complex .. more so than its software.

This is because the hardware needs to meet a stack of conflicting specifications and standards for EMC, robustness, etc. Things need to work at a wide range of temperatures -40degC to 150degC.

Optimising for one standard or parameter causes all your other calibration to go out of whack. Adding a protection element to the circuit can itself fail and be the reason that the circuit fails. At times it can be like the Jenga tower ready to topple over.

Modern standards like ISO 26262 make it even more complicated because now we have to calculate and add up the failure probabilities of every single resistor and capacitor to find the whole-circuit failure probability, and in some applications we would need to prove less than 10 FIT (10 per trillion hours). Usually this can’t be reached without adding redundant components, and then maintaining the document trail to explain why they exist and can't be removed.

And suppose you need to change a component in the design .. you have to find something to do with the 10,000 units of the old component that you bought a year ago before the design was finished because it was on back order for 100 weeks.

I have had feet in both worlds. I started as an EE, and have since moved to pure software.

One of the cool things about writing drivers and firmware, is you can do real-world damage. I’ve blown up $40,000 RF scanners (in 1980s dollars), because I accidentally reversed the position of a switch.

Also, if you think hardware systems are simpler than software, I invite you to open up a modern DSLR (not mirrorless) camera. I worked for a major camera manufacturer for many years, and saw some crazy complex stuff.

But nothing I ever did can hold a candle to the OP.

I think senior engineers should always have good "knowledge gained is proportional to equipment damaged" stories, they're great for helping juniors through their own "learning experiences". (At my current startup, the one about me and the founder/CTO figuring out how to use an API by experimenting on our only robot arm, resulting in slamming a $30k arm into a concrete wall (mmm, polar coordinates) is great for getting across "you'll be more careful next time, because look, really, there will be a next time!")

I’m not here to start any kind of software or hardware war since both are amazing fields that I respect and enjoy learning about.

So rather than take this thread away from the amazing job done by the Juice team, I’ll just say great job and how amazing it all is.

Reply on news.ycombinator.com