Hacker News Reader

Comment by wolverine876 | Hacker News Reader

While I think NASA is the most incredible organization in human history, be wary of simple underpromising-and-overdelivering:

If everything they build lasts longer than expected, not only are they giving bad estimates (which perverts allocation of resources), they are way too smart and experienced to be doing it unintentionally.

Now if Ford Motor underpromised on your car, telling you it would last 2 years, you wouldn't buy it in the first place - you'd never find out that it really lasts 10 years. But I suspect NASA gets away with it because everyone is blown away by even the underpromised result - helicopters on Mars, holy sh-! - and also nobody has experience with how long Martian helicopters typically fly.

View on HN

They never said the helicopter would break down after 5 flights. They said it was designed for 5 flights. Shipping it to Mars is expensive and slow, so they build in a lot of margin to increase the odds that they meet the requirements.

Take a much more common example: expiration dates. A product going bad before its expiration date is an issue. Potential a massive one that could kill people. A product that remains in good condition after its expiration date is fine. As a result, expiration dates almost always occur far before the product actually expires.

> They never said the helicopter would break down after 5 flights. They said it was designed for 5 flights.

That is salesperson talk. NASA has been doing public communication half a century too, and they know what people will take from it.

Meh. I don't think your position makes sense.

Your assessment of the estimates is basically MBA talk.

They designed the helicopter for 5 flights. It is very likely the case that the requirements that a rotary wing aircraft can operate for 5 flights with minimal operator intervention, and only between missions, resulted in an helicopter that had a longer lifespan. Could they have done it cheaper? Maybe, probably, but most likely with a much higher risk of failure, and given the time, distance, and cost, and the organization doing the work, I am pretty confident that the team made the right choice.

They designed the helicopter for at least 5 flights. It's just PR. Classic underpromising and overdelivering. NASA is a an entity that seeks public support and funding. Saying they over delivered by 16x is a good way to get the public, such as people in this thread, to think they're doing great.

It’s more than PR: it’s recognizing that there is a massively greater cost to failure because fixing even a small problem is impossible. When you’re designing something with sufficient robustness to hit that minimum success requirement, it’s almost certain that you’ll be able to exceed it by a large margin. NASA is willing to pay more to avoid failures, and given their funding model that’s almost certainly the right call.

Think about it like having to get across town after work for some reason. If it’s your buddy’s party and it’ll run all night, you might take the bus or call an Uber but it’s no big deal if you’re a few minutes late. If it’s your party, you might pay to park your car nearby or have a ride waiting for you. If it’s the meeting where you’re selling your company or announcing your presidential campaign, you might have a helicopter waiting on the roof. NASA’s stuff is all like that last case because the base costs are so high - shipping anything to another planet and waiting years to do it again, if you are given the chance, costs so much more that it makes sense to pay extra to build everything with deep safety margins just so you can say the odds of failure are comfortably low.

It’s underpromising and over delivering.

They knew internally that it could fly 16x more. They just didn’t want to say it in a statement publicly.

It also means costs and estimates are incorrect and a potential waste of public resources

Again, nothing is incorrect here: a margin is only waste if you’re a Walmart MBA obsessed with lowering costs. If the DOT builds a bridge with a 70 ton rating and it doesn’t collapse when an 80 ton truck drives over it, that’s not waste but rather the engineers prudently including a safety margin in each component to protect against minor defects or inconsistencies meaning that one component being slightly under spec causes the entire system to fail.

NASA and aerospace as a field traditionally have a bias towards that kind of overlapping safety margins because the kinds of failures they get destroy the entire air/space-craft. If some MBA at GE cost-optimizes your dishwasher, they’re better off trimming “waste” until it dies the day after the warranty expires as long as it doesn’t fail in a way which causes you never to buy their brand again. NASA faces far more catastrophic failures and because their systems are all custom with economy of scale for large product runs and their missions by necessity have a hefty base cost so it doesn’t make sense to save $50k on rover parts when you are paying $500M to ship it to Mars.

They knew it would last exactly 80 flights? Wow, they are good.

Actively promising outcomes that push equipment to the brink of failure would be malpractice and the planners would deservedly be fired.

These are novel and truly unprecedented space missions. There may yet be a future in which we can confidently predict outcomes at scale, but we are not there yet. Being conservative in assumptions is a virtue, and hopefully is emulated in every future mission.

The PR is how they're reporting it. "We want an minimum of 20, but it could make upwards of 200 flights before giving out" isn't as punch as they want. They want to say, "We only are planning on 5 flight. WOW, 15 flights?! Wait wait wait...NOW IT'S UP TO 40 FLIGHTS?!?!"

(And if your target is at least 20 but you say it's 5, you can miss your target and still claim you succeeded.)

Look at the reporting from NASA. For instance[1]:

> “Less than a year ago we didn’t even know if powered, controlled flight of an aircraft at Mars was possible,” said Thomas Zurbuchen, the associate administrator of NASA’s Science Mission Directorate. “Now, we are looking forward to Ingenuity’s involvement in Perseverance’s second science campaign. Such a transformation of mindset in such a short period is simply amazing, and one of the most historic in the annals of air and space exploration.” ** > “This upcoming flight will be my 22nd entry in our logbook,” said Ingenuity chief pilot Håvard Grip of JPL. “I remember thinking when this all started, we’d be lucky to have three entries and immensely fortunate to get five. Now, at the rate we’re going, I’m going to need a second book.”

"One of the most historic in the annals of air and space exploration"? Or here[2]:

> NASA's Ingenuity Mars Helicopter completed 71 flights since first taking to the skies above the Red Planet on April 19, 2021, far exceeding its originally planned technology demonstration of up to five flights.

Of course, this makes it to the Wiki article[3]:

> The helicopter's performance and resilience in the harsh Martian environment have greatly exceeded expectations. The aircraft surpassed its required altitude and flight duration specifications soon after beginning operations on Mars. This allowed Ingenuity to perform far more flights than were initially expected of the aircraft.

Or look at how XKCD has a comic about "Wow, I can't believe this rover exceeded our expectations by so much!" for both Spirit[4] and Opportunity[5].

At this point, it's hard to not see this framing as NASA spin, one that people keep falling for over and over again.

[1] https://mars.nasa.gov/news/9146/nasa-extends-ingenuity-helic... [2] https://mars.nasa.gov/technology/helicopter/#Helicopter-High... [3] https://en.wikipedia.org/wiki/Ingenuity_(helicopter) [4] https://xkcd.com/695/ [5] https://xkcd.com/1504/

The XKCD comics do not at all suggest that we are supposed to treat the unexpected longevity of the missions as spin from NASA.

One is a playfully satirical exaggeration of capabilities. To the extent it offers a commentary, is fully crediting a rover for it's toughness and longevity.

The other approaches the longevity from a perspective of sentimental affection for an innocent rover who grows increasingly lonely as it's mission grows longer and longer. It does not imply prior knowledge or PR spin on the past of NASA.

The whole reason why people think of it as being an incredible amount of longevity is because of the NASA spin. Both of those say the planned mission was for 90 days, and so you get things like the exchange in the Opportunity comic of "11 years, wow."/"Wasn't the original mission 90 days?"/"This is starting to get weird.", or things like the Spirit comic having "Day 1944 of 90."

Neither of these really work if NASA says, "We're hoping for a decade, but our minimum goals are 90 days." It works because of the NASA spin, "Wow, this was only supposed to be a 90 day mission/5 flights, but we can't believe how much these things have exceeded our expectations!"

If people are continuously telling you their work is so amazing that they're blown away by how much the results have exceeded their expectations, you should probably question how honest they're being with you.

I've found NASA's communication to be misleading in general. A while back I tried to dive into their spinoff claims, and there were a lot of exaggerations there. There's a reason why you hear press releases from NASA saying things like "Research done on the ISS could help cure Alzheimers!", but you don't hear Alzheimer researchers going "Man, that research on the ISS really helped us piece together what was happening."

Or "NASA says there's going to be a major announcement tomorrow about a discovery on Mars!", and the discoveries are usually "we've found yet more evidence that Mars used to be wet."

Even the missions themselves - there's been a lot of talk about how Perseverance is collecting samples as the first part of a mission to send them back to earth. But the difficulty is in launching a return flight from Mars, something we're nowhere close to being able to do at the moment. You have to wonder how much of the sample caching is really useful, and how much is just a way to look as if were making progress on something that's still incredibly far away.

To be clear, I'm not saying they are wasting resources, I'm saying they just underestimate.

Cyberdyne industries: You were designed to live for eight years.

Cyberdyne industries: I never said you would die at the end! Aieee my eyes!

You are proposing an interesting thought experiment or hypothetical (“to what extent was the design team expecting a 10x longer lifetime?”). That’s worth pondering.

Furthermore, you are answering it with a “they must have known!”

I just don’t see how you can know this. (For context, I work at JPL and interact regularly with some of the Perseverance team, but nobody from the helicopter team, which was very small.)

Some context. The helicopter was originally quite controversial because it had no clear science rationale (mission science requirements did not imply a need for a helicopter). My understanding is that it was put on as a tech demo and (paraphrasing) cool thing. The lab director at the time was a proponent.

Of course it was a huge success.

Because of its precarious status, it was not designed to fly a lot. Planning flights takes a lot of resources, coordination with the rover takes resources, etc. Bandwidth, time, and people. My point is that the “plan” was kept small partly because these ancillary costs had to be kept down. This would hold even if the hardware could last forever. The tech demo must not get in the way of the prime science mission, which had already been planned.

The use of COTS components is another factor. The effects of radiation on microprocessors (in the Mars-surface or deep space environment) are not really well known, as a practical matter. There is just not much engineering experience with how bad it can get. So the longevity of the Snapdragon was (AFAIK) highly uncertain, not to mention whatever other COTS components were on it.

Hope this context is helpful.

Hey - thank you! I really appreciate hearing from someone who actually knows what they are talking about.

I know nothing about the helicopter in particular but as I pointed out, NASA seems to very frequently underestimates longevity and performance (not a crime at all and maybe politically wise); I can't remember when they've overestimated. Forget my impressions in prior posts - how and why does that actually happen?

> tech demo ... precarious ... not much engineering experience

Also, while I've taken the opportunity of spare resources to do experiments (though not usually on Mars), I'm careful about timing and appearances - I don't want to embarass my team or partners with a failure that's visible to outsiders who wouldn't understand it was an experiment, and would think we just failed at our jobs.

Everything NASA does seems to be on a very public stage, interpreted by endless outsiders (like on HN :) ). The whole world watched the helicopter's first flight. How can someone approve the helicopter, and promotion of its flight, and risk the world saying, for the next decade, 'that crazy helicopter embarassment shows we need to rein in NASA', blah blah blah? NASA's reputation is very important to humanity.

Thanks for sharing what you know. You guys do amazing things; people are so jaded that they don't realize how incredible it is.

For very expensive projects like mars landers, it's imperative that everything is very reliable. If the mission plan is 5 flights, that means the systems need to be reliable enough that there's a 99.9% chance that it will successfully complete those five flights. But that also means that there's, say, a 98% chance that it can successfully complete 20 flights and maybe a 80% chance that it could complete over 50 flights. So these extended missions are really a consequence of building things to complete the primary mission as effectively as possible.

As I tried to clarify above, the helicopter was a tech demo ($90M) and not subject to the same risk rules as “very expensive projects like mars landers” (Perseverance: $2.4B). Your first sentence is trying to equate them.

Heli was on board the lander but it was not subject to the same risk constraints.

I think your probabilistic risk assessment has something going for it, but not all risk would scale that way.

I’m surprised at the self assurance of some of these comments!

One key differentiator is if the mission is competed or directed.

Directed (in essence) means that the National Academies (NAS) decadal survey recommended that specific mission. (They specify the what but not too much about the how.) A directed mission is assigned to an implementing center (JPL , GSFC, etc.) And they build it to accomplish their best interpretation of the NAS DS guidance.

Sometimes this can be expensive, more expensive than the NAS expected (JWST). And sometimes the imperative to make it work to accomplish the “down from the NAS” guidance, with near 100% certainty, in a novel environment will result in over performance like you noted.

The noteworthy directed missions are mostly large flagship missions.

The other category is cost capped competed missions. (VERITAS is a recent one, so was New Horizons, but they exist in astrophysics and earth science as well as planetary; see for example https://en.m.wikipedia.org/wiki/New_Frontiers_program). “Tell us what you can build to accomplish some portion of the following high level science goals in $300M or less”

Over engineering is not really possible under those conditions! You will be competing against the absolute best concepts and teams in existence. So the budget is really squeezed.

These two mission types are complementary in the obvious way. The flagships push engineering capacity way forward, and the competed missions have a lot of innovative concepts within a cost cap.

There are also pure tech demonstrations (often piggybacking like the helicopter did) and certain other categories but the competed/directed split is primary.

Thank you, that deepens my understanding quite a bit. Either way, directed or competed, NASA missions hardly ever fail - very impressive in itself.

A 99% chance of lasting 5 flights is likely the same build as a 50% chance of lasting 72.

OTOH there is approximately 0% chance a Ford is going to last 10 years without any maintenance or repairs.

Or the estimates are based around guaranteeing a high statistical chance they hit the estimate which is perfectly fine to do.

Yes, same thing - and I think we can assume that NASA has people smart enough to know it's the same thing.

Yeah, I've often wished organizations I've worked for would define what an "estimate" is. Do they want an estimate I think there's a 50% chance we'll hit, or do they want an estimate I think there's a 99% chance we'll hit? In my experience they want the 50% estimate and expect us to hit that estimate 99% of the time.

Ford knows how to build cars after building them for 100+ years. There's nobody who has ever tried to fly a helicopter on mars before.

NASA built in lots of margin to help ensure they met their objective, especially considering that it's really fucking hard to get a new helicopter to mars if it breaks.

It’s more like Ford sell you a car that’s guaranteed to have no breakdowns or need oil changes for the first two years. It might run a lot longer, but you they only promise two years so you plan to use it for that long.

If it broke in those two years you’d be very upset, but if it does last longer that’s a bonus.

I think it's more like guarantee. If it breaks down in less than 2 years, you get your money back. Still, no one buy a car to only use it for its guarantee period.

Still, no one buy a car to only use it for its guarantee period.

Unlike with software, where they call that "support" and try to scare you into their next version immediately after that ends.

Unfortunately, the automakers are slowly picking up those practices from software.

But we’re not talking about literal cars.

We’re talking about on-off speciality devices designed to perform a task where there is no possibility of repair or service. They’re overbuilt to provide the best chance that the main mission is accomplished. Once that’s done they can keep going until it breaks.

OR, it's design a thing to work for the mission that had been designed/approved/funded. Any science that can continue after the missions actual experiments have concluded is just bonus. Would you want the thing to be designed at any lower caliber just because it was only needing to do 5 short durations? Would it actually be any cheaper? Doubtful.

So, we keep having gear designed that shows us exactly what could happen for longevity while accomplishing the actual mission goals for the same budget

As the expression goes, "Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands."

NASA isn't a business. And while they do have to keep budget in mind, thank fsm they aren't ruled by quarterly profits minded bean counters.

>which perverts allocation of resources

when they get the funding for the mission is when they make the lowball estimates, but with the funding they build equipment that is much more durable. It's hard to see where any perversion takes place

Exactly, that's actually getting more bang for the buck.

Resources are allocated based on return on investment. If estimates are inaccurate, so are ROI projections. It's not just this project, but others too: How do you compare ROI of multiple projects all with inaccurate estimates?

sorry, no, you are ignoring the direction of the errors.

>be wary of simple underpromising-and-overdelivering

Is that actually the case when it comes to the helicopter in question?

Also, why is "under promising and over delivering" a bad thing? There's a missing step in the argument where it would be necessary to explain why that is bad.

If only the rest of the world hit their targets in the same way. The world would be much more reliable place.

Reply on news.ycombinator.com