Spain builds submarine 70 tons too heavy by putting a decimal in the wrong place | Hacker News Reader

Spain builds submarine 70 tons too heavy by putting a decimal in the wrong place

o.canada.com

91 pointsstfu47 comments

In the aerospace business there is a guy known as the Mass Properties Engineer who gets to rule the project with an iron fist. He is responsible for knowing the estimated, calculated or measured weight of every part on the spacecraft from concept through to launch. It's not a very fun job, but it's very important. Some people actually made careers out of it. You simply cannot get this stuff wrong and not pay dearly for it later. There are reviews, arguments, dealings, and handwringing over all of this.

Now, I have no idea if the Spanish sub business runs anything like that, but it surprises me that it apparently doesn't or didn't in this case. It's not like you just let some engineer in the corner calculate the weight and then go build it. This is a systemic fuckup, not a shifted decimal point.

My electrical engineering professors were constantly trying to ingrain in us the tendency to estimate the solutions to problems long before solving them. Why? because if you know the ball park solution you'll be able to catch gross errors before they happen. Thus you will be less likely to pull a Homer Simpson at the power plant or end up transmitting too much power from your antenna. I'm thankful to be in an industry where catastrophic failure does not have such harmful results.

Without more details it's hard to say what exactly went wrong, but it is not at all clear that a ballpark estimate would be of any help here: The submarine weighs 2200 tons, and they think it's 70 tons too heavy.

The bigger point is that 70 tons should not sneak up on you. They should have been managing weights carefully throughout the process.

On the one hand, you are right. Preventing things like this (a 3% budget overrun) in large projects is very, very difficult.

On the other hand, if this really is a single off by ten error, this was not a death by a thousand cuts, but someone must have reported that his 80 ton part only weighed 8 tons. That must have been noticeable, even if it got reported for a 'part' where nobody has a natural feel for its weight (for example, what do living quarters, with kitchen, supplies, and water for the showers weigh?)

Other explanation, more in line with the reality of Spanish reality.

http://www.thelocal.es/20130520/530-million-bill-for-spains-...

> The president of the Navantia board has defended the work of the company's Cartagena shipyard and complained of "meddling" by unqualified people.

> He explained that it had been reported as far back as 2005 that the development process was not being properly followed and that there was a lot of necessary improvisation due to the addition of new elements at the request of the Ministry of Defence.

> “Apparently somebody in the calculations made a mistake in the very beginning and nobody paid attention to review the calculations,” he said.

Supporting the old software engineering folklore that the earlier a bug is introduced (and the later it is found) the harder and more expensive it is to fix.

> Supporting the old software engineering folklore that the earlier a bug is introduced (and the later it is found) the harder and more expensive it is to fix.

I wouldn't call it folklore as there are numerous studies supporting it.

EDIT: I decided to crack open Code Complete for anyone who is interested [1].

[1] "Researchers at Hewlett-Packard, IBM, Hughes Aircraft, TRW and other organizations have found that purging an error by the beginning of construction allows rework to be done 10 to 100 times less expensively than when it's done in the last part of the process, during system test or after release (Fagan 1976; Humphrey, Snyder, and Willis 1991; Willis et al. 1998; Grady 1999; Shull et al. 2002; Boehm and Turner 2004)".

The studies supporting it have apparently been somewhat hard to confirm, a lot of the source material used by McConnell seems to be from secondary sources and self-referential. [1] is a detailed blog post where the author tries to track down the original sources:

[1] http://blog.securemacprogramming.com/2012/09/an-apology-to-r...;

From the headline I thought that somehow they had attached 70 tons more weight onto the thing than they wanted (e.g. accidentally made the hull too thick or something).

But from reading the article, it seems it's actually they just miscalculated the final weight based on the specs. Their miscalculation had it coming in 70 tons under its actual weight (which is 2200 tons). So the sub was built to spec, it's just the buoyancy calculations were wrong.

> And a former Spanish official says the problem can be traced to a miscalculation — someone apparently put a decimal point in the wrong place

Sloppy reporting. Spain uses a decimal comma.

This is an engineering calculation, it may not be guaranteed based on who they're working with and which standard they use.

I'm almost 100% sure engineers use the SI system. (unlike physicist that use a system that seems to be fitting). And the SI system says that 100.000 and 100,000 are the same. The Si system only uses space to group numbers like 12 343,123 12 for twelf thousand and so on.

My thought was that anything that used the decimal comma could be confusing when dealing with vectors.

However, I wouldn't go so far as to say they use SI across the board. I know aerospace engineering in the US can often switch between imperial and metric depending on which contractors work on the project (this resulted in a mission failure on at least one occassion: http://en.wikipedia.org/wiki/Mars_Climate_Orbiter) and it could be the same for other fields.

You'd be right. At NASA we could use whatever we wanted. There was no directive to do one thing or another (at least not one I ever knew about) - only the conventions that allowed you could communicate with your peers. Much of the technology we used had evolved over decades. Changing it to metric would have been insane.

An interesting line from that wikipedia article: This error has since been known as the "metric mixup" and has been carefully avoided in all missions since by NASA.

This is categorically not true, as I worked before and after that, and it had zero impact on the units we used or how we checked them. Most of the time nothing was metric, so it didn't matter.

With the advent of computers and programming languages that only use decimal points, I wonder how long countries will hold out and continue using decimal commas.

I would not be in the slightest surprised to see that engineering would standardize 100% on decimal points (even if the public used commas), just like how in the US engineering (except houses) uses metric, even though the public uses imperial.

(Also, presumably he spoke in Spanish, so translating decimal comma to decimal point in English is not sloppy reporting - it's good translation.)

Decimal points vs decimal commas is a common problem here (and other countries with similar locale details) with software, when exporting numbers in text format to be processed by other software. Over the years I have had to correct for this in 3DSMax and Excel scripts. Very annoying.

It doesn't matter which format of CSV file you choose, or what settings you use, what Excel actually saves will be different depending on the language profile of the individual user. Not useful in a bilingual office, especially if you want a script to parse that CSV.

My simple hack was to create a macro button that saves to CSV, because the macro function ignores the language settings (probably a bug?) and always saves in the English format.

> US engineering (except houses) uses metric, even though the public uses imperial.

I don't know that I'd make this claim, in general. Everything from bikes to American cars can still be found with imperial nuts and bolts. Even the robotics work I did in school was mostly in imperial units due to the greater availability of, for example, 1/4" 6061 Aluminum stock.

People will stick to whatever system is most convenient for most tasks. It makes it difficult to escape local minima.

We've been "fully metric" here in Australia since the early '70's - I still find it amusing when seeing things like the building industries "standard sizing", where everything comes in 900mm increments and lumber all comes in sizes like "50 by 100mm nominal". Everything is still built out of 3, 6, and 12 foot two-by-fours and 3 by 6 foot 1/2 ply sheet - all labeled with metric dimensions of varying clarity.

Same with the Mexican Coke bottles that we get in some US stores (Mexican Coke is made with real sugar). It is a 12 ounce bottle, labeled as 355 ml.

We get 355ml cans here in New Zealand - I had no idea it related to anything at all. Thank you.

When I worked at NASA, nothing was metric unless it came from overseas. Everything we built was imperial. I preferred it for a few reasons, one of which was that it's harder to make a decimal point error with imperial units.

I wonder how long countries will continue to compare the U.S. to Myanmar because of the metric system, while these same countries refuse to switch their primary official language to English and their decimal separator to the '.'.

What's good for one is good for the other after all. ;)

just like how in the US engineering (except houses) uses metric

Copying from a previous comment: This isn't true in my experience. Of course my experience certainly isn't comprehensive, but in my work I interact with a lot of US fields (environmental, manufacturing, civil & mechanical engineering, etc) and I almost never see metric units. The Federal Highway Administration tried to encourage state DOT's to go metric for design and engineering work back in the late 90's but they eventually gave up. If you ever worked on an American car, you still find a mix of metric and Imperial bolt sizes! About the only field where I found that metric units are used extensively is wind energy.

Even my mechanical engineering PE exam used Imperial units exclusively (pounds force vs pounds-mass... sigh)

I imagine that your statement is probably most true for the military though.

The other key figure is that the entire submarine weighs 2,200 tons.

The sub was 3% too heavy and it will cost $14 million to fix, big woop, worse happens 1,000 times per day all around the world.

Just to add to that - it's $14 million on top of a $2.7 billion project.

This is a great example of an attention whoring headline. When you get into the details there's actually no news here.

Didn't they miscalculate a [the?] vital characteristic (buoyancy) on a 2.7B project. Seems news worthy.

"Here is 2.7 billion dollars, build me a super high-tech submarine (with specific specs) and DO NOT make any mistakes." Sounds easy, right?

A military contractor going slightly over budget is not news worth, it's database or list worthy at best. Throw the data on wikipedia with the other million projects that had a hiccup or needed more funding, then be done with it.

no, that's not the cost to fix

the $14M is just for "an assessment of the problem with the S-80 submarine program and the scope of the work that would be required to correct it"

ok, so $14M + parts and labor

I wonder what book time is?

$14 million to find a solution, not to actually put that solution in place. A delay of two years in the $2.7B project will cost at least an order of magnitude more than $14M.

Even if it costs hundreds of millions of dollars, big woop.

How much have we spent on F-35 research, trillions? And those things don't even work in space.

Or under water!

Not really practical for an aircraft to operate underwater, but someday operating in space might not be inconceivable. My point was that for trillions of dollars, the aircraft better do some cool shit. Yes, crash avoidance is awesome, but not trillions awesome.

The sea is arguably more valuable than space in terms of military value, an underwater vehicle is practically invisible and can deliver massive amounts of firepower to any region in the world. Not to mention the secret mission capabilities of submarines such as tapping undersea fiber and special forces delivery.

the article doesn't go on to explain more on how this could have happened, I would assume they run some tests on the specs, or some kind of simulations (I am obviously not an expert in making submarines) but considering they are so expensive, wow, amazing. Someone put a decimal in the wrong place and nobody noticed , what are the odds of that happening!

It's not as unlikely as you'd think.

Even in the U.S. we used to routinely have to put in slop factors to account for contractors that would routinely deliver gear in excess of spec. It wasn't until the most recent SSN class that they were able to finally hold the line against the contractors and sub-contractors... but at least the design itself would have weighed what was expected.

Say they have a CAD model with different types of steel/composites for material. If they select the wrong type of material, or the wrong weight for that material is used, or the thickness of a material is off by even a slight fraction, then an issue such as this could come up. Considering that there are millions of custom parts to a submarine, it's an amazing feat to ONLY be off by 70 tons.

As I've come to believe its always a communication (or generally human) problem rather than technical. Yet another blunder in line with the Mars Climate Orbiter and the Gimli Glider.

Is there such a thing as submarine unit testing?

You can't exactly unit test the buoyancy - the hull is the only part that floats on its own[1], all the other crap you put in would sink. The two need to balance when you combine them, but there's no way to test that at the unit level.

[1] Ok technically a few other parts like air tanks would float.

But you don't have to unit test a radio throwing it to the water. Just sending messages.

As I* understand it the only "traditional" engineering field with anything close to unit testing is electronic engineering. Lots of chip or circuit design packages let you automatically test small parts of your design.

For mechanical/civil projects, this approach is less common. I think that's partly because of the projects' nature (good luck unit testing a submarine!) and partly because of the engineers being more traditional, and further from the software industry.

This said, I can envision some cool things you might do by taking a CAD package and associating some (automatically updating) equations with various parts' critical dimensions. This might act a bit like unit testing. A check of total mass would be another such useful check.

* Engineering student with some work experience, but by no means a blooded engineer.

Depends - as it's designed (using GRC's Paramarine - http://www.qinetiq.com/what/capabilities/maritime/Pages/nava...) you can virtually unit test things like propulsion, stability and buoyancy.

Once it's been assembled into one complete system, it's very difficult to do that - individual systems on-board from a systems perspective could be unit tested, but actual physical items - not so much - generally they only find out about mismatches when they try and assemble the warship/sub from different sections (sometimes built by different companies) and they discover their tolerances were off.

Good thing they found the problem before the sub's first voyage underwater.

Reply on news.ycombinator.com