“Has something ever failed in the field due to a bug in a CAD tool?”
twitter.com
twitter.com
I guess it's not really a bug per se, just hidden behaviour that you have to learn about and take into consideration.
What? But the software was supposed to take that into account, which is why it it supports bend allowance calculations in it!
(I have a tiny bit of experience with this; I designed some sheet metal thing. I somehow clued into bend allowances before doing it, even though it was my first and only one! I figured, you can't just draw the naive shape; the metal has a bend radius. How the hell does that work? [half a day of Googling] Oh. I used a 2D drafting program that had no clue about this, so I manually drew the pattern with the allowance in it already, based on the thickness of the aluminum.)
Structural engineers aren't allowed to treat software as a black box and trust that it works correctly. They're legally responsible for choosing their tools and verifying that they work correctly.
I make a structural engineering program which is used by pretty senior engineering types in Scandinavia and Germany, who can certainly be trusted to know that with some nonsense input, the output may also be nonsense.
The requirements for the UK are completely different - they want the program to be used not as a tool (like a fancy calculator) but as a simple black box that a novice engineer or even a non-engineer can use.
There is no, "sorry the TV blew up in your face; my scope was out of calibration". :)
Wouldn't it be helpful to be able to look at the code to see what it is actually doing behind the scenes?
Like if the software calculates bend radii that will break things, isn't that a bug? What's the explanation for that even happening? Is it just that the formula involved means that you can't always get a "right" answer, so the software will be like "well, this is the best we can do, figure out the rest"?
https://en.m.wikipedia.org/wiki/Comparison_of_computer-aided...
Virtually all of them are proprietary. They are too expensive to move away from, and every one of them has flaws that firms workaround and build there workflows on. Retraining, proprietary file formats and the disruption of migrating an essential part of most businesses is prohibitive.
Frankly, IMO the CAD marketplace is an example of how closed source really bites an industry. Funnily enough, most engineers - who would never accept inaccurate tolerances on items like screws or bolts - seem to accept this is the way it is and despite the stories we read here let the situation continue.
I'm frankly amazed there aren't more failures in infrastructure from bugs in CAD software. That's more a testament to the fact that engineers are engineers though - when they realise there are problems, they dig in, figure out where the problem is and find a solution, and in the case of CAD software I really think they build or design their way around the problems.
Overall, CAD software is, despite all the flaws of being coded source, still pretty amazing and vendors do fix issues eventually. We can do so many more things in society because of it, but as we evolve as a society it becomes apparent there are limitations. Eventually CAD software vendors will be disrupted, but we're a long way from that point right now!
Not so sure we're that far off. Nobody really has much incentive to innovate in this space because it's so niche. If you're a believer in the version of the future where 3D printers are as commonplace as 2D printers are today, then this may all change.
Suddenly, 3D modelling skills are today's word processing skills (either for work or for fun), and there are a whole bunch of people working on coming up with innovative CAD solutions for the masses.
http://www.pwc.com/gx/en/industries/technology/publications/...
Siemens makes CAD packages. Dassault too. Autodesk does nothing but CAD packages. All in the top 20 of "software companies by revenue". PTC is number 42 and Bentley is 70 again CAD packages. It might be a niche, but it's pretty expensive niche then. Pure CAD companies Autodesk, PTC and Bentley made 2,5 billion dollars of revenue in 2014.
Sometimes autocorrect messes up things I type. It's not really a bug per-se, more just a side-effect of how things like autocorrect work, right? But sometimes Chrome crashes, and that's a bug...
The OP seems to say that the bend radius thing is more of the first case than the second, and I'm wondering how true that is. I'm not really sure of the science behind it (maybe it's an impossible constraints problem).
At least with OSS you can look at the code behind the program to work out how it calculates things like the bend radius.
I'm no engineer, so I had to look it up (Wikipedia was no help whatsoever on this occasion) but I located this page which gives the formula and an explanation:
"Calculating the correct flat pattern layout is crucial to getting a good quality finished part from your press brake. Yet, many CAD and CNC programmers have no idea how to calculate the required values. Years ago, the real experts created cheat sheets and tacked them to the wall. They only taught the new apprentice how to apply the results shown on the cheat sheet, not how to calculate the numbers. Well, now those experts have retired and it's time for a new generation to learn the right way to do the calculate the correct flat pattern layout."
http://www.sheetmetalguy.com/bend-allowance.htm
Now admittedly the guy does put the responsibility on the operator, but it appears CAD packages can indeed calculate this for you. So long as the right info was input into the program, the formula should be programmed correctly and the operator should be able to have confidence in the results!
"Should" is the operative term. I can tell you with that having hand checked the automated results of several CAD packages at developing sheet metal flat patterns and finding unexplained variability and errors too many times to count that if you want confidence in the dimensional accuracy of a sheet metal part, you need to develop it by hand. Especially for anything over 5mm in sheet thickness. As far as I'm concerned you really can't engage in sheet metal design without being able to manually verify the development results, so in a sense it is up to the designer regardless of the capabilities of the CAD software.
To be fair, I'm picking on one of the hardest physical manufacturing processes to actually get right - even CNC brake presses have a great amount of variance in calibration (e.g. the amount of bend allowance you need to make for various sheet thicknesses...) across different suppliers, materials, etc to produce physically identical parts. The valuable lesson for a fresh designer is don't trust your 3d representation of the part, don't trust the manufacturer specs on ANYTHING and always prototype. I'm guessing this is why people tolerate it - the world with 3d CAD is still a vastly nicer place to be an engineer than otherwise, so we just rationalise away the shitty parts as a reasonable price to pay.
Yeah, I know. In my experience, engineers are normally immensely pragmatic people. My dad was an engineer, so I have a lot of respect for the profession. It would just be nice if CAD would be more open.
I think there is scope for some disruption from small companies willing to use STEP as the internal data model for their software but just working on small parts of the picture.
Well, it's not like all companies fixes their bugs or care about them. If, as the parent says, all CAD packages have such issue, what you gonna do?
Mostly I can fix the LibreOffice bug and make the document render correctly (unless it physically corrupts the document when saving), once physical material gets an error it's often impossible to fix.
> I guess it's not really a bug per se, just hidden behaviour that you have to learn about and take into consideration.
I'm not that kind of engineer, so I may be missing something, but it sounds like the reseller snowed you into blaming yourself for their faulty software.
I get what others have said, that a structural engineer is responsible for the things they produce and for doing whatever is necessary to verify correctness, but that doesn't mean that software engineers get to pass off responsibility for the things they produce.
In a better world, I'd love for bug fixing, or even just polishing workflows for existing features, to be as high of a priority for CAD companies as new feature. However, only one of those things is seen as growing the install base...
I routed a two-layer PCB (printed circuit board) using a buggy program whose DRC (design rule check) did not catch the fact that a plated through-hole meant went right through an unrelated track in the bottom layer.
Yes, and so the couple of boards I had fabbed failed. Now that is not in the field. But it's not hard to imagine that such a thing could escape into the field; it's almost in the field.
I didn't catch that until I stuffed one board with all the components.
The workaround was to drill out that through hole slightly wider, removing the metal connecting the layers, but leaving plenty of the pad in the correct layer to which that hole belonged.
This was not a case of logic not implemented in the DRC, but a bug. I tested the issue by adding such mistakes on purpose, and DRC was flagging them properly.
In every other regard, the boards worked perfectly; the fix in the artwork is just to move a track a few millimeters over not to intersect with unrelated through-hole.
Of course, the responsibility was mine to check the artwork and not trust the tool. But then, if you can't trust the tool, what good is it? What is the value in a design rule check which fails, say 1% of the time. Does it still increase productivity? I believe that it does, during the development of the artwork. Any time the DRC catches something, it saves you time. It's just that at the very end when you have the thing fabricated, you have to make a couple of passes over it with your own eyes. You don't have to be that careful at absolutely every stage of the routing, because the DRC "has your back" (mostly).
It's the same with compilers. Just because they don't catch every error with a diagnostic doesn't mean diagnostics aren't useful. They are still useful even if they are buggy, in that certain cases of something that requires a diagnostic isn't diagnosed. But then who reads tens of thousands of lines of code to spot where some compiler failed to diagnose an invalid pointer conversion without a cast, you know?
https://groups.google.com/forum/#!forum/comp.risks
From https://en.wikipedia.org/wiki/RISKS_Digest:
"RISKS is concerned not merely with so-called security holes in software, but with unintended consequences and hazards stemming from the design (or lack thereof) of automated systems."
Also, perhaps by culture, we don't trust CAD enough to let it get us into that kind of trouble, because we know that modeling always involves approximations and assumptions. Truth be told, I rarely see CAD used to determine critical design margins, without at least a sanity check.
See: https://www.ima.umn.edu/%7Earnold/disasters/sleipner.html
Here's a slightly more detailed article
http://journals.library.mun.ca/ojs/index.php/prototype/artic...
--
Seems like this could be avoided by automatically adding more elements and rerunning simulation, measure the gradient of the solution as the number of elements change. Exactly the kind of thing a computer should catch. And exactly the kind of thing having alternative analysis should prevent.
* used an unverified model
* trusted software, didn't double check results
* low safety margin
* incomplete risk model, lack of project wide oversightAs for automating it, sure for a model like that with tens or hundreds of elements, it would take seconds on a modern computer, great. Except, model complexity always scales to reach the boundary of reasonable computation time on modern computers. This has to be a corollary to Moore's law or something. Recently I've been working on a model with tens of millions of elements. Takes 12-ish hours to solve a few load cases. A good convergence study might take weeks.
Thankfully, finite element solvers have gotten much better at tolerating bad element geometry, and at warning users for about geometry likely resulting in bad results. Plus things like NASTRAN's integrated extrapolation of stresses at element corners, instead of outputting only element stresses and having the analyst extrapolate. A colleague of mine ran some tests a few years ago at a customer's request, running a very coarse model with corner stress extrapolation enabled. Results were plenty good, well within normal engineering uncertainties and tolerances.
FEM was still pretty young in the early 90s and probably required very experienced/knowledgeable engineers to use it. As with most things, it's gotten more and more user frendly, and more and more software are taking very automated and behind the scenes approaches to FE (ANSYS Workbench, Solidworks Simulations, Simlab, etc). It has its good and bad sides.
How about a bulk simulation service for running HQ models in a couple different packages for a final check? I could imagine that being a regulatory requirement for big buildings, bridges, aircraft, etc.
That said, mistakes happen in construction all the time. With or without cad. So, things failing in the field is an 'accepted' facet of the process - in that context it's probably very difficult to collect precise data which parts are to CAD errors and which are of different sort - too much noise.
That said, any engineering office worth their salt will enforce a single software version for a duration of a construction process. They all know there will be horrible bugs, but then they have a process to take them into account with expert users doing special configurations and vendors suggesting all sorts of workarounds. It's still far more efficient to do it like that rather than with pen and paper.
CAD is totally unlike this modern apps and ecosystems thing in so many ways.
Sure, but a design error is still a design error. The capability to recover from design errors might be there but it does not mean unexpected flaws won't incur an overhead to costs and schedule.
The software failed to prevent the operator from entering invalid treatment programs which then killed patients.
I seem to remember that this was made worse by operators using a shortcut from a previous version which was safe because of some additional hardware, but the hardware lock was removed in the 25, and it killed people.
Clever comment! But wait, is that really the Therac-25 you're thinking of? The Therac-25 was basically just a bug-trap of assembly language race conditions, that's all.
There was another similar case of a radiation machine in which the dose was supposed to take into account the shield. The user could draw the shield with a CAD-like plannning program. The users tried to draw a shield with a cutout, and the CAD display led them to believe that the shapes were being subtracted. (The filled path was drawn with the in-out rule or whatever.) The actual dose calculation though ignored this and added the negative area as a positive. Whoa, you have lots of shielding, crank up the radiation! Or something like that.
This is not actually the radiation machine but the treatment "planning" software. (It meets the definition of CAD: it's computer software, and it's assisting in the design of something: the treatment.)
Gee what was this? Perhaps that circa 2000 incident in Panama?
http://www.ncbi.nlm.nih.gov/pubmed/17199912
This is a very informative report with all the details, including diagrams:
http://www-pub.iaea.org/mtcd/publications/pdf/pub1114_scr.pd...
(See P. 22 for instance)
Thankfully when I did work for that same organisation they had completely changed their culture, and things were considerably better, but still I suspect that if you explore the bush around the facility (which I doubt the police would allow...) you might see some unusual things.
Unless you mean simulation should've picked it up.
[1] The inductors were small spirals laid out on the top metalization layer. I assume finite element analysis of those involves a lot of magic.
In their case once they knew how far off the inductors were with the process being used they were able to adjust the target inductance and then do a re-layout (couple of 80 hours weeks, and another full set of masks, no problem). I think part of the problem was that for the previous geometry and design frequency 1GHZ they were using their tool gave good results. New geometry and 2.4GHZ and it was wrong. No way to test either.
I was going to say "No, ignoring your tools doesn't count as a bug" but now I'm curious if the Disney Hall one (like the more recent building in London http://www.bbc.com/news/magazine-23944679) was caused by only seeing approximation-based global illumination. If you don't get caustics rendered correctly, you don't realize you'll melt a car ;).
They had heard of this story, which is told often, but it's just not that simple. Problems like this should have been noticed and fixed much earlier with the proper processes.
Now blaming Catia for the organizational problems seems too easy.
It happens a lot actually with CNC processes, if you transfer one drawing between systems, but even then I wouldn't consider that a bug.
Has software ever contained a bug because of a bug in an editor?
The news stories focused on testing, but testing is kind of the emergency fail safe of product construction. The fix involved a redesign, not more testing, so it seems safe to say the initial design was a failure.
It is also possible that the requirements were off.
The strut that we believe failed was designed and material certified to handle 10,000 lbs of force, but failed at 2,000 lbs, a five-fold difference.
http://www.slate.com/blogs/the_eye/2014/04/17/the_citicorp_t...
Also, looks like classic project mismanagement made a vital contribution, too:
> The discrepancy between calculated and measured position, resulting in the discrepancy between desired and actual orbit insertion altitude, had been noticed earlier by at least two navigators, whose concerns were dismissed. A meeting of trajectory software engineers, trajectory software operators (navigators), propulsion engineers, and managers, was convened to consider the possibility of executing Trajectory Correction Maneuver-5, which was in the schedule. Attendees of the meeting recall an agreement to conduct TCM-5, but it was ultimately not done.
http://futureboy.us/frinkdocs/
I tend to think the web is more volatile and more oriented towards sales, ads and services, which in my mind are somehow "less important" than industrial software.
Of course every developer makes his own choice, but I'll always be more condescendent towards web techs in general, versus industrial, sometimes C-oriented programming. Maybe it comes from my technical teaching where I learned programming.