High Performance Computing: Are We Just Getting Wrong Answers Faster? (1998) [pdf]
www3.nd.edu
www3.nd.edu
I once sat on the external departmental review board for my alma mater, along with Mike Griffin (former administrator of NASA). We were interviewing recent graduates, and I brought this point up and got the interviewee to admit that he did not understand and could not double check the results of his FEM analysis tool. I was trying to get the board to rcommend beefing up the numerical methods portion of the curricula. Dr. Griffin, on hearing the interviewee admit this, agreed to add the recommendation. But nothing changed in the curriculum.
How exactly do you propose we get those experts, if colleges aren't capable of training them?
In the majority of cases, colleges (undergrad) aren't in fact capable of training them. The material is too specialized for undergrad. However colleges can prepare those who are talented and interested in the subject matter for further studies.
Many experts in numerical analysis start specializing in graduate school, and then acquire their expertise through postdocs, academia and industry experience.
https://www.siam.org/Students-Education/Resources/For-Underg...
First, I'm not proposing that engineers need to write their own FEM tools. I am proposing that engineers understand how FEM works, so that they can understand where it can fail. I would not expect an IT security expert to write his own ssh, but I would expect him to understand the ways in which ssh can fail.
But second, ssh is not like FEM. With ssh, once you have a solid implementation, it sort of doesn't matter what data you put through it. But the issue with FEM analysis is that every FEM model is different, and the generation of those models is almost always algorithmic, not manual. An engineer should have either a freat intuitive feel for how well a FEM model will match reality, or straightforward mathematical ways to double-check the output of an FEM analysis.
I took a number of numerical analysis courses as a graduate student, and we had an entire course on how stuff could go wrong in numerics, e.g. catastrophic cancellation, ill-conditioning, nonconvexity, approximation errors due to discretization, etc. I don't think I could have appreciated all of this as an undergrad because I didn't see the practicality of this stuff until I had to solve large-scale numerical optimization problems as a grad student. Then it all started to come together. Also there was more time in grad school to step back and think more deeply about problems, as opposed to undergrad where you're just rushing to get as many courses done as possible.
Furthermore, numerical computation is as much an art as a science. The only way to learn is through the repeated grind of trying and failing, and there's no way to fit that kind of practice into an undergrad curriculum. Most people learn this stuff on the job or in grad school. The role of an undergrad numerical methods course is usually only as an introduction to the landscape.
In engineering, V&V requires an intuitive understanding of the physical system at hand, which requires experience, which in turn is usually acquired on the job. This is another reason why it is not taught in undergrad. These types of intuitions are best acquired in industry.
IMO this is a deficiency in how programming is treated in engineering courses. It's there, but people are not taught to apply it to everything else, in the way they are with the math courses.
In my degree we did finite element methods, PDEs, and a load of other things, but only in a presentation style. So you'd go into some depth about how the equations worked, but the examples were always little atoms of a real model. So there'd be some diagram of how one element meshes with another and you could do some hand calculations to see what happens.
I think the issue is that you have to be a pretty good programmer to understand both the new FE/PDE/etc material you've been shown as well as be able to abstract it to such a level that you can write sensible code for it. If you're not quite proficient with abstraction, a specific language, as well as the coding toolchain, everything takes a long time and you'll be thinking about how the debugger works rather than how the model works.
I mean not even editing anything, just opened the file to look at it and then allowed saving the file on close.
But yea, your typcial bio analysis is a lot of clicking around in one peice of software, importing that output into the next software, etc and is totally unreproducible. Then in the end you get an average and standard deviation (at least its not usually only p-values anymore) but no one ever checked what the underlying data looked like.
"Is your standard deviation for Group X so large because of low outliers, high outliers or is it a multimodal distribution?" "No idea. Whats a distribution?"
I'd assume any accredited Canadian university would have the same
- Modeling and discretization errors (usually unavoidable but well understood).
- Uncertainties from the inputs (we have tools to get a feeling for their impact on the result).
- Floating point arithmetic (scientist tend to be unaware of its impact and/or consider that it is negligible next to the other sources of error)
My PhD covers methods to quantify the impact of floating-point arithmetic and suffice to say that you probably grossly overestimate the number of significant digits of your simulation. On the subject I recommend "What Every Computer Scientist Should Know About Floating-Point Arithmetic" : https://www.itu.dk/~sestoft/bachelor/IEEE754_article.pdf
In my training, it was basically assumed that floating point errors are negligible these days, so I'll check out your recommendation. If you've published any of your own more recent work, I'd be happy to see that as well.
Yeah, that's what most of us think initially. The problem is that the effects of error can compound in many ways -- nonlinearities, feedback, etc. can all wreck your accuracy enough to make the results meaningless.
A really instructive counterexample for me was solving quadratics. Easy, right? Just try writing a solver for ax^2 + bx + c = 0. Now try breaking it. If all you did was use the quadratic formula, you should find that it's quite easy to break it, and surprisingly difficult to write a robust solver. (If you're completely stuck, consider the most obvious edge case: what if a ≈ 0? Any other scenarios you can think of?)
In the meantime, if you want a way to explore this subject on your own applications, I would recommend Verrou[0]. It has its shortcomings but it is probably the current easiest way to evaluate the impact of floating-point arithmetic on your code (I did not work on it but exchanged a lot with their team).
To elaborate on a point not discussed in the paper: Many methods used in practical engineering won't necessarily converge to the empirically correct answer as the computational cost increases, say, by decreasing the time step, increasing the number of computational particles, and/or refining a grid. This is true even if the uncertainty is quantified. Without this guarantee, additional computational expenditure could simply be wasted. The problem isn't the lack of computing power, it's the model.
One reason is that "reliable" physics like, for example, the Navier-Stokes equations for fluid flows, are not being used as the model because the computational cost for solving those equations directly is far too high. Instead lower cost models are being used which may be loosely based on the reliable model, but ultimately is not the same thing. A numerical implementation of the lower cost model might converge to something (which is not necessarily right) as the computational cost increases, but converging to the right answer is less likely than the reliable model. It depends on the reliability and generalizability of the lower cost model and how well it has been tuned.
(To those familiar with computational fluid dynamics, convergence is one reason to prefer LES to RANS, though it's worth pointing out that not all LES methods are convergent.)
A hypothesis I have is that computing resources have caused many scientists and engineers' theoretical abilities to atrophy. Perhaps they were never that good on average to begin with, but in my academic field I work in theory partly because I've found so much low hanging fruit. I don't see much of a point in making expensive computational models if applying basic regression techniques with a simple theory-based model is similarly or even more accurate. These simple models are much more efficient.
Also: I wasn't aware of attempts to bound the results. I attended an uncertainty quantification seminar before and asked one of the lecturers about this more generally and they seemed skeptical of its utility.
> At this point a special technique has been developed in mathematics. This technique, when applied to the real world, is sometimes useful, but can sometimes also lead to self-deception. This technique is called modelling. When constructing a model, the following idealisation is made: certain facts which are only known with a certain degree of probability or with a certain degree of accuracy, are considered to be "absolutely" correct and are accepted as "axioms". The sense of this "absoluteness" lies precisely in the fact that we allow ourselves to use these "facts" according to the rules of formal logic, in the process declaring as "theorems" all that we can derive from them.
> It is obvious that in any real-life activity it is impossible to wholly rely on such deductions. The reason is at least that the parameters of the studied phenomena are never known absolutely exactly and a small change in parameters (for example, the initial conditions of a process) can totally change the result. Say, for this reason a reliable long-term weather forecast is impossible and will remain impossible, no matter how much we develop computers and devices which record initial conditions.
(emphasis mine)
A small change in initial conditions can totally change the result in chaotic systems such as weather forecast, or a much simpler example — double pendulum.
There are systems where a small change changes the results only slightly. E.g. a single pendulum.
"In 1970, a printer might produce something that looks like this, and take about 30 seconds to do so: [picture] Over forty years later, a laser printer still might take 30 seconds to put out a single page, but technology has improved to allow full-color, high-resolution output."
The book makes a compelling argument that working with floating point numbers is similar to printing thousands of low-quality pages per second, instead of getting a single high-quality page in 30 seconds. A few paragraphs later, the book states:
"Eventually it becomes so obvious that change is necessary that a wrenching shift finally takes place. Such a shift is overdue for numerical computing, where we are stuck with antediluvian tools. Once we make the shift, it will become possible to solve problems that have baffled us for decades. The evidence supporting that claim is here in this book, which contains quite a few results that appear to be new."
For more information, see:
https://www.crcpress.com/The-End-of-Error-Unum-Computing/Gus...
and John Gustafson's home page:
The original proposal that Gustafson had involved a variable-width format, which is going to absolutely kill performance in practice (particularly on the GPU-based supercomputers that are all the rage nowadays). The posit format was essentially a recognition that variable-width doesn't work.
The more serious error is that, well, interval arithmetic turns out not to be useful in practice. Essentially, the potential accumulation of error is exponential in the number of steps in the worst case, so interval arithmetic in practice too often gives you the answer "the result is 1, but it could have been anywhere between -inf and +inf", which is completely useless. He may complain that he doesn't want to use a numerical analyst to get work done, but the rebuttal is that numerical analysis is still necessary anyways, so promising that one isn't necessary is a false promise.
With ordinary floating point, all too often people don't realize their results are nonsense. Adding measurement and computation error bounds to their graphs is illuminating.
Unless you use C, C++, or Fortran. Which accounts for most numerical algorithms code. Yes, most people aren't aware of them, but not being aware of all the functions in the standard library isn't the same as them not being present there.
I think it is interesting to compare this old panic with ML work. Without gazillions of flops the neural network approach was dead. After a few years it became clear that float/double precision was useless, and now for certain problems half precision is standard.
Where one resource (people, CPU, ram, money) is the fixed constraint, other factors will be tweaked to compensate.