Why There Is No Hitchhiker’s Guide to Mathematics for Programmers
jeremykun.com
jeremykun.com
1. "Mathematics is hard and inconsistent, but so is programming".
Well, yes, but this is exactly the problem! I pity poor souls who started their journey into programming with, say, C++ — the champion of consistency and elegancy. But there are other ways and other languages. And, with some effort, it _is_ possible to provide gentle introduction into programming world. The same should apply for mathematics.
2. "You can't learn mathematics on your own, but it's the same with programming".
And here I deeply disagree. Programming as activity is _the_ perfect learning environment. Find some coding sample, compile and run it (or paste to REPL), see if it gets you the same result. Try to change some variables in the code, see how behaviour changes. Can you run it without this function? What this method does? (A quick look to the documentation reveals everything, including additional arguments to try). You can tinker and hack and play and learn with every language and environment out there, it's like a perfect experimental setup where experiments are cheap, and everything is perfectly reproducible, at least on your own machine. ;)
With math, there are no docs, no playground, no explanations, no nothing. Either you are the part of mathematical community and can hope for explanations and knowledge transfer, or you are not. In programmers parlance, it's like being the part of a huge legacy project with long compile times and cryptic documentation, where the only way you can learn is ask your senior colleagues questions like "what this class is doing?" and get answers like "oh, we are not doing _that_ anymore, this is an artefact from like 15 years ago, still widely referred in the docs, though. Here is the right way instead". Sounds familiar?
This is why learning math is way harder than it should be.
what's the difference to CS here, where did you get that coding sample you mentioned? The playground is your mind, pebbles or pen and paper, maths are a projection of the real world.
Mathematics is not inconsistent. Isn't this the premisses of rigor? Math has mathematical consistency and that's not the same as literal consistency, but maybe that's a problem in semiotics and language. The language and the literature is inconsistent.
I learned programming by reading books and trying samples from there, then, step by step, beginning to understand how parts play together and why some things are faster than another. I could _feel_ loops and recursion, tinker with them, before trying to understand them theoretically.
I can't do that in learning math (yet; Wolfram Alpha is changing it). It's either I understand the exact meaning of symbols, or I don't. There are no intermediary steps.
And being rigorous is not the same with being consistent. (There isn't much consistency in programming, either, but this is a minor problem).
> With a program, you can always write test cases and run them to ensure they all pass.
> There is no compiler to tell you that it’s nonsensical to construct the set of all sets
> In mathematics there is no unified documentation, just a collective understanding, scattered references, and spoken folk lore. You’re lucky if a textbook has a table of notation in the appendix.
To implement something you need to have a level of understanding of it that is way deeper than what is required to get a passing grade in your math class. The problem isn't mathematical notation or the programming language (I am quite competent in and have no quarrel with either).
I recently set out to implement Triple Exponential Smoothing (aka Holt-Winters method), but I didn't want to use someone else's code, I wanted to understand the algorithm so that I knew enough to compare various implementations and/or write my own. It took _months_. There is a lot of material on the subject, but it all disagrees ever so slightly on one thing or another, or skips over something essential. I don't think anyone or anything is to blame for this - that's just life.
You can read about it here, btw: http://grisha.org/blog/2016/01/29/triple-exponential-smoothi...
Isn't this precisely what you said wasn't the problem? The resources out there all differ slightly on one thing or another. I suppose you mean small semantic differences? I think when I wrote this article originally I intended superficial semantic differences in the math to count as well as notational differences. Doing math for too long has made me think of these as the same thing, since the true underlying insight doesn't depend on either.
I think the point I was trying to make (not so well) is that perhaps it's not that programmers do not like math, but rather that when you learn to view everything through a programmer's eyes, i.e. "how am I going to actually build this damn thing", you tend to get very wary of abstract or generic approaches which is what scientific books are made of.
For example, if I were given a book with a chapter on Holt-Winters and told that there'd be a quiz later, I could have "mastered" it in a night probably. But when I had to write the code to actually do that, I'd find out that this book does not mention how you come up with initial values, while wikipedia does, wikipedia cites this NIST publication, but who wrote that and is it any good? Then I came across a 2014 paper that says that the forecast can be improved with a slight change in one of the formulas, only when I tried it, my results were worse; no book mentioned how to find best alpha, beta and gamma, because for that you need some kind of a gradient descent or nelder-mead method, and on and on. _That's_ what took months (and it was challenging and fun too), and I think we agree that there isn't a solution to this, it's a dilemma, not a problem. (As someone once said - avoid trying to solve dilemmas and avoid managing problems).
If I were given the postgres docs, I could "master" the syntax pretty quickly. But when I had to write code that actually uses postgres in production, I realized that the docs don't carefully explain 3NF, making 3NF fast and user comprehensible, good practices on writing complex queries, etc.
Is that really a math-specific problem?
The problem extends beyond mathematics, to essentially everything except programming. And in fairness out of all the fields with shitty names, mathematics has the best excuse: unlike the natural sciences and engineering, math is completely abstract. But programming is also pretty abstract, and yet we don't call stacks Hawk-elbower's double-u structure! No, we notice a semantic similarity between our abstract concept and something more familiar, and we come up with a monosyllabic, five-letter word that conveys a lot of information about the construct to someone completely uninitiated. This both makes it more approachable and easier to learn, serving as a mnemonic device. And once you've internalized it it makes no difference whatsoever.
I hear people whine that naming abstract concepts after concrete objects will only give people a harder time thinking abstractly. Bullshit, I say: have you ever heard of a novice programmer thinking that stacks are only good for representing stacks of paper or dictionaries for looking up definitions of human words.
I understand that naming things nicely in math is harder than in programming, and that in some more cutting-edge branches there might simply be no good analogies to draw, but I don't think it's impossible for more mundane topics. Math professors teach by drawing analogies and thinking of applications all the time. In any case we can do a lot better than "Kronecker's delta function".
To pretend that anything above the most fundamental abstractions doesn't involve domain-specific, obtuse, arbitrary lingo is just silly.
This is both for improved civility (so that actual discussions can take place), and so that people flesh out their counterarguments more.
I apologize to adrusi for my response.
A great many things in mathematics do have descriptive names, although it often helps to know a bit of Latin or Greek or to "get" the metaphor. That said, I don't think naming things in mathematics is particularly akin to naming things in programming. In programming, a good name can tell you enough about an object to be able to use it without understanding anything further about it, but in mathematics it's rare to be able to get much use from an object unless you assimilate its definition and the major consequences of its definition in the context you're working in.
There are few "user-facing APIs" in mathematics, and mostly for good reason.
Software development still has plenty of algorithms and data structures named after their inventors. Could you guess what a Duff's device is based on the name? How about a Merkle tree? (Although arguably you could call the Merkle tree a hash tree and get the point across)
That the original article is directly trying to convince people that programming and math really aren't that different, but you do have to learn the other's culture?
'awk '{print $1'/'$2}' | xargs curl | jq .'
None of the those programs things have any meaningful names. but they're all used in a common programming, or scripting, expression.
If someone says "Oh, you should use Hadoop for that", this is a proposal to solving a problem.
Print is semi intuitive, but it could mean so many things. Print in awk vs print in tcsh might have entirely different behavior. Is there a \n appended? How do you delimit it? It might give you a vague idea, but you're still going to have to go to your informational resource of choice in order to figure out: a) the parameters to pass to it, b) the output formatting/delimiters/implicit \r or \n's, etc. (e.g. Hell, in Python 2 you add a comma to prevent a new-line - I'm not even sure that syntax was influenced from (though logically I can understand the comma-suffix implies additional components and thus you want to suppress any line-break), I'd be hard-pressed to the etymology of that).
Brings me right back to the 'functor' issue. But hey, at least a ML functor is entirely different from C++. They overloaded the term (like delta or mu might be overloaded in math) but at least I don't have the risk of using it and getting 'partially consistent' behavior. (For example, having print exhibit the same properties as bash would plus a side-cases added by the author as 'feature enhancements' which work great for him, but if his side-project gains traction will just lead to more ambiguity.)
Oh! "curl" is "c url". It's taken me several years to understand that, and I knew exactly what it did! I imagined my data "curling" off the website, down the intertubes to my filesystem.
:)
There are two delta functions commonly used in mathematics, the Dirac Delta function, and Kronecker's delta. One is a discrete function, the other is continuous.
Kronecker, a mathematician who was obsessed with the finite, who tried to destroy Cantor's Set Theory, and to some extent, Cantor himself, is unsurprisingly the one associated with the discrete version.
Dirac, a physicist, used in in his book on Quantum Mechanics, and finds lots of use in Physics. Unsurprisingly, it's going to be the continuous version of the thing.
In mathematics, the history of ideas is caught up in the function of the ideas. Sometimes, to understand something, the easiest way is to look back at the problems that resulted in the development of the idea. In times before source control and git blame, it's actually quite helpful to attach the inventor's name to the concept. This is not, of course, the primary reason for the attachment (and many times, the history is more complicated and interesting), but it doesn't mean that it isn't useful.
You also have definitions named after common intuitions like openness and closeness in topology that end up confusing people because after some thought, mathematicians have found that some sets are open and closed at the same time, but that's not possible in most physical objects.
Ultimately, I think that the unfamiliar-name problem is just something that people have to get over with. We learn all the time the difference Coca-Cola and Pepsi, Apple and Google, and so many other unfamiliar names.
It's open sets which are badly named. Open sets are used for distinguishing between points - i.e., two points (x,y) are topologically distinct if you can find two disjoint open sets x \in X, y \in Y with X \cap Y == \emptyset. Coming up with a collection of sets which are good for this purpose is totally NOT the opposite of being closed. "Distinguishing sets" might have been a better name.
"Sum of this set from 1 to infinity"
They use a capital Sigma, and scatter the symbols around it. The terms exist, but they're not present in the equations!
Computer science has only had about 50 years to develop - we may have seen our Issac Newtons and Leibniz's but once we have 200 years of Gausses and Riemann's there'll be a lot of additional abstraction that's implicit depending on which subset of study you choose. CS will fork off into various subsets just like any other formal science, and a greek symbol to someone who finished their PhD in complex analysis might mean something entirely different to someone who's been working specifically in combinatorics for 30 years. At that point, our level of abstraction won't be pieces of paper on a stack.
It's not exclusive to CS or mathematics, ask someone who hasn't taken O-chem what "R" means in a Lewis Diagram. Ask a non-linguistic (of either type (ugh pun not intended)) what a 'terminal symbol' is. I'm sure they'll answer differently from someone who works at the airport, that's for sure.
The analogy isn't 100% precise, though: If structures with the same signature (and perhaps equational laws) form the objects of a category, then what exactly are the morphisms? There are several possibilities. My favorite one is pre-ordering structures by the asymptotic complexity of their operations. (Every preorder can be viewed as a thin category.)
I believe the usual story is that Kronecker called it a delta function, then someone other than Kronecker decided to call it a Kronecker delta. It would be ridiculously presumptuous otherwise!
Regular mathematical notation is like what you would get when you run a program through one of those obfuscators that blows away all of the comments and whitespace, reduces variables down to a single letter, and renames all of the functions to single letters.
But we can't change it because it would confuse all of the existing mathematicians who spent years memorizing all of the quirks of the system.
While some algorithms had a pre-existing implementation, wiser than rewriting I simply nabbed them and evaluated. Most of the implementations were unreadable/undocumented and of dubious origin, many of the implementations claiming to provide the same algorithm delivered different results. Some of these work for some countries, some for others. No idea where the differences lie and not enough interest to bother finding out.
Then of course there's the origin of these 'standards', ISO, who insist on charging 88CHF for a 5 page document, thus we can't actually read the damn standard. I would up porting a family of algorithms from Java using code generation.
Why we don't yet have a library delivering smackdown-documented functions in arbitrary languages for named algorithms escapes me...
However, the few times I've really wanted to delve into math and really understand the concepts underpinning that formula I'm using, I need a machine to iterate on. My computer isn't ideal because the software that's available is too expensive for 'hobby' use; there's no library that lets me use code in a language I know to manipulate symbols and equations. The TI series of calculators are not ideal because A) they're intentionally crippled (to prevent student cheating) and B) they have a bit of learning curve to them when you don't use them regularly.
Can I get a touch-device app for mathematical symbol/equation manipulation? That'd be ideal.
PS. Cheating on TI's is trivial. If you're a math professor, adopt a no graphing calculator rule. (You can still cheat on your standard secretary's 20$ calculator, especially in the age of low-power Cortex M0+'s, but it's somewhat less trivial. I'd imagine memorizing 5 or 6 rules on taking the derivative would be easier for someone capable of doing that. (Though if those tools had been around when I was in uni, I might have just done it for the sake of doing it.))
Not at all convinced it's a feature. More like an unfortunate side-effect of tradition.
It's easy to just stick with the traditional notation(s) of every subject, but there could be some cases where an attempt to refactor some notations might be of benefit.
I usually like to criticize the combined use of i and j. Their use in pseudocode for analyzing loops also just aids confusion. I've helped some people by telling them to go through examples and change one of the variables into something more readable, and then it clicks for them.
It has always been very clear for me, and I'm not sure what could be improved. I'd think any attempt to make it less ambiguous or more expressive would eventually devolve in something very similar to what we have now, because of our compulsion to simplify repeated tasks (eg Einstein notation, bra-ket etc..)
A few things that spring to mind:
dx/dy notation, and especially manipulating it like it's a real fraction.
Implicit multiplication, which forces use of single character variable names, which forces use of weird fonts and greek letters.
Writing sin^2(x) for (sin(x))^2
Writing sin^-1(x) for arcsin(x) (especially bad because it conflicts with sin^2(x))
The base for log(x) is ambiguous.
Writing |x| for abs(x) (abs() is obviously a function, it shouldn't have special notation)
Calling abs "modulus"
And the names "imaginary number" and "complex number" seem almost deliberately designed to intimidate outsiders. IMO they should be called "oscillating numbers" and "rotating numbers".
Both of those are because the absolute value of a number is the norm of the one-dimensional space.
https://en.wikipedia.org/wiki/Norm_%28mathematics%29
It allows you to think about |x| when x is a vector (or even other kinds of objects!) rather than just a real number. This kind of generalization can be a really powerful thing about mathematics, making existing insights more broadly applicable.
For all that I know, the only reason to write "|x|" instead of "norm(x)" or "length(x)" is because brevity and because it is a well accepted notation.
Edit: wrong post.
And like nearly every piece of mathematics notation, this one is overloaded: https://en.wikipedia.org/wiki/Determinant
are you serious?
In fact, the reason they're called "imaginary numbers" makes perfect sense if you were a mathematician 500 years ago - one method of solving cubic equation manipulated the square root of negative numbers in intermediate steps, which of course doesn't make sense, but by the final step these square roots canceled out and the final answer was correct. So they named them "imaginary numbers" to kind of tell people "don't worry whether this makes sense, it's only used in an intermediate step, they're not real numbers". So, naming a concept after its most common use is exactly what got us into this situation!
In addition to the modelling-rotation and solving-cubics aspect of complex numbers, last year I used them in a factoring-things context in a number theory class, for example to prove https://en.wikipedia.org/wiki/Proofs_of_Fermat%27s_theorem_o... (read the "Dedekind's Proof" section).
I can honestly say though every single one of those ambiguities I had an issue with in the past. This post reignited latent of ire of mine which I haven't experienced since 16. Denotational semantics are important!
See also Wikipedia on that subject (https://en.wikipedia.org/wiki/Logarithm):
>The logarithm to base 10 (that is b = 10) is called the common logarithm and has many applications in science and engineering. The natural logarithm has the number e (≈ 2.718) as its base; its use is widespread in mathematics and physics, because of its simpler derivative. The binary logarithm uses base 2 (that is b = 2) and is commonly used in computer science.
In higher mathematics, f^-1 does not mean the inverse of a function, unless that function is injective. Most people are aware of that at that point, but it's still a mental hurdle you have to unlearn and can lead to dangerous composition errors. (Edit: since I can't respond to you, poster below me, I mean the latter of course)
Historically, in yore days of sextants, slide rules and log tables, engineers could "lift" multiplication into addition with said log tables which were of convention base 10. For anyone who wasn't a peer in the Royal Society or a professor, that's how complicated math calculations were performed. Banks balanced books and calculated interest payments via tables like this, and the precision of your log tables had a mantissa (the original floating point error, heh).
Do you mean that it has a different meaning for a non-injective function, or that it's undefined for a non-injective function because the non-injective function does not have an inverse?
To the experienced eye, all these usages overlap in subtle ways, so that it all feels reasonable and even justified. But to newcomers, I think the notation just feels like a complete mess.
This has always bugged me. Andrew Ng's notation for machine learning sometimes uses superscripts as indices, and sometimes as exponents. In the same formula.
Years ago, when I was working on a physics engine for animation, I was struggling through a 2-volume treatise on nonlinear differential equations. One formula in Volume 2 had a symbol I didn't understand, and couldn't find in the text. I eventually found it defined in the first volume, about 500 pages back.
Steven Wolfram was annoyed by that, and he created Mathematica partly to define an unambiguous notation for much of mathematics.
This is a fixable problem. By now we should have programs which read mathematical papers and textbooks, and check that all symbols are defined and unambiguous. There would be style sheets for different branches of mathematics. When reading a paper processed through such a system, you would be able to see the expanded, unambiguous form of a formula, and the definitions of all the standard operators and symbols. This might not be possible for truly cutting-edge mathematics where the conventions haven't settled yet, but that's a small fraction of what's published.
I've never read Ng's work, but typically upper and lower indices is a sign of a summation. It's known as "Einstein Summation Notation". At first glance it seems obtuse, but when you start doing covariant/contravariant derivatives on tensors, the notation makes things much easier.
http://mathworld.wolfram.com/EinsteinSummation.html
Anyway that doesn't invalidate the point that mathematics often has highly ambiguous and hand-wavy notation, whereas programming doesn't.
Maths notation has very many problems, most of which stem from the fact that its syntax isn't formally specified. I have no idea why would mathematicians want to continue using this informally specified, ambiguous notation when there are alternatives.
I'm just happy it's not true for Computer Science and programming. Whatever you want to say about a social side of programming, there's no denying it that outsiders and beginners are warmly welcomed in the community. We create and provide for free tutorials and guides for beginners, we design languages to be as easy for people to understand as possible (we don't exactly know what makes languages easy to understand, but we at least try!), we create tools for visualisation and summarizing code, and so on.
Similar activities seem to be unpopular with mathematicians. I can't help but feel mathematicians are just a bunch of elitist pricks, who know they are on their way out and who try to build artificial barriers for entry into their field to make themselves needed for a bit longer. It's actually natural, most people will fight to retain their status (and jobs). It reminds me of "refucktoring", writing convoluted, unreadable code just to make sure nobody, else than you, can work with it. Such programmers are shunned in our community, yet they are heroes in mathematics. Oh well, as an "outsider" it's not my problem.
There is no saving maths notation now. It will die out and it will be replaced with a machine-friendly notation because benefits are too huge to ignore in the long run. Meanwhile, mathematicians will throw tantrums in defence of their notation, like all single-language users did throughout the history when their language was being replaced with something else. I wonder, how do math polyglots - the ones who know both normal maths notation and alternatives (like Mathematica and programming languages) - feel about this.
IME verbose notation makes it easier to "type check" an equation or expression, but checking a proof is much more involved than that.
And while you're obviously right that syntactic issues constitute just a minor part of work (at least after you're fluent in a language), the time you spend on them does add up in the long run. Not to mention such issues increase the barrier to entry for the new users unnecessarily...
Yeah, mathematicians are often ambiguous out of laziness, because they have no incentive to make it terse+unambiguous, and making it terse+unambiguous takes more effort than making it terse.
Also sometimes you cannot get maximal terseness and unambiguity at the same time. For example if f is a function of 2 real variables, you fix the first to be x_0 and take the derivative with respect to the second, I would write this as f'(x_0, y), but I'm sure you agree that this is ambiguous to some people.
But mathematics notation needs to be extremely succinct because the notation is not just for reading. When you're working something out you often need to write pages and pages of formulae and diagrams by hand. So using a notation more verbose than absolutely necessary would be unnecessarily painful.
Programming is different because you can use a text editor with copy, paste, autocomplete etc. But I imagine that if I wasn't allowed to use any of those then my variable names would shrink down to one letter when I was programming too.
Also, I think that making formulas smaller makes it easier for the eye to see the whole formula together. This makes it easier to quickly understand the meaning of the formula, at least once you've got used to the notation for that particular area of mathematics.
It seems like coming up with a format that at least allows definitions and usages to be hyperlinked might be pretty nice, instead of a format whose primary benefit is looking pretty when you print it on paper.
So, from the outside, it seems like there are big problems with communication in the mathematics community, and it's simply accepted that it has to be that way because math is hard.
Yes, a lot of mathematical papers are incomprehensible to others in the field. But it's not just because of notation. You wouldn't ask a content marketer who is great at A/B testing to deal with tweaking something in the linux kernel because it's all "computers".
Poor mathematical writing is pretty rampant, but that's partly due to the fact that it's not taught well and there's definitely some cultural biases towards jargon. I don't think that negates the fact that even within a typewritten document a human needs to read and understand what's going on, so often terseness is desired, as long as the reader understands the notation enabling it.
Not only this, but the notation itself is often intentionally used as a particular abstraction to aid intuition, see bsilvereagle's example of Einstein notation below as a great example. This makes things easier to reason about with our human brains even when dealing with very complex ideas.
The abstraction is usually leaky, however, and some notation that aids intuition in one way often makes some other interpretation horribly verbose and unintuitive. Hence the need for more than one notation for the same situation.
All of which isn't to excuse the many historical accidents that haven't been winnowed out by our predecessors. That's often a cultural problem, however, and while some of it can be realistically fixed by teaching the same way we teach good grammar, some of it is as hopeless to rail against as asking to fix every irregular conjugation in the english language. Good luck with that, honestly.
But we don't do this. We use abstractions. Why doesn't the mathematician? (And you'll say of course they do). In which case we go back to the original question of why dense notation?
When I first started in my adult adventure in math (never doing well with it academically, I started on pre-calc with Khan Academy about a year and a half ago; I'm now working through multivariable calculus on MIT OCW), I used scratch mode in emacs because hey, I'm right there on the computer.
I gave up and started using graph paper as my scratch pretty quickly :)
I can't imagine trying to actually _do_ math with something "descriptive" like LaTeX.
Math, from what I've come to realize, is less about describing how to compute something and more about describing something in such a way that it can be manipulated in order to reveal something new.
And that's why the obtuse iconography is popular in math, and why the obtuse iconography of APL never became truly mainstream in programming.
But I don't think Unix commands are a particularly great example for your side. They are nice for interactive work but terrible for maintenance. Nobody wants to read a thousand-line shell script.
http://www.amazon.co.uk/Mathematical-Handbook-Scientists-Eng...
With a program, you can always write test cases and run them to ensure they all pass. If your tests are solid and plentiful, the computer will catch your mistakes and you can go fix them.
There is no corresponding “proof checker” for mathematics. There is no compiler to tell you that it’s nonsensical to construct the set of all sets, or that it’s a type error to quotient a set by something that’s not an equivalence relation.
I joined college after a gap of an year, and as I faced Physics and Mathematics again, that feeling strengthened. I now sometimes think of a very strict compiler for mathematics that catches errors from the math lingua. Could possibly be extended to Physics too. Don't even know if that is feasibly possible, but I find the idea intriguing.
Is that what Coq (https://en.wikipedia.org/wiki/Coq) is?
I agree that we need a proof 'compiler' though.
What do you think are the specific ways in which a proof compiler would be different?
[I'm joking of course, but only half-joking. There's a serious point to be made, which is that mathematics exists completely independently of notation. When mathematicians don't care about notation, it's because they care about the actual mathematics instead. The reason mathematics notation is non-standardised is because standardising it wouldn't really help mathematicians that much. Unfortunately this means that non-mathematicians who use mathematics suffer.]
This or a variation is on almost every physicist's shelf I know...
Unfortunately, that is a significant part of the problem.
Numerical Recipes was an OK book for its time, and certainly a popular one given the limited material available in the field. However, neither the recipes themselves nor the book's general presentation are ideal today, and in many cases, someone interested in implementing robust, efficient algorithms for various mathematical constructs would do better now to read other sources.
For example, substantial linear algebra computations are probably going to use some variation of BLAS/LAPACK today. There is also a lot of background material available about the algorithms used within these libraries and the underlying mathematical foundations, for those who need to implement something a little different or who are simply curious.
In most fields, and excluding those who are actually writing this kind of mathematical library, a programmer will do better to use the tools that are already freely available today rather than trying to implement their own code based on ideas from Numerical Recipes.
NR is hugely helpful. One example a few years ago I needed to implement technique from a math paper in some C code.
The paper included such classic math terms like "Tridiagonal Matrix" and "Cholesky Decomposition" I'm sure they mean a lot to someone with a math background but baffled me at the time.
It wasn't until I pulled out my handy copy of NR I was able to even slightly wrap my head around just what the hell a "Cholesky Decomposition" even was. Let alone how I would code it. And even then I had to dredge up my old Linear Algebra textbook from Uni to re familiarise myself with basic Matrix operations.
The code I wrote ended up being full of comments like this:
"/* Newton's method to compute positive root f(p)^2 = (u^T)(Q^T)(D^2)Qu and F(dF/dp) = (u^T)(Q^T)(D^2)Q(du/dp) */ "
Which is basically me trying to wrap my head around just what the hell the mathematical paper is talking about whilst writing the algorithm programatically.
Even now opening the .C file some years later it took me a few minutes to make sense of that comment I had to realise that U^T was shorthand for transpose of Matrix U.
Numerical techniques are hugely complicated especially when you don't deal with them every day I have yet to find a book better than NR at breaking them down and presenting them in a way someone like me (with Science/Eng background) can understand them.
I'll gladly take any recommendations you have for updated references. As far as I know NR and the GSL (GNU scientific library) documentation are considered the gold standard by all my peers.
BLAS and LAPACK themselves are constantly evolving to add new or better production-quality algorithms as the field develops, and I highly recommend using the functionality they provide instead of trying to reimplement any of the basic algorithms in-house. There are heavily optimised versions of these libraries available for almost every platform you can imagine, their own documentation is pretty good, and often recent research in the field also gets written up as background papers and incorporated fairly quickly (the magic words to search for are "LAPACK working note").
If memory serves, the GSL actually depends on having a BLAS implementation available for some of its functionality, so if you're not limited by the licensing you might already be using the same sort of code under the hood anyway. :-)
Implementing numeric algorithms yourself after just having read NR is a bit like implementing crypto algorithms yourself after just having read Applied Cryptography.
Recently, after watching R.W. Hamming lectures and reading his book "Art of Doing Science and Engineering" [1] [2] decided that I would try learning mathematics again.
I found it really hard to do. There are so many implicit assumptions, so much ground that is not covered. I was about to give up again, when I found a book [3] by retired computer science professor, who seemed to have written it specifically for me! it really felt that way.
Now I recommend it to almost everyone, especially tech-savvy people. I am no longer afraid of mathematics, with the help of this book, and great advice by Hamming, I am finally "getting it". If you find learning new programming languages easy, but struggle with mathematics, check it out!
The book is not aimed at teaching you mathematics, rather, it teaches you those missing pieces, that almost every other book either assumes you already know, or describes vaguely and badly, gives you context and valuable advice. To me it almost feels like it is teaching you a functional programming language. A strange and very flexible programming language, with its warts and all. You learn the language, and then you can go on to explore programs written in it, algorithms, data structures whatnot.
My ultimate goal is to, at least, learn enough so that I can really understand books by Hamming, especially "Methods of Mathematics Applied to Calculus, Probability, and Statistics" [4]. And, of course, to do interesting projects along the way, now that I am starting to understand some of the research papers.
[1] http://worrydream.com/refs/Hamming-TheArtOfDoingScienceAndEn...
[2] https://www.youtube.com/watch?v=AD4b-52jtos&list=PL2FF649D0C...
[3] http://www.amazon.com/The-Language-Mathematics-Utilizing-Pra...
[4] http://www.amazon.com/Methods-Mathematics-Calculus-Probabili...
I can relate to the funny part on notation. I've been fortunate enough to have an older sister who's Algebraist and I remember when I was in primary school and I'd have a question, she'd explain on paper using different symbols than the ones I was using just to make sure I wasn't a slave of notation.
She insisted I had to think in an "abstract" way (that's precisely the word she used) and not be tied by what the letters are called, and only look at the relations between them and context. This one piece of advice served me well.
It's also funny the author mentions Fermat's Last Theorem because the first time I had heard of it, it was still called "Conjecture de Fermat" and I had read a piece on "Science & Vie" on Andrew Wiles' proof and the fact there still was work to be done to see whether it was correct or not. I was 7 at the time and it seemed strange and amazing how something that can be stated in terms deceivingly simple even I could think I understood, can be so difficult to prove.
I've studied Electronics Engineering in college and we had really cool Maths courses (again, in the spirit of different notation, I noticed a peculiarity: in the U.S., the term "Calculus" is used, whilst we used "Analysis", "Introduction to Mathematical Analysis", "Differential Analysis", etc. Perhaps the influence of French and Soviet Mathematicians who taught at my University. The books are also either Smirnov, Demidovich, Pontryagin, Piskunov, or Dieudonné, so the style taught is kind of different).
I digress. For me, it doesn't matter. I love Mathematics. Not the way many people say they love it. It is something on my top list of things to do and I am self studying currently. Why? First, emotional reason: in retrospect, the happiest years of college where in the "Common Core" (two years of core knowledge (mainly Physics and Maths) every Engineering student goes through before choosing a specialty (except Computer Science students, they only do freshman and then CS). Maths (and Physics) bring me happiness in a way that is hard to describe.
The second reason is practical: when my Maths skills were somewhat sharp, I had no problem understanding other things, because I had the necessary tools to strip down the things and go at it where it was called for (for example, modulation with a transistor and using Taylor series for the current). I then turned utterly incompetent and let everything rust and I'm paying the price right now. I have graduated but the blade is dull and I can't live with myself like that. I've come a long way, especially because I was clueless in time management and focusing my efforts and getting things done and I had to learn how to mitigate my proclivity to be all over the place. The one sure thing is that I'll never let go. I may not become a great mathematician, but every day I am less incompetent than the day before it. The point is, as in everything else: people don't learn because they lack the drive to do so, face the reality of sucking, learning the basics and fighting that embarrassing feeling of the inner voice that says "shouldn't you already know that stuff?".
It was after those core subjects where I fell in love - Thermodynamics and Fluid Dynamics was the 'aha' moment for me. It made all that core stuff make sense when I finally got a sense of the applications. Bernoulli, Euler, Fermi, Dirac - The shoulders of giants...
For applying the stuff, it's sort of easy. But I know that I'll never appreciate the beauty of it until I understand the concept of stability from a "Calculus" stand point and how it relates to systems.
Anyhow, I'm also acquiring Russian. Many Russian books have been translated but a lot have not.
"What do you get if you multiply six by nine?"