Linear algebra for programmers

Oh and for even more linear algebra stuff, here is a 30 min condensed video tutorial that introduced most of topics in a standard LA course, also using SymPy to show demonstrations: https://www.youtube.com/watch?v=2G3PmEZI6n8&list=PLGmu4KtWiH...

Agree. I love the Gilbert Strang series and am working through it myself. Also, the book "Linear Algebra Done Right" by Sheldon Axler[1] is amazing. He has on his website a set of short videos and slides to accompany it[2]. It's incredibly comprehensive and takes an interesting strategy of teaching all of linear algebra in a very rigorous fashion without introducing determinants until the end. His reasoning is explained in a paper he published called "Down with Determinants!"[3].

[1] https://linear.axler.net/

[2] https://linear.axler.net/LADRvideos.html

[3] https://www.axler.net/DwD.html

I never found Strang's lectures out of ordinary nor particularly enlightening. But what managed to give me perspective was Pavel Grinfeld's playlists, starting from https://www.youtube.com/watch?v=Fnfh8jNqBlg&list=PLlXfTHzgMR....

Much of the use of linear algebra in programming is for machine learning. To a first approximation, ML is statistics on huge datasets, and linear algebra makes it possible because matrix operations are massively, if not embarrassingly, parallel.

Fast ai had a computational linear algebra class iirc

Look into making a Doom-like game, tons of linear algebra there.

One place to start might be a tutorial on principal component analysis, which will take you through some of the intuitions for applying SVD.

You can also go in the direction of cryptography; here's, for instance, a really excellent LLL tutorial that builds on Graham-Schmidt: https://kel.bz/post/lll/

Imperial College has a pretty good course on Linear Algebra for Machine Learning on Coursera. It's part of their Mathematics for Machine Learning Specialization. Deep Learning also has a course that appears to be similar.

The specialization as a whole starts simple, and works its way up to PCA.

https://www.coursera.org/specializations/mathematics-for-mac...

I don't know a recommendation to make; probably it makes more sense to find a field your interested in (pretty much any STEM field will do), and learn the more math heavy version of that. Inevitably linear algebra will come up, giving some motivation for the pure theory.

As for why it comes up so much, it concerns itself with solving systems that look like y = Ax + b, where the Ax term works similarly to multiplication in 1-D. The point is these simple equations are ones we can actually understand! Everything else is too hard.

But there's a trick we have for everything else: if you have some y = f(x) where f is super complicated, you can differentiate. The derivative of f at a point x_0 is the best linear approximation to f. i.e. f'(x_0) is the best matrix A such that y ~= Ax + b near x_0. Now your problem is linear and you can understand it (locally)! Then you can integrate your local solutions into a global one.

The purpose of dot products is that they let you talk about things like angles, lengths, and projections. The point is you learn how it works for arrows and shadows and stuff, and figure out some equations that hopefully make intuitive sense in 2- and 3-D, and then it turns out those equations work in higher (even infinite) dimensions too.

Projections are useful because they let you break vectors down and build them back up, and hopefully the broken down version is easier to understand. Understanding projections in high or infinite dimensions gives some intuition for things like the Fourier transform, where you project a function onto simpler waves, maybe study how a system reacts to those waves, and then use that description to build back how up the system reacts to your original function.

Angles give one way to measure closeness. If you have some machine learning model that figures out a way to map text into a 50,000 dimensional space, you might be able to do it in a way where two sentences are intuitively similar if they are mostly pointing off in the same directions, so if the angle between them is small.

So tl;dr the idea is you learn some geometry with arrows and all that, you figure out some equations from that geometry, and then you realize that those equations and that geometric intuition work anytime you have a linear (i.e. f(ax+b) = af(x) + f(b)) system. Calculus gives you ways to turn non-linear problems into linear ones, so you will find examples of linear systems everywhere.

I found a lot of practical CS applications in this one:

"Coding The Matrix: Linear Algebra Through Computer Science Applications" https://codingthematrix.com/

It used to be on Coursera too, has video lectures and a book to go with it.

Boyd & Vandenberghe's new book: https://web.stanford.edu/~boyd/vmls

It has companion code in Julia and Python, and addresses many important applications, such as convex optimization where the authors' previous book is really famous.

I just posted this in another comment: https://archive.org/details/build-your-own-flight-sim-in-c-d...

It turns out dot products, matrix multiplication, etc. are fairly common in projecting a 3D world onto a 2D display (flight simulators, for example).

Find and work through some old OpenGL tutorials from the fixed-function era. It's long obsolete, but there's a nice immediate connection between your matrix algebra and what you see on the screen. Putting chunky triangles on the screen is fun!

Edit: I should mention that this is a good way to learn to appreciate what modern shaders are doing. You could jump into those as well, but I think it's a little harder to get your first triangle on the screen. All this assumes you're comfortable writing C, of course.

Two chapters of my book https://pimbook.org focus on linear algebra, with the first chapter being for SVD (a workhorse of data analysis) and the second demonstrating physics modeling using eigenvectors and eigenvalues.

But if you really want practical applications, you probably want to read a book on numerical analysis, which is where linear algebra really finds its glory.

You might want to watch this short video a couple of times - https://www.youtube.com/watch?v=rowWM-MijXU

A simple way to think about Linear Algebra is as a set of rules i.e. functions (aka Matrices) applied to input points (aka vectors) to give output points in a vector (i.e. multidimensional) space. Thus multi-valued inputs give multi-valued outputs. The key is to keep both the algebraic manipulations and what they may mean in the geometric interpretation simultaneously in mind. The basic example of solving a system of linear equations in two variables will cement this idea when you map it to the XY coordinate system.

You might find the book Practical Linear Algebra : A Geometry Toolbox by Farin and Hansford useful for further study.

In addition to what has already been recommended, I would recommend going to a used book site and typing in "applied linear algebra" or "linear algebra applications". You'll see a ton of books, many under $10 shipped. Usually old textbooks about 5-10 years old, but the math doesn't go out of date.

In my experience it largely depends on the field within Software Engineering, and where it blurs into subfields of Applied Math / Engineering.

Machine Learning needs some things (mainly "easy" things like matrix multiplication and convolution with the occasional truncated SVD), Modeling of Physical Systems needs other things (PDE solvers), Computer graphics different stuff (mainly focusing on small rotation/transformation matrices applied to many many points), Nonlinear Optimization yet another subset (solution of large sparse systems), and that's before you get to signal processing and statistics.

The main commonality is once you have the basic terminology down and understand that if you're explicitly inverting a matrix you're probably doing it wrong, you should just use the existing highly tuned libraries for your use area (at least until you decide it would be cool to try and beat them). Once you need to go beyond that you're more into the realms of matrix analysis and structure exploitation, and firmly have at least one foot in the math camp.

This is the advice for the ages!

However i fear it is lost on the current generation which is always looking for that quick/short article/blog/video/etc. which will teach and make them understand the subject as painlessly as possible. Everything must be "Fun and Enjoyable" (i don't even know what that means anymore!). They have forgotten the maxim "There is no Royal Road to Mathematics"(or any other scientific field). The amount of people (even on HN!) who espouse disdain for Textbooks is astonishing. By definition, the process of learning involves diving into the unknown and hence there will be discomfort/difficulties/effort/time needed and thus is not going to be easy. But the student doesn't want to put in any effort at all; everything is the fault of the Teacher(can't teach properly)/Teaching method(i am a visual learner)/Too rigorous/Mathematical/I have ADHD/ADD/Autism/Whatever.

The real tragedy is that used Books are now so easily/cheaply/universally available that there is no reason not to have one's own library of books on subjects of interest; it is "food" for the Mind.

I take it as analogous to the association of a matrix to a linear transformation. This association is via multiplication.

3 is the following function: 3 == lambda x: 3*x

But I think that the technical, mathematical way to think about it is:

The monoid of linear functions L:R->R is isomorphic to the monoid (R, *)

Meaning, the structure of 1x1 matrices under multiplication is exactly the same as the structure of real numbers under multiplication.

I think the mathematical concept that you are looking for is that of the dual space. Essentially if you have a vector space V, you can construct a dual space V* where the elements of the dual space are functions taking elements of V to the underlying field F, and under certain conditions these spaces are isomorphic (the same) - so there is a 1:1 correspondence between elements of the vector space and the functions in the dual space.

Check out Peano arithmetic.

Intuitively, natural numbers come from and are defined by counting, and that implies that "3" means inherently that something (could be anything) happened or was repeated three times. For example, if you have three apples, that means that you can identify one particular apple that you have, then do that again, then do that again.

Adding a unit to a number is like adding further information on what it is that is being repeated. Three pairs of apples? You just invented the number six!

The meaning of doing something three times (most abstractly: applying the successor function) is already inherent in the meaning of three, so multiplication isn't something that has to be added on top. It's already in there.

Hmm you're right, I didn't quite explain it well. The idea is that you can think of `3` as a function that makes thing three times bigger, and think of `*` as a function application operator. So `3*x` is equivalent to `three(x)`. I'll think on this more and change the wording to try and make this more clear.

IMO it is much clearer to justify something like matrix multiplication via a simple real life example like markov chain computations, it might add a bit of complexity in understanding the application but motivates the definition much better.

i think it's a rushed draft

it hasn't gone through a spell-checker, with words in there like hight and strage

True.

Moreover, here as in most other writings about linear algebra that I have seen, there is the very bad habit of describing the more complex operations as being composed from dot products.

On modern CPUs, dot products must be avoided, because like all reduction operations they consist of one chain of dependent operations, so their speed is limited by the latency of the fused multiply-add operations, instead of being limited by the much higher throughput of the FMA operations.

When vectors are multiplied with vectors, there is no alternative for dot products, so the only way to accelerate them is to reorder the operations into a tree, to be able to overlap a part of them, which are independent.

When arrays with more dimensions are multiplied, e.g. matrices with vectors or matrices with matrices, the multiplications correspond with nested for loops, i.e. 2 nested loops for matrix-vector multiplication and 3 nested loops for matrix-matrix multiplication.

The nested loops can be reordered arbitrarily. In each case, one of the possible loop orders has in the innermost loop the computation of a dot product.

This is the only order mentioned in the parent article and in most other linear algebra manuals. However, this order is exactly the worst possible computationally.

For 2 or more nested loops, there is always another order where the innermost operation is a so-called AXPY operation (the BLAS function name). AXPY means the scalar A multiplied by the vector X Plus the vector Y, with the result stored in Y. AXPY operations are always better than dot products, because the FMA operations are independent, so they can be pipelined, but for 3 or more nested loops there are better orders, where the innermost loop needs much less load and store operations than for AXPY.

For 3 or more nested loops, there is always an order where the innermost operation is a tensor product of 2 vectors. This is a more attractive operation than both AXPY and dot products. If the tensor product of 2 vectors with N elements is stored in registers, then its computation needs N+N loads from memory, but N*N FMA operations, so if there are enough registers so that N>2, there will be much more FMA than loads, allowing the full utilization of the execution units of a modern CPU.

In conclusion, matrix-vector products must not be described as being composed of N dot products that are executed separately for each element of the result vector, but as being the sum of N AXPY operations, which are accumulated into the result vector.

Similarly, a matrix-matrix product must not be described as being composed of N^2 dot products, one for each element of the result matrix, but as being the sum of N vector-vector tensor products, which are accumulated into the result matrix.

Such descriptions would be much more useful in practice, where the definitions based on dot products are just a hindrance.

Many devs who "zone out" because they won't "need math" still manage to do just fine?

Software is a big field and there's room for folks with different interests and talents.

That’s how I was in high school, and immediately regretted it the minute I found interest in a domain where strong math ability was required.

I don't understand why you take issue with that statement, it sounds descriptive as an attitude that may be common in the audience but perhaps you are interpreting it as the author endorsing that view? i.e. I imagine they might agree with you.

Or write a flight simulator from scratch.

I followed a book in the 90's to create a flight simulator from scratch. Besides learning Bresenham's line algorithm, I learned a lot of linear algebra.

Probably this book: https://archive.org/details/build-your-own-flight-sim-in-c-d...

Author here.

The point is to teach it to people who never understood its value but now want to learn, and to do that it helps to relate to their experience.

As a programmer, I like maths and find it very useful, albeit difficult at times.

However, the naming conventions and syntax overloads/ambiguities really can be a pain as a non-mathematician.

I get it. Everything has warts, and the mathematical syntax is as much for thinking and experimentation as it is for communication, and verbosity hinders understanding. I'm also aware computing has its horrible dark corners. But from the outside, it seems like maths practitioners often oppose any attempts to make it any clearer, even to people in other mathematical subfields.

Papers leave variables undefined because everyone working in that subfield is expected to just know what that variable means in that context, possibly derived from on some book that everyone in that field knows about so nobody feels the need to specify it. I mean just a stupid but simple example, I learnt maths from an applied engineering school so the imaginary unit was j. Always j. Never specified that it could be anything else nor was it ever specified anywhere that j was the imaginary unit. Then I start exploring the topic outside textbooks and everyone's using i. Again, no indication that this is the imaginary unit, it's just a given. Obviously in this particular case it's easy to notice that it's been swapped over, but there are plenty of other much more subtle ambiguities that can make trying to understand anything a real slog.

It's not a criticism over math. It's a criticism over high school math education.

> criticize everything from naming conventions

A programmer criticizing naming conventions. a.k.a another Tuesday.

I was nodding in agreement until "naming conventions". Naming conventions and common notation is really bad. To the point, that it's sometimes easier to invent your own for personal use.

The thing about math is that developers live in a well specified world. The moment they have to deal with mathematical notation, not only is there no way to look it up, it is inherently ambiguous and arbitrary and nobody tells you that. No math teacher on this planet is going to tell you "alpha beta gamma usually refers to angles, by the the way this is completely made up. In fact, it is the result of a popularity contest. If you see anyone else use the same symbols it is sheer coincidence as all mathematic notation is made up on the spot. The moment you read another book, they are free to do things as they see fit."

But here is the thing. As a software developer, there IS a "higher being" aka a designer for the programming language or library and they try and try and try their best to maintain "rhyme and reason". With math? There is no such thing, or rather, everyone pisses in the pool of math notation but nobody wants to admit that and so you get confused people, people who mistakenly look for the "rhyme and reason" and they find none, they find no pattern.

So the first lesson about math notation that you need to learn is that there is no such thing. There is this chicken scratch that makes it easier to write on the blackboard or on paper and that is about it.

Math isn't "hard..." it is abstract.

I did this awhile ago, but found myself simply copying and tweaking graphics algorithms, rather than gaining any intrinsic understanding of what linear algebra is really doing. I guess I just didn’t have enough computer graphics background? Yes, I can use lighting equations to define a pixel shader, but I’m basically copying and translating the algorithm from a book.

Deep learning! It's all "just" (more or less) high school calculus (partial derivatives, chain rule) and matrix multiplication.