The Art of Linear Algebra [pdf] | Hacker News Reader

The Art of Linear Algebra [pdf]

raw.githubusercontent.com

358 pointsallending42 comments

Wow this is really well done. It's like a visual-algebraic approach... I've never seen this before.

I like how the author sets up a "grammar of matrix multiplications," and then reuses the same patterns in the rest of the document.

For people who might not be familiar, these visual notes are inspired by and complement Prof. Strang's new book https://math.mit.edu/~gs/everyone/ and course https://ocw.mit.edu/resources/res-18-010-a-2020-vision-of-li... https://www.youtube.com/playlist?list=PLUl4u3cNGP61iQEFiWLE2... see also https://news.ycombinator.com/item?id=23157827

Your books are pretty amazing too Ivan. Have em all.

Ivan, thanks for sharing the background concepts Prof. Strang has told us.

Yes, I am a big fan of him, and I came up with this idea during several email exchanges with him.

It seems like the "github.com" link would be a better and more canonical url rather than link an opaque url to "githubusercontent.com"

https://github.com/kenjihiranabe/The-Art-of-Linear-Algebra

And, "Graphic Notes on Gilbert Strang's Linear Algebra for Everyone" would be less ambiguous than "The Art of Linear Algebra."

Also check out 3Blue1Brown's Essence of Linear Algebra

https://youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFit...

And here is linear algebra as path counting [1]. This is closely related to path integrals in quantum physics. The rules for combining quantum amplitudes are the rules for combining path counts [2]. This is also how graphical linear algebra works [3]. And since we are all hackers here, if you replace the underlying number system with the min-sum semiring you get things like Dijkstra's algorithm. I really like how all these ideas are related.

[1] https://www.youtube.com/watch?v=ei6RfbplYZM

[2] https://www.feynmanlectures.caltech.edu/III_03.html

[3] https://graphicallinearalgebra.net/

I watched the path counting one and I want my time refunded please. It had nothing to do with linear algebra. Nothing to do with quantum physics either, except that they used bracket notation for some inexplicable reason.

What am I missing?

Thanks for the feedback. Do you want your time refunded, or do you want answers to your questions? The two seem to be in contradiction. If it's the former then what is your hourly rate for watching well-intentioned youtube videos made by experts in the field ?

Here's part 1 of Pavel Grinfeld's linear algebra series: https://www.youtube.com/playlist?list=PLlXfTHzgMRUKXD88IdzS1....

I never understood why we don't do more of this stuff in school, and how calculus instead became the defacto advanced math curriculum in most high-schools. Students grow up working on their basic algebraic operations, solving equations, etc. Liner Algebra introduces them to the universe that lies just beyond those techniques, has very readily applicable uses, lends itself excellently to simulation/connections to computer science (which is super popular to teach now), etc.

I suspect the answer is partly historical; without a computer, calculations in linear algebra are a pain.

Both calculus and linear algebra are fundamental for further studies, but I'm not sure linear algebra is more approachable?

I personally find geometry and algebra more interesting, but it seems to me that derivatives are more fundamental than matrices. But maybe that's just my bias from being educated like that.

What if I told you the derivative is a linear operator? (matrix meme style)

Of course, but there are plenty of linear operators, that property alone doesn't define it.

I was responding to derivatives being more fundamental than matrices. My rebuttal is that the derivative is but one in a sea of interesting linear operators.

This begs the question: more fundamental to what?

At a high school level, you have students who are either engaged and will likely learn both subjects eventually, or students who aren’t as engaged and are just looking to take their “last math class”. In my opinion, it makes more sense to offer a choice or at least focus on the curriculum that keeps students are that age to most engaged.

As someone who has studied both subjects, I’d go with linear algebra 9 times out of 10, the exception being if someone wanted to also study physics.

The question seems to be: should integration and differentiation be taught before matrix operations, or after?

IMHO, since linear algebra is largely a tool for solving differential equations, I think calculus should be taught first, as the fundamental knowledge.

I’m not really sure where you got the idea that it’s largely a tool for solving differential equations. Certainly that is an application, but that isn’t the only use case.

That's how it was introduced to me. Which is probably part of the discussion about what should be taught first.

I was introduced to the topic on its own, so that colors my perspective as well I suppose.

If I were to think about what a high school student would get the most value out of, it would realistically involve a combination of linear algebra and statistics (getting into basic linear modeling/ols). You’d have to hand wave away some of the proofs which require knowledge of calculus, but high school classes aren’t very rigorous anyways.

Linear algebra is incredibly useful, basically for anything you might want to do, differential equations is pretty far down the list of applications I would think of.

But calculus is also very very useful, and probably easier to understand, derivatives and integral are quite intuitive concepts compared to eigen vectors...

> [...] integration and differentiation be taught before matrix operations [...]

When worded this way, it sure doesn't sound like something worth doing. Matrix algebra (computational rules for matrix-vector and matrix-matrix products) is just an "implementation detail" of the general idea of a linear transformation.

The notion of a linear transformation T(x) = y where x is an input vector and y is an output vector is a really good thing to know ASAP so I'm all for the LA before CALC... or rather, if I had to choose between one XOR the other, I'd go for LA for sure!

1/ For practical considerations, the notion of a linear transformation is super useful if you'll be studying any complex process (as soon as you have multiple input variables, you'll want to put some coefficients in front of them, and what is the simplest math model you can use? In high school math we learn about proportionality relations, i.e. y = mx, where the output y depends on the input x multiplied by coefficient m (the slope if you think geometrically in the xy-plane).

Extending the notion of proportionality to transformations with n inputs and k outputs, instead of the single slope m you need k*n coefficients to describe the proportionality relations between input component j and output component i.

Linear models are pretty good in the bang-for-your-buck metric for math models since: (1) k*n params for an R^n --> R^k transformation is reasonable amount of parameters, and (2) using "tomography*" you can easily estimate each of the coefficients. This is why LTs are used in many fields of science/computing use linear transformations (Biology, Chemistry, Economics, Statistics, NNs, etc.).

tomography*: input n "probing inputs" to T e1, e2, ..., en and record the outputs T(e1), T(e2), ..., T(en) (each of the outputs is a k-dimensional vector) --- if T is a linear transformations, then the info you've collected is enough to know all the k*m coefficients of the linear transformation (<=> entries of the matrix).

2/ From a theoretical point of view, I think teaching linear transformations and matrix-vector products (the boring row-times-column arithmetic rules) is a really good thing since it introduces learners to representation theory. You have one thing in math land y = T(x) and another thing in math land y = Mx and you know their behaviour and properties are identical (isomorphic?). This means you can understand the properties of one of the things by studying the properties of the other thing, e.g., Ker(T) <=> Nullspace(M).

For me, this first contact with representation theory concepts feels like a really valuable thing to have. (A good knowledge buzz moment to get learners more interested in learning math). And it's not just linear transformations and matrices that have a "is a representation of" relationship between them. There are lots of representations in LA:

    - vectors <-> coordinates
    - system of equations <-> matrix equation
    - row ops <-> elementary matrices
    - linear transformations <-> matrix-vector products
    - composition of linear transformations <-> matrix-matrix products
    - graph <-> adjacency matrix
    - conditional prob p(y|x) <-> matrix whose columns are p(y|x_i)
    - function in time <-> Fourrier coefficients
    - quantum state <-> vectors with complex coefficients
    - quantum operation <-> unitary matrices
    - quantum measurement <-> list of projection matrices that sum to the identity

In particular for devs, it's a really easily transferrable analogy. The linear transformation T is the "spec" while the matrix M and matrix-vector product rules together represent a particular implementation of the spec. See `T` and `T_impl` near the bottom of this notebook https://github.com/minireference/noBSLAnotebooks/blob/master... (binder https://mybinder.org/v2/gh/minireference/noBSLAnotebooks/049... or colab https://colab.research.google.com/github/minireference/noBSL... )

3/ From a pedagogical point of view, if we can deliver the "representation theory buzz" from linear algebra in high school, then this will be a good chance to review some important high school ideas:

Integer representations:

    - integers in terms of decimal digits with place value
      a = dn...d3d2d1d0 = dn*10^n + ... d3*1000 + d2*100 + d1*10 + d0
    - integers in terms of prime factorization
      a = 2^a2*3^a3*5^a5*...

Rational representations:

    - fractions as m/n where m in Z and n in N\*
    - reduced fractions as m/n where m in Z and n in N\* where GCD(m,n)=1

So all in all, if we were to define LA as "representation theory knowledge buzz," and someone asks me if I recommend learning LA before CALC, I'd say yes.*

Sure, linearity is a very useful concept, but very few high schoolers are able to understand such concepts in the abstract, which is when it's the most useful.

I remember when we were introduced to abstract vector spaces in high school, and we were all pretty confused, even though this was a high school dedicated to mathematics, the foremost in the country.

Even complex vectors had us scratching our heads, which in retrospect seems absurdly trivial. It's just that we were used to thinking in very concrete terms, anything purely abstract is 10 times harder to grasp, so you probably can't teach things like linear transforms without matrices.

Derivatives and integrals on the other hand are very easy to visualise.

Great thoughts .... I haven't thought of this "representation theory aspect" of LA ! thanks. (And also introduction to SymPy. interesting)

I would say that calculus is more relevant to the sciences, is it not?

Possibly not Calculus 2, but at least getting an intuitive understanding of rates of change, second derivatives, and area under a curve seems pretty critical for all the sciences.

There is a nice "opinion piece" by Strang that echoes what you're saying here: Too Much Calculus (and not enough LA) https://siags.siam.org/siagla/articles/Strang2001.pdf

I think highschool tries to expose you to as much as possible. I did get a tiny bit of linear algebra at the end of my senior year in calc, but back then PCs pretty much didn't exist. I think today LA is more applicable to programming than integrals, series & expansions, continuity, etc... so maybe times have changed.

My guess is that it's just easier to find Calculus teachers vs Linear Algebra teachers. Maybe it's less obvious in the US but when I have visited schools in poor countries you really get a sense of where knowledge in certain subjects tops out. I find people in the US seem to assume they can magically drum up a supply of linear algebra teachers, or the like. STEM teachers are technical professionals.

Probably historical. Modern linear algebra is extremely recent as math goes. Definitely much younger than calculus.

Also I would add that linear algebra, calculus, and differential equations all go pretty much hand in hand. We could probably stand to teach anyone with an inclination for STEM all of those much sooner.

I think it's historical. Linear algebra applications fall into two categories:

1. Theoretical (as in vector spaces, etc). These mostly are useful in advanced courses in engineering/science, and a lot of their applications involve calculus (e.g. Fourier series, function spaces, etc). So calculus needs to be taught first.

2. Computational. These can be subdivided into applications that involve calculus (e.g. differential equations) and everything else (graphics, etc).

Many of the latter's applications are relatively recent (last few decades). Whereas calculus was needed in virtually all types of engineering and science. So it made sense to teach calculus.

Imagine it's the 1970's. Your in HS. What will you do with all the linear algebra knowledge that won't require calculus? Assume you have no access to computers.

Hello, I'm the author of the article, thanks for the nice comments.

This article should have been titled as "Graphic Notes on Linear Algebra for Everyone", and Prof. Strang kindly suggested this big name. I was lucky that this drew this attention.

There are some other visuals I'm trying around the area. - Eigenvalues https://anagileway.com/2021/10/01/map-of-eigenvalues/

-Matrix classification https://anagileway.com/2020/09/29/matrix-world-in-linear-alg...

When I was an undergraduate, I didn't get this understanding of linear algebra... but after watching all the Prof. Strang's 18.06 classes in MIT OpenCourseWare, now I have much clear view of this area... So I really appreciate his way of teaching.

BTW, I even made a T-shirt and sent him ! https://anagileway.com/2020/06/04/prof-gilbert-strang-linear...

Instantly fell in love with the presentation! Might give this a read-through just out of pure curiosity.

In undergrad, my mnemonic for these operations was visualizing the matrices animated in my head. The more complex ones, it was actually easier for me to remember Scheme functions that represent the algorithm (all expressed via higher-order functions so it was pretty concise); this was unique to my circumstances as an undergrad, not something I can pull off today without reviewing a lot of material.

Presenting the operations with color and blocks just gives a more natural "user interface" (lacking a better term) for remembering it!

On the first page they say "if neither a or b are 0", but they haven't defined what it means for a vector to be 0.

Also, they say "rank 1 matrix", but they haven't defined the concept of rank yet.

Some readers might find this kind of presentation acceptable, but personally I strongly dislike it when concepts are used before they are defined.

To be fair, it does not look like this text is intended to be a self-contained introduction to linear algebra.

The concept of an additive zero is built into what it means to be a vector space though.

Nice, I was just reviewing SVD and PCA, I grok them but then I forget, this material is useful for remembering the big picture.

I keep seeing the same material like this (and love it btw) but I keep thinking that it's all only scratching the surface. From what I've seen in abstract algebra this stuff goes way deeper and becomes way more beautiful. I would love the "homemade" explanations and visualizations to start enlightening us about that. E.g. so DIY machine learning folks like the amazing GAN art community that's cropping could get more tooling to pluck easy hanging fruit.

Would love to see other things like physics, chemistry, etc. done with this kind of representation. Visualized, concise descriptions.

This is great for those studying algorithms (DP, DC, FFT, etc.). Very useful.

Great visuals

good notes, very useful!

Reply on news.ycombinator.com