Linear Algebra Done Wrong (2004)
math.brown.edu
math.brown.edu
I know there are Google engineers here, possibly search. Do you guys not also feel it? I mean you dogfood, right? I know it is a hard problem but boy has it become harder to use, especially in the last 6 months.
So now, to the maximum extent possible, I work to undermine their interests and to not participate with their products or addons or browsers.
Great strategy on their part.
There is no single entity Google cares about. There is no "average user" (nor median). Trying to make the product best for an "average" user makes it bad for everyone. You have made the assumption that this solution space is smooth and relatively uniformly distributed.
Consider this: on a uniformly distribution, if you interpolate between two points, those interpolations are representative of the distribution. But if you do so on a gaussian distribution, this is not true. This is because the density of these distributions are not evenly distributed (this is what "uniform" in uniform distribution means). For more complicated distributions you need different interpolations. Many real world distributions are the agglomeration of varying distributions (you probably have clusters and pockets).
The distribution of searches (or customers) is neither normal nor uniform.
To understand some further complexity, it is important to remember that the burden of success is the requirement for further nuance. Pareto. It may take few resources to get to "80%"/"good enough" but significant to increase that another 10%. Think about a Taylor Series (Fourier Approximation, or pick your preferred example), the first order approximation will only take you so far. It may be good enough to start, but the need for higher order (and this more complex calculations) approximations are increasingly necessary for precision (non-linearly). So simplify where you can, but don't forget what you've done. If you do, you'll be stuck. Or worse, you will compound your error (also very common).
I wish it had a yes/no prompt that you could click and when you click no it would refresh with the results you wanted, and possibly use those answers to update whatever algorithm does the suggestions.
I understand the problem is difficult, but it appears to me that the signals they have that can help are being ignored.
I don't need a different variation of the same result, I need a different result. While the former can be certainly useful, it is absurd to think this is all there is.
If you ever search anything with any bit of nuance, you'll have experienced this. I'll give an example: try searching "out of distribution" and try to find a result that is not about ML based OOD detection. They do exist and you'll find some better queries, but that result still dominates (and frankly, they are poor results).
For our example query `statistics out of distribution -"machine learning"` has a top result of mathworks on neural networks and all results on the front page work. "statistics" is ignored before variations of "Machine learning." Top few results for `statistics out of distribution -"machine learning" -"deep learning" -"artificial intelligence" -"neural network"` returns to me an ICLR paper who's title that ends in "Deep Neural Networks" and many results are still ML based...
How do I tell Google (or any engine) that I want to exclude results!?
And before someone says GPT/Claude or some other LLM, the problem persists. Yes, I've paid. Yes, for the most advanced models. Derailing these tools from the wrong track is difficult. Even harder the longer the conversation goes (i.e. if you don't recognize you're headed in the wrong direction). The nuance matters.
https://www.math.brown.edu/streil/index.html redirects to https://sites.google.com/a/brown.edu/sergei-treil-homepage/h...
But OP's link, https://www.math.brown.edu/streil/papers/LADW/LADW.html does not redirect. So it seems like a problem that the site owner can fix.
> "Linear Algebra Done Wrong site:sites.google.com" finds the page you linked.
As in, if you didn't know a second newer version existed, would you find it? Certainly the new version should prioritize the older version.
FWIW, my query "Linear Algebra Done Wrong" yields the newer version (2017), but I've just interpreted OP's point as being illustrative and helpful to readers (and it looks like the link now does point towards the new version. Possibly this made my query work)
Does he ever answer this question? It's posed at the start of the book, but then immediately ignored.
Sheldon's book is more suitable for a second course in linear algebra. It's an excellent book, that my more advanced students have enjoyed.
Sheldon, however, approaches the subject with some rigid dogma: Determinants are evil. His view is they're not practical to compute at scale. It can be good to learn theory with one's determinant hand tied behind one's back, just as it can be good to learn "constructive" mathematics, but this should not be one's only approach.
I used to attend an academic sponsor's day at the math institute MSRI (now SLMath). Afternoons reached to keep us busy. One year Sheldon's wife demonstrated some academic teaching software, and my group was tasked to solve a linear algebra problem that Sheldon had provided. Unaware that everyone was watching us on the big screen, I solved it instantly, using determinants. I still don't understand Sheldon's longer solution.
His view is that they are presented as an arbitrary piece of magic that people really don't understand. And, exactly because they don't really understand the determinant, they wind up not properly understanding fundamental concepts in linear algebra. Such as Jordan normal form.
If your goal is actually to properly understand linear algebra, he's absolutely right. If your goal is to use linear algebra for a wide variety of practical purposes, though, determinants are often essential.
When I hear, for example, ‘upper triangular matrix’ I have to translate it in my head into something like ‘matrix representing a linear operator that preserves the standard flag’ in order to actually feel like I understand it.
Of course, I’m not a programmer or an engineer of any kind, so I have the luxury of not needing the computational efficiency.
I would think "upper triangular" would be a weird/bad notion to talk about in a basis-independent setting (because it depends on the basis).
What I’m saying is that an upper triangular matrix preserves each of these subspaces because it sends each basis vector e_i into the span of the e_0, …, e_{i-1}.
You’re absolutely right that upper triangularity is basis-dependent and so is somewhat ‘weird’/‘evil’ (in fact, not even well-defined) as a purported property of maps rather than matrices. What I meant to say was ‘triangularisable’ by analogy with ‘diagonalisable’ — matrices which represent such a map in some basis. Given a linear operator, its matrix representation is triangular when expressed in a basis B if and only if it preserves the standard flag in basis B.
Determinants are totally basis-invariant, however, similar matrices have the same determinant.
Seeing the determinant as the unique solution to the problem of assigning a scalar to each linear map/matrix in such a way that several natural axioms are satisfied is a very nice way to motivate it. Along with the fact that it, along with the trace, can be calculated from the eigenvectors (from which the basis-invariance is clear).
Or, if you're using exact arithmetic (e.g. cryptography), you should use something like the Bareiss algorithm - Gaussian elimination is only polynomial if you use floats: https://en.m.wikipedia.org/wiki/Bareiss_algorithm
The Leibniz formula is totally impractical for calculating determinants, but it is sometimes useful in proofs due to its symmetric form.
I guess if you just throw a definition of the determinant at people and leave it at that, I can understand that it doesn't make sense. But that's not how I was introduced to the concept (I learned about abstract vector spaces and transformations and how they are represented by matrices - and only then about determinants and their properties and different ways of computing them). That's why Axler's critique rings somewhat hollow to me - maybe it's different if you learn about linear algebra mostly in an engineering context as opposed to pure maths.
For theoretical purposes, determinants are important because they constitute a homomorphism between the general linear group and the multiplicative group of the underlying field, and the kernel of this homomorphism is the special linear group. This leads to some very short and elegant proofs.
Volume is base x height. Height is a linear function in the vector that rises above the base; the usual properties of the determinant follow from this.
Linear Algebra Done Wrong [pdf] - https://news.ycombinator.com/item?id=999494 - Dec 2009 (12 comments)
The product of eigenvalues definition is also somewhat intuitive to me ("How much does the matrix scale vectors in each direction? Now multiply those numbers together."), but it's harder to motivate the fact that adding rows together doesn't change the determinant, which is kind of important to actually computing the determinant.
The second half of LinAlg was a bunch of stuff about Eigenthis and Eigenthat, with no particular explanation as to what you'd use it for or why you would care. I pass the class with an A and was even recommended by the instructor to be work as a tutor for other students, but 23 years later I couldn't tell you the use of any of that stuff.
T-matrices, though - I've used that stuff my whole career, working in medical imaging and related fields. I still couldn't tell you what an Eigenvalue or Eigenvector is useful for, as I never had any reason to apply that stuff.
It's not particularly abstract if you can explain why and how this stuff is used - and I know it's heavily used in many fields - just not those that I've worked in.
Say for example you rotate something in 3D. The rotation axis remains unchanged. That's an eigenvector. Or say you mirror something in 3D, then all the lines lying in the mirror plane remain unchanged (all eigenvectors with eigenvalue 1), and the line orthogonal to the mirror plane remains unchanged - or rather, flipped, so it's an eigenvector with eigenvalue -1.
I'm not sure that linear algebra is an elementary topic, if the word "elementary" is as in math of elementary school. That said, I don't think there's much confusion about linear algebra either. It's more likely that linear algebra is substantially more abstract than high-school algebra, so many students have a hard time deeply understanding it, just like many students already bail out on high-school math. When I was a TA in college, I also observed that many students were not prepared for the fact that college-level math is fundamentally different from high-school ones. College-level maths has far less time for students to grasp the concepts through sheer brute-force practice. College-level maths requires students to focus on intuitive understanding of they key concepts so the students won't get bogged down by hundreds and hundreds of concepts or corollaries or theorems. Of course, high-school maths requires intuitive understanding too, but because high-school maths is so simple that many students get the intuition naturally so they are not aware of how important such intuitive understanding is.
This book used to help my students build intuitions: https://www.amazon.com/Algebra-Through-Geometry-Undergraduat.... It starts with 2D geometry to teach linear algebra, and then moves to 3D, and then 4D. The author also chooses to use calculations and constructions to prove most of the theorems instead of more abstract or algebraic approach.
A lot of the stuff you learn initially, about systems of equations and how they relate to the matrix inverse, are very interesting because it's clear how this can be applied.
But as you move forward into vector bases and change of coordinates there's a very long dry spell that you sort of have to slog through until much later when you start to see how it is actually useful. I'm not sure how to fix this -- maybe take a step back from the symbolic and do some numeric computations because that's where they start becoming useful again.
IMHO there should be two versions of linear algebra. One for computer science majors and one for mathematicians. I regularly run into stuff at work where I say to myself, "self, this is a linear algebra problem" and I have next to no idea how to transform the problem I have into a matrices or whatever.
But I can write a really stinkin' fast matrix multiplication algorithm. So there's that I guess.
Modern CPUs with big ass SIMD registers are incredibly fast at slogging through a linear algebra problem. Lots of incredibly intelligent people (ie, not me) spend an incredible amount of effort optimizing every last FLOP out of their BLAS of choice. For several years the only question Intel asked itself when designing the next CPU was, "How much faster can we make the SPECfp benchmark?" and it shows. Any time you can convert a problem using whatever ad-hoc algorithm you came up with into a linear algebra problem, you can get absurd speedups. But most programmers don't know how to do that, because most of their linear algebra class was spent proving that the only invertible idempotent nxn matrix is the identity matrix or whatever.
Discrete math has the same problem. When I took discrete math in college, the blurb in the course catalog promised applications to computer science. It turns out the course was just mucking through literally dozens of trivial proofs of trivial statements in basic number and set theory, and then they taught us how to add two binary numbers together. The chapters on graphs, trees, recursion, big-O notation and algorithm analysis, finite automata? Skipped 'em.
Unfortunately this has been the debate for thousands of years. One of Euclid's students asked him what practical use geometry had. In response, Euclid instructed a servant to give the student a coin, saying, "He must make gain out of what he learns."
Legend aside, I'm actually surprised to see, many times actually, that people on HN criticized the "absence of practical examples". If we compare the textbooks written in the US and those in China or Europe, there's a sharp contrast. The textbooks from the US are thick, full of discussion of motivations and practical examples across multiple domains (Thomas' Calculus, for instance). In contrast, Chinese and European textbooks are terse and much thinner. They focus on proofs and derivations and have only sparse examples.
Personally, maths itself is practical enough. I'd even venture to say that those who can naturally progress to college-level maths should be able appreciate high-school maths for its own sake.
I get that 2^32 simultaneous equations may be a straightforward linear algebra problem but I am now going deep to understand the exact mechanism to solve this.
I've actually stopped asking questions about Cayley-Hamilton and Jordan form (both covered in year 1 of any decent BS math course in Europe) when I used to do inteviews for trading/quant positions because so many people failed.
It’s mostly because, as Axler explicitly tries to address, linear algebra is a subject that can be viewed through at least two different lenses: the intuitive, (possibly higher-dimensional) geometric lens, or the algorithmic and numerical lens. Of course they are equivalent, but almost every course in linear algebra teaches the second and barely even touches on the first. It’s like learning Euclid’s algorithm before you’ve learnt to count. No wonder everyone’s so confused.
Apart from "argument by year number", what's actually wrong with the book?
2. What are examples of better terminology, notation, or pedagogy that are improvements over this book? (Doesn't need to be total, could retain the same terminology and notation but just have better pedagogy for example)
However, I'll give you that interactive texts (or texts with supplemental interactive material) are often better for some things (and some parts of linear algebra) than a plain dead tree format.
In some ways I like the idea of replacing traditional indexing with something more like the Einstein summation notation, and moving away from the arbitrary feeling that you get around how matrix multiplication is structured, and provide an immediate route into tensor theory.
But so many things that are very important in linear algebra, like matrix inverses, are awkward to express in this notation.
Every lecture was so eye opening. I couldn't believe that linear algebra could be taught in such a context with such a variety of application domains.
The lectures are all available online with the assignments : https://ee263.stanford.edu/archive/
[1] https://www.amazon.com/Data-Driven-Science-Engineering-Learn...