The matrix calculus you need for deep learning (2018)
explained.ai
explained.ai
The matrix calculus you need for deep learning (2018) - https://news.ycombinator.com/item?id=26676729 - April 2021 (40 comments)
Matrix calculus for deep learning part 2 - https://news.ycombinator.com/item?id=23358761 - May 2020 (6 comments)
Matrix Calculus for Deep Learning - https://news.ycombinator.com/item?id=21661545 - Nov 2019 (47 comments)
The Matrix Calculus You Need for Deep Learning - https://news.ycombinator.com/item?id=17422770 - June 2018 (77 comments)
Matrix Calculus for Deep Learning - https://news.ycombinator.com/item?id=16267178 - Jan 2018 (81 comments)
Ultimately, the point of using matrix calculus (or matrices in general) is not just concision of notation but also understanding that matrices are operators acting on members of some spaces, i.e. vectors. It is this higher level abstraction that makes matrices powerful.
For people who are familiar with the concepts but need a concise refresher, the Wikipedia page serves well:
Matrices themselves form non-commutative rings too; and based on this, you can think of a 4N x 4N matrix as a 4x4 matrix whose elements are NxN matrices [1] :D
[1] https://youtu.be/FX4C-JpTFgY?list=PL49CF3715CB9EF31D&t=1107
You already know whose lecture it is :D
I love math.. I should have become a mathematician ...
Block MM demonstrates the equivalence.
[1]: https://pdfs.semanticscholar.org/2e43/477e26a54b2d1a046c2140...
https://arxiv.org/abs/1802.01528
---
EDIT: It turns out explained.ai is the personal website of one of the authors, so there's no need to change the link. See comment below.
I prefer reading on the web unless I'm offline. The latex its super handy for printing a nice document.
Thank you for doing this with Jeremy and sharing it with the world!
I tried driving these formulas from scratch using what I learned from OP's post but it felt like there was something missing. I think it boils down to me not knowing how to aggregate those element wise derivatives into a matrix form. Afaik the Matrix cookbook and certain notes from Stanford cs231n that helped me grok it fully
Always wanted to dip my toes into ML, but I've never been convinced of it's usefulness to the average solo developer, in terms of things you can build with this new knowledge. Likely I don't know enough about it to make that call though.
https://github.com/williamcotton/chordviz
Labeling software in React, CNN in PyTorch, prediction on app in SwiftUI. 12,000 and counting hand labeled images of my hand on a guitar fretboard!
You don't need math to make a model perform well, but you do need math to know why your model is wrong.
It seems that these topics are covered by the first one or two semesters of a Math degree. Of course university is a bit more advanced.
I'd like more resources geared for people that are done with Khan Academy and want something as well made for more advanced topics.
If you're REALLY rusty (maybe you've been out of school for a while 5+ years), or maybe you just never learned the material that well in the first place, then you might want to start with one of our Mathematical Foundations courses that will scaffold you up to the level where you can handle the content in Mathematics for Machine Learning. More info can be found here: https://mathacademy.com/courses
The Mathematics for Machine Learning course would be ideal for anyone who majored in a STEM subject like CS (or at least has a solid mathematical foundation) and is interested in doing work in machine learning.
I only see pictures, I'm curious the extent of the interaction in the linear algebra/matrix calc specifically
I wouldn't say that is everything, but it is a useful trick.
The answer for why it is so is pretty trivial(just do the indexing for each element) if you know the definition of the kronecker product and what the 'vec' operation is.
For an intuitive explanation, try thinking of how the matrix multiplication would work and consider how the kronecker product pattern would apply to the vector.
This honestly isn't a super interesting result, and I would say the original commenter was overstating its importance in the matrix calculus. It really is more useful for solving certain matrix problems, or speeding up some tensor product calculations if you have things with a certain structure. Like if we have discretization of a PDE then depending on the representation the operator in the discrete space may be a sum of kronecker products, so applying those can be fast using a matrix multiply and never storing the kroneckered matrices.
If you take the result of transforming the columns vectors in the C matrix by AB and vectorize it you get the same as vectorizing first C and then transforming it by a block matrix obtained as the Kronecker product of B transposed and A.
The significance is that it performs a reduction of matrix calculus to vector calculus (i.e., it shows that you can convert any matrix calculus operation/formula/statement into a vector calculus operation/formula/statement).