Hacker News Reader

Comment by jph00 | Hacker News Reader

Jeremy here. Here to answer any questions or comments that you have.

But more importantly - I need to mention that Terence Parr did nearly all the work on this. He shared my passion for making something that anyone could read on any device to such an extent that he ended up creating a new tool for generating fast, mobile-friendly math-heavy texts: https://github.com/parrt/bookish . (We tried Katex, Mathjax, and pretty much everything else but nothing rendered everything properly).

I've never found anything that introduces the necessary matrix calculus for deep learning clearly, correctly, and accessibly - so I'm happy that this now exists.

View on HN

Terence here. Jeremy's role was critical in terms of direction and content for the article. Who better than he to describe the math needs for deep learning. :)

Glad to see you in this space Terrance! Been a long time since the traveling parser revival and beer tasting festival!

Question; when I learned Vector Calculus back in college, I used Marsden & Tromba as our text book where they equate the derivative of a function from R^n->R^m with the Jacobian. Is matrix calculus the same thing, just a slightly different notation?

Typographic advice: the body text has very long lines in a desktop browser, which makes it a bit slow and tiring to read. I’d say the ideal is somewhere between 1/2 and 2/3 this length. I’d recommend keeping the same width on screen but bumping the font size up by 30%.

As an extra minor nit, italicizing functions like sin, etc. is also somewhat unconventional in mathematical typesetting.

Thanks for the advice!

I agree that the font should be bigger. I need to learn more CSS in order to switch between font sizes per platform. The font of the text is easy but all of the images were generated from latex using a specific font size. I need to scale the in-line equation images as the font size bumps up.

the magic incantation here is probably media queries

@media (max-width: 768px) {

    p {

     font-size: 1rem;
    }
}

We have to also adjust the image sizes for the in-line equations. That’s what I need to figure out :)

it's a variable name for a polymorphic function, taylor series, euler formular, etc. depending on context. This is a significant difference to singular types, but being a variable name to an abstract concept is in principle no different to typesetting x. This goes neatly with "everything is an objectreference" and might be more of a programmer's perspective.

In mathematical notation it gets confusing because several italic letters in a row are otherwise interpreted as separate variables.

This is great! my graduate advisor a long time ago made a really great matrix calculus study sheet for me that was absolutely invaluable in learning ML ( i mean really this is great for not just DL but all sorts of reasoning in ML)

We originally had that generic ML target in mind but figured a DL bent would make it a wee bit more interesting.

Wondering if you still have that with you? Would be super useful for folks here.

What do you think about the index notation physicists use for tensor calculus?

I don't find it very accessible, myself - but I'm not a physicist, so the materials using or about that notation aren't aimed at me.

The only tensor notation I've been happy with is that used in J (http://www.jsoftware.com), which is simple, flexible, and concise.

There's also some nice-enough modern notation used in this excellent review: http://www.cs.cmu.edu/~christos/courses/826-resources/PAPERS...

The notation is very simple: you write tensor expressions with indices, and repeated indices are implicitly summed over. For example, matrix multiply:

   A_ij = B_ik C_kj

Differentiating this with respect to variable l:

   ∂_l A_ij = ∂_l (B_ik C_kj) = (∂_l B_ik) C_kj + B_ik (∂_l C_kj)

By writing out indices you can just use the rules for scalar derivatives.

Oh in that case I'm just confused - that's what I know as Einstein notation. Modern physics papers seemed to use much more complex notation, but I probably just misunderstood.

If we're talking about Einstein notation, then I'm a fan - `np.einsum()` is often a great way to create fast tensor computations with minimal code.

index notation isn't useful for calculation it's useful for algebra i.e. if you want closed form solution to tensor equations. my distinct impression is that in ML no one cares about that because eventually everything gets a numerical treatment.

There are discrepancies between the HTML and pdf version.

Eg. In the first line in 'References'

HTML: "When looking for resources on the web, search for “matrix calculus” not “vector calculus.” "

Latex/pdf: "When looking for resources on the web, search for “elements” not “elements” "

Whoops. thanks. translator error. I'll fix it.

Hi Jeremy, thanks for creating so many great learning resources. Myself and a fellow ML enthusiast are starting a deep learning meetup here in Phoenix next month. Do you have any advice for creating a welcoming environment for people to learn/teach? Thanks!

Probably better asking that at http://forums.fast.ai , since a lot of participants there have set up local meetups, and some of them have gotten pretty big. I haven't created a meetup myself - my focus has been on the online course and community, frankly.

In the article, you say the gradient of 2x + y^8 is [1, 7y^8]. Shouldn't it be [2, 7y^8]?

Oops. yeah. thanks

Now you are just replacing one error with another, you mean [2, 8y^7]. ;)

ERRATA: In the start of Matrix Calculus section, shouldn't the first value in gradient of "g" be 2 instead of 1 i.e., [2,8y^7]

This is really great, thanks.

Reply on news.ycombinator.com