Higher Order Derivatives of Transforms | Hacker News Reader

Higher Order Derivatives of Transforms

nosferalatu.com

Don't think I'll ever get used to stars in LaTeX when someone means simple multiplication.

The preceding blog post[1] seems to contain the more interesting parts though. This is just (d/dt)^n e^At x = A^n e^At x, which is kind of obvious from the definition of e^At.

[1]: https://nosferalatu.com/DerivativesLogarithmsTransforms.html

The article (even more the previous article) really misses a reference to matrix exponentials [1]. Everything about X(t) = X^t becomes a bit clearer. Also I think using the term transform is a bit loose here. IIRC Matrix exponentials only work with linear transforms that are represented by a square matrix.

[1] https://en.m.wikipedia.org/wiki/Matrix_exponential

The previous post in the series[1], helpfully linked to in the introduction, has a lot more details of the underlying concepts.

It also set my mind wandering to the not-technically-related functional derivatives[2], where you vary the function slightly rather than the argument value.

I'm not great at math, but I do love this what-if exploration you can do in math. Due to the various proofs underlying it all it seems sometimes more fruitful than similar exploration in programming, where one might quickly stumble upon obscure compiler errors or similar obstacles.

[1]: https://nosferalatu.com/DerivativesLogarithmsTransforms.html

[2]: https://en.wikipedia.org/wiki/Functional_derivative

If you're looking for interesting derivative-adjacent ideas, I would also recommend Clarke derivatives [1]. They occasionally show up in ML papers, e.g., [2], [3]. Unrelated bu tangential, another place where you need derivatives but don't have access to them (standard or otherwise) is in the area of black-box optimization. Within this area, Bayesian Optimization (BayesOpt) has picked up quite a bit, which I've successfully used quite a bit in my work - I've an introduction here [4]. There is also a good book available online for free on the topic [5].

[1] https://en.wikipedia.org/wiki/Clarke_generalized_derivative

[2] https://proceedings.neurips.cc/paper/2021/file/70afbf2259b44...

[3] http://proceedings.mlr.press/v202/lee23p/lee23p.pdf

[4] https://blog.quipu-strands.com/bayesopt_1_key_ideas_GPs

[5] https://bayesoptbook.com/

Functional derivative : Frechet derivative :: gradient : directional derivative

What is meant by a Transform in this context? The author doesn't seem to give a definition.

They give a brief statement in their previous post (see link at start of post), essentially it's a linear transformation also known as linear map[1].

[1]: https://en.wikipedia.org/wiki/Linear_map

I read their previous post, and nowhere do they explicitly say "a transform is a...". One might assume that it is indeed a linear transformation, as you suggest, but it shouldn't be up to the reader to do detective work just to understand the objects the author is talking about.

What I meant was that based on what they said in their previous post ("[g]iven a transform T and a point x, we can find the transformed point with T∗x") and the interactive graphics, I felt certain they meant a linear map.

I agree it's sloppy, at least a reference or something should be given if one doesn't want to spend time on the full definition.

Maybe the author could have used a more general notion, then, if omission and brevity were going to be present? Like, instead of a linear map or transform, he could have said an operator or something.

I don't know what is the general form of a transform or linear map. I think it's something like operator, though.

True, however I don't know how well versed the author is. Back in my late teens when I was deep into 3D graphics and ray tracing, I knew a lot about that specific math but not much beyond it. To me, "transform" was crystal clear to mean some kind of linear transform, and I hadn't yet learned of the more general operator notion[1].

So I can see myself writing something similar thinking it was clear.

[1]: https://en.wikipedia.org/wiki/Operator_(mathematics)

Game developers, physicists, and other engineering-adjacent types don't know much higher math. If any.

It takes a special type of engineer to explore advanced abstractions and get familiar with them. An exceptional engineer, really.

Author here. My background is game development. Now I work on 3D modeling software. I wouldn't say I know "higher level math", because that seems like a very deep well...

What would be the best way to define "transform" here? What I mean is something that can be applied to a point, like a linear map. So translation, and/or rotation, and/or scaling, and/or skewing, are all things that can be done by this "transform" in 3D.

In computer graphics these are often expressed as 3x3 or 4x3 (or sometimes 4x4) matrices. But a "translation+quaternion" can also be a transform, or just a quaternion (a unit quaternion can be used to rotate points for example). So I'd be happy to use a better definition for "transform" than 'given a transform T that can be applied to a point' but I'm not quite sure what the best definition would be.

Honestly, this is a chance to be creative and informative, here. You could have provided a background and motivation for transforms. A "why we are here in the first place." There's actually no need to be formal or a need for a worry about good definitions.

You could have provided your own perspective for why transforms exist and why they need to be used. That would have involved explaining why a point or a vector needs to be transformed into another vector, in the context of 3D graphics.

Or why transformations in 3D space work so well and reliably.

You basically elaborated on the how, like a simple technician, while neglecting the fundamental context of your study. That's great for recording some notes and for helping to reinforce a memorization requirement and routine. But going down the deep wells of abstract subjects like the intimidating ideas of advanced mathematics calls for some unusual approaches to looking at things standing in front of you.

Engineers know how things work. And mathematicians know why those things work. The smarter person sees through the black boxes.

operators are usually defined as linear maps from spaces to themselves

    V -> V

so they are less general than a linear transformation

In robotics, it's pretty widely used to refer to a 6-degree of freedom pose or relative pose in space, for example it's widely used in ROS, a de facto standard https://docs.ros.org/en/melodic/api/geometry_msgs/html/msg/T.... Not sure if there they're just using it as an example in the graphics.

I meant a "transform" as a linear map. I'm using the word as it is used in computer graphics (my background), so it's something that translates, rotates, scales, etc. other things (such as points). That is often a 3x3 or 4x4 matrix, although it can also be a vec3 translation and a quaternion, or just a quaternion. I think "Transform" is clear in the context of computer graphics, but I see what you mean about it being vaguely defined in my blog post.

Terence Tao has a book “Higher order Fourier analysis”, can anyone chip in on whether the two are related?

Here is a video, it seems to focus on aaplications in number theory, but I cant tell as a layman https://www.youtube.com/watch?v=nr2Xv9v9CZc

For example, this is the tradFourier version of what hes trying to do

https://en.wikipedia.org/wiki/Hardy–Ramanujan–Littlewood_cir...

The techniques in this article would be considered common knowledge for any of the prerequisite classes you'd need to take to be able to follow a course on Higher order Fourier analysis.

I wouldn't say they're unrelated, but if you want to know whether both uses of the phrase "higher order" have any relation then no.

Thank you!

I remember seeing in the Tao book a Fourier operator that looks something like the “integral” of exp(ik^2 x).. (or exp(ik x^2), haha)

As a followup, is it possible to ELI5 roughly what Tao meant by “higher-order”?

The first paragraph of the preface on page 7 in the book gives an explicit example of “higher order”: https://terrytao.files.wordpress.com/2011/03/higher-book.pdf

Lollol thats what i half remembered.. i guess then what Terence Tao meant by higher order is lower order in the featured artiicle!

I like this summary of Lie theory in the context of robotics: https://arxiv.org/pdf/1812.01537.pdf

I like this tutorial because it doesn’t get too bogged down in abstractions and has numerous examples. When I’ve tried to learn differential geometry in the past, standard texts get very abstract very quickly and it’s hard for me to envision what the generalization is doing for me.

Reply on news.ycombinator.com