The physics principle of diffusion inspired modern AI art
quantamagazine.org
quantamagazine.org
Gradient descent uses the gradient, but is itself not the gradient.
The gradient itself is just the vector of partial derivatives with respect to each parameter.
Gradient descent is just following the gradient "downwards" towards a minima.
The article even describes it pretty well in non-mathematical terms.
> the gradient of the distribution (think of it as the slope of the high-dimensional surface).
I always hear devs say they never use calculus but I suspect it's because you can't use what you don't understand.
Yes, we do, like a decade ago at school when we were forced to learn it. Then the knowledge atrophies unless you're actively using it in real life.
The majority of engineers I know, including myself, work on some form of web services, mobile apps or FOTM UI frameworks, and you don't need calculus to style a progress bar or query a database
Could be a handy thing to have if, for example, you were designing some progress animation...
I've seen a lot of cases where progress bars actually jump backwards because these predictions fail, whether it be games or applications, which feels like the universe is making fun of your existence when it happens.
Sometimes it feels like a 'this action is not stuck' indicator would be more appropriate.
Any self-respecting CS theorist would tell you that accurate progress bars are generally impossible to make because that would reduce to some version of the halting problem; and any self-respecting software engineer would tell you that even in the simple cases where the halting problem is not a problem (eg. simple copying of files), in practice many things outside of the software's control could stall the progress.
Even if you could get an accurate progress value, you don't need a "gradient" to animate it. Just update the bar with the current value as a percentage at 60FPS and you're already done. (Progress bars generally ask the programmer to supply the current progress value, not a stream of progress rates in real time) If you tried to predict the progress "gradient" past the present point, you'd end up having to "lie" (more progress than actual) or have the progress bar jump back when it turns out the actual progress was slower than predicted.
If all you had was a hammer, everything looks like a nail. That almost always applies for those enamored with advanced maths.
Imagine you have a huuuuge landscape, with peaks and valleys. This landscape defines the density of the distribution. Now, imagine that you have a bunch of samples drawn from the distribution, this is the dataset.
Sampling an image is a mixture of dirac delta functions (or a mixture of Gaussian with variance approaching zero).
We can increase the variance to smoothen the landscape. This sort of builds an empirical estimate of the true landscape.
The height of this landscape is “the score”.
It turns out that we can actually compute an approximation of the gradient of the score.
The gradient of the score always points towards where the score (aka density) will increase, and you essentially apply gradients to “walk” towards a high density region.
This idea is actually very similar to using gradients to create poisoned inputs that are falsely predicted.
I am being a bit handwavey here, we don’t just increase the variance, but this is a decent enough approximation of what’s happening and what the gradient refers to in this case.
For all intents and purposes, you can think of the gradient as a vector pointing towards the direction that the density of the dataset increases.
In other words, for each tuple of parameters you can estimate the direction in which the reduction in the resulting error would be steepest.
Gradient descent is then a method whereby you iteratively move towards better and better parameterisations, by following the direction of the (negative) gradient at each step, which guarantees you will reach a local minimum much more efficiently than if you sampled new parameters in the area of the previous iteration at random.
And if your error function is convex, the minimum error is a global minimum.
This is made even more prominent by the lack of any mention of the competitors.