Interactive Gradient Descent Demo
blog.skz.dev
blog.skz.dev
The "step size" parameter seems the same as given a sample size with what appears to be a random walk. I lack the background, but the example shows that you already know the function, so instead of picking an arbitrary point and descent rate, you pick a number of random samples, then apply the secretary problem to them (https://en.wikipedia.org/wiki/Secretary_problem#1/e-law_of_b...)
Top down from an outsiders perspective, (think product manager or customer for ML solution level) what advantage do I get from the compute cost of gradient descent over random sampling?
Random sampling becomes prohibitive in higher dimensions due to the curse of dimensionality [0]. Gradient descent doesn't have this problem and will always converge to a local (but, as can be seen here, not necessarily an absolute) minimum.
The step size effectively controls how far from the "real" local minimum you can get: too big a step size and you end up repeatedly "jumping over" the minimum.
Be careful with this. Understanding "edge cases" for an algorithm is just as important if not more than the core idea. With "plain language descriptions" you tend to get the latter but not the former which is _very_ dangerous.