Hacker News Reader

Comment by jey | Hacker News Reader

"Just stir the pile until [the answers] start looking right" is actually a pretty decent description of gradient descent.

View on HN

It's clearly simulated annealing. You stir less as you get tired. :)

Simulated annealing also exploits gradient information, so it is not equivalent to random search, although it is a stochastic algorithm.

With gradient descent you are walking in the direction of negative gradient. Stirring the pile implies that you are walking randomly from point to point.

So more like a genetic algorithm then?

GAs also exploit structure in the search space.

To see this most clearly, read up on estimation of distribution algorithms (EDAs), which generalise the ways in which GAs work.

No, by stirring you're just making the pile to move, and things descend down the gradient of gravity. :).

The little things moreso. Brazil nut effect!

When you stir, you're stirring in a pattern, probably a circular one. Plus you're probably not paying too much attention to the shape of the pile, so it's gradient-free. It's pattern search. Except it's only approximately a circular pattern. There are random variations. Stochastic pattern search.

It is more like tuning the hyper-parameters, which is guided by anything but logic.

It actually seems almost like the definition of RANSAC.

https://en.wikipedia.org/wiki/Random_sample_consensus

Gradient descent is not random search, so this is an inaccurate description. It works by exploiting the gradient in a structured landscape of solutions.

I'm not so sure. Gradient Descent refers to reaching the best solution available. Xkcd seems to be referring to more like the Bonferroni principle where you are looking for patterns without a hypothesis and post facto justifying it.

AKA half the work done in Machine Learning nowadays.

I think the point is about how data manipulation is about 80% of machine learning work. If an algorithm is giving you crap results for some data set, once you've fiddled with its hyperparameters through cv and so on there's not much you can do besides data manipulations like PCA, ICA and the like, to try and get a better result. Most algorithms work pretty badly with raw data.

I'm guessing that, in the sciences, that is a big no-no. Imagine if doctors, seeing that a new drug being trialled is failing to cure a disease, simply started chucking out the sick subjects until all the ones that were left where healthy, declaring the sick ones to be "noise" and the trial a success. Somehow, I don't think that would fly...

> I think the point is about how data manipulation is about 80% of machine learning work. If an algorithm is giving you crap results for some data set, once you've fiddled with its hyperparameters through cv and so on there's not much you can do besides data manipulations like PCA, ICA and the like, to try and get a better result.

Data manipulation is different from data transformation. Manipulation changes the nature of data, transformation does not change the nature of data. PCA is data transformation, not data manipulation.

Tell me what you mean by "the nature of data" and I'll tell you if I agree with your definition. I don't see why the dimensionality of a dataset is not its "nature" for example.

This is confusing "cherry picking" (selecting some data and discarding other data) with "data transformation". Transformations are applied for perfectly good reasons, such as dimensionality reduction, or to enable the subsequent application of certain statistical tests that rely on assumptions not valid for the data in its original form.

To be precise: the best solution reachable from a given starting point, which may be a local minima rather than a global minima.

(iterative) stochastic gradient descent to be specific. Data is stirred for each pass to prevent cycles.

Reply on news.ycombinator.com