An introduction to Machine Learning

It's free. They have run it a couple years in a row now starting in January.

The linked page says "Free".

Nothing wrong, quite the opposite scikit-learn is awesome.

I think the comment was a word play on "script kiddie" (ppl w/o real "security chops" but who know enough to run an exploit "script" of some sort).

The reason they call it error function in perceptron learning is that it relates to how the perceptron is taught a correction for an error. Loss functions are more general and usually the word people use when they're talking about optimization problem.s

RMSE and MSE are monotonic transformations of each other, so minimizing one is equivalent to minimizing the other. You can think of linear regression as minimizing RMSE if you like; it's just cleaner to do the math without the extra square root.

- Is the term 'error function' used as a special function (like sin, cosine, etc) or is it a generalized term ?

For backpropagation I call "error function" a function which takes the training data and the parameters of the model and returns an empirical approximation of how good is the model to be able to differentiate it.

- Are we using the term 'loss function' as a generalized term? whose special case is 'error function'? e.g., in linear regression loss function is 'error function' (MSE like function) but in perceptron, loss function is max(0, -xy)?

I used a squared error loss to define the error function of the linear regression model. This is arbitrary and the choice of the loss depends on the meaning we want to give to prediction errors. Here a big error is penalized way more than a small error.

- Are we using the term 'loss function' as a generalized term? whose special case is 'error function'? e.g., in linear regression loss function is 'error function' (MSE like function) but in perceptron, loss function is max(0, -xy)?

The way I understand the difference between error and loss is by thinking about regularization methods (where you also add a penalty in the error depending on the parameters of the model). In that case error = loss + regularization. When applying gradient descend you would derivate the error and not the loss. The error is an empirical value that is supposed to estimate the performance of the model on unseen data. This is by consequent not formally defined.

- Using final output of perceptron for error function makes it a "hard problem" agreed. But what about using just the function from linear regression (the MSE-like one) instead of a using a brand new function max(0, -xy). (It's not very intuitive to reason what's so special about max(0, -xy)).

The loss function used is a hinge loss. If the point is correctly classified no penalty is added, otherwise the penalty is the distance to the hyperplane.

- You can think of error, loss, and cost functions as the same. In fact, two textbooks in front of me say that the loss function is a measure of error. If "loss" is a confusing word, think of it as the "information loss" of the model -- if your model is not perfect, you lose some of the information inherent in the data.

- There is no particular function used for error and loss. Different functions can be chosen based on the model, problem type, ease of theoretical analysis, etc. In practice, the final loss function is often experimentally determined by whatever yields the best accuracy.

- The perceptron uses a different loss function because it is a binary classifier, not a regressor. In this case, because there are only two classes (1 and -1), the loss function max(0, -xy) is 0 if x and y are the same class and 1 if they are different. Then, the error function just sums these losses together. (Note this is quite similar to MSE.)

- RMSE is also valid -- adding the square root will not affect minimization. MSE is likely more common for minor reasons, such as slightly better efficiency and cleaner theoretical proofs.

I liked the UW Coursera class that gave a broad overview of these topics with some applications: https://www.coursera.org/learn/ml-foundations

It's part of a Machine Learning Specialization on Coursera (5 courses + a capstone project) which goes deeper on some areas after the foundations course: https://www.coursera.org/specializations/machine-learning

I am taking this specialization and I have learned a lot so far. The material seems like it's at exactly the right level of depth (balances giving a high level overview of the field, with enough depth in specific areas to understand how things work and be able to apply them). Disclaimer: I work at Dato, and the CEO of Dato is also one of the instructors of this course.

You know... I believe I started this class once and didn't finish it due to time constraints. I think it's time to try again...

Thanks! I'll checkout it out.