Hacker's guide to Neural Networks
karpathy.github.io
karpathy.github.io
"Andrew Ng's Standford course has been a god send in laying out the mathmatics of Machine Learning. Would be a good next step for anybody who was intrigued by this article."
Has it been supplemented since then?
The backwards-moving pattern of "backpropagation" is really just a side-effect of the derivative chain rule application order, but a lot of intro materials treat backprop as if it is some fancy thing specially-designed for neural nets. I suppose "compute the gradient of this function using basic vector calculus" just isn't sexy enough. I complain mostly because it took me a while to figure out whether backprop was exactly the same as gradient descent, or if there were subtle differences.
https://en.wikipedia.org/wiki/Automatic_differentiation
which is really cool stuff and should be included more often when talking about backprop.
100 iterations give best_x = -1.76, best_y = 2.91, best_out = -5.15
10,000,000 iterations (less than 1 sec CPU) give best_x = 16,657, best_y = 16,662, best_out = 277,541,583
And if you're serious about learning it, you'd plop down the $149 for the Home edition of Matlab plus the $45 for the neural net toolbox. Or take a statistics class and grab the $99 student edition, which comes with the machine learning toolbox.
If you're gonna learn some shit, learn some shit. Calling Math.* to calculate rise over run in Javascript isn't where it's at.
Or maybe I'm just grumpy because I interviewed a Caltech CS grad last week who flat out did not know C. Never touched it in his studies. World's gone nuts.
Why no Python ? Why not C ? What about Lisp ? And Haskel ? Hey did I mention D ?
I can agree that JS is not the best suited languages for this task but, JS is extremely well know so it is perfect to explain something.
Matlab... Humm is not so widely know...
And yes, that was a mandatory course for CompSci. You mean people get through CS without touching Matlab and such? That's bonkers.
Did they also make you implement operating systems in assembler?
He did seem sad that SpaceX wouldn't touch him. Apparently you need more than friends at the Jet Propulsion Laboratory and an ability to talk endlessly about Haskell to send shit to Mars. So that's good news at least.
I'm personally of the opinion that knowing how to select and apply the right algorithm is far more important to learn for undergraduates. Implementing numerical algorithms correctly and efficiently is serious high level PhD territory, and if you're not going to learn how to do it right you're much better off not doing it at all and leaving it to the experts.
I was, however, surprised starting out to find out how many of my fellow students had never programmed in a compiled language before - different focuses during undergrad, I suppose.
It wasn't before the intermediate courses that Fortran (77 and 95),C and assembly where taught. And even there Matlab was used to illustrate most higher level concepts.
Python, because I know it already, and its my day to day language. As far as I know it has decent libraries.
GNU Octave, as it was reccomended on the Coursera course.
R as it is an open source alternative, no expensive licensing cost, so I can try it out a no risk.
Would I gain anything going to Matlab, as I can see advantages with the others being open source?
"One of the biggest new features for the Octave 3.8.x release series is a graphical user interface. It is the one thing that users have requested most often over the last few years and now it is almost ready."
If you're planning on spending more than a few hours in the tool, consider spending the $200 on the one that's from this century.
I picked up Matlab late and wish I'd done so sooner. It's a REPL for higher-level thinking and quick experimentation. It makes some hard things very very simple and some simple things very hard. They've dropped the price and fixed a lot of shit in the last 5 years. It's replaced a lot of goofing around in Python and Excel for me.
Edit: I have a love/hate relationship with R and more of a love relationship with Python but they don't do ANYTHING like what Matlab does.
http://en.wikibooks.org/wiki/MATLAB_Programming/Differences_...
Love open source, glad they're doing it, much love to the devs -- and life is just too fucking short. If you're broke or screwing around--ok, maybe. If you're sitting in it more than an hour a week, maybe Photoshop is worth the $10/month vs GIMP, dig?
And if you're in school and get it for free or are interested in getting up to speed with the state of the art in machine learning and qualify for the personal license at under 200 bucks? I can't think of a better tool.
I've used both before. Matlab in my CompSci BS and MS and Octave in my professional career.
I'm just making the point that your critique of Octave seems a little harsh, and to the Leyman skews the reality - that Octave is an open source alternative. Yes, a little rough around the edges if you intend to use the visual aspects of the tool, but for matrix multiplication etc it has its use cases.
The parts you say are good about Octave are EXACTLY the stuff I use Python and R for. If you want to use a Matlab library or toolkit, maybe cut in paste some code from a paper, do a quick 'what if' thing -- Octave just doesn't cut it.
(My point in speaking up here isn't to diss Octave it's to re-center the reality around Matlab.)
I've not used Matlab but I've used Octave (during the ml class from Andrew Ng) and it wasn't too awful. Occasionally very frustrating though so if Matlab solves those frustrations I could see it being worth it if you're going to spend time in it!
That being said Matlab is a pretty poor language for anything beyond prototyping and testing proofs of concept. And when I'm doing anything that might turn out become a longer lived project I always choose Python and numpy.
If you want more rigor, check out the coursed he co-created: http://vision.stanford.edu/teaching/cs231n/
This is "on web where all books should be" and claims to "contain very little math" despite being nothing but an endless list of formulae and absurd javascript math that looks like it was ripped out of a BASIC language manual for "how to draw circles" in text mode circa 1978.
Nothing wrong with JavaScript. And maybe this prof is working out some issues his students are struggling with.
But this ain't exactly the Feynman lectures:
var x_try = x + tweak_amount * (Math.random() * 2 - 1); // tweak x a bit
var y_try = y + tweak_amount * (Math.random() * 2 - 1); // tweak y a bitActually, I think a CS grad should be able to read any language as long as it's not completely bonkers like J or brainfuck or malbolge or something. But maybe I'm weird.
Sure, you might not understand all the intricacies of each, but I think the basic idea should be understandable when expressed in any of those languages. But I'm biased, I was exposed to Prolog in college and I learned Lisp and Haskell on my own (to an extent) so it's really difficult for me to judge what is and isn't readable.
But if you can read: foo.map(function (d) { return d+1; }), you should be able to read (map foo (+1)) imho.
Never seem Erlang, Prolog or Smalltalk code.
I am capable of deciphering Perl code as long as its not too obfuscated.
Mind, at the time I graduated undergrad, I had no real idea about Smalltalk. But that was 4 years ago.
Because both seem like odd complaints for a tutorial like this. No, you're not going to do real world work this way, but you're not going to write a game in shadertoy either.
The latter shouldn't be replaced with an apology note for wasting everyone's time and link to UE4.
I've heard stories about some students in France, who are doing a whole year and half of C projects before anything else. Maybe these are just far tales of unicorns.
When the students understand the concepts and what they're interested in, then we can nudge them to focus on choice of languages. Even then it's a dangerous domain as there's so many options.
Deep learning and want to focus on algorithms or only previously had high level experience? Python with Theano is a good bet and can take advantage of the CPU or GPU. Even Python + numpy.
Replicating existing work in the literature and want to take advantage of the some of the existing libraries? Much of it is in Matlab.
Doing something crazy on the GPU? C for OpenCL ...
The list keeps going, but before getting to any or all of those details, the first step is understanding the concepts.
Matlab is a very poor language in and of itself, but it doesn't help that the culture in Matlab (at least whhen I was last exposed to it) doesn't use revision control or write tests. And if you want to productionize a service you have to screw around with licenses on deployment machines, use the painful Matlab compiler, or just reimplement the relevant work. It's just a dead end technical credit card -which may be good for some! Don't get me wrong. But for many many people it just makes a mess.
a) Lots of people are being introduced to programming via something related to the web; if neural networks are 'hidden' in a technology that they won't be using for a long time, a whole lot of people won't get into using neural networks.
b) JavaScript's everywhere - so if something can be shown in JavaScript, a lot more people can see it. And they can try it out themselves, tinker with it, all within software that their computer came with. No $149 software package required.
That said, Andrej Karpathy is a pretty smart guy, and so even if I had serious reservations against using JavaScript for this kind of thing , I'd probably be taking whatever he's doing seriously :)
2) If you want to be grumpy and complain about "what has the world come to". I'd rather use Fortran than C for this specific domain. I'd also rather use Fortran than Matlab.
If it's a bigger project and not the typical "academic toy example" I'd very much prefer Python+NumPy over Matlab (if I was a better Fortran programmer I'd use that but I lack software engineering expertise with Fortran). I can very much live with the performance loss (which isn't even major) if I gain engineering benefits. You can always measure + optimize later if need be.
+I'd rather not waste time thinking about licensing
I've also seen Matlab shitheads who couldn't do it in C either, so it goes both ways.
The point of mentioning that wasn't to "neener neener" a recent grad it was to open the greater discussion you see here regarding CS trends.
Caltech is brutal and spending five years there without touching C seems... odd. How can you spend half a decade driving past the Jet Propulsion Laboratory on the way to Trader Joe's and not pick up the skills to get an interview with SpaceX?