Comment by agibsonccc | Hacker News Reader

Comment by agibsonccc | Hacker News Reader

Parent Full thread

I would seriously look in to deep learning. http://deeplearning.net/ I am doing everything with it now. This includes principal component analysis/compression, face detection,hand writing recognition, named entity recognition, clustering, topic modeling, semantic role labeling, among other things.

There is a very common structure to this. Despite neural nets having their own baggage, they're worth understanding.

The structure you're wanting is definitely in there. Edit: Yes a bit of self promotion here. Just making a point with patterns I've found as I've built this out.

See:

https://github.com/agibsonccc/java-deeplearning/blob/master/...

https://github.com/agibsonccc/java-deeplearning/blob/master/...

Half the battle is understanding the linear algebra going on here. Beyond that you can pretty much do everything with one set of algorithms and terminology.

For those who go WTF java are you insane? The core idea I'm linking to here is the fact that deep nets are composed of singular neural networks with slight variations having a very common structure for both the singular layer as well as the deep nets themselves.

I wouldn't claim the Neural nets don't work. We've seen many use cases where they do work (And I have looked at deep learning just a little, it may superior in ways but it doesn't seem in any way fundamentally different from the other stuff).

I would add that Support Vector Machines also work and they are similar and have a much clearer math to them [1]. But SVM and neural nets are ultimately just linear matchers on a nonlinear pattern space, they ultimately involve adhoc choices that experts learn over time.

As I said above, once you learn the maths (linear algebra, statistics, functional-analysis or whatnot), it become less basic understand and more "understanding how", a series of tweaks that experts "with a feel for this stuff" do. But this "feel" level understanding seems exactly what stands in the way of serious, rational progress on the topic.

[1]http://en.wikipedia.org/wiki/Support_vector_machine

Just today, I finished a deep-learning example in C# with Accord .NET. It solves a simple XOR function, as well as classifying ascii digits.

If you're looking for a basic example in .NET, I think the XOR one is as simple as it gets.

Deep-Learning XOR: https://github.com/primaryobjects/deep-learning/blob/XOR/Dee...

Deep-Learning Digits: https://github.com/primaryobjects/deep-learning/blob/master/...

Very cool stuff. A .NET implementation is a great project. If you want any advice, I'm working with a friend on a hadoop version as well.

https://github.com/jpatanooga/Metronome

We have a wiki and some resources put up.

Very interesting!

Is that Word2Vec implementation you have roughly equivalent of the Google version[1]?

Any examples of how to use deeplearning4j generally?

[1] https://code.google.com/p/word2vec/

Binary compatible yes. Star the repo and watch it in the next few days. Example apps are on the way. I plan on implementing a full "easy to use" machine learning lib around this.

Edit: Poke around in the tests, Here's an example of it learning a compressed version of MNIST: https://github.com/agibsonccc/java-deeplearning/blob/master/...

I have a lot more example usage in each of the tests. Test coverage was a higher priority above the documentation, but example usage is there. I'm more than happy to answer emails around the usage of the library as well. I also take feature requests.

I plan on implementing convolutional nets, recursive neural nets and some other ones based around that same structure; That includes the scale out versions with akka for easy multi threading or clustering ( I have built in service discovery with zookeeper among other things in there.

Advice question - how far into the linear algebra should I go? I'm currently working my way through a book - and it's not too bad, but I really don't want to invest more than I really need. Any suggestions?

The main thing to understand is the basics. For example element wise multiplication vs full blown matrix operations. Think something like this: http://en.wikipedia.org/wiki/Matrix_multiplication

Matlab/Octave is a great way to practice this due to the native data types. If python is your thing numpy's arrays are also pretty easy to digest.

Subtle little tricks like this: https://www.youtube.com/watch?v=evF-3ykjRU0

And understanding the dynamics of scalar operations vs matrix - vector operations.

The machine learning class has some good fundamentals if you need a refresher on how something works.

There will be more complex things like some optimization algorithms have different uses for eigen values: http://see.stanford.edu/materials/lsocoee364b/11-conj_grad_s...

See: http://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors

One other thing might be understanding different ways you can manipulate data. In this case, numerical representation here is an example per row when I toss in one matrix for training. This is applicable to many machine learning problems.

Very much appreciated. That's really helpful and feels very doable. :)

You could look at the appendixes in texts by Bishop, MacKay, Barber and Rasmussen/Williams to see what they expect (and they expect a pretty thorough understanding. The last 3 are freely available content

http://www.inference.phy.cam.ac.uk/itila/book.html

http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf

http://www.gaussianprocess.org/gpml/chapters/RW.pdf

Also, an excellent list of ML resources: http://www.reddit.com/r/MachineLearning/comments/1jeawf/mach...

Reply on news.ycombinator.com