How Convolutional Neural Networks Work

I setup an automated script to setup a AWS g2 instance, train my neural net using tensorflow, copy my model to my personal computer, and spin down. It costs like $5-$10 to train/test most neural network models. My most expensive model cost like $100 and required a ton of time and resources. It took like 4 days or something.

You really dont need $2k workstation..

Of course, for personal use I do have a gtx 1080 because I like to game and play tensorflow/caffe

It's a better start than having no visibility into the topic at all.

With that number you can train a model from scratch. Alternatively, you can retrain the last layers (it requires much less data).

The easiest way is to use some well-known architecture (e.g. VGG16) and go. See: https://github.com/leriomaggio/deep-learning-keras-euroscipy...

Oh, if you don't want to get into the nitty gritty of it, you should use Digits (https://github.com/NVIDIA/DIGITS).

This is the easiest way to setup a CNN and train it with your sample images (at least compared to Caffe, Tensorflow, and Theano). I say that because it's all GUI based! Real convenient.

OT, I'm interested in this field as well. Are all these pictures USA cars or international?

Do you mind disclose where you got them?

Thanks!

As I said in another thread, I used that (https://www.tensorflow.org/versions/r0.9/how_tos/image_retra...) which seems to work well. I think especially with your problem since it is similar to the type of classification of ImageNet.

One thing that might work well with a CNN is geospatial data (easy to consider as an image).

It comes down to the characterizing architecture of convolutional nets, that is weight sharing, and the assumption on data this makes. If by image one means something where you can expect any pattern (at some level in the hierarchy) being equally likely to occur anywhere across an input dimension, then yes this is true. Personally I would say that this is too narrow of a definition of an image (too great of an assumption), and, interestingly enough, perhaps too broad too. I am not a pro.

[Edit] Too broad in the sense that, intuitively, there is perhaps an implied assumption of continuity of the input function defining the image. Note that such assumptions can be made explicit with various so-called statistical priors incorporated in the network.

If you google "Hinton machine learning" on youtube, you will find hinton's lecture's they are non-mathematical, he is a psychologist/math guy, and he is the inventor of almost all this stuff, backprop, drop-out,

You will find his lectures to be very entertaining and easy to understand, being a psychologist whose desire is to make a computer operate like a human brain, he's more interested in how the brain actually works, than hacking ML code.

Hinton describes backprop, why he invented it, and exactly how it emulates the way the human brain works.

Hinton now works at microsoft, he is considered the modern day 'godfather' of DeepLearning/ML

The underlying math was figured out a long time ago, but it's only been in recent years that we've had the computing power to test these out on lots of complicated, real-world classification problems, and had some incredible success.

Computing power, but also some implementation tricks that turn out to make things a lot better.

For example, activation via ReLU instead of sigmoid/tanh significantly improves the performance of deep neural networks.

Then there's stuff like BatchNorm, Pooling, Dropout etc...

1. Computing power 2. Data availability 3. Fast, large local storage

Decades ago I played around with neural nets but was frustrated because I either had to preprocess and normalize my inputs to the point where I didn't need a network anymore or I had to train a large network with so much data that it was not practical.

Having a cookbook approach with a catchy name and orders of magnitude more processing power have revived neural nets and now they are finally doing something useful.

Now everyone is jumping on the bandwagon so the field is progressing very quickly. Just because it's hyped doesn't mean it's not worth giving it a second look (although I'm still on the sidelines myself.)

The basic CNN structure was in place, but as the saying goes, "The Devil's in the details." Early CNN's were applied to problems such as handwritten character recognition with rows of small grayscale image cells as inputs, and were much shallower, smaller models. Today's CNN's operate on full resolution, multi-channel images and video, and can be orders of magnitudes deeper and larger. For instance, ResNets have been proven to demonstrate monotonic performance improvements out to 1200 layers on benchmark datasets. This would have been unthinkable even a couple years ago. By way of comparison, even the state of the art VGG network architecture of a couple years ago originally had to be trained in stages to reach 16 and 19 layers for submission to ILSVRC 2014 (Xavier / MSRA initialization makes this unnecessary now). At the time, VGG and GoogleNet (22 layers) were considered to be extraordinarily deep CNN's.

I argued back in 2000 (a year after I got my computer engineering degree) that AI wouldn't take off until computing moved from single threaded/single core to multithreaded/ multicore processing. The fact that we are only hearing about this stuff 15 years later makes me feel that that assertion was largely right.

The biggest problem I see in AI is that the algorithms are generally fairly straightforward, but people haven't had the computing power to explore the problem space. We are seeing drastic improvement in things like video cards (routinely 1000+ cores) and data processing locality (map reduce). But processors have stagnated.

If we really want AI in any reasonable timescale, we need large arrays of general-purpose cores with a sane communication protocol that doesn't fixate on things like caching, we need a hybrid between Go and Erlang to do concurrent functional programming in a readable way with automagic scaling over a network, and we need all this yesterday. The fancy schmancy AI algorithms will become apparent when processing power is no longer the primary limitation, and at that point we can optimize them.