AI and Deep Learning in 2017 – A Year in Review | Hacker News Reader

AI and Deep Learning in 2017 – A Year in Review

wildml.com

188 pointsMrQuincle6 comments

the fact this thread got over 100 points without a comment or discussion proves what I suspected.

most of us are observers when it comes to this tech. sure, there is some quick tutorial to build a "learning model" somewhere or read an article to grasp what is being discussed...

but when it comes to contributing something remotely significant, we got nothing. whether it is due to lack of resources (giant datasets and computing power) or lack of knowledge and experience to even theorize something plausible.

this is even more frustrating knowing how important of a role this tech will play in the future.

being a passive observer is not enough.

I'm looking forward to 2018 AI summary.

The summary illustrates another point - the overwhelming majority of DL applications are on visual and sound datasets. Outside of the Facebook/Google bubble, images and sounds are not that important. On Kaggle, for example, gradient boosted trees tend to dominate the structured data challenges, but the gradient boosted tree "revolution" is happening unnoticed.

I think the combination of a) more processed datasets than google/Facebook can sometimes have, b) smaller cpu/gpu resources that google/Facebook have, and c) the incentive to squeeze every last bit of log loss out, all encourages you to ensable all your independently trained models under a gradient tree.

Often the challenge is getting a big data set to play with. My personal round-to-it is using my home security system cameras to create a training set for licence plate collection, and gait recognition.

I think for most people that hurdle of understanding is immense, and isn't likely to get any better. ML/DL infrastructure will continue to improve and deploying AI systems will become more accessible as time goes on. But I don't see familiarity with (non) convex optimization, statistical learning, topology, etc. becoming mainstream. Without these its hard to see a reasonable path forward for AI.

For me this year in DL was a lot narrower. it mostly revolved around systems becoming more human in capability. I think most of this is in the article already:

We can now hopefully say that ImageNet has been solved.

AlphaZero tackled structured games with a single algorithm [1].

Tacotron2 produced completely passible TTS [2]

GANs improved dramatically and saw some good theory to back it up [3] [4].

We also started to care more about how well our models do in general, which to me shows maturity:

Adversarial examples showed their teeth, and hopefully convinced everyone to care about robust models [5] [6]. Reinforcement learning algorithms were shown to have poor transferability [7].

In 2018 I hope to see a new kind of CNN. Residual style networks are the norm now, since they mostly solve problems with gradient flow. But take away all the skip connections and we're mostly left with a vgg-style linear net with box filters. I'd be really excited to see a network with image-sized conv filters that could adapt their shape (and therefore representational power) to a given feature or signal.

Hopefully 2018 is the year where people stop calling AI a one-trick-pony. I don't hear it as often these days, but I think its time to put that phrase to rest.

[1] https://arxiv.org/abs/1712.01815

[2] https://research.googleblog.com/2017/12/tacotron-2-generatin...

[3] https://arxiv.org/abs/1710.10196

[4] https://arxiv.org/abs/1701.04862

[5] https://arxiv.org/abs/1707.07397

[6] https://arxiv.org/abs/1710.06081

[7] https://arxiv.org/abs/1709.06560

If hinton pivots too like Jeremy Howard to cryptocurrencies, we will know a new ai winter is upon us.

Reply on news.ycombinator.com