Hacker News Reader

Comment by stiff | Hacker News Reader

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any understanding of anything particular about deep learning, just general derogatory comments "not used in industry", "overhyped", pointers to whatever someone heard in an undergraduate ML class on older types of networks etc.

Maybe if you don't have anything on topic to say, just do not comment? You really are not obliged to have an opinion on everything.

(Waiting for the downvotes)

View on HN

>not used in industry

Well, that ones about me. Yes, I have a plenty of experience in machine learning, including undergraduate research in neural networks, a graduate degree in machine learning, and more than five years of industry experience (including several years building some of the most utilized neural network models in industry). I have read many of the deep network papers in detail, and have played around with them on actual data.

And yes, I think your comment deserves to be downvoted; unlike those of us with insight into the issue you added nothing to the discussion other than derision. It bothers me that comments like yours end up at the top of so many threads like this.

edit: and I'd like to point out as someone in the industry I have a good reason to temper expectations. Undeserved hype leads to bubbles, and bubbles create collateral damage when they pop. The AI industry has dealt with this at least twice already, and I don't want to see it happen again. The results so far are extremely exciting, but deep networks still need to prove they deserve the hype.

What kind of insight into whether deep networks are interesting or not does the fact that they are (supposedly) not used in the industry bring to the table? To me, insight would be something like "they do not generalize well unless you have really a lot of data" or anything concretely related to the research done on them.

By the way, I am not asking anyone what their degree is, I am just asking to bring arguments and experiences or stay neutral.

You missed the point of my comment. I was speaking about the hype, I wasn't putting down deep networks or making a technical analysis of the algorithms. Anyway, here are my major reservations, including my technical analysis. You're oddly demanding in a casual non-technical online discussion about a topic you don't seem qualified in, so if this isn't enough for you we really aren't going to find common ground.

There isn't great library support for deep networks. This is a big deal, I don't want to spend tons of time building my own library or working with buggy/poorly supported/infant libraries. In production systems we prefer extremely well-established libraries that work in our language/environment of choice. Also deep belief networks are a couple orders of magnitude slower than linear models (probably the most commonly used type of model in industry). They require more parameter selection. They're not even useful for a lot of tasks - if I'm already spending tons of time building useful features (often a requirement in industry for non-technical reasons, like reporting or legal constraints) deep networks aren't going to be very useful. Much of their utility is taking raw, unstructured data and creating useful features for a supervised model. You can't easily interpret them as models unless you are working with visual data, they are a black box.

Instead of doing an obvious ad-hominem (Yes :D, this is one of the times it is genuine!), can you address what the other commenters are pointing out?

I have degrees in Physics, EE and a PhD in CS. I munched axiomatic set theory and infinite ordinals during the course of my PhD. I dabbled in theoretical machine learning for 3 years. See, I can play the credentials game too.

But does that address the fact that they are not used in the industry? AI is full of charlatans and broken promises. Sadly, by listing "deep learning" alongside Deep Blue and Watson, it seems more charlatany.

I am not playing the credentials game, quite the contrary, I remember trying to understand DBFs while at the university and failing miserably due to the complexity of the subject. I am also not defending deep learning in any way or having any stance about the subject myself. I just think in a place like HN you should not criticize technology without having practical experience or technical arguments, and of course "not used in industry" is not a technical argument. With Google just yesterday hiring George Hinton who lead most of the deep learning research and with Jeff Dean working on it there already[1] it is also overall a rather weak one.

[1] http://research.google.com/archive/large_deep_networks_nips2...

1. Google hiring X is not the same as X's model being scientifically valid.

2. Model X for intelligence being highly technical or cool in some mathematical way is not scientific validation.

3. Google hiring X is not the same as X's model being successful in the industry. I have long switched over to DuckDuckGo for technical queries.

Anyways, what AI people should first address always is point number 2.

AI has always jumped from one cool thing to the next without answering whether that cool thing has any scientific basis.

Don't bring another AI winter ;)

It is always cool to see excitement over research in AI! (As long as it does not drown out other competitive approaches which might bear fruit in the long run.)

1. Google hiring X is not the same as X's model being scientifically valid.

That's exactly what I am complaining about.

2. Model X for intelligence being highly technical or cool in some mathematical way is not scientific validation.

I never said that, it just narrows the amount of people that can comment on it with merit.

3. Google hiring X is not the same as X's model being successful in the industry.

It was just a side-comment.

Nobody has ever convinced me humans are intelligent. Every specific example of human intelligence put forward, it eventually ends up that machines do it better.

How is one to scientifically validate against something that can't even be defined?

>Nobody has ever convinced me humans are intelligent. Every specific example of human intelligence put forward, it eventually ends up that machines do it better.

First of all that is completely wrong even for simple things like image recognition (try building a face recognizer which works under all possible conditions).

But more starkly consider the following question:

Is Geoff Hinton a machine?

The downvote is amusing. Care to engage in an interesting conversation rather than resorting to drive-by-downvoting? I guess I touched an sensitive point with the Google fan boys here. Sigh.

>of course "not used in industry" is not a technical argument. With Google just yesterday hiring George Hinton

Again, you're quoting me on that. Yes, 'not used much in data science' is a valid argument that it's not one of the biggest breakthroughs in data science.

And if you want to discuss the topic (while blanket criticizing people for not knowing what they're talking about) at least get the father of deep network's name right: it's Geoff Hinton, not George.

The OP title include "The Biggest Data Science Breakthrough of the Decade", the topic has already stopped being "is this interesting" but rather "is there something to justify the hype" or, surprise, "is this 'The Biggest Data Science Breakthrough of the Decade'"

Sure, I only have a graduate math degree and only follow these latest developments casually and perhaps I just miss exact way this newest artificial neural network stuff is really that different than the older stuff. But the only thing that's being touted is a NYTimes article. As another poster said, if you'd like to add to the conversation, give us some "meat" here.

My small exposure to ML also left me feeling the whole train, test, operate cycle is a pain in the neck.

All (good) statisticians validate their models. Nothing new there.

The main difference between these newer networks (besides much improved performance) is that the algorithms can handle "deeper" networks better (more hidden units). If we're talking about Deep Belief Networks, they're not much like the old ANNs. DBNs are generative probabilistic graphical models using Bayesian inference.

Conceptually, going deeper (LOL) allows the networks to learn higher level concepts. For example, a 1 layer ANN (perceptron) can only learn linear functions, while a deep network is able to internally form a belief of what, say, a cat is.

More technically: Much of the work in ML is deciding what your inputs (features) should be. When classifying text documents, should you use word counts, bag of words, word stemming, character counts, etc. Should the model be linear, polynomial, gaussian, trigometric, etc. Deep learners try to automatically do feature selection and control the degrees of freedom in the model for you.

Also, deep learning is catching on in some industries. It has recently had huge successes in speech recognition, and all major companies developing this technology have started using it (e.g. Siri for one).

At the risk of being a bit annoying... regarding speech recognition, as said here

https://news.ycombinator.com/item?id=5376319

they use DNNs not DBNs (DBNs only used for pre-training, sometimes). Also if you read Microsoft's paper Table2, 7 hidden layer networks, which clearly qualifies as deep, work just fine with Back-propagation. Just a bigfat-MLP, no preprocessing! but 17.4 Word Error Rate (WER) vs 17.0 WER for DBN pre-training.

>they're not much like the old ANNs. DBNs are generative probabilistic graphical models using Bayesian inference.

MLPs (DNNs) can also be interpreted probabilistically. Just a directed model where inference is attained by marginalization of the hidden binary nodes in a layer-wise manner and by using a naive mean field approximation. All that to say the classic "forward-pass" ;).

Also could you indicate me a source confirming that Siri (Nuance) also switched to DNNs?. I am interested in that.

>Conceptually, going deeper (LOL) allows the networks to learn higher level concepts.

That is the really interesting part!. Now, I have not seen a proof for that. Wondering at individual neurons modeling individual features of e.g. a face or so is also a trend of the 90s and does not count as proof. I said this because it is what I usually hear.

Until now the justifications I saw for multiple layers of perceptrons being suitable for modeling arbitrary high level abstractions are reduced to

1) MLPs are universal approximators. This in my opinion is a superficial argument. GMMs also allow modeling "any" distribution and Taylor series any linear function, but in reality there are physical limitations to this argument. Maybe is true if you had a billion layer net, but will you get there?. If you had that computing power maybe a more realistic modeling of the brain might work better

2) They resemble how brain architecture works and similar arguments. Which I am fairly sure is not true. There are more human-brain based approaches to AI like e.g. cortical learning algorithms and those just seem to model that stuff to a certain extent.

It is purely a matter of opinion whether DBNs are "overhyped" but I hope you would agree that they are currently being "hyped". And I hope you understand how this can actually damage the potential of what is likely some very good technology like say like what happened to neural nets. :)

It is also a matter of opinion how widely they are being used in industry. Certainly they are being studied in many companies but they do not a appear to used much in production because of their complexity and high training cost. This is still cutting edge technology.

In my experience most professionally trained mathematicians and statisticians are still pretty skeptical of these claims. Wouldn't you agree?

It is OK to be skeptical, all I am trying to say is that most of the comments leave the impression of the poster trying to lean over backwards to say something at all related to the topic, often ending up with generic truisms. For example how is model building being a small part of practical ML criticism of deep learning? How is it on-topic at all?

Not my comment but I think they mean that it is a relatively small part of the overall process and thus not "The Biggest Data Science Breakthrough". Maybe not a fair criticism but they have a point.

I would love to see a breakthrough in data cleansing or how about just standardized coding, labeling and formatting. Unfortunately I've used lots of 3rd party data sources and wasted more time on these brainless activities than I want to think about. Consider yourself lucky if you only work on web logs where you control what they look like.

rather than being so aggressive and waiting for downvotes you could choose to be constructive ;) and answer a concise technical question to a sub-thread that you yourself started

https://news.ycombinator.com/item?id=5377101

I haven't commented although my field of study is ML, but I have to say this is one of the shallowest infomercials I've seen hitting the front page here at HN.

They say absolutely nothing of relevance other than how awesome it's supposed to be, and oh by the way this is Cloudera and it's great, and I happen to work in Kaggle and it's magnificient. After preying your personal data to let you listen to the infomercials.

After 10 minutes, not a single slide apart from the cloudera ad has been shown. This is a podcast.

Oh that's annoying. When I did the webinar I was sharing my screen and showing both slides and scribbles in onenote. Is there no button to view that in the recording? Sorry I can't check myself now, I'm on my phone.

Reply on news.ycombinator.com