Hacker News Reader

You're seeing hate for deep learning on HN!?! I'd love to see something approaching critical thought! Most of the comments I see are silly fluff about the singularity and/or AGI and killer robots.

But seriously: I think you're overstating the case. I've enjoyed watching the fast.ai videos, and I think deep learning is a clear choice for some areas right now. Many others, no.

If you're working in a data-limited domain, in particular (i.e. most of them) there are probably better first choices. I know plenty of folks who have used deep learning to achieve no better than comparable results to the simpler methods they were using before.

But yes, the initial engineering costs of a deep learning system have come down a lot further than I thought before I watched your videos.

It is world changing tech. If skeptical, wait and see.

> You're seeing hate for deep learning on HN!?!

Most of the hate for deep learning on HN is usually against using it in titles as clickbait. (i.e. just simply taking a pretrained/predefined model and changing the source dataset without any optimization)

The in-depth explanations of mechanics discussed in deep learning as noted in fast.ai courses and this submission do justify the use of the deep learning moniker, though.

A pre trained deep net can work very well in a data limited domain if there's sufficient overlap.

Yes, but that's true of any ML approach.

Deep learning is great if you're dealing with a hugely dimensional problem, and you have the data to train the model. If one of those things is not true (and usually, one of those things is not true), you're better off starting simple.

That's an excellent point and one that kept me from trying the deep learning approach for a long time. But in the end the machine + rudimentary deep learning provided its own dataset and that really made it work.

So even if I didn't have the data to train the model I had enough data to bootstrap the process and sometimes that's all you need.

Yes, agreed. Your use case was interesting to me in the way that you bootstrapped the problem. If only more domains were like that!

>Every time I've heard from someone who has switched from a manual feature engineering approach to deep learning I've heard the same results ... dramatic improvements in accuracy, generally within a few days of work

It bothers me how often otherwise rational individuals continue to be susceptible to survivorship bias.

Of course you'd only hear about the successes, the failures are either embarrassed or know better than to tell you their story.

The people who tried it for several months, saw accuracy cap out at something like 60-70% for processes that need something like >90% confidence to justify expenses, and which proceed to get ignominiously shifted into a different team (or fired) for building what the higher-ups view as a massive waste of money at your salary and the other engineers see as a data-guzzling black box. No, these guys aren't likely to tell their story. Or, at least THIS story.

The story you instead get from these guys is how

>After having had some fun times doing deep learning and data analysis, I'm excited get into the big new field of $THING_THAT_WAS_HIRING

and all the other dross that modern vocal programmers use to mask any possible scent of failure and actual on-the-job difficulty experienced.

This goes pretty much for every tool though. Applying a tool to a job it isn't well suited to is always going to come out as a frustrating experience, be it a hammer or a piece of software.

What I always try to do is to get a feel for the problem space trying different methods with as little investment as possible. That way you will - once you decide to go full power after a certain solution method - at least have a feeling that you are on the right path.

Dogmatically trying to shoehorn every problem into the toolset that you know how to use is a way to stay reasonably productive but it rarely leads to optimal outcomes, sometimes you simply have to learn how to use a new tool in order to get to the maximum.

I'm the last person to jump on new bandwagons, still have a dumb phone, don't use facebook and still run my own mailserver. Even so, when a tool has a significant and most importantly measurable advantage compared to the tools I'm already familiar with I'll adapt.

I see plenty of stories about "$POPULAR_THING is not the solution and here's why". If it's true that HN has been hating on Deep Learning for the past year, there should be a market for articles explaining why or when it doesn't work. If someone started out using Deep Learning and then switched to something else, that seems like an excellent subject for an article.

Good points. How about you the following though: driven by all the hype surrounding deep learning and AI, consultants are currently jumping on this new thing they can sell to management without understanding much of the intricacies behind it, so even after facing many disillusions just a few years ago when the talk was all about big data, companies are now jumping towards investing in products promising some of that sweet deep learning magic.

That's fine, but the thing is they once again plan to apply it on their badly maintained data warehouses (now often somewhere in their dusty hdfs stack) to build traditional predictive models. Customer churn, next best offer, propensity modeling, that sort of thing.

I try to keep telling them that with structured data sets, and simple classification problems, a random forest or God forbid, a logistic regression model, would do just as well given that you spend some solid time in feature cleaning and preprocessing. Am I wrong in this setting? Perhaps I should also jump on the deep learning bandwagon and start selling 10 layer binary Classification networks :).

Not disagreeing with you, but I do fear that many traditional industries will be jumping on board pretty soon without any of the actual use cases to warrant deep learning.

Jeremy your fast.ai course was amazing. Any idea when/if there's going to be a part 2?

Part 2 recording is finished! Just need to get it online :)

Can't wait to see it! Part1 was one of the best courses I have ever taken, it is so rewarding to have a motivated teacher like yourself.

I can't recommend this comment enough! So many people give outdated advice with regards to deep learning like "it requires too much data and takes too much time" and "start with something theoretically grounded (tm) like an SVM".

The truth is that an imagenet pre trained network will often work astonishingly well. The only other mark against deep learning is computational complexity, which I expect to go away soon (yours truly ;) )

> And there's been a strong reaction from those who are heavily invested in SVMs/kernel methods, bayesian methods, etc to claim that deep learning isn't theoretically well grounded (which is not really that true any more, but is also beside the point for those that just want to get the best results for their project.)

Could you elaborate more on what has changed recently in the theoretical grounding of deep learning?

It's not so much that there's just one or two specific results, but rather that there's just far more researchers working on this now, and quite a few are working on more theoretical stuff. Sometimes, that theory results in really practical outcomes - a good example would be https://arxiv.org/abs/1701.07875 . Or another by the same researcher: https://arxiv.org/abs/1506.00059 .

We've seen, in the last year or two, interesting results in nearly every area of theoretical research of deep learning, including generalization, optimization, generative modelling, bayesian models, and network architecture.

I've really enjoyed the Fast.ai videos and am in the process of using it to build some hardware/software solutions. Even 2 years ago, it was not possible to get started on a problem like this and solve it in a short time.

Except that the guys system doesn't work, at present. He will probably need both feature engineering and machine learning approaches.

What gave you that impression? It works just fine for my use-case, and I will most likely be able to add more use-cases by virtue of being able to enrich the data from other data sets out there once I have the part identified.

It is possible that at some point to get to the last .1% or so I may have to embed features that were engineered but right now there is no indication that that will be the case.