Hacker News Reader

Comment by minimaxir | Hacker News Reader

The title is clickbait, but the problem with the religious fervor of machine learning/deep learning is that entrepreneurs/VC keep arguing that deep learning is magic and can solve any problem if you just stack enough layers. (see also: https://www.reddit.com/r/ProgrammerHumor/comments/5si1f0/mac...)

Meanwhile, statistical methods for non-image/text data with identifiable features can often work better than neural networks, but they are not as sexy. (Good discussion on HN about this: https://news.ycombinator.com/item?id=13563892)

View on HN

It's worth noting that ML doesn't just mean neural networks - those just get the most hype. I'm also curious if you could give an example of traditional models (presumably logreg, SVM, decision forest/trees) outperforming DNN on classification tasks. I read your thread and agree with the points, but I'm curious to see if anyone's outlined examples of when DNN fail in particular (not necessarily RNN/CNN).

Aside - I think it's really ironic for people with no technical know how to go on a talk show and say "learn ML or you're toast". First, the inertia of the world economy is much greater than 3 years. Second, it's unrealistic to ask every 7-11 cashier to go learn ML. Third, programming is a complex art and we're surely not 3 years away from the singularity. Most people I know who have some ML knowledge also realize how far we are from a computer program that can read code, read specs, and then decide how to implement or fix existing code. And last, even if all of the above were moot, I think it's more likely that we write a program that successfully helms a Fortune 500 company before we write one that produces or improves on software. The former could be more easily reduced to a game of chess while the latter requires some massive novel research into "creativity". Maybe it's Mark Cuban who should be learning ML!

>Most people I know who have some ML knowledge also realize how far we are from a computer program that can read code, read specs, and then decide how to implement or fix existing code.

That's exactly the wrong way to think about it IMO. My vision of Machine Programming is that it's a fluid OS/UI that adapts application architecture to how people actually use them. So for example you have "Deep Excel" which is trained first to give the basic functionality of a spreadsheet, but as each person uses it, it gets data about most used/combined UI elements, follow on actions etc... that "rewire" through RL the UX to behave in the way that the user indicates it wants to behave.

So each instance of the program running would be unique to each user and constantly changing to make it more adaptive to their needs. Network these in a deeper system and you can iterate on design and functionality of products way faster.

Sounds cool but have you ever given support to a non tech savvy Excel user? Now add the dimension of an ever changing UI.

Excel is probably a bad example, but you get the idea hopefully.

> Now add the dimension of an ever changing UI.

Didn't MS add a "dancing UI" to office at one stage?

Sounds like a take on the future of programming yc request for startups item. Cool thought.

I totally agree. I don't agree, however, think that this is the only future of programming. The statistical UI morphing may work for UX, but when it comes to algorithms writing novel algorithms to solve mathematical equations without external stimuli... That's just another story entirely, one seemingly far beyond the reach of Super Excel.

I would argue we are already inside the singularity.

As far as ML, we have seen this before. When the tools get cheaper and easier to use, adoption goes up. RoR + cheap cloud servers and easy VC financing greatly increased the number of startups.

Now TensorFlow and the Nvidia Pascal architecture have made ML accessible and fast.

If there is one thing history has shown us, it is that the Hacker News comment section is full of myopic pessimists with an axe to grind (no offense peers!) -- just go back and look at the Dropbox launch thread here.

Even if ML doesn't meet all the hype (it won't for a time, and then later on it will probably exceed it and create things we can't even dream about right now) there will be an entire wave of ML startups, products and jobs.

There is a ton of money to be made on the ML train guys. It's the new social-local-mobile. Much will be shit, some will change all our lives for the better.

I'm old enough now to know how this works.

Edit PS.

I've heard listenable, interesting music completely generated with tensorflow. I have seen artwork that is arguably as good as anything any of the greats have put out generated through similar methods, and we are in the infancy of machine learning.

Machines have beaten humans at chess, go and most impressively in my opinion Jeopardy!

ML is real and it's here and you should leverage it as much as you can, all the knowledge and tools are free.

I'm not saying "don't go learn ML because it's garbage", I'm criticizing Cuban's argument, particularly that the singularity is coming and it's going to replace programmers tomorrow. I gave good reasons. I also _know_ some things about ML, and that gives me insight. Cuban's lack of knowledge affects his arguments on the topic as well - not understanding Calculus and Linear Algebra is tantamount to not understanding deep learning. Not understanding what we are doing today does not prime one for knowing what will happen tomorrow.

It's patently ridiculous to say that we are currently living in the singularity, though, it defies the very meaning of the term.

Cuban is wrong when he says programmers will be replaced any time soon. AGI is still pretty far away, but machine intelligence doesn't have to be general to make a huge dent.

"The accelerating progress of technology and changes in the mode of human life, give the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, can not continue"

- John von Neumann

I would argue that the intelligence explosion has already occurred. Human affairs as we knew them when that line was uttered have been fragmenting and crumbling all across the world.

The internet itself, the current 10 billion or so machines networked together have resulted in a fundamental change in the human condition.

All of us are operating with amplified brains now. We have access to knowledge from the entire corpus of human information at the touch of our fingers, or now with our voice.

I would say human intelligence is already machine amplified.

The runaway reaction has already begun.

Yeah, we just came across someone on HN trying to write a testing DSL with NLP. [0] He's apparently gotten funding for this.

To be frank, the tech community has really been leaving a sour taste in my mouth the last few years. Everyone is anxious to bandwagon on these buzzwords that provide little to no benefit or are applicable only to a small segment of the tech population.

In most cases, even a superficial understanding of the problem space should make it obvious that $This_Weeks_Sexy_Solution is a very bad fit. Do you need a server that persists data and is individually addressable? Then why are you using Kubernetes and Docker, which are still struggling to figure out these very basic things? k8s and Docker have very specific uses, but unless you're Google, they're probably not the right fit for your production environment right now.

This phenomenon seemed to hit a critical mass with document databases and single-page apps, and it continues to iterate with every open-source release that comes out of Google or Facebook. Since Google released TensorFlow last year and the compounded hype of [much worse than advertised attempts at] conversational speech recognition in Siri and Alexa, the "machine learning" bandwagon is starting to try to edge into the spotlight, and it sounds like it's already a mandatory part of any VC pitch.

Fortunately, machine learning is pretty hard and you get into hairy math practically right away, so I don't think the legs on this one will last as long. But we'll see. We'll at least have a lot of faux-ML going around and a lot of people making spurious claims on their resumes.

The economic collisions that make Silicon Valley and the tech industry in general a hive for inexperienced and insecure youth are bearing some really interesting effects this way. How can a company that's not "blown about by every wind of [tech fad]" fully exploit its relative sanity for competitive advantage?

[0] https://news.ycombinator.com/item?id=13572415 ; my reply downthread at https://news.ycombinator.com/item?id=13573978

I've noticed this and I think it's an expression of a more general human urge to feel more important / significant than you really are.

It's the same urge that makes us all think we're better drivers than average, better looking than average, etc. It's the urge that make people move to Silicon Valley/Hollywood/etc., because everyone's going to "make it" even though statistically, that isn't possible. It's why every parent with the means pays through the nose to get their kids into Harvard, why SUV sales are high even though most people don't off-road, and why people buy technical outerwear for climbing, etc. if they never go near a mountain.

Everyone has a constructed image of themselves and we're lying if we say we don't try to burnish it. I think using all this tech is an expression of the desire for status. The problem is, it really can be a time-consuming, frustrating, customer-data-losing, unsupportable, waste of resources. That's bad for business, bad for your health (spending time fixing shit rather than eating well or going to the gym), and it can make your friends and family irritated with you, too.

I'm building a product right now on the most boring stack possible: python, postgres, pyramid. I've been there/done that on the whole sharded-MongoDB-Redis-Memcache-900 machines-AWS-Docker-startup-ops nightmare-use all the things (we actually used every single thing in that list at my last job). It was unreliable, a huge pain in the ass, a little bit fun, but ultimately quite counterproductive. These days, my bets are on things like the JVM, the .NET CLR, and Postgres. As Knish from Rounders would say, "My kids eat".

Another thing to keep in mind: a lot of this is resume padding. Yes, this is completely unprofessional, reckless behavior, but it happens, many recruiters are keyword-driven, and as interviewers many of us are impressed with a keyword-laden resume. We do this to ourselves as an industry in how we hire, who we let speak at events, and who we praise. It's nobody's fault but our own.

I was amazed to see this comment, like a meteorite of sanity crashing into HN.

My sibling post is on the right lines - a lot of people in tech are chasing a delusional self-image - though I don't think that's true of everyone. People can make the effort to be self-aware and honest with themselves. I think the rapid expansion of tech in the last few years attracted many, many youngsters, hucksters, delusional types, and so on, and people with maturity and experience became a minority.

For whatever reason, statistics seems to be in the midst of a serious branding problem. Maybe traditional frameworks of inference are too conservative for non-researcher's tastes?

I'm curious what Fisher, Neyman, and Pearson would say about the current state of the field. Especially considering how often they and other statisticians disagreed with each other throughout the 20th century.

I believe your branding hypothesis is correct, and all of this hype around neural nets and machine learning is creating a startup ecosystem ripe for market correction. I've heard a million times that most machine learning problems are solvable by regression, and I expect to hear that a million times more.

Regression is great if your output is a linear function of your inputs. Neural networks don't have that limitation. For a long time people have assumed that linear functions are the rule and nonlinearity is the exception, but the fact that nonlinear predictors are trouncing linear ones in many domains seems to be proving that false.

As for deep neural networks, in my opinion they are just an ensemble of neural networks, and ensemble methods have been shown to produce best-in-class results almost regardless of the base learning algorithm.

The only real advantage linear modeling has is marginal effects can be determined easily. This matters for some business intelligence / decision making applications but if you want to build a model with as small errors as possible there's really no reason to restrict your model to be linear in the age of massive computing power.

Well it's also easy to model. NN needs lots of data from what I've read.

Also the whole sport analytic prediction industry mostly uses linear regression... But next hype thing is medical data for them but they're super slow in adopting new thing. Industry seems boring if you're not into sport and into modeling in general not just linear regression.

Regression beats neural networks when you care about your model parameters.

> For whatever reason, statistics seems to be in the midst of a serious branding problem.

Many of them don't know how to code. And the stat program of most of these universities doesn't require any real coding classes. The professor give you code templates and you tweak it to get the result. But the stat classes are really good to handle data.

Data science in general like Chapman Hill and UC Irvine seems like a really wide range of stuff from data analyst, a lil programming, data visualization, etc... I don't feel like it goes deep enough in what Statistic does imo.

Most are stuck with Tableau and SAS. Pharmacy companies uses SAS. Startup companies and tech companies uses Python and R.

I'm speaking from one data point sample, my current stat grad program. My undergrad is comp sci and I did a few years in the tech industry before coming back to grad school.

> I'm speaking from one data point sample, my current stat grad program. My undergrad is comp sci and I did a few years in the tech industry before coming back to grad school.

If you don't mind sharing details; why'd you choose your current grad program?

I have no experience with the entrepreneur/VC side of this coin, but Data Scientists know the differences between traditional regression based ML and CNNs. We know that doing image recognition or NLP is best suited to nvidia/tensorflow/keras/deeplearning/buzzword/buzzword. We also know that SVMs & Random Forests & regression lines work just as well as they always have to make predictions based on your click-thru data.

Maybe we should explain the cost/benefit of the buzzwords vs. the science?

Your comment sounds wise, but with all those acronyms and technical terms I can't be sure :-)

It's simple. More buzzwords equals more money. So whatever it is you're really doing, just find some way to semi-plausibly attach the latest hot buzzwords to it.

Most of the "dinosaurs" around here should know by now that you don't get paid to do what works; you get paid to do what the boss/customer wants. (And that's not always going to be exactly what they asked for, either.)

As long as you continue to throw money at me, I'll continue to chase your wild geese, hunt your snipes, fish for your red herrings, and joust with your wind turbines. Most of us here can learn fast enough to always stay one or two steps ahead of the boss on whatever topic they might latch on to. And that becomes genuine experience if it somehow manages to graduate past the business fad phase.

To say something in defence of the deep learning religion: it's a very flexible framework for expressing lots of different ideas and the ability to train all those ideas jointly if you can frame them in terms of SGD updates is very appealing.

And even if you don't want to use SGD to fit your model, the deep learning machinery can be reused for other things like bayesian inference, eg PyMC3 and Edward are built on Theano/TensorFlow respectively.

There have even been papers showing differentiable trees integrated into the deep learning framework. There is work on how to input graphs.

Maybe this is just my own desire for unifying frameworks shining through, but joint training seems to have real benefits and feature learning has clearly had a large impact on reinforcement learning.

Maybe it's not the best tool for day to day data science work, but that tends towards what tools are accessible and good enough, rather than what is best.

It's not even that clean. A lot of the buzzword impact causes mass confusion on the applicability of techniques. Lots of people are trying to apply deep learning techniques on too small amounts of data (and truth) with fairly predictable results ... and throwing around at best questionable performance numbers with little validation.

CNN's are great for some image recognition/classification problems, but that's just the start of the conversation.

Sexy is in the eye of the beholder. Seriously, I think the problem of ML is that it often goes against the so-called principle parsimony, which plays a critical role in statistical modelling. Especially in many areas where the whole point isn't strictly to predict future outcomes, but to understand how things work and to test hypotheses. Understanding leads to better prediction anyways. If you can predict something but you don't know why or how, I would take such predictions with a grain of salt.

I think you have the cart before the horse regarding understanding and prediction. Prediction precedes understanding in importance and sometimes methodology, since if you can't predict outcomes correctly your understanding, however well fleshed-out, will be inaccurate.

Interestingly, the parsimony principle is lent empirical weight by bad results obtained from overfitting.

Machine learning enables us to multiply many more entities than we can using our conscious thought processes. The various image recognition models use many variables that enable results impossible with other techniques. It does not violate parsimony if you can obtain better results on a large data set.

I can predict that if a car drives by it will not teleport to some other location, but will rather continue along its path. I've been able to predict this since I was a child. I did not need to study the laws of motion to do so. I would not take such a prediction with a grain of salt.

In many cases, an acceptable prediction plus understanding is better than a more accurate prediction alone. And I'd say the example in your last paragraph is an example of understanding, by the way, or you would not have been able to express it in abstract terms. (For a young child, an animal or any other simple mind it may be possible to get the prediction without understanding, I guess).

I think that's because the majority of folks using ML care only about discriminative ML, not generative. A black box ML technique that delivers low predictive errors is good enough. To those ends, parsimony is largely irrelevant.

I suspect that others using ML, who are inclined to test hypotheses, induce cause, and discover unknowns (via generative ML), are far fewer.

So it's hardly surprising that the eye of discriminative MLers dominates in defining 'sexiness'.

Just call it "statistical machine learning" or some other vacuous term and people who know enough will respect it (maybe after rolling their eyes a bit), and people who don't will be happy to hear you are using machine learning.

I call it the Jimmy Johns method of machine learning - just keep stacking those layers.

Once it's good enough to solve as many problems as VCs claim, you won't need anyone to program it :p

Reply on news.ycombinator.com