Hacker News Reader

Comment by ma2rten | Hacker News Reader

I think this field is suffering from some confusion of terminology. In my mind there are three subfields that are crystallizing that each have different goals and thus different methods.

The first one is Data Science. More and more businesses store their data electronically. Data Scientists aim to analyze this data to derive insights from it. Machine Learning is one of the tools in their tool belt, however often they prefer models that are understandable and not a black box. Sometimes they prefer statistics because it tells you if your insights are significant.

The second one is Machine Learning Engineering. ML Engineers are Software Engineers that use Machine Learning to build products. They might work on spam detection, recommendation engines or news feeds. They care about building products that scale and are reliable. They will run A/B tests to see how metrics are impacted. They might use Deep Learning, but they will weight the pros and cons against other methods.

Then there are AI Researchers. Their goal is to push the boundaries of what computers can do. They might work on letting computers recognize images, understand speech and translate languages. Their method of choice is often Deep Learning because it has unlocked a lot of new applications.

I feel like this post is essentially someone from the first group criticizing the last group, saying their methods are not applicable to him. That is expected.

View on HN

> I feel like this post is essentially someone from the first group criticizing the last group, saying their methods are not applicable to him.

Probably more accurate to say that it's the first group criticizing others in the first group who try to act like people in the third group. Data scientists who use deep learning for everything, when a more interpretable model would do just as well.

Or, someone in the first group criticizing observers who fail to distinguish among groups before making predictions.

Sounds very familiar in pattern to those people who were part of the 'big data' hype who would attempt to use Hadoop for everything.. even work that could have been accomplished with an Excel sheet and an hour.

Looks like us Engineers have a common pattern in running towards the next shiny thing ;-)

Somewhere along the journey/career some of us wise up and learn when to say No, and use the right tools for the job, and ignore the "cool" factor.

It's also suffering from hype.

And the criticism you note isn't one-directional in the field at large. I'm finding that ML/AI researchers deriding ML/Data engineers and "scientists" as not doing "real" ML or AI is becoming a thing, similar to how some computer scientists deride engineering as not doing real computing.

I hate this. Remember they did it to Michael Faraday in 1813, deriding him as not a real scientist as he was about to play a part in advancing one of the greatest challenges of mankind (unification).

It is not suffering from hype. There is too little hype. People are vastly underestimating what is about to happen.

See my comments here:

https://news.ycombinator.com/item?id=13079598

under our recent article " Artificial Intelligence Generates Christmas Song".

Basically, if there is no pixie dust that makes humans intelligent, and instead it is a matter of the architecture of the brain and the first few years of supervised sensory input, then neural net breakthroughs (which use a similar architecture/topology) have the potential at any moment in time to break through and match general human intelligence.

What I mean is that if someone sent back source code from 80 years from now, but we had to run it on a bunch of Amazon / Google servers in a server farm, we're pretty much guaranteed to have enough computing power to do so!

(This is a combination of the number of neurons, their number of connections, and their very slow speed.)

We have the hardware.

Now: we do not actually have the source code from 80 years from now that we can go ahead and run on those machines.

So, we're at like heavier-than-air travel right before the Wright Brothers flew at Kittie Hawk. Except we have like jet engines already -- just no way to design them into something that flies.

I think that AI is vastly underrated and I watch with incredible interest every single breakthrough.

For the second category above, ML built into products or engineering solutions, Alphago surprised me, because Go had an intractibly large possibility space, it's not in any sense subject to brute-forcing or exhaustive search.

Dragon's Natural dictation surprised me in that using its model its able to get basically perfect dictation. I've never worked as a transcriptionist, but a small search reveals it has basically annihilated the industry of medical transcription.

These are not the big, general breakthrough.

But the big, general breakthrough is right there, somewhere. The results researchers are coming up with are astounding, and they're doing it in many cases with neural nets, quite similar to the wiring of the human mind.

The hype is faaaaaaaaaaaaaaar less than warranted for the stage that we're at. At any moment someone can put together something that achieves higher-level intelligence and can be set loose upon the world corpus of culture.

True, there are no clear indications that this is about to happen (For example: people do not extract innate language algorithms from the human genome which encodes them), so we are not exactly taking many steps that we could be trying to. As far as I know we're not even genetically engineering people to see what different parts of DNA do -- which is obviously a very, very good thing, and who would allow anyone to bear to term a child made as an experiment to see what DNA does.

But despite not going from a human starting point, the results that we are achieving in many cases match and surpass human ability - while we do know that in many cases some of their architecture is similar. I feel quite strongly that we have more than enough hardware for general intelligence - and I see advances every day that could end up going past the ppint of no return on it.

EDIT: got a downvote, but I would prefer a discussion if you think I'm wrong.

I didn't downvote you, but the TL;DR of the article is that most ML demonstrations to date have been toys and there are no known real-world applications that would justify the "40% of jobs lost!" hype.

And you're trying to rebut this by referencing an AI-generated Christmas jingle. I think the author rests their case...

just toys? https://en.wikipedia.org/wiki/Deep_learning#Applications

No, I just meant to reference my discussion from there (i.e. for people to read through my comments there, after clicking.)

IOW I meant to transclude that discussion here. (Perhaps within that comment thread a good specific summary comment is: https://news.ycombinator.com/item?id=13090869)

Obviously it is hard to know when that magic moment will happen that some kind of general AI is created that can learn in some sense similarly to how humans do. My every indication and astonishment at the results that are being produced strongly suggests "at any moment". The results are absolutely astonishing every day and we have vastly more than enough firepower.

You might also be interested in this separate thread where I dealt with questions of consciousness and pain. I again reference it here:

https://news.ycombinator.com/item?id=13026020

and you can click through at the top to follow my reference.

We are way past the point of no return here, and in my estimation it is a question of years or at the most decades - not centuries.

absolutely dude.. for example, this paper just popped up in the last week https://arxiv.org/abs/1611.02167

Interesting. The concept seems to be very similar to this one: https://arxiv.org/abs/1611.01578

  > Obviously it is hard to know when that magic moment will happen that some kind of general AI
  > is created that can learn in some sense similarly to how humans do. My every indication and 
  > astonishment at the results that are being produced strongly suggests "at any moment".

As an IT guy with a basic but solid neuroscience education (which isn't even needed for what I'm about to say): Yep, you are on the hype train, and very deeply. I really like reasonable discussions, I have no idea what this is right here. We will see more amazing results, sure - but applications will be specialized narrow subjects. From creating "some kind of general AI is created that can learn in some sense similarly to how humans do" we are still very far away. Your statements remind me of 1960s "future" hype,a nuclear reactor in every car by the year 2000, stuff like that.

My hype is different, because in my estimation we already have the hardware. You write:

>As an IT guy with a basic but solid neuroscience education

-- could you go ahead and take a few minutes (maybe will take you 5-10) to read through my above-referenced links referencing my previous discussion and tell me whether I'm correct in your estimation on the bottom-up aspect - i.e. the amount of computation that human neural nets can likely be doing, and how it compares to server farms with fast interconnects today?

I'm not an expert in neuroscience so your feedback might be helpful there.

If P=NP then we already have the hardware to crack RSA encryption.

The above sentence is true, but it has no bearing on anything.

don't you think it would have a lot more bearing if you had 7 billion devices nonchallantly walking around cracking RSA every day using the same or less hardware? (but we couldn't reverse-engineer them, because they were obfuscated in biology)?

The fact that they weren't reverse-engineered (yet) would still have huge bearing on everything.

By 7 billion samples I mean the humans walking around. Your analogy with an RSA crack is fundamentally different beccause biology doesn't do it in 3 pounds of grey goo in seven billion different bodies already.

so you would have to come up with an analogy that uses something we cant use, to say, okay fine it exists and fine, we have the hardware to also do it, but the former doesn't have any bearing on us doing the latter.

Interesting observation. If brains routinely cracked RSA, that could be evidence that P=NP.

Still it wouldn't help us find the P-time algorithm in question. We could say "it seems to exist", but that would not imply "we'll discover it any day now".

  > If brains routinely cracked RSA

For brains numbers have a completely different meaning and internal representation than for computers. Brains don't "think in numbers". Doing the kind of math we invented is a major effort for the brain, it's not what it developed for, and it is very poorly equipped to do explicit numerical calculations (emphasis on "explicit"). So looking at brains to "crack RSA" seems like waiting for a hammer to be useful in driving screws.

How much computing hardware does it take to beat a human at chess or at go? (Less than you think, Deep Blue was hopelessly inefficient.) And how long did it take for those much simpler things to happen?

  > i.e. the amount of computation that human neural nets can likely be doing

We don't know nearly enough what they are really doing! We only know a few selected bits and pieces! You are basing your assumptions on nothing, so according to logic any conclusion is possible from a faulty premise. On which you promptly deliver spectacularly.

Also, the "computation" a brain does at one moment leaves out the time aspect: Lots of things lead to constant changes. The wiring changes all the time. The "computation" metaphor has little use for describing or understanding this major aspect of "brain". The more I learned about neuroscience the more unhappy I got with the computing metaphor that I had had going in (as a CS graduate, naturally, I think). The brain is so very, very different from my pre-neuroscience-courses notions.

How much neuroscience do you know? If the answer isn't at least an undergrad introductory course (the accompanying book is over a thousand pages), why do you get the idea you can make any predictions?

Read this to read about complexity in biology vs. engineering and what scientists in the field think how well we are dealing with it:

- http://biorxiv.org/content/early/2016/05/26/055624

- http://www.cell.com/cancer-cell/fulltext/S1535-6108(02)00133...

Test yourself: Do you understand what he's talking about? http://inference-review.com/article/the-excitable-mitochondr...

Fortunately you don't have to sign up at university these days just for such knowledge:

Free courses (if you ignore the certificate nonsense):

- https://www.mcb80x.org/ (This is linked to from edX as "The Fundamentals of Neuroscience" Parts 1, 2, 3)

- https://www.coursera.org/courses?languages=en&query=neurosci... (Especially "Medical Neuroscience": https://www.coursera.org/learn/medical-neuroscience)

- https://www.edx.org/course?search_query=neuroscience

Thank you so much for those insightful links!

A lot of revolutionary technologies begin as toys. In the 9th century two brothers created a machine called, "the instrument which plays by itself." It uses a rotating cylinder with "teeth" that triggers keys which were encoded to play music. One can say it's a programmable music machine.

Fast forward to the 1700s and artisan engineers in France were creating similar devices that automatically played music. Toys for the entertainment of the aristocracy. Some insightful engineer noticed the use of pinned cylinders to recreate music, and realized the device might also be used to create textiles with a loom. You could program a machine to create textile patterns. While the initial implementations didn't work very well, eventually Joseph-Marie Jacquard created a version that revolutionized textile manufacturing in the early 1800s, relying on punch cards instead of expensive-to-create cylinders.

By the mid-1800s Charles Babbage became interested in Jacquard's work using punch cards to create complex textile patterns. "The major innovation was that the Analytical Engine was to be programmed using punched cards: the Engine was intended to use loops of Jacquard's punched cards to control a mechanical calculator, which could use as input the results of preceding computations." - https://en.wikipedia.org/wiki/Charles_Babbage#Analytical_Eng...

Toys are amazing for inspiration and innovation.

I think your story does more to reinforce the parent post's point. The chain of events you outline spans an entire millennium. Over that time, yes, a lot of technology developed, and eventually we got to computers, and there was even the movement that gives us the term Luddite. But it was still one long chain of gradual, evolutionary developments over the course of a very long expanse of time. At no point was there anything like the kind of sudden turmoil that futurists like to claim that deep learning will produce.

We never know, that is what the original parent comment was about. Regarding gradual progression, remember - we are now in a highly connected world, where things move at tremendous pace. And a lot more population have access to the latest information. Just like the invention of the wheel, or the invention of steam engines, or the computers, transistors etc, there could be a tipping point, which could happen today as I write or tomorrow or even after a decade. We can only guess. But looking at the speed of things, I would say the probability of it happening very near is very high and is worth the hype.

If the world evolved in linear terms, you would be right. However, the rate of scientific advancement has multiplied exponentially over the history of mankind: our scientific understanding has advanced more in the last 100 years than it has in the previous 10000 combined (nuclear physics, quantum physics, molecular biology, information technology, astronomy/cosmology - you name it.)

Remember, 10 years ago smartphones were in the realm of science fiction. Now, everyone has one in their pockets.

Today we can train a computer for a week to generate a cute little christmas jingle. How many years do we need to train a human child to do the same? AGI won't happen overnight, sure, but its effects of that research will be felt continuously along the way.

The '40% of jobs lost' figure is reachable without any Singularity-style event. That's what you can get just by applying modern automation techniques to low hanging fruit. Self-driving vehicles, automatic checkouts at supermarkets, fully automated fast food places, robotic warehouse storage/retrieval etc.

If you used to have 10 people working at a supermarket and you now have 6 with self-checkouts, that's your 40% jobs lost (for that supermarket) right there.

Self driving vehicles are not a low hanging fruit. We are far, far away from fully autonomous vehicles that don't need a human backup driver. This is despite massive investment by Google and others.

True, I shouldn't have grouped it in with self checkouts etc. When we do achieve reliable self driving capability it will be arguably the single most impressive engineering feat we've accomplished.

Tesla are describing their 'hardware 2' self driving platform, fitted to all of their new cars, as "full self-driving hardware", and they've posted several videos recently of fully autonomous journeys on public roads. They've said they plan to roll out an OTA update to enable full self driving next year.

Mercedes-Benz have a limited self-driving mode on their 2017 model.

Uber-owned Otto, a company focusing on self-driving technology for trucks, plans to start offering its services next year.

And of course Google is quietly continuing their self-driving project, although they've gone a bit quiet recently.

Now, it does look to me like we've still got a fair few corner cases to straighten out, but either all of these companies are badly wrong, or we'll be seeing self-driving vehicles on sale to the public in the next few years (even if not next year).

I agree with this. People don't understand or appreciate the potential of generative AI, especially when guided by human hands.

>if there is no pixie dust that makes humans intelligent [...] then neural net breakthroughs [...] have the potential [...]

You're assuming neural nets are the right model.

Like a 19th century person saying, "if there is no pixie dust ... then eventually Newtonian mechanics will explain these unexpected wobbles we see in the planets' orbits."

What do you mean "assuming" it's the right model - a known-good model is the model in human brains, which is definitely a series of neurons, no? what else would it be?

Here's one: neurons are cells, and cells are really complex and have all sorts of internal state on different scales (organelles, molecules, etc). Neural network models in computers are an extremely simplified abstraction by comparison. And that's still ignoring all the other things that may or may not be happening in the brain beyond the connections betweens neurons themselves.

I think you've fed majorly into the hype but to address your point there was a great comment on neurons (from someone who knows much more than I do) just a week ago: https://news.ycombinator.com/item?id=13030766

Yes. Furthermore, neural nets are just one small part of the solution to [edit] general intelligence. A truly scalable intelligence that learns on its own, and doesn't rely on "the right answers" through training data by human experts. Without this training data, neural nets can't do much ...

I kind of doubt if neural nets have to be the right model.

Any model that gives reasonably similar results to human thinking will be useful. Planes don't have feathers and don't eat bugs, but we say that they "fly" anyway, and they carry far more cargo / passengers (and faster) than birds ever could.

80 years is very conservative. Rich Sutton has a very well articulated argument for achieving human level hardware (using Moores law, and an estimate of the computational power of the brain derived from measurable computations in the eye (retina?).

But as you said, it's the algorithms that are by far the long pole. Current supervised learning is much akin to simple rote learning. This is the promise of reinforcement learning - and the ability to truly learn on your own through experience. That's scalable. *that said ... I'm biased being one of Rich's students :(

Historically speaking, this isn't the first time someone has said any of that.

I read your consciousness post and will reply here because that thread is already locked. I think the problem of consciousness comes from its definition. The word consciousness is too large, encloses too many aspects of cognition, religion and philosophy to be useful. It's a suitcase word, reifying various meanings into one name.

What we need instead of "consciousness" is a set of concepts:

- "perception" - means representing inputs in such a way as to be able to adapt to the environment to achieve goals (I place qualia here)

- "judgement" - the ability to select the best action for the current state

- "reward" - comes from time to time, and is used as a signal in behavior learning

These three concepts : perception, judgement and reward are simpler, clearer and less ambiguous. They have implementations in AI, not human level yet, but getting there. These concepts make the problem concrete instead of bringing up 2000 years of attempts to "get consciousness" based off armchair philosophising.

A more interesting question than "what is consciousness?" would be: what are the reward signals that train the human brain. We do Reinforcement Learning in the brain, but the reward signal is complex, made of multiple channels, which are evolutionarily optimized for our survival. It's the engine of drive that puts us in motion. We need to reverse engineer that in order to replicate human consciousness in silico.

Consciousness is a replicator, a system that is concerned with survival and maintaining balance in the face of perturbations and entropy. What it does is to make us go find food, make babies and protect from danger - essential actions without which the human species would disappear, and consciousness with it. Or, in other words, it maximizes its rewards over time, reward being having food, shelter, company, access to learning and a few more. But it's just limited number of reward types, and they are much simpler to understand than consciousness itself.

So consciousness is just that thing that maintains itself instead of becoming disorganized and dissipating out like most physical processes that lack a self stabilizing, self replicating dynamics.

Consciousness is a balancing system for the colony of cells that just want to pass their genes into the next generation.

>Consciousness is a balancing system for the colony of cells that just want to pass their genes into the next generation.

is there anything in that which requires that it be a physical biological substrate rather than a model? Can't computer models follow the same behavior as long as that's how their environment / system is set up? (More specifically, can't they also have brains made up of cells connected to each other, in a way closely analogous or even an exact simulation of chemical pathways?)

I think there is absolutely no need for the system to be biological. A computer model can work just as well. As long as it solves the same behavior problem, there is no difference between brain reinforcement learning and artificial reinforcement learning.

>It's also suffering from hype.

I'm imagining someone tweeting at Robin, several years back:

"Good Prediction Markets expert says: Most firms that think they want advanced Prediction Markets really just need linear regression on cleaned-up data."

;)

It's also suffering from hype.

If you mean hype in media and general public, I agree with you. Research is inherently risky and uncertain, often that is not conveyed correctly. Also research results are often oversold.

If you mean that big tech companies are overinvesting in Machine Learning then I have to disagree. It's not a coincidence that the companies that invested the most in Machine Learning (Google and Facebook) are companies that have a lot of data and already use Machine Learning in their products. It provides things like better feed ranking and better signals for search. These "invisible" improvements to existing products are often overlooked.

And the criticism you note isn't one-directional in the field at large.

Yes, like I mentioned I think there is confusion about terminology. Many ML Engineers (in my definition) call themselves Data Scientists. This leads to misunderstandings, when people don't recognize that others have different goals.

You mean, some tech companies understand the difference between Data Science, AI in general, ML and Deep Learning and try to utilize the best tools in hand to handle their data.

But what is driving this hype, as well as the Blockchain hype, is not engineer-driven companies like Google, but rather MBA-driven buzzword-friendly tech companies (imagine Balmer-era Microsoft), non-tech companies who want to share the cake, the media and finally the investors who are misled by the rest, but end up creating a capital-based feedback loop for them.

As far as the people driving the hype train right now are concerned, Machine Learning is the way you do AI, and Deep Learning is just a more powerful (ahem deeper) version of Machine Learning. This is what AlphaGo used to defeat Lee Sedol, so it's obviously superior, and we should use it to process all data, in the same way we should strive to store everything on a blockchain, which is clearly superior to hash tables and databases.

This is the first time I'm hearing that computer scientists deride engineering as not-real-computing. Any references? It doesn't check out based on my background. In fact anecdotally, I've heard the reverse. I've heard EE and stat algorithms folks criticize CS ML/CV researchers for using algorithms as black boxes compared to the rigorous standards of EE/Stat (aka reviewer standards in IEEE Transactions in Information Theory, IEEE Trans Sig Proc, majority of stat journals). Now obviously there are a vast number of CS researchers who are competent at stat-level rigor. But there are also a large number of ML researchers (applied ML?) in CS departments who use algorithms as black boxes.

My understanding is that the grandparent comment was referring to _software_ engineers.

The prof of my applied ML class this semester jokingly refers to ML research's approach to statistics as "punk stats", which I find to be pretty accurate.

This and the above comment kind of says my thoughts on the matter. I think there is a very real emerging field that combines mathematics, traditional computer science, and AI into a big family resemblance of quantitative methods and research, but that we're still in the early stages and we're still sorting things out. I remember hearing someone on hacker news joke that when he got out of school people called him a statistician, then people started saying he was a data miner, and now he sells himself as a data scientist, but he hasn't actually done anything differently apart from just keeping up with the field. Sometimes it feels as if we're just playing with words and people get excited for advances and what it means for the future, but the reality is people just keep plodding along make incremental progress, finding new ways to leverage the tools they have, and continuing to publish papers. For all of the tech industry's worship of "disruptive innovation" and massive changes, the reality is that we've been developing and refining our quantitative abilities and figuring out automation since the industrial revolution. AI may become less exciting to people 5 years from now, but we will still continue doing what we've been doing.

A lot of standard automation is being called 'AI' too, because that sells.

Not a bad way to put it. You could make his argument valid though by discussing the economic value of the problems solved by the first category of people vs. the third. Right now I agree with him that the first is bigger than the third. But I believe that balance is starting to tip and the potential value of the third will keep increasing relatively to the first to the point of dwarfing it (hence no bust).

I don't think that there is a "balance" to tip, it's just that the previously theoretical work of the latter group is starting to become practical to the former in a business setting. Neural networks have been a topic of research for many decades before consumer demand for graphical video games drove the technology to make them practical. Backpropagation was first described as a solution to efficiently training multi-layer neural networks in 1975 but only in the last decade have we developed the infrastructure for non-CS researchers to use complex neural networks using a sub-$1000 teraflop GPU and an iPython notebook.

Off the top of my head the closest analogy would be number theory, especially the study of prime numbers. Before information technology in general, number theory was esoteric and considered useless by many pure and applied mathematicians (I'm simplifying a bit for argument's sake) but all of that accumulated research proved massively useful once we started to communicate electronically. WWI and II cryptographers didn't become or absorb number theorists as a group, they just adapted the knowledge to their field under the umbrella of electronic warfare. I think this is happening with data scientists, who are starting to experiment with ML but it's still just another tool in their toolbox. The media hype train focuses on the flashy AI contests and muddles the terminology but the real work [1] is happening behind the scenes in data science.

[1] By "real work" I mean work that directly translates into value on a company balance sheet. The theoretical work is important in and of itself and has been happening for decades.

This was a really nice breakdown, thanks.

You mentioned that that Deep learning was a method of choice for AI researched because Deep it has unlocked a lot of new application.

I have a question - is it also a "method of choice" for researchers because its not well understood yet why Deep Learning actually works?

Good question. Just one anecdotal data point here ... So take it for what it's worth. I'm a grad student focusing on reinforcement learning but have a lot of interaction with many deep learning folk. I'd have to say that they seem mostly motivated to learn how to make deep learning even more powerful and how to apply it. Not so much solving what's going on inside the box.

We do understand in general, just not every neuron and input weight in particular, because they are too many for limited human working memory to hold at once.

An automated labyrinth could be solved by hand up to a size, but if it becomes larger, it would become "impossible to understand". We'd need to rely on computers to find the route for us.

It's not computer magic, just an ability to hold more data at once. We're limited to 7-8 things - try to remember a string of more digits and see if you can - it's hard, we just have that kind of limitation. So any computer algorithm that can't be broken down into 7-8 understandable parts is hard to grasp.

Of course we can make ML tools synthesize the preferred input of any neuron. That does help a little.

For many researchers, "understand" relates more to the underlying math. Things like "what kind of convergence properties and guarantees are there" and "formality, what can this model learn and not learn"?

Reply on news.ycombinator.com