AI winter is well on its way | Hacker News Reader

AI winter is well on its way

blog.piekniewski.info

993 pointswei_jok492 comments

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjusting number of categories could be used in any image classification task, likely with state-of-art results. I was surprised when I applied it to another, completely unrelated dataset and got >92% accuracy right away.

If you think this is a symptom of AI winter, then you are probably wasting time on outdated/dysfunctional models or models that aren't suited for what you want to accomplish. Looking e.g. at Google Duplex (better voice synchronization than Vocaloid I use for making music), this pushed state-of-art to unbelievable levels in hard-to-address domains. I believe the whole SW industry will be living next 10 years from gradual addition of these concepts into production.

If you think Deep (Reinforcement) Learning is going to solve AGI, you are out of luck. If you however think it's useless and won't bring us anywhere, you are guaranteed to be wrong. Frankly, if you are daily working with Deep Learning, you are probably not seeing the big picture (i.e. how horrible methods used in real-life are and how you can easily get very economical 5% benefit of just plugging in Deep Learning somewhere in the pipeline; this might seem little but managers would kill for 5% of extra profit).

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought.

Understand that in pop-sci circles over the past several years the general public is being exposed to stories warning about the singularity by well respected people like Stephen Hawking and Elon Musk (http://time.com/3614349/artificial-intelligence-singularity-...). Autonomous vehicles are on the roads and Boston Dynamics is showing very real robot demonstrations. Deep learning is breaking records in what we thought was possible with machine learning. All of this progress has excited an irrational exuberance in the general public.

But people don't have a good concept of what these technologies can't do, mainly because researchers, business people, and journalists don't want to tell them--they want the money and attention. But eventually the general public wises up to the unfulfillment of expectations, and drives their attention elsewhere. Here we have the AI winter.

I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment.

Further, I have repeatedly heard people who should know better, with very fancy advanced degrees, chant variants of "Deep Learning gets better with more data" and/or "Deep Learning makes feature engineering obsolete" as if they are trying to convince everyone around them as well as themselves that these two fallacious assumptions are the revealed truth handed down to mere mortals by the 4 horsemen of the field.

That said, if you put your ~10,000 hours into this, and keep up with the field, it's pretty impressive what high-dimensional classification and regression can do. Judea Pearl concurs: https://www.theatlantic.com/technology/archive/2018/05/machi...

My personal (and admittedly biased) belief is that if you combine DL with GOFAI and/or simulation, you can indeed work magic. AlphaZero is strong evidence of that, no? And the author of the article in this thread is apparently attempting to do the same sort of thing for self-driving cars. I wouldn't call this part of the field irrational exuberance, I'd call it amazing.

> Deep Learning makes feature engineering obsolete

I think even if you avoid constructing features, you are basically doing a similar process where a single change in a hyper-parameter can have significant effects:

- internal structure of a model (what types of blocks are you using and how do you connect them, what are they capable of together, how do gradients propagate?)

- loss function (great results come only if you use a fitting loss function)

- category weights (i.e. improving under-represented classes)

- image/data augmentation (self-driving car won't work without significant augmentation at all)

- properly set-up optimizer

The good thing here is that you can automate optimization of these to a large extent if you have a cluster of machines and a way to orchestrate meta-optimization of slightly changed models. With feature engineering you just have to do all the work upfront, thinking what might be important, and often you just miss important parts of features :-(

> But eventually the general public wises up to the unfulfillment of expectations, and drives their attention elsewhere. Here we have the AI winter.

And more importantly, business and government leaders wise up and turn off the money tap.

This is why it's wise for researchers and business leaders to temper expectations. Better a constant modest flow of money into the field than a boom-bust cycle with huge upfront investment followed by very bearish actions.

> AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits.

I think they also happen when the best ideas in the field run into the brick wall of insufficiently developed computer technology. I remember writing code for a perceptron in the '90s on an 8 bit system, 64 k RAM - it's laughable.

But right now compute power and data storage seem plentiful, so rumors of the current wave's demise appear exaggerated.

I wonder, though, what will happen with the demise of Moore's law... can we simply go with increased parallelism? How much can that scale?

I wish GPUs were 1000x faster... Then I could do some crazy magic with Deep Learning instead of waiting weeks for training to be finished...

> AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits.

A symptom of capitalism and marketing trying to push shit they don't understand

I don't think the claim is that AI isn't useful. It's that it's oversold. In any case, I don't think you can tell much about how well your classifier is working for something like cancer diagnoses unless you know how many false negatives you have (and how that compares to how many false negatives a radiologist makes).

There are two sides to this:

- how good humans are in detecting cancer (hint: not very good) and if having an automated system even as a "second opinion" next to an expert might not be useful?

- there are metrics for capturing true/false positives/negatives one can focus on during learning optimization

From studies you might have noticed that expert radiologists have e.g. F1-score at 0.45 and on average they score 0.39, which sounds really bad. Your system manages to push average to 0.44, which might be worse than the best radiologist out there, but better than an average radiologist [1]. Is this really being oversold? (I am not addressing possible problems with overly optimistic datasets etc. which are real concerns)

[1] https://stanfordmlgroup.github.io/projects/chexnet/

Alright. What is the cost of a false positive in that case?

The problem AI runs into is that with too much faith in the machine, people STOP thinking and believe the machine. Where you might get a .44 detection rate on radiology data alone, that radiologist with a .39 or a doctor can consult alternate streams of information. The AI may still be helpful in reinforcing a decision to continue scrutinizing a set of problem.

AI's as we call them today are better referred to as expert systems. AI carries too much baggage to be thrown around Willy nilly. An expert system may beat out a human at interpreting large unintuitive datasets, but they aren't generally testable, and like it or not, it will remain a tough sell in any situation where lives are on the line.

I'm not saying it isn't worth researching, but AI will continue to fight an uphill battle in terms of public acceptance outside of research or analytics spaces, and overselling or being anything but straightforward about what is going on under the hood will NOT help.

Is there a prevailing approach to thinking about (accounting for?) false negatives in ground truth data? I'm new to this area, and the question is relevant for my current work. By definition, you simply don't know anything about false negatives unless you have some estimate of specificity in addition to your labeled data, but can anything be done?

I don't get the sentiment of the article either. I can't speak for researchers but software engineers are living through very exciting times.

  State of the art in numbers:
  Image Classification - ~$55, 9hrs (ImageNet)
  Object Detection - ~$40, 6hrs (COCO)
  Machine Translation - ~$40, 6hrs (WMT '14 EN-DE)
  Question Answering - ~$5, 0.8hrs (SQuAD)
  Speech recognition - ~$90, 13hrs (LibriSpeech)
  Language Modeling - ~$490, 74hrs (LM1B)

"If you think Deep (Reinforcement) Learning is going to solve AGI, you are out of luck" --

I don't know. Duplex equipped with a way to minimize his own uncertainties sounds quite scary.

Duplex was impressive but cheap street magic: https://medium.com/@Michael_Spencer/google-duplex-demo-witch...

Microsoft OTOH quietly shipped the equivalent in China last month: https://www.theverge.com/2018/5/22/17379508/microsoft-xiaoic...

Google has lost a lot of steam lately IMO. Facebook is releasing better tools and Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google does remain the master of its own hype though.

> Microsoft, the company they nearly vanquished a decade ago, is releasing better products.

Google nearly vanquished Microsoft a decade ago? Where can I read more about this bit of history :) ?

IMO, Axios [0] seem to do a better job of criticizing Google's Duplex AI claims, as they repeatedly reached out to their contacts at Google for answers.

0: https://www.axios.com/google-ai-demo-questions-9a57afad-9854...

My thoughts on AGI (at least in the sense of being indistinguishable from interaction with a human) are the same as my thoughts on extraterrestrial life: I'll believe it only when I see it (or at least when provided with proof that the mechanism is understood). This extrapolation on a sample size of one is something I don't understand. How is the fact that machine learning can do specific stuff better than humans different in principle than the fact that a hand calculator can do some specific stuff better than humans? On what evidence can we extrapolate from this to AGI?

We haven't found life outside this planet, and we haven't created life in a lab, therefore n=1 for assessing probability of life outside earth (which means we can't calculate a probability for this yet). Likewise, we haven't created anything remotely like animal intelligence (let alone human) and we have no good theory regarding how it works, so n=1 for existing forms of general intelligence.

Note that I'm not saying there can be no extraterrestrial life or that we will never develop AGI, just that I haven't seen any evidence at this point in time that any opinions for or against their possibility are anything more than baseless speculation.

This is what we know from Google about Duplex:

"To train the system in a new domain, we use real-time supervised training. This is comparable to the training practices of many disciplines, where an instructor supervises a student as they are doing their job, providing guidance as needed, and making sure that the task is performed at the instructor’s level of quality. In the Duplex system, experienced operators act as the instructors. By monitoring the system as it makes phone calls in a new domain, they can affect the behavior of the system in real time as needed. This continues until the system performs at the desired quality level, at which point the supervision stops and the system can make calls autonomously." --

If the dollar amounts refer to the training cost for the cheapest DL model, do you have references for them? A group of people at fast.ai trained an ImageNet model for 26$, presumably after spending a couple hundered on getting everything just right: http://www.fast.ai/2018/04/30/dawnbench-fastai/

Thats what you get with Google TPUs on reference models. The ImageNet numbers are from RiseML, the rest is from here - https://youtu.be/zEOtG-ChmZE?t=1079

"Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training."

OK, but 83% ROC/AUC is nothing to be bragging about. ROC/AUC routinely overstates the performance of a classifier anyway, and even so, ~80% values aren't that great in any domain. I wouldn't trust my life to that level of performance, unless I had no other choice.

You're basically making the author's case: deep learning clearly outperforms on certain classes of problems, and easily "generalizes" to modest performance on lots of others. But leaping from that to "radiology robots are almost here!" is folly.

Yeah, but the point here was that radiologists on average fared even worse. 83% is not impressive, but better than what we have right now in real-world with real people, as sad as it is. Obviously, best radiologists would outperform it right now, but average ones, likely stressed under heavy workload might not be able to beat it. And of course, this classifier probably works on certain visual structures better than humans and other ones easier detectable by humans would slip through.

There is also higher chance that next state-of-art model would push it significantly over 83% or best human radiologist at some point in the future, so it might not be very economical to train humans to become even better (i.e. dedicate your life to focus on radiology diagnostics only).

I think you're missing a very important part here that maybe you've considered: Domain knowledge. I'm assuming your radiologic images were hand labeled by other radiologists. How did they come to that diagnosis? By only looking at the image? This was a severe limitation of the Andrew Ng paper on CheXnet for detection of pneumonia from chest x Rays. CheXnet was able to outperform radiologists on detection of pneumonia from the chest x Rays, but the diagnosis of pneumonia is considered a clinical diagnosis that requires more information about the patient. My point is that while your results are impressive and indicative of where deep learning could help in medicine, these same results might be skewed since you're testing the model on hand labeled data. What happens if you apply this in the real world at a hospital where the radiologist gets the whole patient chart and your model only gets the x Ray?

"Yeah, but the point here was that radiologists on average fared even worse."

Except they don't. See the table in the original post. Also, comparing the "average" radiologist by F1 scores from a single experiment (as you've done in other comments here) is meaningless.

Unless my doctor is exactly average (and isn't incorporating additional information, or smart enough to be optimizing for false positive/negative rates relative to cost), comparison to average statistics is academic. But I don't really need to tell you this -- your comment has so many caveats that you're clearly already aware of the limits of the method.

> Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training

Of course! Using DenseNet-BC-100-12 to increase ROC AUC, it was so obvious!

Would you mind sharing which other, unrelated dataset you have used the model on?

You can take a look at: https://github.com/sfikas/medical-imaging-datasets or http://www.radrounds.com/profiles/blogs/list-of-open-access-...

I can't unfortunately, proprietary stuff being plugged into existing business right now.

Next winter will probably going to be going over that 92% across all domains.

Possibly, but will it be called AI winter, if e.g. average human has 88% accuracy and best human 97%?

Yea this sounds extremely unlikely unless the other dataset has a fairly easy decision boundary. The kind of cross-domain transfer learning you seem to think deep neural networks have is nothing I've observed before in my formal studies of neural network

How much of this can we pin on IBM's overhype of Watson?

ROC AUC is fairly useless when you have disparate costs in the errors. Try precision-recall.

I mentioned F1 in some later comment.

This is a deep, significant post (pardon pun etc).

The author is clearly informed and takes a strong, historical view of the situation. Looking at what the really smart people who brought us this innovation have said and done lately is a good start imo (just one datum of course, but there are others in this interesting survey).

Deepmind hasn't shown anything breathtaking since their Alpha Go zero.

Another thing to consider about Alpha Go and Alpha Go Zero is the vast, vast amount of computing firepower that this application mobilized. While it was often repeated that ordinary Go program weren't making progress, this wasn't true - the best, amateur programs had gotten to about 2 Dan amateur using Makov Tree Search. Alpha Go added CNNs for it's weighting function and petabytes of power for it's process and got effectiveness up to best in the world, 9 Dan professional, (maybe 11 Dan amateur for pure comparison). [1]

Alpha Go Zero was supposedly even more powerful, learned without human intervention. BUT it cost petabytes and petabytes of flops, expensive enough that they released a total of ten or twenty Alpha Go Zero game to the world, labeled "A great gift".

The author convenniently reproduces the chart of power versus results. Look at it, consider it. Consider the chart in the context of Moore's Law retreating. The problems of Alpha Zero generalizes as described in the article.

The author could also have dived into the troubling question as of "AI as ordinary computer application" (what does testing, debugging, interface design, etc mean when the app is automatically generated in an ad-hoc fashion) or "explainability". But when you can paint a troubling picture without these gnawing problems appearing, you've done well.

[1] https://en.wikipedia.org/wiki/Go_ranks_and_ratings

>Deepmind hasn't shown anything breathtaking since their Alpha Go zero

They went on to make AlphaZero, a generalised version that could learn chess, shogi or any similar game. The chess version beat a leading conventional chess program 28 wins, 0 losses, and 72 draws.

That seemed impressive to me.

Also they used loads of compute during the training but not so much during play.(5000 TPUs, 4TPUs).

Also it got better than humans in those games from scratch in about 4 hours whereas humans have had 2000 years to study them so you can forgive it some resource usage.

It's not like humanity really needs another chess playing program 20 years after IBM solved that problem (but now utilizing 1000x more compute power). I just find all these game playing contraptions really uninteresting. There are plenty real world problems to be solved of much higher practicality. Moravec's paradox in full glow.

>Also it got better than humans in those games from scratch in about 4 hours whereas humans have had 2000 years to study them so you can forgive it some resource usage.

Most humans don't live 2000 years. And realistically don't spend that much of their time or computing power on studying chess. Surely a computer can be more focused at this and the 4h are impressive. But this comparison seems flawed to me.

> The chess version beat a leading conventional chess program 28 wins, 0 losses, and 72 draws.

In a not equal fight, and the results are still not published. I'm not claiming that AlphaZero wouldn't win, but that test was pure garbage.

And didn’t they just do all of this? It’s not like 5 years have passed. Does he expect results like this every month?

> Also it got better than humans in those games from scratch in about 4 hours whereas humans have had 2000 years to study them so you can forgive it some resource usage.

Few would care. Your examiner doesn't give you extra marks on a given problem for finishing your homework quickly.

oh wow, it can play chess. can it efficiently stack shelves in warehouses yet?

there is no human that has studied any of those games for 2000 years. So I think you mean 4 hours versus average human study of 40 years.

I'm sure the same could be said for early computer graphics before the GPU race. You don't need Moore's Law to make machine learning fast, you can also do it with hardware tailored to the task. Look at Google's TPUs for an example of this.

If you want an idea of where machine learning is in the scheme of things, the best thing to do is listen to the experts. _None_ of them have promised wild general intelligence any time soon. All of them have said "this is just the beginning, it's a long process." Science is incremental and machine learning is no different in that regard.

You'll continue to see incremental progress in the field, with occasional demonstrations and applications that make you go "wow". But most of the advances will be of interest to academics, not the general public. That in no way makes them less valuable.

The field of ML/AI produces useful technologies with many real applications. Funding for this basic science isn't going away. The media will eventually tire of the AI hype once the "wow" factor of these new technologies wears off. Maybe the goal posts will move again and suddenly all the current technology won't be called "AI" anymore, but it will still be funded and the science will still advance.

It's not the exciting prediction you were looking for I'm sure, but a boring realistic one.

> Funding for this basic science isn't going away.

What make this 3rd/4th boom in AI different?

The other AI winter, the funding for these science went from well funded to little funding.

I'm skeptical, with respect of course, on your statement because it doesn't have anything to back that up other than it produce useful technologies. Wouldn't this statement imply that the other previous AI which experience AI Winter (expert system, and whatever else) didn't produce useful enough technologies to have funding?

I'm currently on the camp of there is going to be an AI Winter III coming.

> None_ of them have promised wild general intelligence any time soon.

The post talk about Andrew Ng wild expectation on other things such as radiologist tweet. While it's not wild general intelligence. What I think the main article and also I am thinking is the outrageous speculation. Another one is the tesla self driving, it doesn't seem to be there yet and perhaps we're hitting the point of over promise like we did in the past and then AI winter happen because we've found the limit.

> BUT it cost petabytes and petabytes of flops, expensive enough that they released a total of ten or twenty Alpha Go Zero game to the world

Training is expensive but inference is cheap enough for Alpha Zero inspired bots to beat human professionals while running on consumer hardware. DeepMind could have released thousands of pro-level games if they wanted to and others have: http://zero.sjeng.org/

Bleh, no it isn't.

I am 100% in agreement with the author on the thesis: deep learning is overhyped and people project too much.

But the content of the post is in itself not enough to advocate for this position. It is guilty of the same sins: projection and following social noises.

The point about increasing compute power however, I found rather strong. New advances came at a high compute cost. Although it could be said that research often advances like that: new methods are found and then made efficient and (more) economical.

A much stronger rebuttal of the hype would have been based on the technical limitations of deep learning.

> A much stronger rebuttal of the hype would have been based on the technical limitations of deep learning.

I'm not even sure how you'd go about doing that. You could use information theory to debunk some of the more ludicrous claims, especially ones that involve creating "missing" information.

One of the things that disappoints me somewhat with the field, which I've arguably only scratched the surface of, is just how much of it is driven by headline results which fail to develop understanding. A lot of the theory seems to be retrofitted to explain the relatively narrow result improvement and seems only to develop the art of technical bullshitting.

There are obvious exceptions to this and they tend to be the papers that do advance the field. With a relatively shallow resnet it's possible to achieve 99.7% on MNIST and 93% on CIFAR10 on a last-gen mid-range GPU with almost no understanding of what is actually happening.

There's also low-hanging fruit that seems to have been left on the tree. Take OpenAI's paper on parametrization of weights, so that you have a normalized direction vector and a scalar. This makes intuitive sense for anybody familiar with high-dimensional spaces since nearly all of the volume of a hypersphere lies around the surface. That this works in practice is great news, but leaves many questions unanswered.

I'm not even sure how many practitioners are thinking in high dimensional spaces or aware of their properties. It feels like we get to the universal approximation theorem and just accept that as evidence that they'll work well anywhere and then just follow whatever the currently recognised state of the art model is and adapt that to our purposes.

> A much stronger rebuttal of the hype would have been based on the technical limitations of deep learning.

Who's to say we won't improve this though? Right now, nets add a bunch of numbers and apply arbitrarily-picked limiting functions and arbitrarily-picked structures. Is it impossible that we find a way to train that is orders of magnitude more effective?

Overhyped? There are cars driving around Arizona without safety drivers as I type this.

The end result of this advancement to our world is earth shattering.

On the high compute cost. There is an aspect of that being true but we have also seen advancement in silicon to support. We look at WaveNet using 16k cycles through a DNN and offering at scale and competitive price kind of proves the point.

The brain most likely has much more than a petaflop of computing power and it takes at least a decade to train a human brain to achieve the grandmaster level on an advanced board game. In addition, as the other comment says, they learn from hundreds or thousands of years of knowledge that other humans have accumulated and still lose to AlphaZero with mere hours of training.

Current AIs have limitations but, at the tasks they are suited for, they can equal or exceed humans with years of experience. Computing power is not the key limit since it will be made cheaper over time. More importantly, new advances are still being made regularly by DeepMind, OpenAI, and other teams.

https://www.quora.com/Roughly-what-processing-power-does-the...

Unsupervised Predictive Memory in a Goal-Directed Agent

https://arxiv.org/abs/1803.10760

Sure, but have you heard about Moravec's paradox? And if so, don't you find it curious that over the 30 years of Moore's law exponential progress in computing almost nothing improved on that side of things, and we kept playing fancier games?

What, no progress six months after achieving a goal thought impossible even just few years ago? Pack it up boys it’s all over but the crying.

I was thinking just that when reading the paragraphs about the uber accident. There's absolutely nothing indicating that future progress is not possible, precisely because of how absurd it seems right now.

Retrospectively it might sound that the Japanese were partially right in pursuing "high performance" computing with their fifth generation projects [1] but the Alpha Zero results are impressive beyond the computing performance achieved. It was a necessary element but not the only one.

[1] https://mobile.nytimes.com/1992/06/05/business/fifth-generat...

> petabytes and petabytes of flops

Why not petaflops of bytes then?

>> Makov Tree Search

You mean Monte Carlo Tree Search, which is not at all like Ma(r)kov chains. You're probably mixing it up with Markov decision processes though.

Before criticising something it's a good idea to have a solid understanding of it.

We very well might be in a deep-learning 'bubble' and the end of a cycle... but I don't think this time around it's really the end for a long-while, but more likely a pivot point.

The biggest minds everywhere are working on AI solutions, and there's also a lot in medical/science going on to map brains and if we can merge neuroscience with computer science we might have more luck with AI in the future...

So we could have a draught for a year or two, but there will be more research, and more breakthroughs. This won't be like the AI winters of the past where it lay dormant for 10+ years, I don't think.

Moore's law (or at least, the diminishing one) is not relevant here because these are not single threaded programs. Google put 8x on their TPUv2 -> v3 upgrade; parallel matrix multiplies at reduced precision are a long way away from any theoretical limits, as I understand it.

Totally agree but why on earth down voted?

The first generation TPUs used 65536 very simple cores.

In the end you have so many transistors you can fit and there are options on how to arrange and use.

You might support very complex instructions and data types and then four cores. Or you might only support 8 bit ints, very, very simple instructions and use 65536 cores.

In the end what matters is the joules to get something done.

We can clearly see that we have big improvements by using new processor architectures.

A different take by Google’s cofounder, Sergey Brin, in his most recent Founders’ Letter to investors:

“The new spring in artificial intelligence is the most significant development in computing in my lifetime.”

He listed many examples below the quote.

“understand images in Google Photos;

enable Waymo cars to recognize and distinguish objects safely;

significantly improve sound and camera quality in our hardware;

understand and produce speech for Google Home;

translate over 100 languages in Google Translate;

caption over a billion videos in 10 languages on YouTube;

improve the efficiency of our data centers;

help doctors diagnose diseases, such as diabetic retinopathy;

discover new planetary systems; ...”

https://abc.xyz/investor/founders-letters/2017/index.html

An example from another continent:

“To build the database, the hospital said it spent nearly two years to study more than 100,000 of its digital medical records spanning 12 years. The hospital also trained the AI tool using data from over 300 million medical records (link in Chinese) dating back to the 1990s from other hospitals in China. The tool has an accuracy rate of over 90% for diagnoses for more than 200 diseases, it said.“

https://qz.com/1244410/faced-with-a-doctor-shortage-a-chines...

Hi, author here:

Well first off: letters to investors are among the most biased pieces of writing in existence.

Second: I'm not saying connectionism did not succeed in many areas! I'm a connectionist by heart! I love connectionism! But that being said there is disconnect between the expectations and reality. And it is huge. And it is particularly visible in autonomous driving. And it is not limited to media or CEO's, but it made its way into top researchers. And that is a dangerous sign, which historically preceded a winter event...

I agree that self-driving had/have been overhyped over the previous few years. The problem is harder than many people realize.

The difference between the current AI renaissance and the past pre-winter AI ecosystems is the level of economic gain realized by the technology.

The late 80s-early 90s AI winter, for example, resulted from the limitations of expert systems which were useful but only in niche markets and their development and maintenance costs were quite high relative to alternatives.

The current AI systems do something that alternatives, like Mechanical Turks, can only accomplish with much greater costs and may not even have the scale necessary for global massive services like Google Photos or Youtube autocaptioning.

The spread of computing infrastructure and connectivity into the hands of billions of global population is a key contributing factor.

Hi, why in your analysis you spoke only about the companies that are not doing so well in self driving leaving out waymo success story? They are already have been hauling passengers without a safety pilot since last October. I guess without the minimum problem otherwise we would have heard plenty in the news like it happened for Tesla and Uber accidents. Is it not too convenient to leave out the facts that contradict your hypothesis?

Making cars that drive safely no current, busy roads is a very difficult task. It is not surprising that the current systems do not do that (yet). It is surprising to me how well they still do. The fact though that my phone understands my voice and my handwriting and does on the fly translation of menus and simple requests is a sign of a major progress, too.

AI is overhyped and overfunded at the moment, which is not unusual for a hot technology (synthetic biology; dotcoms). Those things go in cycles, but the down cycles are seldom all out winters. During the slowdowns best technologies still get funding (less lavish, but enough to work on) and one-hit wonders die, both of which is good in the long run. My friends working in biology are doing mostly fine even though there are no longer "this is the century of synthetic biology" posters at every airport and in every toilet.

How can something be biased when it's listing facts?

Those are actual features that are available today to anyone, that were made possible by AI. Do you think it would be possible to type "pictures of me at the beach with my dog" without AI in such as short time frame? Or to have cars that drive themselves without a driver? These are concrete benefits of machine learning, I don't understand how that's biased.

" letters to investors are among the most biased pieces of writing in existence. "

Maybe true but they are words that are about things which are either true or not true. Has nothing to do where the words were shared. Saying they are on an investment letter so not relevant seems very short sighted.

But just looking at the last 12 months it is folly to say we are moving to a AI winter. Things are just flying.

Look at self driving cars without safety drivers or look at something like Google Duplex but there are so many other examples.

When I saw the Google demo of a CNN using video to split a single audio stream of two guys talking over each other, I became a believer.

Hey, a small advice for the future: never build your belief entirely on a youtube video of a demo. In fact, never build your belief based on a demo, period.

This is notorious with current technology: you can demonstrate anything. A few years ago Tesla demonstrated a driverless car. And what? Nothing. Absolutely nothing.

I'm willing to believe stuff I can test myself at home. If it works there, it likely actually works (though possibly needs more testing). But demo booths and youtube - never.

A rigorous evaluation with particular focus on where it doesn't work would be better.

Do you have a video of this?

> understand images in Google Photos

This is one of the areas I’m most enthusiastic about but … it’s still nowhere near the performance of untrained humans. Google has poured tons of resources into Photos and yet if I type “cat” into the search box I have to scroll past multiple pages of results to find the first picture which isn’t of my dog.

That raises an interesting question: Google has no way to report failures. Does anyone know why they aren’t collecting that training data?

They collect virtually everything you do on your phone. They probably notice that you scroll a long way after typing cat and so perhaps surmise the quality of search results was low.

> understand

what is this 'understand'?

Well, the way I see it: mostly, these are “improvements”, huge ones, but still. They ride the current AI tech wave, take it an optimize apps with it.

For most things, that people dream of and do marketing about need another leap forward, which we haven’t seen yet (it’ll come for sure)

Almost anything that has to do with image understanding is entirely AI. Good luck writing an algorithm to detect a bicycle in an image. This also includes disease diagnostic as most of those have to do with analyzing images for tumors and so on.

Also, while a lot of these can be seen as "improvements", in many cases, that improvement put it past the threshold of actually being usable or useful. Self-driving cars for example need to be at least a certain level before they can be deployed, and we would've never reached that without machine learning.

>caption over a billion videos in 10 languages on YouTube;

Utterly useless. And I don't think it is improving.

This is less useless than you think. Captioning video could allow for video to become searchable as easily as text is now searchable. This could lead to far better search results for video and a leap forward in the way people produce and consume video content.

I disagree, even with the high error rate, it provides a lot of context. Also, a lot of comedy.

I find the auto captions pretty useful.

I'm a scientist from a field outside ML who knows that ML can contribute to science. But I'm also really sad to see false claims in papers. For example, a good scientist can read an ML paper, see claims of 99% accuracy, and then probe further to figure out what the claims really mean. I do that a lot, and I find that accuracy inflation and careless mismanagement of data mars most "sexy" ML papers. To me, that's what's going to lead to a new AI winter.

I'm in the same situation and it's really worrying.

Deep learning is the method of choice for a number of concrete problems in vision, nlp, and some related disciplines. This is a great success story and worthy of attention. Another AI winter will just make it harder to secure funding for something that may well be a good solution to some problems.

You hear Facebook all the time saying how it "automatically blocks 99% of the terrorist content" with AI to the public and governments.

Nobody thought to ask: "How do you know all of that content is terrorist content? Does anyone check every video afterwards to ensure that all the blocked content was indeed terrorist content?" (assuming they even have an exact definition for it).

Also, how do they know how much terrorist content they aren't blocking (the 1%), since they by definition haven't found it yet?

I'm 100% convinced it can block 99% of all terrorist content that hasn't been effectively SEOed to get around their filters because that's just memorizing attributes of the training set data. Unfortunately, the world isn't a stationary system like these ML models (usually) require. I still get spam in my gmail account, nowhere near as much as I do elsewhere, but I still get it.

> Does anyone check every video afterwards to ensure that all the blocked content was indeed terrorist content?

They might not, but they could sample them to be statistically confident?

Yes, they have a test set. They don’t check every video, but they do check every video in a sample that is of statistically significant size.

FYI This post is about deep learning. It could be the case that neural networks stop getting so much hype soon, but the biggest driver of the current "AI" (ugh I hate the term) boom is the fact that everything happens on computers now, and that isn't changing any time soon.

We log everything and are even starting to automate decisions. Statistics, machine learning, and econometrics are booming fields. To talk about two topics dear to my heart, we're getting way better at modeling uncertainty (bayesianism is cool now, and resampling-esque procedures aged really well with a few decades of cheaper compute) and we're better at not only talking about what causes what (causal inference), but what causes what when (heterogeneous treatment effect estimation, e.g. giving you aspirin right now does something different from giving me aspirin now). We're learning to learn those things super efficiently (contextual bandits and active learning). The current data science boom goes far far far far beyond deep learning, and most of the field is doing great. Maybe those bits will even get better faster if deep learning stops hogging the glory. More likely, we'll learn to combine these things in cool ways (as is happening now).

Honestly as much as it is slightly irritating to see deep learning hogging all the glory, there's a lot of money being sloshed around and quite a bit of it is spilling over to non-deep learning too. Which is great. An AI winter may be coming, though I think it's at minimum several years off, since big enterprises are just getting started with the most hyped things. If the hype doesn't return on its promises enough for sustained investment (that's a rather big if since the low hanging fruit aren't yet all picked) then the companies and funding will eventually recede, maybe even trigger another winter, but just as it takes a while to ramp up, it will also take a while to course correct. In the meantime all the related areas get better funding and attention (and chance to positively contribute to secure further investment) that they'd otherwise not have since we'd still be stuck in the low funding model from the last winter.

I think the problem is the definition of AI. It appears most in the field define it as a superset of ML, encompassing all kinds of statistical methods and data analysis. For the general public, AI is a synonym for deep learning. When large companies speak about AI they always mean deep learning, never just a regression (probably also because many don't see a regression as intelligent). So AI in the public's perception could face a winter but much of the domain of machine learning would be unaffected.

>For the general public, AI is a synonym for deep learning

I'd contend for the general public, AI is a synonym for machines like: HAL; The Terminator; Star Trek's "Data"; the robots in the film "AI"; and so on.

We're nowhere remotely in the vicinity of that, and no-one even has any plausible ideas about how to start.

A random person outside of tech probably doesn't even know what deep learning is. They might have heard of it somewhere in passing.

Bayesian can be seen as a subset of deep learning or hell a superset.

AI is a superset and Machine learning is a subset of AI and most funding is in deep learning. Once Deep Learning hit the limit I believe there will be an AI winter.

Maybe there will be hype around statistic (cross fingers) which will lead to Bayesian and such.

>Bayesian can be seen as a subset of deep learning or hell a superset.

eh-hem

DIE, HERETIC!

eh-hem

Ok, with that out of my system, no, Bayesian methods are definitely not a subset of deep learning, in any way. Hierarchical Bayes could be labeled "deep Bayesian methods" if we're marketing jerks, but Bayesian methods mostly do not involve neural networks with >3 hidden layers. It's just a different paradigm of statistics.

How can Bayesian stuff be seen as a subset or superset of deep learning?

Forget about self driving cars - the real killer application of deep learning is mass surveillance - there are big customer for that (advertising, policing, political technology - we better get used to the term) and its the only technique that can get the job done.

I sometimes think that there really was no AI winter as we got other technologies that implemented the ideas: SQL Databases can be seen as an application of many ideas in classical AI - for example its a declarative language for defining relations among tables; you can have rules in the form of SQL stored procedures; actually it was a big break (paradigm shift is the term) in how you deal with data - the database engine has to do some real behind the scenes optimization work in order to get a workable representation of the data definition (that is certainly bordering on classical AI in complexity).

these boring CRUD applications are light years ahead in how data was handled back in the beginning.

The BBC recently requested information about the use of facial recognition from UK police forces. Those that use facial recognition reported false positive rates of >95%. That led some to abandon the systems, others just use it as one form of pre-screening. Mass surveillance with facial recognition is nowhere near levels where it can be used unsupervised. And that's even before people actively try to deceive it.

For advertising, I'm also not sure if there's been a lot of progress. Maybe it's because I opted out too much but I have the feeling that ad targeting hasn't become more intelligent, rather the opposite. It's been a long time that I've been surprised at the accuracy of a model tracking me. Sure, targeted ads for political purposes can work very well but are nothing new and don't need any deep learning nor any other "new" technologies.

Where I really see progress is data visualisation. Often dismissed it can be surprisingly hard to get right and tools around that (esp for enterprise use) have developed a lot over recent years. And that's what companies need. No one's looking for a black-box algorithm to replace marketing, they just want to make some sense of their data and understand what's going on.

Aha, yeah I saw this in the news - pretty classic use of people horribly misunderstanding statistics and/or misrepresenting the facts. Let's say one person out of 60 million has sauron's ring. I use my DeepMagicNet to classify everyone, and get 100 positive results. Only one is the ringbearer, so I have a 99% error rate. Best abandon ship.

> Those that use facial recognition reported false positive rates of >95%.... Mass surveillance with facial recognition is nowhere near levels where it can be used unsupervised.

Is this deep neural networks with the latest technologies?

While yes, deep learning isn't going to solve everything, we'll probably see significant changes in the products available as this technology discovered the past few years makes into the real world.

Most scanners that do OCR and most forms of facial recognition isn't using deep neural networks with transfer learning, YET.

This is not to say that discoveries will continue, winter is probably coming :)

95% false positive rate is extremely good for surveillance, as the cost of false positive is low (wasted efforts by the police). That means for every 20 people the police investigate, one is a target.

I disagree. Even if there are a number of big customers for mass surveillance, self driving cars fundamentally changes the platform of our economy for everyone.

That sounds like US-centric PoV. For large portion of (most?) Europeans self-driving won't change much in their lives, definitely not anything major.

> Deepmind hasn't shown anything breathtaking since their Alpha Go zero.

Didn't this just happen? Maybe my timescales are off, but I've been thinking about AI and Go since the late 90s, and plenty of real work was happening before then.

Outside a handful of specialists, I'd expect another 8-10 years before the current state of the art is generally understood, much less effectively applied elsewhere.

I had the same response. AlphaZero was published like 5 months ago. Saying they've reached the end of the line because they haven't matched AlphaZero in six months is lame.

It also marked the end of a major multi-year project. With Deepmind moving that team to focus on other problems I wouldn't expect immediate results.

Also why does every single result has to be breathtaking? Here's a quick example, at IO they announced that their work on Android improved battery life by up to 30%. That's pretty damn impressive.

> Also why does every single result has to be breathtaking?

If you build the hype like say Andrew Ng it better be. Also if you consume more money per month than all the CS departments of a mid sized country, it better be.

Because it's the only time we see it in action. The speech recognition of my Amazon Echo is still subpar (and it feels like it's getting worse each week) and ad targeting also hasn't really improved. Of all those claims that came with deep learning, Go was the only time where you really saw a result. I'm not sure which version of Android will bring the improved battery life (and which manufacturers) but I wouldn't be surprised if the 30% were a bit optimistic.

I get that a lot of services we use on a daily basis make use of deep learning to accomplish tasks. But I don't really see what has fundamentally changed over the past 5 years in the way I use services. Siri was introduced 7 years ago and while we have clearly made progress in voice recognition, it's nowhere close to what many had hoped.

Warning 23 year old CS grad angst ridden post:

I'm very sick of the AI hype train. I took a PR class for my last year of college, and they couldn't help but mention it. LG Smart TV ads mention it, Microsoft commercials, my 60 year old tech illiterate Dad. Do any end users really know what it's about? Probably not, nor should that matter, but it's very triggering to see something that was once a big part of CS turned into a marketable buzzword.

I get triggered when I can't even skim through the news without hearing Elon Musk and Steven Hawking ignorantly claim AI could potentially takeover humanity. People believe them because of their credentials, when professors who actually teach AI will say otherwise. I'll admit, I've never taken any course in the subject myself. An instructor I've had who teaches the course argues it doesn't even exist, it's merely a sequence of given instructions, much like any other computer program. But hey, people love conspiracies, so let their imagination run wild.

AI is today what Big Data was about 4 years ago. I do not look highly on any programmer that jumps bandwagons, especially for marketability. Not only is it impure in intention, it's foolish when their are 1000 idiots just like them over-saturating the market. Stick with what you love, even if it's esoteric. Then you won't have to worry about your career value.

You can finish a CS undergrad without taking any AI course? Or just haven't taken one yet? It's very helpful to go through even a tiny bit of AI: A Modern Approach to cut through a lot of the hype. What annoys me is that when people say "Machine Learning" these days they almost invariably mean deep learning, ignoring all the rest of AI.

> I can't even skim through the news without hearing Elon Musk and Steven Hawking ignorantly claim AI could potentially takeover humanity.

Have you considered that their claims may not in fact be ignorant, just the reporting around them? For some details perhaps you would start with this primer from a decade ago, section 4 & 5 if you're in a hurry: http://intelligence.org/files/AIPosNegFactor.pdf

Or if you want a professor's opinion, from one of the co-authors to the previously mentioned AI:AMA check out some of the linked pointers on his home page: http://people.eecs.berkeley.edu/~russell/

I know right, the school I went to wasn't exactly the best.

I skimmed through sections 4 & 5, optimization processing was difficult to understand.

When I was in elementary school, I remember pitying the mentally disabled children, knowing their financial success was destined, so I connected with the g-factor definition. I really think general intelligence is more of a sense of all clusters awareness, whether it be social or cognitive. I've met tons of great students in Math courses who simply cannot converse with the general public. I've also met tons of people on the streets of my city who would have a difficult time understanding high school algebra.

As for Section 5, I do think the rise of AI over humanity is completely in our grasp. I really should take a course on the subject before I sound like the people that I'm criticizing for ignorance, but from a general perspective, I cannot see it outside of our control. As Eliezer said, we can make predictions, but only time will clear the fog.

> What annoys me is that when people say "Machine Learning" these days they almost invariably mean deep learning, ignoring all the rest of AI.

But that's not people's fault. Companies only say AI if they mean deep learning. I've yet to hear a company advertising AI if they accomplished it with a linear regression. Maybe experts should stop talking about AI and use specific terms instead (Deep Learning in the case of this article).

Your post has two different points, and I think they should be separated. Yes, there's an AI hype train, and it's pretty tiring. I'm starting to wonder when the first AI-enhanced shoes are going to come out. Or an AI enhanced blockchain, right?

That said, the rest of the rant about AI is less solid. Sure, AI today is fairly boring and run of the mill data processing / optimisation stuff (I sort of know, though I only have an MSc in the topic). Much of the promise of near-future AI is that we can go from human designed, or shall we say human-bootsrapped AI to self-bootstrapping kind. The fact that AlphaGo pretty much does this (in a very limited capacity), and accomplished something which classical programming and game "AI" couldn't, should show us that we're pretty close to this type of AI being highly effective. How exactly the future unfolds from there is anyone's guess, but outright calling it ignorant is... pretty ignorant, IMHO.

> I'm starting to wonder when the first AI-enhanced shoes are going to come out.

Wait no more:

The first connected cycling insole with artificial intelligence: https://www.digitsole.com/store/connected-insoles-cycling-ru...

The World's First Intelligent Sneaker: https://www.kickstarter.com/projects/141658446/digitsole-sma...

I never claimed AI was boring or run of the mill, just that it's not in my current interest. It's when I hear Hawking make claims like this, I call ignorance.

"Computers can, in theory, emulate human intelligence, and exceed it,' he said. 'Success in creating effective AI, could be the biggest event in the history of our civilization. Or the worst. We just don't know. So we cannot know if we will be infinitely helped by AI, or ignored by it and side-lined, or conceivably destroyed by it.”

I feel like I'm in a high school stoner circle, yeeesh!

I'm in same boat, except now I have to deal with both this and blockchain.

It's pretty clear to everyone pushing this is being dishonest. Sometimes I wonder if they intentionally being dishonest or if they just don't know what they don't know. Very few people are doing useful true machine learning, and the applications are very specific with its own set of quirks. It's just to make some quick money and get out.

What somehow is never being said with all this hype is that you need to hire good software devs to make a great solution to a problem. All these buzzword driven ideas tend to confuse a bunch of people and die out after wasting everyone's money. In my region I keep seeing a bunch of business suits pushing X idea being solved with Y with no justification or engineers behind them.

Yesterday I have found out I have to have yet another meeting to explain why "blockchain" does not translate to a full solution given to a very superficial poorly thought out problem.

The only reason I've had a voice in my region with all this noise is because I've made things people see that actually work.

I work for a consultancy and I was brought in to talk about ‘our blockchain strategy’, because I was one of the few people at the company who actually read the white papers and had been invested in it. This was towards the end of the last bull run. I think they expected me to tell them to go all in on it, but I essentially said it was a bunch of bullshit hype and a solution in search of a problem and then I didn’t get invited to any more meetings. A few weeks later they announce we’re going to start selling block chain solutions right as the crypto market crashes.

Meanwhile in the real world they can’t even figure out the basics like containers and Cicd which we’re actually dealing with in our actual contract.

> Very few people are doing useful true machine learning, and the applications are very specific with its own set of quirks.

I worked on nudity detector in 2017. Deep learning works, and is useful. Although you are right it's very specific and quirky.

I found "How HBO's Silicon Valley built Not Hotdog" article very interesting, because it's basically the same problem. They found MobileNet better than SqueezeNet and ELU better than ReLU. You know what? We found SqueezeNet better than MobileNet and ReLU better than ELU for our problem and data. Who know why.

https://medium.com/@timanglade/how-hbos-silicon-valley-built...

But this is normal with any new technology -- there used to be car movie places and people thought of putting miniature nuclear reactors in everyday appliances, it doesn't mean there's a car winter coming or a nuclear winter coming(pun intended).

It's absolutely human nature to think of crazy ways of using things in new ways, probably most of those ways don't work out in the end.

The history of humanity is full of examples of people using technology to prevail over other groups of people. Applications of AI and ML will be no exception: computer vision, game theory, autonomous systems, material design, espionage, cryptography... You name it.

Supraintelligent AI is not required to cause severe problems.

I'm sure they have, but none of those fields have been exploited for marketability, at least nowhere near this degree.

AI taking over is the biggest threat facing humanity, but I don't think Hawking ever claimed it was imminent; it's likely 1000+ years away.

It should be obvious why a superior intelligence is something dangerous.

"it's likely 1000+ years away."

That seems like a pretty high number when you consider the exponential rate of technological advancement.

1000 years in the future is probably going to be completely unrecognizable to us given the current rate of change in society/tech.

The day we create a superior intelligence will be the greatest day of humanity. It will be fantastic if there's something beyond humanity. A logical, evolutionary conclusion to us.

All our Darwinian ancestors never had the capacity for intellectual fear of their superior successors...

Hopefully we as a species can get beyond our fear.

It would be great if people feared climate change the same way they fear AI takeover.

The biggest threat facing humanity is 1000+ years away?

AI is already taking over humanity. The news you read, the products you buy, the advice you take, the friends you meet are all partially or fully supported by machine-learned algorithms.

Personally, I do not doubt AI-based methods are changing our language, our communication patterns and our transport infrastructure.

The problem with the statement "AI will take over humanity" is actually in:

- What exactly is AI? There are many definitions. Most researchers adopt the weakest forms, whereas the general public adopts the strongest form.

- What exactly is 'take-over'? Does this mean: in control? Like a dictator is in control over a country? Or: adopting us as slaves? As a gradual change, when does it 'take-over'? At 50%? Does this need to be a conscious action by an AI actor, or would an evolutionary transition suffice?

- What exactly is humanity? I would go for the definition: "the quality or state of being human", but most people probably read in it: "the human race". In the former case, technology is a part of the quality of being human. In Heideggerian fashion, we become the technology and the technology becomes us. Technology, and AI as part of it, has been taking over humanity since we started permanently adjusting our environments.

> People believe them because of their credentials, when professors who actually teach AI will say otherwise

While some experts like Andrew Ng are sceptical of AI risk, there are lots of others like Stuart Russell who are concerned.

Here is a big list of quotes from AI experts concerned about AI risk: http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-ri...

I mean at least you don't have to worry about Steven Hawking bothering you any more...

>I get triggered

You should be more considerate than to be throwing around this term.

Author's reasons:

1.Hype dies down (which is really good! Meaning the chance of burst, is actually lower!)

2.Doesn't scale is false claim. DL methods have scaled MUCH better than any other ML algorithms in recent history (scale SVM is no small task). Scaling for DL methods are much either as comparing to other traditional ML algorithms, where it can be naturally distributed and aggregated.

3. Partially true. But self-driving is a sophisticated area by itself, DL is part of it, it can't really put full claim on its potential future success or ultimate downfall.

4. Gary Marcus isn't an established figure in DL research.

AI winter will ultimately come. But it is because people will become more informed about DL's strengths and limits, thus becoming smarter to tell what is BS what is not. AGI is likely not going to happen just with DL, but that is no way meaning it is a winter. DL has revolutionized the paradigm of Machine Learning itself, the shift has now complete, it will stay for a very very long time, and the successor is likely to build upon it not subvert it completely as well.

Author here: I'm using deep learning daily so I have a bit of an idea on what I'm talking about.

1) Not my point. Hype is doing very well. But narrative begins to crack, actually indicative of a burst... 2) DL does not scale very well. It does scale better than other ML algorithm because those did not scale at all. If you want to know what scales very well, look at CFD (computational fluid dynamics). DL in nowhere near that ease in scaling. 3) self driving is the poster child of current "AI-revolution". And it is where by far most money is allocated. So if that falls, rest of DL does not matter. 4) Not that this matters, does it?

CFD is good at using big machines "efficiently", but the cost of DNS scales as the cube of the Reynolds number which will never be tractable for most engineering problems. Apart from niche basic research on the edge of tractability, all the effort goes into modeling (RANS, DES, wall, etc.) to deliver statistically calibrated estimates of functionals of interest at feasible cost. Those methods actually don't "scale as well" (though the state of research is ahead of commercial software), but also don't need to because they can solve the problem in less time with less hardware. This situation is actually pretty similar to your DL analogy where more hardware provides diminishing returns for solving the actual problem.

The scaling argument in the article doesn't make any sense. There are rhetorical queries like "does this model with 1000x as many parameters work 1000x as well?" but what it means to scale or perform are not clearly or consistently defined - let alone defined in a way that would make your point about the utility of the advances.

OpenAI's graph shows new architectures being used with more parameters because people are innovating on architecture and scale at the same time. Arguing that old methods "failed to scale" is like arguing that processor development was a failure because Intel had to develop a 486 instead of making a 386 work with more transistors (or more something).

And what does CFD have to do with anything, except maybe an odd attempt to argue from authority? Can you formalize from CFD a notion of "scaling well" well that anyone else agrees is useful for measuring AI research?

2)Why do you think DL doesn't scale? I am curious. It can easily leverage thousands of GPUs, training on 300 millions of images (https://ai.googleblog.com/2017/07/revisiting-unreasonable-ef...). No other methods is even close to leverage that amount of computational power. I don't really know about CFD, but at least in ML land and dealing with ML problems, DL is very scalable, maybe only next to random forests style algorithm, where they effectively share nothing.

3)It does matter. In fact most valuable startup around DL are CV based startups, they are mainly located in China though.

What do you think of advances like those in major DeepMind papers? They seem to represent significant shifts in capabilities once matured.

Here's a recent example:

Unsupervised Predictive Memory in a Goal-Directed Agent

https://news.ycombinator.com/item?id=17177442

"Author here: I'm using deep learning daily so I have a bit of an idea on what I'm talking about."

Very weak to appeal to authority. The only true argument I can find against DL/ML/AI atm is the continuing appeal to authority by PhDs who have zero engineering knowledge, zero business sense and zero understanding of risk assessment.

I’m not sure how the hype wagon started but I for one am glad it’s about to pop.

I am working (founded) a startup and while we have AI on the roadmap for about a years time, it isn’t something that’s central to our product. (We already use some ML techniques but I’d not confidently boast its the same thing as AI).

Cue an informal lunch with a VC guy who takes a look, says we’re cool and tells us just to plaster the word AI in more places - he was sure we could raise a stupendous sum of cash doing that.

As an AI enthusiast I was bothered by this. We have everyone and their mother hyping AI into areas it’s not even relevant in, let alone effective at.

A toning down would be healthy. We could then focus on developing the right technology slowly and without all the lofty expectations to live up to.

> ...just to plaster the word AI in more places...

I bet that's how the new "AI-assisted" Intellisense in Visual Studio got greenlit:

https://blogs.msdn.microsoft.com/visualstudio/2018/05/07/int...

If an AI-infested text editor isn't a sure sign that the bubble is going to pop soon then I don't know ;)

I think part of this ridiculousness may just be that the common lexicon doesn't have enough words for AI. "AI-assisted Intellisense" at the moment seems to boil down to a marginally novel way of ordering autocomplete suggestions, and like...

It's not wrong? It's neat, it's potentially useful, and it's powered by something under the umbrella of "AI". The problem is that that umbrella is gigantic, and covers everything from the AI system providing routes for the AI system in a self-driving truck on AI-provided schedules for a hypothetical mostly-automated shipping business, to a script I whipped up in ten minutes to teach something 2+2 and literally nothing else.

So we get to the nonsense position where there isn't a better way to describe a minor improvement to what is essentially the ordering of a drop-down list except by comparison to the former example.

AI winter is not on its way. We constantly get new breakthroughs and there's no end in the view. For example, in the last year a number of improvements in GANs were introduced. This is really huge, since GANs are able to learn a dataset structure without explicit labels, and this is a large bottleneck in applying ML more widely.

IMO, we are far away from AGI, but even current technologies applied widely will lead to many interesting things.

I sure agree there are many interesting things going on, there is no question about that. Also most of them are toy problems focused in some restricted domains, while a huge bag of equally interesting real world problems is sitting untouched. And let me tell you, all those VC's that put in probably way north of $10B are not looking forward to more NIPS papers or yet another style transfer algorithm.

>Also most of them are toy problems focused in some restricted domains, while a huge bag of equally interesting real world problems is sitting untouched.

It always starts with toy problems. Recognizing pictures from imagenet was also a toy problem back then.

"Deepmind hasn't shown anything breathtaking since their Alpha Go zero"

... what about when the Google assistant near perfectly mimicked a human making a restaurant reservation .... the voice work was done at DeepMind.

All the problems in AI haven't been solved yet? Well no, of course not. Limitations exist and our solutions need to be evolved.

I think perhaps the biggest constraint is requiring huge amounts of training data so solve problem X. Humans simply don't need that, which must be some indication that what we're doing isn't quite right.

what about when the Google assistant near perfectly mimicked a human making a restaurant reservation

Any sufficiently advanced technology is indistinguishable from a rigged demo

The Google Duo demo was definitely very impressive. The only caveat I can think of is that they didn't say how often it works. That might have been the one success from hundreds of calls.

> what about when the Google assistant near perfectly mimicked a human making a restaurant reservation

That seemed pretty staged.

> Humans simply don't need that, which must be some indication that what we're doing isn't quite right.

Not really. DNNs are much simpler and need to be much more specialized towards specific tasks than the human brain. They're more like a nematode that was optimized by evolution for millions of years to tell cats and dogs apart because it preferentially infects dogs. It's the only thing it does. It does it via some shortcuts and it won't be able to learn chess without a radical redesign.

Not to mention that primate brains take some time to get bootstrapped. A toddler has to be fed visual input for many months before they can be left unsupervised for reasonable amounts of time. And have you seen what those "optical illusion" things do? It's a miracle that those humans can self-navigate at all considering those failure modes!

Disclaimer: I am a lay technical person and don't know much about AI.

I find this article somewhat condescending. I look at all the current development as stepping stones to progress, not an overnight success that does everything flawlessly. I imagine the future might be some combination of different solutions, and what the author proposes may or may not play a part in it.

I agree. Overhype is annoying, but it happens with every technological advance. So does the inevitable backlash jump from "some people claim too much" to "this whole field is a bubble with no real gains". Both are cliched viewpoints that give a false sense of being "in the know" without helping anyone navigate change effectively.

it's not a stepping stone, if you look closely it's a dead end

I don't see how systematically accurate image classifiers and facial recognition systems built on deep learning is a 'dead end'. Products are products. If deep learning has led to actual profits in actual companies, it's not a dead end. As to whether this leads to AGI is a completely different question.

What do you see exactly when you look closely and how do you know it is dead end?

An AI "winter" is a long period in which [edit: funding is cut because...] researchers are in disbelief about having a path to real intelligence. I think that is not the case at this time, because we have (or approaching) adequate tools to rationally dismiss that disbelief. The current AI "spring" has brought back the belief that connectionism may by the ultimate tool to explain the human brain. I mean you can't deny that DL models of vision look eerily like the early stages in visual processing in the brain (which is a very large part of it). Even if DL researchers lose their path in search for "true AI", the neuroscientists can keep probing the blueprint to find new clues to its intelligence. Even AI companies are starting to create plausible models that link to biology. So at this time, it's unlikely that progress will be abandoned any time soon.

E.g. https://arxiv.org/abs/1610.00161 https://arxiv.org/abs/1706.04698 https://www.ncbi.nlm.nih.gov/pubmed/28095195

No, AI winter was when the AI people oversold the tech, then failed to deliver, and lost their funding. This is well documented in histories of the field.

I think the scientific pessimism preceded the funding cuts:

https://en.wikipedia.org/wiki/AI_winter

The brain has a number of functional parts that we don't understand all that well. Research on the brain hits a wall every now and then, but you never hear the phrase "Neuroscience Winter".

We're starting to train models that match biological brain behavior, at least in some crude functional/structural sense. Maybe this is just a string of coincidences, but my guess is that discoveries of analogous biological/model components will continue to happen, and we'll be able to learn more about AI and the brain by linking related phenomenon in vitro and in silica.

A few more recent examples:

https://deepmind.com/blog/grid-cells/

https://www.nature.com/articles/nature04485

Neuroscience has been in constant winter, mainly because the methods are too crude and small scale, and that by observing 100 neurons out of billions makes it impossible to tell the whole story. The deep mind papers are interesting, but they are just a start imho. It may be that they are focusing on a mere coincidence. Nevertheless it is exciting to see progress in that direction, thats why i don't think DL research is going to lose steam soon. It would require a major show-stopper discovery (like the minsky-papert paper). Tesla crashes are no such thing.

It seems CS people are sick and tired of the hype, but i feel neuroscientists are now warming up to it.

That’s not what the AI Winter was. It was when anything that used the term “AI” (including academic research) was unfundable.

From what I’ve read of the last AI winters, funding was more centralised, & projects were larger and fewer. Hardware was scarcer.

These days anyone with a few dollars to spend on compute time and some free software can do machine learning. I can’t see that going away.

Author here: seriously I'm here at the front page for the second day in the a row!?

The sheer viral popularity of this post, which really was just a bunch of relatively loose thoughts indicates that there is something in the air regarding AI winter. Maybe people are really sick of all that hype pumping...

Just a note: I'm a bit overwhelmed so I can't address all the criticism. One thing I would like to state however, is that I'm actually a fan of connectionism. I think we are doing it naively though and instead of focusing on the right problem we inflate a hype bubble. There are applications where DL really shines and there is no question about that. But in case of autonomy and robotics we have not even defined the problems well enough, not to mention solving anything. But unfortunately, those are the areas where most best/expectations sit, therefore I'm worried about the winter.

Do you think there's a balance to be struck between connectionism and computationalism?

The fact you use the word connectionism makes you alright in my book :)

The argument is that self-driving won't work because Uber and Tesla had well-publicized crashes. But I don't see how this tells us anything about other, apparently more cautious companies like Waymo. There seem to be significant differences in technology.

More generally, machine learning is a broad area and there's no reason to believe that different applications of it will all succeed or all fail for similar reasons. It seems more likely there will be more winners along with many failed attempts.

Yes. I've been saying this for a while. Waymo's approach is about 80% geometry, 20% AI. Profile the terrain, and only drive where it's flat. The AI part is for trying to identify other road users and guess what they will do. When in doubt, assume worst case and stay far away from them.

I was amazed that anyone would try self-driving without profiling the road. Everybody in the DARPA Grand Challenge had to do that, including us, because it was off-road driving and you were not guaranteed a flat road. The Google/Waymo people understood this. Some of the others just tried dumping the raw sensor data into a deep learning system and getting out a steering wheel angle. Not good.

A lot of companies fear the ship's leaving without them. They try to rush ahead without thinking and then crashes. That's what it feels like every time a car company or another tech company say they're going to build the next self-driving car. I don't know why but I always feel like waymo is already 10,000 miles ahead.

Seriously. Frankly, based on Uber's culture I would have been surprised if they didn't kill at least one person with their self-driving efforts. It's a total non-data point. The fact that Uber got as far as they did without killing anyone is strong evidence that the problem is tractable.

As for Tesla - Tesla isn't even trying to make proper self-driving cars. Tesla's goal has always been assisted driving. However you feel about that, it's really not relevant to the success or failure of self-driving cars.

OP can't possibly have been ignorant of the fact that Waymo is the clear leader here with a substantial head start, and a proven record (and an actual fleet of self driving cars now on the road), and yet he chose not to mention it. That really undermines his credibility for me - he seems clearly more interested in making his point than in accurately engaging with reality.

Honestly, I think this is a good thing for both AI researchers as well as AI practitioners. One mans AI-winter is another mans stable platform.

While the number of world-shattering discoveries using DL may be on the decline (ImageNet, Playing Atari, Artistic Style Transfer, CycleGAN, DeepFakes, Pix2Pix etc), now both AI researchers and practitioners can work in relative peace to fix the problem of the last 10%, which is where Deep Learning has usually sucked. 90% accuracy is great for demos and papers, but not even close to useful in real life (as the Uber fiasco is showing).

As an AI practitioner, it was difficult to simply keep up with the latest game-changing paper (I have friends who call 2017 the Year of the GAN!), only to later discover new shortcomings of each. Of course, you may say, why bother keeping up? And the answer is simply that when we are investing time to build something that will be in use 5-10 years from now, we want to ensure the foundation is built upon the latest research, and the way most papers talk about their results makes you believe they are best suited for all use cases, which is rarely the case. But when the foundation itself keeps moving so fast, there is no stability to build upon at all.

That and what jarym said is perfectly true as well.

The revolution is done, now it's time to evolution of these core ideas for actual value generation , and I for one am glad about that.

AI winter? Hardly. Current methods have only been applied to a very tiny fraction of problems that they can help solve. And, this trend will only accelerate until computing resources become too expensive.

As long as there is ROI, AI projects will continue to be financed, top thinkers around the world will be paid to do more research, and engineers will implement the most recent techniques into their products and services to stay competitive. This is a classic feedback system that results in exponential progress.

This seems over negative. Just the opening argument, that companies were saying "that fully self driving car was very close" but "this narrative begins to crack"

Yet here they are self driving https://www.youtube.com/watch?v=QqRMTWqhwzM&feature=youtu.be and you should be able to hail one as a cab this year https://www.theregister.co.uk/2018/05/09/self_driving_taxis_...

You sure about that? https://www.youtube.com/watch?v=8IqpUK5teGM

Tesla self driving cars have crashed too. Arrogant people like Elon Musk are making a bad name for the hardworking AI developers who are actually trying to make self driving cars faulty proof.

That vid of the Uber crash - just because one company has a crap product doesn't mean they are all bad. Waymo is about the only one that seems about ready to go and I think partly because they don't just count on deep learning.

The video is PR.

So far, no independent journalist or reviewer was allowed to test the system with the exception of a few tightly controlled rides.

Don't take it too seriously.

I thought there would be more of a backlash / winter onset when people realize that Alexa is so annoying to deal with (and you basically have to learn a set of commands) because AI isn't that clever yet. Also, when people realize that autocorrect took a dive for making edits when Google started putting a neural net in charge. (No! Stop deleting random words and squishing spaces during edits).

In other words I figured it would be the annoyances at what "should be easy by now" that would get Joe CEO to start thinking "Hm. Maybe this isn't such a good investment." When measurements are made and reliable algorithmic results attract and keep more users than narrowly trained kind of finicky AIs.

I don't want there to be an AI winter, and it won't be as bad as before. There are a lot of applications for limited scope image recognition, and other tasks that we couldn't do before. Unfortunately,I do agree with the post that winter is on its way.

The OP is obviously not keeping up with the field and has lot to learn about scientific approach. He basically uses the count of tweets from AndrewNg and crashes from risk-taking companies as indicator of "AI winter". He should have tried to look in to metrics such as number of papers, number of people getting in to field, number of dollars in VC money, number of commercial products using DL/RL etc. But you see, that's a lot of work and your conclusion might not align with whatever funky title you had in mind. Being an armchair opinion guy throwing link bait titles is much more easier.

I'll happily read your next post where you will include all of those. In fact amount of VC money spent in that field would only support my claim. And the number of papers is irrelevant. There were thousands of papers about Hopfield network in the 90's and where are all of them now? You see, all the things you point out is the surface. What really matters is that self driving cars crash and kill people, and no one has any idea how to fix it.

I think the most important question is what 'winter' really means in this context. The new concepts in AI tend to follow the hype cycle so the disillusionment will certainly come. One thing is the general public see the amazing things Tesla or Google do with deep learning and extrapolate this thinking we're on the brink of creating artificial general intelligence. The disappointment will be even bigger if DL fails to deliver its promises like self-driving cars.

Of course the situation now is different than 30 years ago because AI has proved to be effective in many areas so the research won't just stop. The way I understand this 'AI winter' is that deep learning might be the current local maximum of AI techniques and will soon reach the dead end where tweaking neural networks won't lead to any real progress.

AI winter is not "on its way". There is AI hype and anti-AI hype, and then there is actual practice. This article is anti-AI hype, just as bad as its opposite. In practice there are tons of useful applications. We haven't even begun to apply ML and DL to all the problems laying around us, some of which are quite accessible and impactful.

The hype cycle will pass with time, when we learn to align our expectations with reality.

> AI winter is not "on its way". There is AI hype and anti-AI hype, and then there is actual practice. This article is anti-AI hype, just as bad as its opposite. In practice there are tons of useful applications.

I don't think the article was saying that AI isn't useful, but just that deep learning specifically is not an AI panacea, and that the current hype around AI is on its way out. The hype dying down and the associated buzzwords starting to repel money instead of attract it is all that's meant by AI winter, I believe, not that we'll run out of places where the techniques would be useful.

Give us a list then, because what I’m seeing is shoehorning “AI” into everything but not with significant results.

It's not my job to supplement the press or to do paper reading for you. If you're really interested in AI and don't just read the clickbait press, then open Arxiv and look there.

http://www.arxiv-sanity.com/

If, on the other hand, your opinion that AI is in a winter has been already decided without reading the latest scientific papers, then there's nothing I can say to you that will change your mind.

I think you will see the "hype" cycle last for a very, very long time.

It is a very different thing when the computer can start doing things that only humans could do.

Look at all the drama after Google did the Duplex demo.

We have barely even got started with self driving cars.

But ultimately it is about money and the profits that can be made from AI/ML are just huge. So that will fuel the hype.

But also AI changes the calculus of companies competing. Before a product was sold and deteriorate over time.

Now a product is sold and gets better over time. Increasing moats and making it much more difficult to compete. So AI becomes a far more valuable thing than maybe we ever had in the past.

I think that we will have AI Winter once we see the true limitations that face us having a level 5 fully autonomous self driving car. The other thing we will see happen is the deflation of the AdTech bubble. Once we see both of these events occurring that should start the AI Winter.

I agree. The AdTech sphere is keeping the current hype alive more than anything else. There's some obvious imbalances in AdTech that should lead it to a damning end soon enough.

AI and machine learning is a tool. Like any other tool it's perfect for some problems and doesn't work well for other. Pick the right tools for the problem that you are working on. Don't follow the hype and don't use AI/ML just for sake of using it.

Judea Pearl sees a way out of the winter.

https://www.theatlantic.com/technology/archive/2018/05/machi...

I think a lot of GOFAI approaches ought to be revisited to see whether they benefit from the new perceptual and decision capabilities of Deep Learning systems. Alex Graves's papers are particularly good at this.

Things like this reinforcement learner for theorem proving are pretty exciting possibilities. https://arxiv.org/pdf/1805.07563v1.pdf

I can recommend his new book, "The book of Why" very highly. Even though I am very familiar with Bayes nets, I discovered that that a lot of progress has been made in that side of AI.

There's a lot of good stuff coming from research in AI these days. Still, I think the author's right.

As with the onset of the previous AI winter a generation ago, the problem is this: Once a problem gets solved (be it OCR or Bayesian recommendation engines or speech recognition or autocomplete or whatever) it stops being AI and starts being software.

As for self-driving cars: I recently took a highway trip in my Tesla Model S. I love adaptive cruise control and steering assistance: they reduce driver workload greatly. But, even in the lab-like environment of summertime limited access highways, driverless cars are not close. Autosteer once misread the lane markings and started to steer the car into the side of a class 8 truck. For me to sit in the back seat and let the car do all the work, that kind of thing must happen never.

Courtesy is an issue. I like to exit truck blindspots very soon after I enter them, for example. Autosteer isn't yet capable of shifting slightly to the left or right so a driver ahead can see the car in a mirror. Maybe when everything is autonomous that won't be an issue. But how do we get there.

Construction zones are problems too: lane markings are confusing and sometimes just plain wrong, and the margin for error is much less. Maybe the Mobileye rig in my car can detect orange barrels, but it certainly doesn't detect orange temporary speed limit signs.

This author is right. AI is hype-prone. The fruits of AI generally function as they were designed, though, once people stop overselling them.

While I basically agree, really it ought to be called "AI autumn is well on its way", since I'm not sure we're into actual winter (i.e. dramatic reduction in $$ available for research) quite yet. But, probably soon.

Author here, yeah, it is the autumn. But I guess not many people would recognize the meaning, winter on the other hand is not ambiguous...

True.

"it is striking that the system spent long seconds trying to decide what exactly is sees in front (whether that be a pedestrian, bike, vehicle or whatever else) rather than making the only logical decision in these circumstances, which was to make sure not to hit it."

That is striking. It always sort of bothered me that AI is really a big conglomeration of many different concepts. What people are working on is deep learning for machines, but we think that means "replicating human skill/behavior". It's not. Machines will be good at what they are good at, and humans good at what they're good at. It's an uphill battle if your expectation is for a machine that processes like a human, because the human brain does not process things like computer architectures do.

Now, if some aspiring scientist wanted to skip all that and really try to replicate (in a machine) how the human brain does things, I think such a person would be starting from a very different perspective than even modern AI computing.

That's why Augmented Intelligence is a better term. It doesn't scare up visions of Skynet or Hal 9000 run amok. Nor does it promise utopian singularity right around the corner.

It just means better tools to increase human capacity. But it's not nearly as good at getting headlines in the media.

We call it deep learning, but it is deep pattern matching. Extremely useful, but don't expect it to result in thinking machines.

Are our brains magic? If they aren't then surely they must be doing something that we can reproduce. We've built so many things that we considered "thinking machines" in the recent past (realistic speech synthesis, image recognition and captioning, human-level translation, elaborate recommender systems, robust question answering) on "deep pattern recognition".

Brains are not magic, and will be reproduced eventually, but DNNs are a fundamentally weaker architecture and won't be enough. Neural nets can solve some problems that brains can solve easily and lots of other ML methods couldn't solve, which is great. But the space of problems that brains can solve and neural nets can't is still rich, and will remain so until better methods are developed.

I believe we can make thinking machines. And "deep pattern matching" will be part of it. But it will need to fit in another architecture, most likely a lot like (deep) reinforcement learning. Probably with strong focus on modeling and predicting the environment. But also with some kind of goal modeling.

See also for this not-magic: https://psyarxiv.com/387h9

The discussion on radiology is extremely sloppy.

Andrew Ng claimed human level performance on one radiology task (pneumonia). This claim seems to hold up pretty well as far as I can tell. Then the person criticizing him on twitter posts results on a completely different set of tasks which are just baseline results in order to launch a competition. These results are already close to human level performance, and after the competition it's very possible they will exceed human level performance.

Yes it's true that doing well at only Pneumonia doesn't mean that the nets are ready to replace radiologists. However, it does mean that we now have reason to think that all of the other tasks can be conquered in a reasonably short time frame such that someone going into the field should at least consider how AI is going to shape the field going forward.

Well, the breathless hype around deep learning (with and without reinforcement learning) is bound to subside sooner or later, and attendance to staid academic conferences like NIPS sooner or later will revert back to a smaller group of academics and intellectuals who are truly interested in the subject over the long term.[a] That much is certain.

But we're still in the early stages of a gigantic wave of investment over the next decade or two, as organizations of all sizes find ways to use deep learning in a growing number of applications. Most small businesses, large corporations, nonprofits, and governments are not using deep learning for anything yet.

[a] https://twitter.com/lxbrun/status/908712249379966977

Well, now that the cat's out of the bag in regards to AI/ML, we can all get in on the ground floor of the next hype wave - quantum computing!

IMHO Quantum computing is as well hyped as Cold Fusion and shares some of its properties. Until "quantum supremacy" occurs or something that will show a real speedup we won't hear that much from it.

Cold Fusion was outright scientific misconduct. I'm not optimistic about QC working as intended, but I think the hope around it is honest.

Stopped reading after the first half. The evidence for the idea that deep learning is failing is that Deep Mind haven't produced anything revolutionary since Alpha Go Zero which was published not even a year ago? And that preformance doesn't scale linearly with the number of parameters? And speculation about why Lecun made a certain career decision? Not very convincing.

You're forgetting that Andrew Ng is tweeting 30% less this year! Isn't that enough to convince even the staunchest critic?

Only tangentially related to the article, but it's always struck me as a little unethical that Demis Hassabis' name goes on every paper that's written by Deepmind. No-one produces that much research output.

No, but wait! We're just on the verge of replacing doctors! ;-)

There's still a lot of space for the improvement of "curve-fitting" AI in the workplace. The potential of existing tech is far from being thoroughly exploited right now. I believe the next big improvements will come more from better integration in the workplace (or road system) than new scientific advances, so that might seem less sexy. But I also believe this will be a sufficient impetus to drive the field forward for the years to come.

I would not call it the “AI winter”. If you look at what people have called AI over time, the definition and the approaches have evolved (sometimes drastically) over time.

Instead of being stuck on the fact that deep learning and the current methods seem to have hit a limit I think I am actually excited about the fact that this opens the door for experimenting other approaches that may or may not build on top of what we call AI today.

Yeah, the problem is deep learning sucked a bunch of money - essentially took a loan against the future in the form of VC investments. And if that loan does not get payed, for the next few years you may not afford to explore all that other stuff.

Technically, VCs are not loaning you money. They’re more likely betting on you. It’s true that maybe they’ll be more reluctant to place big bets, but as with everything in life the VC optimism is cyclical.

Perhaps it'd be more correct to call it a "Strong AI Winter". We're no closer to "aware" machines. We've simply gotten very good at automating tasks that were once difficult to automate.

A friend that’s more optimistic about Strong AI once said that the ML that goes on today will probably serve the purpose of driving the peripheral sense organs of a future AI. Although it stretches a bit what’s possible today I could see that. I would call this a win if this ends up happening although I still belive we’re hundreds of years away from Strong AI.

There never was a Strong AI Summer to begin with.

This reminds me of a recent Twitter thread [1] from Zachary Lipton (new machine learning faculty at CMU) arguing that radiologists have a more complex job than we, as machine learning enthusiasts, think.

[1] https://mobile.twitter.com/zacharylipton/status/999395902996...

I think all talk about computer intelligence and learning is bullshit. If I'm right, then AI is probably the most /dangerous/ field in computer science because it sounds just likely enough that it lures in great minds, just like a sitcom startup idea[0].

[0] http://paulgraham.com/startupideas.html

You could actually make a reasonable argument for the opposite of a winter, that we are heading into an unprecedented AI boom.

The article's main argument for a winter is that deep learning is becoming played out. But this misses the once in history event of computer hardware reaching approximate parity with and overtaking the computing power of the human brain. I remember writing about that for my uni entrance exam 35 years ago and have been following things a bit since and the time is roughly now. You can make a reasonable argument the the computational equivalent of the brain is about 100 TFLOPS which was hard to access or not available in the past but you can now rent a 180 TFLOP TPU from Google for $6.50/hr. While the current algorithms may be limited there are probably going to be loads of bright people trying new stuff on the newly powerful hardware, perhaps including the authors PVM and some of that will likely get interesting results.

Deep learning maybe not the complete answer to gai, but it’s moving down the right path. Computers though are still years/decades away from approaching human brain power and efficiency, so my take is that current ai hype is 10 years too early - a good time to get in.

> but it’s moving down the right path

Time will tell. I think DL is amazing, but is no the right path towards solving problems such as autonomy. I think if you enter this field today, you should definitely take a look at other methods than DL. I actually spent a few years reading neuroscience. It was painful, and I certainly can't tell I learned how the brain works, but I'm pretty certain it has nothing to do with DL.

A lot of people said the same thing in the '90's about "Neural Networks". Senior colleagues of mine have vivid memories of consultants coming in then saying much the same thing "in 10 years this will completely revolutionize your industry (manufacturing)."

When I first start working in 2004 "data mining" was the big thing and it was going to solve all our problems. Nowadays I'm hearing the same thing again about "Machine Learning".

It's pretty natural to be skeptical people make big promises it ends up being a lot of hot air.

Great essay but this "Deep learning (does not) scale" I think is missing an important point.

There are many ways to think about scale.

If you think about a learned skill then that skill actually scales extremely well to other machines and thus to other industries that might benefit from the same skill.

The primary problem with technology is that society doesn't just implement it as fast as it gets developed so you will have these natural bottlenecks where society can't actually absorb the benefits fast enough.

In other words, Deep Learning scales as long as society can absorb it and apply it.

Has anyone done something genuinely useful with ml/ai/whatever outside of advertising or stock trading? I am genuinely curious if it has really been applied to real commercial applications.

Improvements in search, translation, image recognition and categorization, voice recognition, and text to speech off the top of my head. I'm sure there are a lot more.

Yeah but those are all pretty terrible to the actual end consumer. They might be cool technologies but at the end of the day, I am a user that hates dealing with them. IVRs are terrible a experience. Image recognition is iffy at best. Text to speak is terrible. In 10 years maybe they will have it hashed out... just like 10 years ago, or 20 years ago

Have you spent real money on something because it used Deep Learning?

Me not.

Maybe indirectly, ads have become more targeted, but we cant be sure how much. It might be just a dtandard small optimization

A lot of problems that involve risk or uncertainty, and that have measurable inputs. Thinks like market segmentation, customer retention and conversion, identifying potential bad debts and fraud, targeting products, identification of product features etc.

Not as sexy as the headlines, and less likely to involve DNN, but still profitable.

Sure the thing is overhyped, but the problem is that we cannot be sure about the next big thing. The advances are slow but then a giant step forwards happen all of a sudden.

Everyone dropped their jaws when they saw the first self driving car video or when alpha go started to win. This was totally unthinkable 10 years ago.

Some guy may come up with a computer model that incorporates together intentionality, some short term/long term memory, and some reasoning, who knows?

AI is favorable for big companies to better scale their services. It seems that Facebook have also faced AI scaling drawbacks and they are developing there own AI hardware for it https://www.theverge.com/2018/4/18/17254236/facebook-designi...

AI has a lot to offer to the industry right now I think where you don't need good worst case performance (ex., information retrieval, optimization, biology, etc.). The big problems in terms of application start appearing when you try and remove humans from the loop completely. That's not even close to possible yet but that doesn't mean the economic utility of even current AI is close to being maximized.

I know this about the state of Deep Learning but I like to point out:

While autonomous driving systems aren't perfect, statistically they are much better at driving than humans. Tesla's autonomous system has had, what, 3 or 4 fatal incidents? Out of the thousands of cars on the road that's less than 0.001%.

There will always be a margin of error in systems engineered by man, just hopefully moving forward fewer and fewer fatal ones.

Depending upon your statistical sources, US traffic fatalities are around 1.25-1.50 per 100 million miles. [1] All forms of real-world autonomous driving across all manufacturers across the world are still somewhere below 200 million miles, conservatively estimated. [2] [3] Between the Tesla and Uber fatalities, by these rough back-of-the-envelope numbers, autonomous of various grades is still roughly 2X higher than human drivers. Maybe 1X if you squint at the numbers hard enough, but likely not orders of magnitude lower. I don't anticipate rapid legislative and insurance liability protections for autonomous systems until we see orders of magnitude differences on a per 100 million miles driven basis, and that will take time.

Waymo racks up about 10,000 miles per day across about 600 vehicles spread in about 25 cities. [4] Roughly 3.6 million miles per year if they stay level, but they're anticipated to rapidly add more vehicles to their fleet. In the US alone, about 3.22 trillion miles were driven in 2016. [5] Don't know what a statistically valid sample size is based upon that (I get nonsensical results below 2000 miles, so I'm doing something stupid), though. If Waymo puts two orders of magnitude more cars out there, they'll still "only" rack up about 365 million miles per year, and not all the miles on the same version of software.

[1] https://en.wikipedia.org/wiki/Transportation_safety_in_the_U...

[2] https://www.theverge.com/2016/5/24/11761098/tesla-autopilot-...

[3] https://www.theverge.com/2017/5/10/15609844/waymo-google-sel...

[4] https://medium.com/waymo/waymo-reaches-5-million-self-driven...

[5] https://www.npr.org/sections/thetwo-way/2017/02/21/516512439...

Woah, I was prepared to be all gung-ho for this post, given that I've suspected the winter was going to be here for quite a while now. But strangely, this post actually caused the opposite effect for me. The winter will probably come one day, but is all the evidence the poster can find? Andrew NG tweeting less and a statement that DNNs doesn't scale based on flimsy data is not at all convincing to me.

Is this AI Winter 2.0? I was hopeful that logic programming would have developed more and spread to a larger audience at this point.

As a beginner in deep learning space, I am a bit baffled about the case "You need a lot of computational power". Good models learn fast, so if potential model looks promising on local machine, one can do training on gcloud for 100$ on high end machines. Where am I wrong in this line of thinking?

No, you are absolutely right. And modern transfer learning improves this even more in many domains.

Thank god. We're definitely not ready and perhaps could never be ready for true general purpose ai.

This is something I always wondered about AI and it promises. Sometimes, the last 1% is the hardest or can be even impossible. Self-driving cars, in particular, are a good case. We get to solve 99% of the use cases but achieving full autonomous vehicles might be just out of reach.

But, they're getting more and more data every year, right? All those almost millions of teslas running around could provide enough video input for the training data

Besides "Good software takes 10 years", according to Joel Spolsky. As I see it, we're, what 5 year into ML.

5 years ha! Wow the rebranding has worked well.

Another case in point. http://www.latimes.com/local/lanow/la-me-ln-tesla-collision-...

> Nvidia car could not drive literally ten miles without a disengagement.

From the same source as the author cites, that's because their test runs are typically 5 miles and resuming manual control at the end of a test counts as a disengagement.

Deep Learning was a noticeable improvement over previous neural models, sure. But deep learning is not the entire field of AI and ML. There has been more stuff going on like neural turing machines and differentiable neural computers.

We are beginning to see some sweet differential embeddings of discrete things like stacks and context free grammars. This is where deep learning gets really fun because it is learning to program.

https://twitter.com/jvmancuso/status/1002387357776207872

For me Google is attacking on 2 main fronts 1. Quntam computing 2. Machine Learning/AI

If they are able to combine the 2. A big if though the cost analysis will change for AI quite dramatically.

Number of tweets as reliable data points? Very dubious. Simple explanation: They are busy working, so less time to tweet.

Maybe they're working on something so cool, that the AI winter may not even come. Sure, there's a lot of marketing-speak around AI at the moment.

But this wave of AI seems a lot stronger with better fundamentals than 20 years ago. At the very least, at least we have the hardware to actually RUN NN's cost effectively now as oppose to grinding your system to a halt back then.

Before AlphaGo, it wasn't even clear when a computer could beat a top professional in go, let alone crush humans in the game - low bound guesses were 50 years.

https://en.wikipedia.org/wiki/Lighthill_report (1973)

Low hanging fruits are scarce now. With 3 orders of magnitude difference in power (MW over few watts), clearly this is not the right way for reaching the tree top.

/shrug people need time to research;

Anyway i also don't get what the issue is with the model from radiology. It is already that good?! This is impressive. One model is close to well trained experts.

Just today i had an small idea for a new product based on what google was showing with the capabilities to distinguis two people talking in parallel.

At the last Google IO i was impressed because in comparision to the previous years, ML created better and more impressive products.

I was listing for years at key nodes about big data and was never impressed. I hear now about ML and im getting impressed more and more.

Google IO is a developer's conference with an emphasis on marketing its own products and tools. We have to take the news from it with a grain of salt.

If only there were some technology that might enable us to discern patterns so that we could better predict fluctuations in demand for AI software.

Truly, I agree.

I've long been interested in learning about AI and deep learning, but to this day haven't done much that truly excites me within the field. It feels more or less impossible to make anything significant without Google-scale databases and Google-scale computers. AI really does make it easier for the few to jump far ahead, leaving everyone behind.

I also agree that a lot the news around AI is just hype.

Honestly, I'm yet to see anything practical come out of AI.

But hey, if something eventually does, I'm all for it.

Speaking of Google-scale databases, I've always felt the unhyped hero of deep learning was the data sets themselves. I'd be happy to see one less book about deep learning, and one more book on how you can gather and build a quality dataset yourself. Granted, I think gathering and preparing the data is a broader and most difficult task in many ways.

> be Google

I'm kinda curious now where those giant datasets will come from now that there's a big push for privacy with things like GDPR preventing some random researcher from just buying data off whatever data mining corp is most relevant to their AI's purpose

Many traditional businesses (e.g. banks, insurance) collect customer data internally, and don’t share it with anyone. That’s not likely to change much. For many of these businesses, the customer is other businesses, and privacy rules aren’t even applicable.

You won’t hear about these projects outside industry specific publications, if at all.

I’m kinda curious now which researchers have been buying PII from data mining corps.

Bet your house on it if it’s “well on it’s way”

Yeah this is a dumb article. Number of tweets by AndrewNg? Really? All those articles denying the reality of the revolution brought by AI have an emotional basis, but I don't understand what it is. Are they feeling threatened? Or is it an undergrad/early 20s thing, like a complete lack of understanding of the dynamics coupled with abnormally strong opinions?

This reinforces the need to benchmark any 'human expert equivalent' project against the wattage of the human brain.

How much of this can we pin on IBM's overhype of Watson?

Yawn. Contrarianism is easy and this article offers little. The real world application you’re speaking of has a comically small amount of data (a few million miles?). You hear about a handful of accidents that still average to better than human performance and suddenly the sky is falling.

When machine learning stops successfully solving new problems daily, then maybe a thread like this will be warranted.

without being an expert just by reading articles it seems to me that some people wish foe an AI winter. It makes them feel better somehow

Oh, I thought "AI winter" would refer to a state of ruin after AI had come into existence and destroyed everything, analogous to nuclear winter.

AI Winter is a very well-known term in the industry referring to a general lack of funding of AI research, after the last time AI was overhyped.

I guess it could be used about anything that experiences a low level of interest, then.

A real Winter is a lack of warmth. An AI winter is a lack of ______

If we would stop calling this stuff "AI" it would make all our lives a lot easier, but people can't resist.

When computers first came on the scene a lot of people had a very poor conception of what it was the human mind did, computationally. So when computers turned out to be good at things that were challenging "intellectual" tasks for humans like chess and calculus many were duped into thinking that computers were somehow on a similar level to human brains and "AI" was just around the corner. The reality was that one of the most important tasks that the human brain performs: contextualization, categorization, and abstraction was taken for granted. We've since discovered that task to be enormously computationally difficult, and one of the key roadblocks towards "true AI" development.

Now, of course, we're at it again. We have the computational muscle to make inference engines that work nothing like the human brain good at tasks that are difficult to program explicitly (such as image and speech recognition) and we've built other tools that leverage huge data sets to produce answers that seem very human or intelligent (using bayesian methods, for example). We look at this tool and too many say "Is this AI?" No, it might be related to AI, but it's just a tool. Meanwhile, because of all the AI hype people overpromise on neural networks / "deep learning" projects and people get lazy about programming. Why bother sitting down for 15 minutes to figure out the right SQL queries and post processing when you can just throw your raw data at a neural network and call it the future?

One of the consistently terrible aspects of software development as a field is that it continues to look for shortcuts and continues to shirk the basic responsibilities of building anything (e.g. being mindful of industry best practices, understanding the dangers and risks of various technologies and systems and being diligent in mitigating them, etc.) Instead the field consistently and perversely ignores all of the hard-won lessons of its history. Consistently ignores and shirks its responsibilities (in terms of ethics, public safety, etc.) And consistently looks for the short cut and the silver bullet that will allow them to shirk even the small vestiges of responsibility they labor under currently. There's a great phrase on AI that goes: "machine learning is money laundering for bias", which points to just one facet among so many of what's wrong with "AI" as it's practiced today. We see "AI" used to sell snake oil. We see "AI" used to avoid responsibility for the ethical implications inherent in many software projects. We see "AI" integrated into life critical systems (like self-driving cars) without putting in the effort to ensure it's robust or protect against its failures, with the result being loss of life.

AI is just the latest excuse by software developers to avoid responsibility and rigor while cashing checks in the meantime. At some point this is going to become obvious and there is going to be a backlash. Responsible developers should be out in front driving for accountability and responsibility now instead of waiting until a hostile public forces it to happen.

I've always understood the claim that deep learning scales to be a claim about deployment and use of trained models, not about training. The whole point is that you can invest (substantial) resources upfront to train a sufficiently good model, but then the results of that initial investment can be used with very small marginal costs.

OP's argument on this front seems disingenous to me.

His focus on Uber and Tesla (while not even mentioning Waymo) is also a truly strange omission. Uber's practices and culture have historically been so toxic that their failures here are truly irrelevant, and Tesla isn't even in the business of making actual self driving cars.

I'm the first to argue that right now AI is overhyped, but this is just sensationalist garbage from the other end of the spectrum.

Hi, it appears that "sensationalist garbage" triggered quite a bit of a discussion. This is typically indicative that the topic is "sensitive". Perhaps because many people feel the winter coming as well. Maybe, maybe not, time will tell.

And FYI, Tesla is in the business of making self driving car. If you read the article, you might learn that Tesla is actually the first company to sell that option to customers. You can go to their website right now and check that out.

Uber, like it or not is one of the big players of this game. I agree they may have somewhat toxic culture, but I guarantee you there are plenty of really smart people there who know exactly the state of the art. And their failure is therefore indicative of that state of the art.

I also omitted Cruise automation and a bunch of other companies, perhaps because they have more responsible backup drivers that so far avoided fatal crashes. But I analyze the California DMV disengagement reports in another post if you care to look. And by no means any of these cars is safe for deployment yet.

> Hi, it appears that "sensationalist garbage" triggered quite a bit of a discussion.

Yes. Sensationalist.

> I also omitted Cruise automation and a bunch of other companies, perhaps because they have more responsible backup drivers that so far avoided fatal crashes.

So your explicit reason for omitting Waymo, as I understand it, is that it didn't support your argument?

One of the more silly articles on HN in a while. Waymo has cars as I type this driving around Arizona without safety drivers.

People were freaked out by the Google demo of Duplex a couple of weeks ago as it was just too human sounding.

Can give so many other example. One is foundational. The voice used with Google Duplex is using a DNN at 16k cycles a second in real-time and able to offer at a competitive price.

That was done by creating the TPU 3.0 silicon. The old way of piecing together was NOT compute intensive and therefore doing it using a DNN requires proprietary hardware to be able to offer at a competitive price to the old way.

But what else can be done when you can do a 16k cycles through a DNN in real-time? Things have barely even got started and they are flying right now. All you have to do is open your eyes.

DNN - Deep Neural Network.

It's the same story again like exaggerating the influence of IoT 5 years ago. The whole thing is exaggerated to raise money from investors and attract customers instead of actually buidling superior product

It's 99% marketing and places like HN and reddit eat it up and try to hype it up even more. When you confront these characters about the basis on which they claim AI will solve whatever problem or evolve to whichever point, they only reply "it'll only keep getting better (given time, data, resources, brains, etc)"

It's a buzzword people brainlessly use to fetishize technological progress without understanding the inherent limitations of the technology or the actual practicality and real-life results outside of crafted demos or specific problem domains (for example Alpha Go beating a grandmaster has almost no bearing on a problem like speech cognition).

It's turned me off a lot from reading about advances in the field because I know like a lot of science releases that most of it is empty air that won't really have bearing on the actual software I use (I've watched the past two Google's I/O where pretty much every presentation mentions AI, but the Android experience still remains relatively stale).

Deep Recession ‘18

Winter is coming.

What a relief.

The inconvenient but amazing truth about deep learning is that, unlike neural networks, the brain does not learn complex patterns. It can see new complex patterns and objects instantly without learning them. Besides, there are not enough neurons in the brain to learn every pattern we encounter in life. Not even close.

The brain does not model the world. It learns to see it.

This post is very uninformed.

"It can see new complex patterns and objects instantly without learning them."

Except, it doesn't. It is clearly false. When animals grow up in an environment without certain patterns, they will be unable to see these patterns (or complex combinations of these) at a later stage. We see complex patterns as combinations of patterns we have seen before and semantically encode them as such. This is very similar to how neural networks work at the last fully connected layers.

"Besides, there are not enough neurons in the brain to learn every pattern we encounter in life."

There is a lot of self-similarity in our environment. Compression algorithms (and NN auto-encoders) are able to leverage this self-similarity to encode information in a very small number of data-points / neurons.

"The brain does not model the world. It learns to see it."

Except, it doesn't. Your brain continually makes abstractions of the world. When you 'see' the world you see a (lossy) compressed version of it, compressed towards utility. Similar to how MP3 compression works: the information gain of higher frequencies is low, so your brain can safely filter these out.

We learn to see patterns, but we see through physical and cultural action patterns that are simply present, not learned.

It’s like a river flowing... yes, the water molecules each “discover” the their path, but the path of the river is a property of the landscape. It is not learned.

OK. See you around.

You’re close.

“learns to act in it”

we don’t even see the world, we see a hyperdimensional action space. Anything we can’t relate analogically back to some embodied action will literally be invisible to us.

Reply on news.ycombinator.com