AI Update, Late 2019
blog.piekniewski.info
blog.piekniewski.info
There is without doubt something novel in the successes of convnets for sensory perception, deep Q-learning for decades-old and new game problems, artificial curiosity, recent machine translation, generative models and their various applications. Recent models also found their way into for-profit companies. It’s legit to be fascinated by this, and I’d rather stand on the side that doesn’t remain in their cave.
AI research may have picked all current low-hanging fruits or hit a wall either soon or in ten years, nobody can know yet, so there is no reason to run around predicting the future painted in only positive or negative light.
It's still better than what I can read from "LinkedIn influencers" in my feed like "Logistic regression is still the best" or "Self-driving cars will never work because of long tail"...
Note that AI has a history of being stalled by overly pessimist evaluations (Minsky / Papert on the perceptron, Lighthill report).
> The whole field of AI resembles a giant collective of wizards of Oz. A lot of effort is put in to convincing gullible public that AI is magic, where in fact it is really just a bunch of smoke and mirrors. The wizards use certain magical language, avoiding carefully to say anything that would indicate their stuff is not magic. I bet many of these wizards in their narcissistic psyche do indeed believe wholeheartedly they have magical powers...
Although artificial intelligence is actually spot on. We just understand the wrong side of the ambiguity. Its not really intelligence that we have reproduced artificially - since it isn’t intelligence - but a fake intelligence, the artificial kind. We’ve created the artifice of intelligence, through statistics, but not intelligence.
People knew long before newton that an apple would drop to the ground when released. Statistical experience has allowed us to have knowledge of this very early on. But it took newton to explain what was going on, so that instead of predicting through experience, he could predict by reason and logic. Thus saving him many lives of accumulating experience to make his next prediction ever more precise.
"Statistical intelligence" allow us to do a bunch of neat things though. Many problems are best approached statistically (because noise, lack of formal understanding etc), and these some of these methods achieve impressive results in a wide range of situations.
So a RL chess algorithm tells your statistically a move (action) from a state S to a new state S’ such that you are expected to maximize your reward. Whereas a chessmaster (probably) designs his next sequence of moves based on logic (my opponent will respond in such a way because etc). This is different from « statistically, this move right now has the best odds of leading to a win » a la monte carlo. Now what is surprising, is that statistical algos are better than our best logicians at this particular task. But its the action at a given state is still statistically designed.
Finally, you need your data mining to be representative of the underlying distribution you are trying to model. So you need your simulator to be the most real whereas they are in fact approximations in most useful cases (landing a plane for instance).
So for instance if you want an algo to design the flight path of a rocket landing on an asteriod, you could recreate a simulator modeling spacetime from observations and model its dynamics from eintein’s equations, but then what’s the RL for, why not just use an off the shelf optimization algorithm like we have for decades? [1]
The bellman equation and DQNs are nice and all, but they’re still statistical algorithms, producing - in my mind - statistical intelligence about a particular system. An RL agent will not tell you WHY such an action was taken, but it’ll tell you that statistically, it is the action to take.
Very neat results in RL however.
[1] i worked on a RL based agent to control trafic lights, and it wasnt clear whether our solution was better than a classical optimization one. Actually, classical optimization (minimizing an analytical model of the system) seemed to scale much better to larger meshes.
I have no idea what's the moral here, if any.
Sure, AI has its faults; the tantalizing cost savings of automation has created some negative feedback loops - might that be more deserving of the question "what in the hell are we trying to accomplish and how exactly did we get here in the first place?"
A Rubic's cube solver is the problem? Really?
OpenAI (of now infamous Rubic's cube failure :p) released a hide-and-seek demo a few months back that gave me literal goosebumps. Little AI agents facing off in a game of hide and seek start evolving with seriously clever strategies. According to the author's bio (dynamic, time-aware ML systems, etc.) that sort of thing should be right up their ally!
Instead we get some sort of selective self-promotion hit piece - highlighting anecdotal failures while claiming some better AI based robotics startup is coming soon(tm).
There may be genuine criticisms of that particular project, but 'only the actual solving is done via symbolic methods' is a non-sequitur. The Rubik's cube is just a generic physical task that requires dexterity, they could have done the same research with dominoes or blocks or playing Tic-Tac-Toe with random pens in various adverse conditions -- the point wouldn't be that the ML solves or doesn't solve the actual Tic Tac Toe!
to me it did not read like an objective post, but more like just a "all AI is bullshit" style blog post
All new technology is overestimated short term, but underestimated long term.
We might not have autonomous driving that takes our kids at school, but we have cars that can recognize lanes and other cars and complain if something is dangerous. We also have facial recognition, voice recognition and almost turing level chatbots to sell you stuff.
I suspect AI might be analogous to computer graphics. Purists in the beginning said the holy grail was ray tracing. However, people still worked on the problem, marching the state of the art forward with smaller building blocks, and now that ray tracing is appearing, a practiced eye is needed to see the difference.
Well, no, some technology is overestimated short term and also overestimated long term.
For example, flying cars. Nuclear fusion (though that one could still come through). Gallium arsenide (still one of my favorite names for a speed-metal band, and still available as far as I know).
The question is, which category is AI going to be in? AI for specific tasks seems likely to be underestimated long term. AGI? My guess is that it's overestimated long term, because it isn't going to happen. That's a guess. Evidence? Don't have any. Guesses are like that.
Anytime there's new progress in AI, you will see many comments or posts some variations of "but humans do it more efficiently" (in arbitrary dimension) or "what about the other problem AI didn't solve". More often than not these are just some lazy layman criticism that makes the posters feel smart without offering anything new or substantial.
> I mentioned in my previous half-year update, Open AI came up with a transformer based language model called GPT-2 and refused to release the full version fearing horrible consequence that may have to the future of humanity. Well, it did not take long before some dude - Aaron Gokaslan - managed to replicate the full model and released it in the name of science. Obviously the world did not implode, as the model is just a yet another gibberish manufacturing machine that understands nothing of what it generates. Gary Marcus in his crusade against AI hubris came down on GPT-2 to show just how pathetic it is. Nevertheless all those events eventually forced Open AI to release their own original model, and much to nobody's surprise, the world did not implode on that day either.
HN crowd is often intellectual elite; imagine regular persons reading what GPT-2 produces when they can't understand what a regular grad student writes. I can use e.g. talktotransformer.com to complete some quotes like "Intel CEO said that the new 10nm CPUs will...", then post that to some Reddit thread, it would get picked up by search engines, and at some point somebody would use it in some serious work or it would spread like wildfire on sites that don't check their references.
John Carmack did not invent the fast inverse square root algorithm. (I'm still rooting for him, though!)
https://en.wikipedia.org/wiki/Fast_inverse_square_root#Histo...
No, you're generalizing the marketing department at IBM over a deeply passionate, hard-working, brilliant community of scientists and engineers.
But really, what did AI do to this guy? ML really does have real world applications. Though many self driving start ups are overblown, my Tesla really does drive me to work everyday.
As always, things are easy to critize, and hard to create.
He's a founder of an ML startup with published papers.
Seems to hate ML.
He does not hate symbolic ML which is based on logic/knowledge (and do understand the world).
Yeah everything is harder than the first wave of hype made it seem, no this list of ridiculous hype proved to be ridiculous doesn't mean it's all useless or doomed. I get the impression the author knows this from reading the about page though, which makes me think I just missed the joke.
I'd also like to point out that very little of what you can see in those Boston Dynamics videos is "AI". It's mostly just good old fashioned control systems, just very sophisticated.
- Academics need to create PR and hype to increase their chances for grants
- PhD students need to publish papers, and thus convince reviewers, with unnecessarily complex and hype-filled language, that their papers are good. They are also more incentivized than ever to create their own personal brand (via hype-filled blog post or videos) to increase future employment opportunities. More PR also means more citations, which is metric academics are often evaluated on. After all, if you work on something related, you're pretty much obligated to cite research that everyone has heard about, right?
- Startups, as it has always been, jump on the latest trend to increase their chance of raising money from investors. They slap AI/ML onto their pitch decks to differentiate themselves from others, or to become eligible for AI-focused funds. In reality, none of them will ever use any of the new ML techniques because they are too brittle to work in real-world products or require many orders of magnitude more data then the startup will ever have.
- Big companies want to brand themselves as "thought-leaders" in AI to drive up their share prices, hire better talent, improve their public image, convince investors, etc.
- The general media has no idea what they are talking about and wants to generate clicks. Same as always.
Put all this together and you get the current AI hype cycle. We've seen this happen with lots of other technologies in the past, what's kind of new this time is the entrance of academia into the cycle. When I first started in (ML) academia I was under the naive impression that I would be doing hard and cold science - I was so wrong. Everyone is optimizing for their own objective (grants, salary, publications, etc, see above), which makes most of the published research completely useless or simply wrong. One of the, sometimes unspoken, criteria of choosing ML projects in many of these labs is "how much PR will this create". This useless "research" is then treated as if it was a proven method and picked up by startups to convince clueless investors or customers with "look at this latest paper, it's amazing, we will monetize this, we're at the forefront of AI!", or by the general media to create more hype and drive clicks.
One important point that the blog post makes that is always overlooked is this:
> Now what this diagram does not show, is the amount of money which went into AI in corresponding time periods.
With all the hype over the last few years, just think about how many billions of dollars and tens of thousands of some of the smartest people on this planet got into the field, often to make a quick buck. With this many resources invested, would you expect there to be no progress? Obviously there will be, but most of it is smoke and mirrors. People think that the progress comes due to new AI techniques (Neural Nets), but in reality, if you were to take the same people and money and forced them to make progress on the same problems using some other technique, let's say probabilistic models or even rule-based or symbolic systems, they would've done just as well, if not better.
It's possible that the human baselines were bored and so performed sub-optimally when picking between the 1K classes. But the argument has now become a subtler one, much less clear cut.
As an example of categories that may be difficult to distinguish between, do you feel confident that you can reliably distinguish between the Norwich terrier [4] and the Norfolk terrier [5]? These are two separate categories in ImageNet1k.
[1] https://i0.wp.com/blog.piekniewski.info/wp-content/uploads/2...
The first diagram shows exponential growth in the compute usage of state of the art deep learning architectures.
[2] https://i1.wp.com/blog.piekniewski.info/wp-content/uploads/2...
The second diagram shows diminishing returns on Imagenet1k top-1 accuracy from doubling the size of Resnext.
[3] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.725...
[4] https://www.google.com/search?tbm=isch&as_q=norwich+terrier&...
[5] https://www.google.com/search?tbm=isch&as_q=norfolk+terrier&...
1. Google "difference between norfolk and norwich terrier".
2. Click first link: https://www.terrificpets.com/articles/10290165.asp.
3. "The Norwich terrier has prick ears, or ears that stand up, seemingly at alert, while the Norfolk has drop ears, or ears that seem to be folded over".
SOTA models are merely doing black-box pattern matching on who-knows-what, and are highly likely to fail dramatically outside of the training dataset confines.