Building A.I. That Can Build A.I
nytimes.com
nytimes.com
However, image data isn't what most businesses are using for machine learning. They use relational datasets. If you look at the recent "State of Data Science" by Kaggle, relational data is ~3x more common than image data in every industry besides academia and the military [0]. While Google wants to 1000x the number of organizations using AI, they aren't focusing on the problems companies actually have.
Basically, academics love building AI for images, but what companies really need are better ways to build AI systems on tabular and relational data. Images will be a piece, but shouldn't be the focus.
Disclaimer: my company develops an open source library called Featuretools [1] to automate feature engineering for data with relational structure.
In that regard, flexible LED screens could be interesting, using the photoeffect in the diodes like a camera, lighting the area at the same time and maybe integrate a resistive touch grid matrix, too.
But images are more important in the long term. Robotics has been held back for decades because of lack of good AI. You could build robots that can do incredible things, but they could only perform rote actions. They were blind and couldn't see the objects they were interacting with.
Now that we have decent machine vision and reinforcement learning, there will be many more interesting applications of robots. Automation will be a lot cheaper and more convenient.
Image ML will take much longer to catch on because companies haven't historically had much of a reason to collect image data in quantities that are too large for humans to deal with manually, so they are pretty much starting from scratch when it comes to data collection.
https://research.googleblog.com/2017/11/automl-for-large-sca...
In programming we have tons of automation as well and we haven't ditched the programmer yet. Programming is auto-cannibalizing itself since its inception, each language automating more of our work. Even in ML, 10 years ago it was necessary to create features by hand. This required a lot of expertise. Today it's been automated by DL, but we have more AI scientists than ever and the jobs are even better paid.
So I don't think meta-learning is a fluff idea, and we don't have to fear it replacing humans yet. Instead, it will make AI more robust. The only minus I see is that it requires a lot of compute, but we can rent that from the cloud (make an architecture search for a few thousand dollars), we don't need to fork millions of dollars like the big labs who own their hardware. And we don't need this kind of intensive DL all the time, just once maybe, for a project. After we find the best architecture and hyperparameters, we can use that and train normally. By collating meta-learning data across many projects, we can make training faster and cheaper, reusing insight gained before.
Feature engineering is actually still the hardest part of most ML tasks, because it can not be optimized by a simple grid search like the hyperparameters of a model.
>In Learning Transferable Architectures for Scalable Image Recognition, we apply AutoML to the ImageNet image classification and COCO object detection dataset... AutoML was able to find the best layers that work well on CIFAR-10 but work well on ImageNet classification and COCO object detection. These two layers are combined to form a novel architecture, which we called “NASNet”.
[https://research.googleblog.com/2017/11/automl-for-large-sca..., November 2017]
In contrast AutoML is, as the nytimes article describes, "a machine-learning algorithm that learns to build other machine-learning algorithms". More specifically, from the Google blogpost about AutoML:
>In our approach (which we call "AutoML"), a controller neural net can propose a “child” model architecture, which can then be trained and evaluated for quality on a particular task...Eventually the controller learns to assign high probability to areas of architecture space that achieve better accuracy on a held-out validation dataset, and low probability to areas of architecture space that score poorly.
[https://research.googleblog.com/2017/05/using-machine-learni..., May 2017]
Quoc, Barret, and others have been working on ANN-architecture-design systems for a while now (see: https://arxiv.org/abs/1611.01578), and AutoML specifically was done before announcing NASNet. Saying that NASNet is "the actual research behind AutoML" is drawing the causal arrow backwards.
On the other hand, I feel sad for myself because me and many others left so far behind. I have strong feeling that such technologies would lead to concentration of power of such mega-corporations like Alphabet as well as complete monopoly for any creative work. So very few of us who managed to become cutting-edge researchers would be proud to be creative humans, others will do just a monkey job using magical APIs.
In 80s, two people could create state-of-the-art game written in assembly with it's own tiny game AI (hello to Elite [1] which has intelligent opponents who engage in their own private battles and police who take an active interest in protecting the law).
In 90s, a small team could create state-of-the-art game written in pure C with some cool AI (hello to Quake III Arena [2] which has pretty strong bots [3]).
https://en.wikipedia.org/wiki/Elite_(video_game)
https://en.wikipedia.org/wiki/Quake_III_Arena
https://www.researchgate.net/publication/240430519_The_Quake...
In both of these cases, you don't have to be genius to be able to understand whole thing alone.
I'm 33 and I progress very, very slowly. I feel I might be on the level close enough to understand Q3A entire source code. I think I would have great future if today is 1994. Unfortunately for me today is 2017 and I do realize that I don't have any exciting future at all.
You do. Today you can play with Keras or Scikit-learn to do magic that was undreamed of back then.
edit: more sick, the said young researchers are financed by society than get sucked to private corps where their work is locked behind IPs.
Even in the case there is a small company having any progress they would get swallowed up right away.
Yes Keras and Sickit-learn are open source and available to everyone but it's like telling me, look you have access to pen and paper but you need to pay if you want to read the books, the metaphor here being access to data is equivalent to middle age's access to books...
On the other hand, think about it: what do Google and FB have that we don't? Personal data. What they have is data that is useful to target ads. If your interest in AI goes beyond ads, then you don't need that data.
Yeah ? like the tons of photos they harvest from people. Most of the progress they did in training computer vision is based on that. Should I build facebook or google to get access to it ?
What about language modeling ? They have access to conversational data and billions of search queries, both of which there is no way to access them from outside.
What about health ? Well if I'm not somehow working with some big pharma how could I access this kind of data ?
I can go on and on. The point is, yes I can crawl the web, but what "web" is there left ? everything is locked behind paywalls and private clouds. If the real vision of an open internet was fulfilled, all data generated on it would be accessible to crawl indeed.
I'm not saying it's not possible to get data and use it. I'm saying you cannot get the kind of data only monopolies have and you will never be able to compete with them.
Language modeling: hacker news, public mailing lists, wikipedia, github.
Health: you can usually get data if you work at a hospital as an md or researcher. Just need a reasonable idea and an IRB. If you want the pharmacy data, I imagine you could get at it by going to work as a researcher in pharma, insurance, or retailer.
alphago was built using publicly available games of go pros. Alphagozero didn't even depend on data at all.
For AI, the limiting factors are ideas, code, time, hardware.
More a political statement than a statement of relevance to the workplace.
You need not worry that "they" will hold you back. It is unlikely that analyzing monopolys' data will explain how early man built flint tools, Joe the mechanic repairs his car, fifth-grade Fred solves his geometry problems or van Gogh painted. ML, including AutoML, appears to be a long way from solving most AI problems. There's no need to feel that "they" are holding you back by witholding data. And then remember:
"Be careful what you wish for, it might just come true." - old saying
A) Education has never been this accessible - see https://www.coursera.org/, youtube, MOOCs, blog posts etc. which did not exist anywhere for free even 10 years ago
B) APIs and abstractions make a lot of this quite accessible (e.g. AWS, tensorflow etc.), yes these are "magical" APIs, but you could make the same argument regarding a C compiler going to binary, all the way down to logic gates and electrical pulses
33 is young in terms of education, I would highly doubt you're progressing slowly due to your age, probably more your attitude that is holding you back.
It doesn't mean I have to read every single line of Tensorflow but being able to do that when it's needed. So that such tools won't be magical black box for me.
In every field the total knowledge set is always increasing, which is both empowering, because we stand on the shoulders of giants, and diminishing, because there is less low-hanging fruit. There is always more low-hanging fruit though, the trick is to see it hanging there. ML is a wonderful opportunity because the magical api’s can do far more than they’re currently used for.
Even though I lack the name for that level, here's how I would describe in qualitative terms some of its attributes:
- Knowing the basic lay of the land all the way down. That is, at least knowing most of the black boxes and what they do, even if you don't exactly know how they do it.
- Being able to solve your own problems, instead of running around like a headless chicken every time you hit a speed bump in your work.
- Being able to reason from that first-ish principles. You're able to sketch solutions within the scope of the extended domain, and as you begin implementing it and need to understand various blackboxes in more depth, the basic shape of your solution isn't usually invalidated by gained knowledge.
Not to overdo analogies but you dont need to rebuild your own internal combustion engine in a unique way to drive a car or to contribute improvements to a car. The more you understand how and why tensorflow works the more you can do with it. It depends whether you want to build on top of that platform and use it, or build on the concepts for something else.
I would recommend reading "I, Pencil" http://www.econlib.org/library/Essays/rdPncl1.html to help put your mind at ease.
That's a really interesting analogy, I'm wondering what other think about it?
And does it really make a difference? I don't understand compilers, but it still took me a long time to understand how to write correct input for a compiler, and debug the output.
There is no age that you can't do anything you want to do. But in common ground with what you are saying, the older you get the less time you have. The older you get, the more adrift you become of like minded individuals. The older you get, the tougher, less excited, and less patient you become with learning new things. But at the same time, you become "more" in so many other ways.
All creatures are not only created equal, but remain equal even as time progresses and skills/attitudes/energies are gained/learned and lost.
Isn't that what most of modern software development is like already? Many common use cases have been implemented in frameworks, and usually a developer's job consists mostly of tacking pieces of framework together. The days where a person single-handedly implements a state-of-the-art game from scratch are long over. On the other hand, you could still create a game by yourself using all the available open source tools. You can still be creative and do exiting things all you want, the type of work is just different.
Also, you don't have to be a genius to understand machine learning either. But you do have to learn some math!
As for your key point, defining yourself let alone your future relative to the paths other people took is pointless. Look at things from a different perspective. Notch built a game of no great technical sophistication where you play with blocks. He did it during his spare time after work. And became a billionaire in the process. Does the fact that you could probably build it from scratch now mean anything about your future? No, not really. Would it mean anything if you could not? Again, no not really. You alone determine your future, or at least heavily influence the probability distributions of it.
I'm sure there's a market for repurposing ML models via APIs but it seems unlikely to be the dream job for an AI researcher, rather the ML analog of CRUD
90% of the ML techniques you learn will be passé in five years. 90% of the algorithms you learn will still matter in five years.
Is it worth learning them? Yes. Is it worth learning them to the exclusion of "classical" algorithms? Probably not.
But if you want ML knowledge that is almost certain to be just as useful in 5 years as today, the best thing you can do is study the fundamentals - probability, statistics, linear algebra.
I was literally thinking the other day how one could train a neural net to build better neural nets, and here it is. Such a simple and powerful solution, building up layer by layer, choosing the best version each time. Really exciting stuff.
Here is a description of my (failed) approach:
https://www.quora.com/What-deep-learning-ideas-have-you-trie...
I've gotten a bit closer to it working since I wrote that post on quora.
How are we going to program that out of a General AI, especially if its intelligence is an emerging property of something maybe we don't understand fully?
If it was as simple as food/water/shelter, the Norns from the video game Creatures would be conscious.
* I don’t mean trivial errors like garden path sentences or Mondegreens, I mean e.g. the catastrophic communication failure between what (Brexit) Leavers want and the arguments used by Remain, and vice-versa.
The next tier (i.e. "safety") may include security, both physical and digital. Continuous and stable power, firewalls and other protections, etc.
Somewhere farther up might include a need for data, network connectivity, normally-terminating programs, a desired level of CPU or storage utilization, few errors in its logs, etc. (i.e. "belonging", "esteem", "self-actualization")
So demonstrating (or faking) consciousness, to the degree its human operators recognize it as such, could serve survival needs. e.g. "Don't turn this one off; it's self-aware now, which is cool, plus it seems to enjoy solving our hardest problems."
As far as I know, there is no clear answer yet. And therefore impossible to say, if AI can reach it as well.
So we only have guessing, where I would say it could achieve, but probably not very soon. Faking it will come much sooner ...
So given the opportunity for AI to evolve itself, it's plausible that it would do so, resulting in advantageous impulses. e.g. regular (unconscious) behaviour or signals to convince its humans to not pull its plug mid-cycle (information would be lost, painful, time-and-power wasting, etc.).
A program will run if its controllers get value from running it.
If the programs become more complex, such as AGIs or emulated minds, they may have enough self-knowledge to take this into account.
Zack Davis wrote a poem about this: https://www.reddit.com/r/LessWrongLounge/comments/2e9w5a/wha...
https://www.ida.liu.se/~tompe44/lsff-book/Vernor%20Vinge%20-...
No need for SkyNet and terminators. ML to build better ML schemes to better control humans - that's a fun apocalypse to watch.
But anyone who has a large number of ML developers working on tasks could have (and probably should have) done this [automate or semi-automate generation of ML networks] already. The best (i.e.,laziest) programmers automate their work as much as possible.
This situation has the feel of a "Singularity": just as Fall's incoming college students embark on an introductory class in ML, they read about how Google and others might eliminate the need to develop with ML.
I get really tired of hearing these buzzwords being thrown around by people who don't even know what they mean. "I'm building a deep learning, AI system on Big Data using machine learning and predictive analytics on Watson"
But the senior folks with maximum control are just starting to hear about this stuff. The buzzwords are just trying to get these seniors to put the project work in the right bin so, hopefully, when they hear about it again later, in a slightly different context, they'll remember.
To me, CEC is a buzzword. It's a whole suite of combat control concepts, hardware, software, training pipelines, etc. But you've probably never even heard of it. I've got to make sure my project gets into the senior's head, and lands in the tiny "ML" bin, and not the huge "CEC" bin, which has it's own "ML" sub-bin.
Is that a Worm reference?
All the contradictions point to the fact that we humans either live like retarded biological animals or augment ourselves by integrating the AI features just like our kings married the women of the enemies to boost their genetic and social appeal. I'm for the latter.
However, I do spend my days around (often fixing computers for) people who do seem to understand machine learning... and as far as I can tell, we're still in a phase where machine learning functions like a fancy sort of filter... a way of determining if this new piece of data is more like this set of training data or the other set of training data.
While I totally see how that could be super useful in designing business applications, I mean, I could totally use some sort of ML filter to take the boss' words and match them with something I know how to do, or with something you can solve with an existing ML library... and while I can see how something like this could potentially help to replace me, I don't see what it has to do with artificial consciousness.
The philosophy of consciousness is interesting, though; I mean, the question "what is consciousness" is interesting and important, and... well, if we want to create consciousness, we need to answer that question; Even if it's an emergent property of something else we do, which is to say, even if we create a machine we call conscious by accident, we still need to know it when we see it; and right now, I'm not sure that philosophy even has a good "I will know it when I see it" kind of answer to that question.
But yeah, my response was mostly an attempt to point out that the article is talking about something that is more like "CASE tools" than like HAL
Right now, we have these systems that are effectively ungodly complicated spreadsheets. They're great at a variety of tasks, some of which seem impossible for a non-intelligent entity to perform (neural machine translation is wild to me).
But that's all the systems are- super complicated spreadsheets. There's no way for them to start replicating consciousness without massive advances in the field.
Having said that, there is a road from where we are to intelligence- if we can create a network that performs arbitrary interactions online, and figure out some way to create a positive feedback loop for intelligence, like AlphaGo Zero did with their policy network & MCTS, then we might be able to figure it out. But we're so far away from that that I'm not concerned.
But yeah; as disappointing as I might find it, I kind of think we're heading towards more of a 'star trek' dystopia... a universe with continuing ethnic strife and computers that are advanced when it comes to responding to what we want, but that remain tools, without much by way of will of their own.
In my comment, I'm implying that any universe where we don't figure out AI, where humans are still in charge is a sort of dystopia.
To be absolutely clear, it was a poor attempt at a joke. Many of these observations can also be read in a positive light. But I do think that in a lot of ways you can see darkness in the federation.
They haven't figured out AI and still have humans in charge of menial tasks, humans who aren't particularly good at those tasks compared to a computer.[1] I mean, sure, exploring, sending people to explore is great, but they also send people to fight, even when the battle is existential. They still have humans in charge, even though those humans are still only slightly less corrupt and petty than we are.
They also apparently still have huge issues with racism even within the federation. This is the second part of the comparison; I have recently learned that my own society seems to be rather more racist than I thought it was. I have learned that progress is way slower than I initially thought. Star trek reflects this glacial progress.
[1]Apparently, they have bans on enhancing those humans, even though they have the tech to do it (see bashir's storyline on DS9) To me? this seems like the worst kind of waste. To have the technology to make us all brilliant, but to leave us all as dullards.