Marc Andreessen on the atomization of AI
techcrunch.com
techcrunch.com
That said, these startups will have to be extremely research heavy — how does one draw the most talented researchers to work for a small group of people with minimal pay over working for a large company with a high salary and all the computational resources you might need? And what kind of product could theoretical research produce in the short term that would keep a startup afloat long enough to come up with ground breaking ideas?
I disagree with Andreessen that Google's Tensorflow propaganda on Udacity is "opening the kimono" or having a large effect on AI. TF isn't better than pre-existing tools in many ways. And the ways that Google uses it aren't really transferable in an online course, since Google does have the data, architectures, infrastructure and talent -- some of which it can't share by definition, and some of which it won't share, for obvious reasons.
The smartest startups outsource research to universities and government funded bodies. It's a way of derisking themselves as investments. That's no different for AI. Theoretical research and product development are almost (but not quite) totally different things. Which is something that graduate students who think they want to do a startup should remember, because if they love research, they probably won't have much time for it.
[Disclosure: I'm associated with a rival DL library that I won't plug here.]
But by "vanilla deep learning", I think you are referring to this vs research into new models and algorithms. I think "research" in the context of "research-heavy start-ups" is not about new algorithms, but about finding ways to use even "vanilla deep learning" on their available datasets. Quite a lot of papers at conferences, for example, are not about new algorithms, but about new ways to use old algorithms.
To be honest, most of the funding and deeply interesting work for ground breaking Ai research lies in the public and defense sector. They have more of an idea and vision for what Ai can be applied to than the private sector. Leaps and bounds beyond.
In the private sector, I get asked questions like 'Can it be used to predict if someone will click on an ad'. In the public sector, I get a 40 page technology acquisition pdf outlining a forward thinking application of Ai.
People forget that self driving cars arose from Darpa research. The latest innovations in computer security utilizing Ai came from a Darpa competition. IBM's neuromorphic chip : darpa funding. Darpa and other defense programs also have a slew of Ai development projects open to researchers.
Ultimately I find this a bit ridiculous given how vocal some individuals in the tech field are about the 'dangers of Ai'. They harp and harp about what could happen if more capable Ai gets developed but you don't see their money anywhere near the groups doing deeper development in Ai.
Lastly.. Yes, better solutions and more deeply inspired approaches to Ai will trump hoards of data and compute power any-day. Thank goodness for the public sector pushing forward fundamental research and development in Ai.
...because if you think AGI is dangerous, you should fund groups trying to develop it? I am puzzled by the presentation of this proposition as if it is an absolutely obvious sequitur requiring no defense, such that failure to follow through on it is "ridiculous".
Don't stretch your mind to far on this one. If someone or various groups develop AGI and you have no stake in them, you will have no say in how it gets used. For once, people need to be honest with themselves about this.
You're not going to be able to dictate from a third party consortium of PhDs how someone should run their company or develop their software.
In fact much of the deep learning advances we (including Google) enjoy today were done with Canadian (University of Montreal) and European (IDSIA) taxpayers money over the last couple of decades
Given that the public funding of basic research most always results in the ushering in of new technological paradigms that make corporations tens of billions if not hundreds of billions of dollars, more should be done to make sure they pay back into the system that ensures their constant success.
DeepMind was VC funded before Google acquired them. A number of AI/ML startups would like to pursue AGI in the long-term, but why not make revenue along the way? Still, there is at least one startup - Vicarious - focusing purely on long-term research towards AGI that VCs poured plenty of money into, and there are probably others.
>A number of AI/ML startups would like to pursue AGI in the long-term, but why not make revenue along the way? That's exactly the dilemma that many capable researchers faced. Some chose to make money along the way. It shapes the way you then go about solving AGI down the road. It saps up your resources, imagination, and potential and it redirects it to shorter term thinking. Being in the space, there should be an understanding of Short-term vs. long-term thinking based on a reward system that targets short-term success.
> Still, there is at least one startup - Vicarious - focusing purely on long-term research towards AGI that VCs poured plenty of money into, and there are probably others. I'm familiar with Vicarious. You are correct and they notably publish far less details than others. They are funded by top names and are probably the least mentioned when people mention Ai. Why, given the impact that AGI can have, aren't more groups funded? A solution is seemingly right around the corner... Do people not see this?
As someone miles away from the field, what are those forward thinking applications (ie things that are within the reach of MI today but are basically ignored by private sector)
Most AGI focused ventures are not even in the U.S. A prominent one is in China. Another is in Europe. The ones in the U.S are privately funded beyond the big names. Furthermore, where is the U.S's equivalent of : https://www.humanbrainproject.eu/ ?
Where are the seasoned software engineers in the ranks at these highly funded silicon valley Ai companies? I typically see a roster of 'big names' and PhDs... Whereas, I look at the companies specifically pursuing AGI and I see a whole range of individuals.. PHD neuroscience, Senior game developer, Robotics Engineer, Senior firmware developer, Guy from down the hall who can code circles around silicon valley's most decorated engineer, etc etc.
Look at the Geohot story, that's the kind of individual and company who ushers in a new paradigm ....
If the solution requires you to 'think different', how do you expect to achieve it by padding your company with a bunch of individuals who are all centered on the same techniques from academia?
I do not. It seems like researchers who call themselves AGI researchers have made no more significant progress than researchers developing specific analysis techniques. Do you have a list of these groups? I would be very interested to read about their approach and progress.
On the human brain initiative it seems like their is such a huge gap between AGI and current techniques that these HBI projects (and the US response[0]) are a bit misguided into giant neural simulations.
> ... companies specifically pursuing AGI and I see a whole range of individuals ...
I completely agree this is the necessary approach to pursuing AGI. Who are these companies? They sound awesome and I would like to learn more. Are you talking OpenAI? Some other companies?
Geohot. Will check him out.
EDIT: oh, George Hotz. He is using deep learning[1]
Agreed deep learning will be a tool, but not the end of AGI. Companies padding with deep learning PhDs are looking more at specific tasks that can be tackled with deep learning. Also, I would argue that deep learning has become an umbrella term for all neural network based approaches (great marketing) and there are still great advances to be made with DL building blocks.
[0] https://en.m.wikipedia.org/wiki/BRAIN_Initiative
[1] http://www.bloomberg.com/features/2015-george-hotz-self-driv...
I mentioned George Hotz to highlight a capable individual with vision, passion, and capability who 'thinks different'. With many individuals, the capability is there. However, often times the vision, passion, or capability to think different isn't. Funny then that there becomes this hiring norm which attempts to find the 'most capable' individual and ignores the other more important characteristics.
Time will tell. I'm personally looking elsewhere beyond the names everyone mentions when it comes to development in this space.
Your post implied that you have some special insight into research groups that are doing AGI research and are making steady progress. I have done plenty of Google searches and not run across any notable AGI progress.
The one reference you provided was for a deep learning implementation, which is what you seemed to be dismissing.
Yes. Given the attitudes of people in the space and beyond and lack of funding why shouldn't the more valuable and fundamental research be kept secret? They're taking all of the risk to pursue something that everyone is saying is impossible. Why would they publish details? If more funding and support would come through maybe they would. Otherwise, they're seemingly developing a fundamental AGI that has the ability to operate under its own free-will and mechanisms. It is being said that it's impossible... Once it's completed no one will be able to deny it at that point. Maybe the money will come then... Maybe the openness will too.
> Your post implied that you have some special insight into research groups that are doing AGI research and are making steady progress. I have done plenty of Google searches and not run across any notable AGI progress.
Yes and I will not disclose where my insights derive from. I'm sure people have their ways of discovering it nonetheless. Some groups and individuals are developing conscious self-willed software solutions. I'm referencing them. Is that not what 'life' which intelligence springs from fundamentally is? As will be true for AGI... Not programs that mimic human behavior or optimization algorithms.. Software solutions that are truly aware.
If it's secret, how do you know about it? If you don't have specific knowledge, how can you claim there is "fundamental AGI" research done in secret, as opposed to it just not being done, as all appearances indicate?
>It is being said that it's impossible...
Nobody who understands recent neuroscience claims any such thing is impossible. We just claim it hasn't been done in the lab yet.
For example, no smarts would ever confirm the Higgs in the tevatron in its 10 year run even though the machine produced thousands of Higges; its a subtle signal and you just don't have the sample size to be more than 3 sigma sure there's something there (nor where exactly).
What is key, however, is being able to leverage more than just explicit training sets, like with transfer learning and unsupervised learning, as well as making algorithms scale well with more data. The latter seems promising with deep learning, the former still a research thing.
"more data beats better algorithm" is obviously false when the algorithms don't scale with data as well, as deep learning has convincingly demonstrated, being just as mediocre as anything else on small datasets, but being leaps above everything else on sufficiently large ones.
This is interesting, and something that needs further explored IMO. I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics.
We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information.
That's awesome! Can you link me to your research? I've been writing a similar paper on the information theory behind probabilistic programming and (possibly) deep learning.
>We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information.
Definitely! Not only is the Shannon entropy of large or high-dimensional datasets computationally obscene to estimate nonparametrically, what we actually care about is the conditional entropy given an untrained model, and the degree to which we can reduce it by training the model.
Someone who actually DOES AI is Fei Fei Li at stanford. She's focusing on creating datasets like imagenet and more recently the visual genome: https://www.technologyreview.com/s/545906/next-big-test-for-...
A lot of AI startups' best assets are their data which google also has plenty of. Major advancements in AI will come from less glamorous things like labeled data available for models to learn from.
The code without the data is useless. The algorithms are only part of the equation here...and giving them away without the requisite data is useful, but not the critical path from profiting from AI. It's also not going to advance AI much. These AI labs publish subsets of the research they actually do (even if it is still a generous amount which is great).
We can even see this from OpenAI's gym efforts. These environments are creating fundamental infrastructure for pushing the boundaries on reinforcement learning.
That being said, research is a component of the problem, but even most "AI" startups just git clone some open source code and run something pretrained (say: opencv, kaldi for audio,..) and then wrap it in a nice gui.
The main things these startups focus on is delivery of the product just like the rest of these startups, very few are actually building novel algorithms. A lot of it is just them collecting data.
That being said, this is also why a lot of the novel research happens in the major for profit ad tech companies. They have the data to do research on and they can choose what to publish and what to profit from in products.
Disclosure: I work at an AI startup.
I think the short answer is, you don't. Between Google, OpenAI, Facebook, Baidu, Microsoft, IBM, and every other startup that is putting "deep learning" in their pitch deck to see what happens, there's enough money being pumped into the area. 90% of the results of that money being spent will be made available to everyone in the world for free. IE, if you're investing for competitive advantage, you will lose. If you're not already on one of those teams or studying the subject, I don't see any opportunity for at least a year, maybe 2.
To be clear, I'm talking about opportunities around starting a new company. If your business is consulting or you're working as an engineer for an existing company, there are many applications for deep learning, but none that stand on their own.
In 2 years, content generation will be a big thing. Like pandora with purely algorithmically generated music, or games with assets that create themselves in a way far superior to what we see in the current generation of "procedural generation". The big cos will have very good versions of Siri, but no startup is going to be able to touch that near term.
Offer them a PhD
I imaginne that the winner of the self-driving car race will be determined by the product and not the technology per se. Tesla, Google, Uber, comma.ai, etc. all have quite different conceptions of what the product is.
Universities seem to be doing alright at this.
Probably the trick is to offer an exciting problem and lots of freedom.
It will take much more than a "clever idea" to overcome the reliance on data of the current AI state of the art- that is to say, machine learning.
If you think about it, machine learning algorithms are essentially clever search procedures for some optimum in a heap of data (that's optimisation algorithms, so about 90% of the field). You can always get smarter in the way you search, but unless you search far enough what you find won't generalise very well to unseen data. And if "unseen data" is the real world, you need immense amounts of data to get anywhere- hence, the reliance on big data.
What we need is much more than a clever idea: it's a paradigm shift. There's probably some sort of way to make a system that can learn from very few examples and at a very short time, like we do ourselves. Machine learning is certainly not that way. We need to find another way.
I'm not sure startups with clever ideas have a very good chance to just stumble on such an other way while looking for a way to do "Facebook with deep learning".
Startups generally have a short runway before they run out of cash, which isn't long enough to sustain this kind of exploratory work.
That's more like probabilistic methods, which also fight overfitting by regularizing (with a prior distribution) and allocating probability density in a smear around the training data rather than concentrating it as hard as it can (as optimization methods usually do, so to speak). And then hierarchical Bayes is what's used in many of the neat one-shot learning demos, and in theoretical neuroscience these days.
Maybe. I wonder if we can pick up identifying platypus with only a couple of examples, because we spent a long time identifying bird or cat as a toddler. We have a complicated model of the world, and we can take short cuts, because we just identify the differences compared to stuff we already know.
You may be right, no one knows for sure. I think it's one of those P = NP kinda problems. Lots of clever people have thought about it for years. Maybe there's a trick. It seems unlikely there's a trick though. Absence of evidence starts to become evidence of absence.
You mean how does one draw people to graduate school?
Thankfully we live in an age in which the most important research to developing fundamental things like Ai is 'time'. If you have an internet connection and a reasonably decent computer, you have enough resources to create AGi.
If you passionately care about seeing your idea through, save up money, move to a cheaper location, wire your computer to the internet, and get to work.
Once you've proven your vision, who can deny you then? Slam it on the boardroom table, disrupt present day business models, and be the next tech giant.
I wonder if it will be a chicken and egg problem though? Maybe you cannot even create decent gameplay without already having a trove of data, but bad gameplay will not attract more data from potential players?
Just needs integration with popular game engines like UE and unity3d, at the moment it mostly works with 8bit game emulators
Here's what you need to make an Ai Gym :
> Physics engine (pick your open source package)
> Simulator (pick your open source package)
> Write a thin wrapper around it
It can be done and has been done by many individuals in less than a week or two of dedicated work.
Maybe I am also not being very clear, but I am actually visualizing a game which is popular enough to attract a lot of gameplay, but has gameplay elements which are actually the kinds of things which the AI problems are trying to solve. E.g. imagine capturing gameplay from a popular multiplayer drone game to improve drone navigation.
Drones apparently don't fly very well autonomously.
http://spectrum.ieee.org/automaton/robotics/drones/skydio-ca...
Now, imagine if it were possible to get open data where any researcher or group could work on this problem using a gameplay dataset. Perhaps someone with an unexpected take on the problem could make some important contribution?
Big private datasets are important only in narrow domains - if you want to train an ANN to do ad prediction or something in medical imaging, sure.
But longer term and for the most valuable applications, the required data is plentiful and free. The data advanced robots (ex. self-driving cars) and AGI need to learn is all around us and it is free. Progress here is limited by compute/algorithms, not data.