Microsoft is investing $1B in OpenAI
openai.com
openai.com
> Mr. Nadella said Microsoft would not necessarily invest that billion dollars all at once. It could be doled out over the course of a decade or more. Microsoft is investing dollars that will be fed back into its own business, as OpenAI purchases computing power from the software giant, and the collaboration between the two companies could yield a wide array of technologies.
https://www.nytimes.com/2019/07/22/technology/open-ai-micros...
> It could be doled out over the course of a decade or more.
The NYT article is misleading here. We'll definitely spend the $1B within 5 years, and maybe much faster.
We certainly do plan to be a big Azure customer though!
excited about the announcement.
That's great, one question where can I use GYM or universe in the cloud with the render() option.
I've spend many hours trying to set up the environment in cloud [1] without success.
[1]: https://stackoverflow.com/questions/40195740/how-to-run-open...
- https://www.quora.com/Why-did-Elon-Musk-quit-OpenAI-Is-he-af...
- https://www.bloomberg.com/news/articles/2019-02-17/elon-musk...
From the article:
> The most obvious way to cover costs is to build a product, but that would mean changing our focus. Instead, we intend to license some of our pre-AGI technologies, with Microsoft becoming our preferred partner for commercializing them.
Talent will be the hardest challenge for OpenAI in order to reach their goals.
Will they fight?
Azure AI layers and say private company AIs like FBs (Ono Sendai), GCP, AWS, etc... where these AIs start learning how to protect themselves from attack....
Obv it super trivial for API mods to the FW/Access rules in all these systems... so it will be trvial for them to start shutting down access (we have had this for decades, but it will be interesting to see it at scale.)
Can you do an AI CTF like the Stripe distributed systems CTF sometime?
Since you work for OpenAi, are you looking at actual brain processes at all? I read the article and understand you guys will be a big customer with Azure, I wonder if you guys will be conducting some brain research though. I believe for AGI to happen we need to understand the brain.
I work with Cerebral Organoids, Consciousness studies, physics (quantum),
Love to share / connect, we are currently launching the Cerebral Organoids into space today! SpaceX rocket, 6pm EST, there are some thunderstorms so we're hoping there aren't any further delays. DM me?
Congrats to the team, and break a leg!
[1]: https://papers.nips.cc/paper/5346-sequence-to-sequence-learn... [2]: https://github.com/applecrazy/reportik/
Personally, I think it is only a matter of time. Though I suspect that we will probably 'cheat' our way there first with the wetware from cultured neurons that various groups are developing, before we manage to create it in completely synthetic hardware. Also, it might just be the wetware that leads us to the required insights. This is very problematic territory however. I think we are very likely to utterly torture some of the conscious entities created along this path.
The first main issue is that of compute capacity. Human brain has equivalent of at least 30 TFLOPS of computing power and this estimate is very likely 2 orders of magnitudes off.
Assume that somehow simulating 1 synapse takes only 1 transistor (gross underestimate). To simulate number of synapses in a single human brain then would need same number of transistors as in 10,000 NVidia V100 GPUs, one of the largest mass produced silicon chip!
The second main issue is of training neurons that are far more complex than our simple arithmetic adders. Back prop doesn't work for such complex neurons.
The 3rd big problem is that of training data. Human child churns through roughly 10 years of training data before reaching puberty. The man-made machine perhaps can take advantage of vast data already available but still there needs to be some structured training regiment.
So current AI efforts in relative comparison of human brain are playing with toy hardware and toy algorithms. It should be surprising that we have gone so far regardless.
I don't think an AI would help the Left/Right divide in this way because certain news outlets would still have the same incentives to manipulate what people desire in a more extreme directions.
If I were a billionaire, that's exclusively where I'd be putting my money, selfishly.
https://en.wikipedia.org/wiki/Octopus_(yacht)
https://en.wikipedia.org/wiki/Allen_Institute
https://www.cs.washington.edu/building
In fact this tradition of rich people founding universities and research is nothing new. Stanford University was founded by a couple who said "The children of California shall be our children" after their child died. Andrew Carnegie founded the Carnegie Technical Schools, and John Harvard donated money and his library to a college founded two years earlier.
[1] https://www.philanthropy.com/article/Paul-Allen-s-2-Billion-...
While not research, those things can have profound impacts on people. Several years ago a Star Wars exhibit came to the Indiana State Museum here in Indianapolis, they had an entire section dedicated to both the prosthetic devices in the film and in real life, one of the video segments playing next to some props from the film and real prosthetic devices was a clip of one of the inventors of the real technology talking about how watching the film version directly led to him pursuing his career and working directly on various prosthetic devices trying to make it a reality.
These sorts of experiences could have profound impact on the creative process for one or more individuals that might have far more profound effects for society than active research.
Billionaires buy cars and boats because they're stores of value. For instance, a Mclaren from the 90s is worth more today than when it was sold.
This shows that in the 12 months to the end of June the value of classic cars as a whole was up by 28%, which compared with a rise of 12% for the FTSE-100 index of leading shares and a 23% slump in the price of gold.
https://www.theguardian.com/business/2013/sep/07/luxury-inve...
HN had more positive comments regarding that announcement.
Does this mean that OpenAI may not disclose progress, papers with details, and/or open source code as much as in the past? In other words, what proprietary advantage will Microsoft gain when licensing new tech from OpenAI?
I understand that keeping some innovations private may help commercialization, which may help raise more funds for OpenAI, getting us to AGI faster, so my opinion is that could plausibly make sense.
> I understand that keeping some innovations private may help commercialization, which may help raise more funds for OpenAI, getting us to AGI faster, so my opinion is that could plausibly make sense.
That's exactly how we think about it. We're interested in licensing some technologies in order to fund our AGI efforts. But even if we keep technology private for this reason, we still might be able to eventually publish it.
Glad to see that the team is taking a pragmatic safety-first approach here, as well as towards the near-term economical realities of funding a very expensive project to ensure the fastest possible progress.
In the early days of OpenAI, my thoughts were that the project had good intentions, but a misguided focus. The last year has changed that, though. They absolutely seem to be on the right track. Very excited to see their progress over the next years.
No one knows how far off true AGI is, just like no one in 1940 (or 1910) knew how far off fission weapons were.
EDIT: I quite liked this article from a few years back [0], and the fission weapon prediction example is stolen from there.
This is what happened when it became known nuclear weapons were a viable concept. The technology shifted power to such an extreme degree that it was impossible not to invest in it, and the delay from «likely impossible» to «done» happened too fast for most observers to notice.
We don't have any such understanding, or even a definition, of 'AGI'.
Leo Szilard had more plausible philosophical musings in the early thirties, that did not have root in any workable practical idea. The published theoretical breakthroughs you mention didn’t happen until the late thirties. Nuclear fission, the precursor to the idea of an exponential chain reaction, happened only in 1938, 7 years before Trinity.
He didn't have internal combustion engines, but that's a practicality, other mechanical power sources already existed (Alexander the Great had torsion siege engines). They would never be sufficient for flight, of course, but the principle was understood.
But he could never have even begun to build airfoils, because he didn't have even an inkling of proto-aerodynamics. He saw that birds exist, so he drew a machine with wings that flapped. Look at the wings he drew: https://www.leonardodavinci.net/flyingmachine.jsp
That's an imitation of birds with no understanding behind it. That's the state of strong AI today: we see that humans exist, so we create imitations of human brains, with no understanding behind them.
That lead to machine learning, and after 40 years of research we figured out that if you feed it terabytes of training data, it can actually be "unreasonably effective", which is impressive! How many pictures of giraffes did you have to see before you could instantly recognize them, though? One, probably? Human cognition is clearly qualitatively different.
The danger of machine learning is not that it could lead to strong AI. It's that it is already leading to pervasive surveillance and misinformation. (idlewords is pretty critical of OpenAI, but I actually credit OpenAI with taking this quite seriously, unlike MIRI.)
Nuclear weapons required enriched uranium, and the gaseous diffusion process of the time was insanely power-hungry. Like non-negligable (>1% ?) percentage of the US's entire electrical generation power-hungry.
On the 2nd of December, 1942 he led an experiment at Chicago Pile 1 [1] that initiated the first self-sustaining nuclear reaction. And it was made with Uranium.
In fairness to Fermi, nuclear fission was discovered in 1938 [2] and published in early 1939.
0: https://books.google.com/books?id=aSgFMMNQ6G4C&pg=PA813&lpg=...
1: https://en.wikipedia.org/wiki/Chicago_Pile-1
2: https://en.wikipedia.org/wiki/Nuclear_fission#Discovery_of_n...
You are moving goalposts. You mentioned in the first place "fission weapons" and now you take a quote about "nuclear fission reactor" which is a whole different thing.
A nuclear reactor was also required for the production of Pu-239, which is what 2 of the first 3 bombs were made from.
But the fact that Fermi was doing such a calculation in the first place proves that we knew in principle how a fission weapon could work, even if we didn't know "how far off [they] were". As soon as we figured out the moon was just a rock 240,000 miles away, we knew in principle we could go there, even if we didn't know how far off that would be.
By contrast, we don't know what consciousness or intelligence even is. A child could define what walking on the moon is, and Fermi was able to define a self-sustaining nuclear reaction as soon as he learned what nuclear reactions were. What even is the definition of consciousness?
I am of course not saying you're wrong that "we don't know". We obviously don't know. It's possible, just like it's possible that we could discover cheap free energy (fusion?) tomorrow and then be in a post-scarcity utopia. But that's worth taking about as seriously as the possibility that we'll discover AGI tomorrow and be in a Terminator dystopia, or also a post-scarcity utopia.
More importantly, it's a distraction from the very real, well-past-imminent problems that existing dumb AI has, such as the surveillance economy and misinformation. OpenAI, to their credit, does a good job of taking these existing problems quite seriously. They draw a strong contrast to MIRI's AI alarmism.
Have you ever read idlewords? Best writing I know of on this subject: https://idlewords.com/talks/superintelligence.htm
I have problems agreeing with that specific claim, knowing that both "the rock" and the distance were known to some ancient Greeks around 2200 years ago.
https://en.wikipedia.org/wiki/Hipparchus
Hipparchus estimated the distance to the Moon in the Earth radii to between 62 and 80 (depending on the method he used, as he intentionally used two different). Today's measurements are between 55 and 64.
Once we had Newton's law of gravity though, we knew the distance, radius, mass, and even surface gravity of the moon. Would you say it's fair to say that by then we knew in principle we could go there and walk there?
(P.S. I assume you know this but the way you wrote your comment makes it seem like our measurements of lunar distance are nearly as inaccurate as Hipparchus's, when we actually know it down to the millimeter (thanks to retroreflectors placed by Apollo, actually). The wide variation from 55x to 64x Earth's radius is because it changes over the course of the moon's orbit, due to [edit: primarily its elliptical orbit, and only secondarily] the Sun and Jupiter's gravity.)
I think you’re not only wrong but even Kepler and Newton already knew that better than you:
https://en.m.wikipedia.org/wiki/Elliptic_orbit
“Strictly speaking, both bodies revolve around the same focus of the ellipse, the one closer to the more massive body, but when one body is significantly more massive, such as the sun in relation to the earth, the focus may be contained within the larger massing body, and thus the smaller is said to revolve around it.”
But maybe you have some better information?
> due to its elliptical orbit with varying eccentricity, the instantaneous distance varies with monthly periodicity. Furthermore, the distance is perturbed by the gravitational effects of various astronomical bodies – most significantly the Sun and less so Jupiter
https://en.wikipedia.org/wiki/Lunar_distance_(astronomy)#Per...
> Once we had Newton's law of gravity though, we knew the distance, radius, mass, and even surface gravity of the moon.
I think it was more complicated than what you assume there. Newton published his Principia 1687 but before 1798 we didn't know the gravitational constant:
https://en.wikipedia.org/wiki/Cavendish_experiment
However...
> Would you say it's fair to say that by then we knew in principle we could go there and walk there?
If you mean "we 'could' go if we had something what we were sure we haven't had" then there is indeed a written "fiction" story published even before Newton published his Principia:
https://en.wikipedia.org/wiki/Comical_History_of_the_States_...
It's the discovery of the telescope that allowed people to understand that there are another "worlds" and that one would be able to "walk" there.
Newton's impact was to demonstrate that there is no any "mover" (which many before identified as a deity) that provides the motion of the planets but that their motions simply follow from their properties and the "laws." Before, most expected Aristotle to be relevant:
https://en.wikipedia.org/wiki/Unmoved_mover
"In Metaphysics 12.8, Aristotle opts for both the uniqueness and the plurality of the unmoved celestial movers. Each celestial sphere possesses the unmoved mover of its own—presumably as the object of its striving, see Metaphysics 12.6—whereas the mover of the outermost celestial sphere, which carries with its diurnal rotation the fixed stars, being the first of the series of unmoved movers also guarantees the unity and uniqueness of the universe."
Almost nobody really knows how developed is the state-of-the-art theory / applied technology in confidentials advances that the usual suspects may have already achieved. I.E. deepmind, openai, baidu, nsa, etc.
AGI could have already been achieved - even theoretically - somewhere, and like when Edison got to make work a light bulb, we're still using oil and not knowing anything about electricity, or light bulbs or energy distribution networks / infrastructure.
The actual current - new, mostly unimplemented yet - technology level.
Back then you wouldn't have believed if someone had said you "hey, city nights in ten years won't be dark anymore"
There's no way to disprove it, but given that in the open literature people haven't even found a way to coherently frame the question of general AI, let alone theorize about it, it becomes just another form of magical thinking.
There are several public examples of radically more advanced theory/technology than the publicly known possible at a certain time/year, kept secret by governments / corps for a very long time (decades).
Lockheed achieved the blackbird decades before it was even admitted that a technology like that could even exist. But, looking backwards, it just looks like an "incremental" advance, but it wasn't, the engineering required to make fly the blackbird was revolutionary for the time when it was invented (back in the 50s / 60s ).
The Lockheed F-117 and its tech had a similar path, just somewhat admitted in late 80s (and this was 70s technology, probably based on theoretical concepts from the 60s).
More or less the same could be said about the tech in Blechtley Park: current tech / theory propelled to extraordinary capabilities by radical improvement achieved by new top secret advances in engineering. The hardware, events and advances ocurred in Bletchley Park were kept secret for years (I think just in the 50s they started to be carefully mentioned but not fully admitted, but nothing even close to the details currently found in the Wikipedia).
At any given time there could be a lot of theory/technology jump-aheads being achieved out there, several decades ahead of the publicly published/known, supposedly current, theory/technology.
It was hard to predict when or if such a thing could be made, but everyone knew what was under discussion.
Compare this to AGI, some vaguely emergent property of a complex computer system that no one can define to anyone else's satisfaction. Attempts to be more precise what AGI is, how it would first manifest itself, and why on earth we should be afraid of it, rapidly devolve into nerd ghost stories.
All these things have surged incredibly in less than a decade.
It's always a long way off until it isn't.
Not at all, these are all one-trick poneys and bring you nowhere close to real AGI which is akin to human intelligence.
Extrapolating as you seem to be here, when should I expect to see a total conversion reactor show up? I want 100% of the energy in that Uranium, dammit - not the piddly percentages you get from fission!
Seriously, I think you overestimate how predictable nuclear weapons were. Fission was discovered in 1938.
We haven't even had the AGI equivalent of the Rutherford model of the atom yet: what's the definition of consciousness? What is even the definition of intelligence?
However, we are not getting impressively close to AGI. That's why we need to stop the AGI alarmism and get our act together on the enormous societal ramifications that machine learning is already having.
1932 neutron discovered
1942 first atomic reactor
1945 fission bomb
1897 electron discovered
1940's vacuum tube computers
1970's integrated circuits
1980's first AI wave fails, AI winter begins
2012 AI spring begins
2019 AI can consistently recognize a jpeg of a cat, but still not walk like a cat
???? Human level AGI
1943 First mathematical neural network model
1958 Learning neural network classifies objects in spy plane photos
1965 Deep learning with multi-layer perceptrons
2010 ImageNet error rate 28%
2011 ImageNet error rate 25%
2012 ImageNet error rate 16%
2013 ImageNet error rate 11%
2017 ImageNet error rate 3%
2019 Pre-AGIOur study of (automated) intelligence is based on science too.
> A computer ... will never be able to think by itself.
Turing wrote an entire paper about this (Computing Machinery and Intelligence), where he rephrases your statement (because he finds it to be meaningless) and devises a test to answer it. He also directly attacks your phrasing of "but it will never":
> I believe they are mostly founded on the principle of scientific induction. A man has seen thousands of machines in his lifetime. From what he sees of them he draws a number of general conclusions. They are ugly, each is designed for a very limited purpose, when required for a minutely different purpose they are useless, the variety of behaviour of any one of them is very small, etc., etc. Naturally he concludes that these are necessary properties of machines in general.
> A better variant of the objection says that a machine can never "take us by surprise." This statement is a more direct challenge and can be met directly. Machines take me by surprise with great frequency. This is largely because I do not do sufficient calculation to decide what to expect them to do, or rather because, although I do a calculation, I do it in a hurried, slipshod fashion, taking risks.
Can you elaborate which part of sciences you are talking about here?
Any AI curriculum worth its salt includes the many scientific and philosophical views on intelligence. It is not all alchemy, though the field is in a renewal phase (with horribly hyped nomenclature such as "pre-AGI", and the most impressive implementations coming from industry and government, not academia).
And eventhough the atom bomb was based on science too, there is this anecdote from Hamming:
> Shortly before the first field test (you realize that no small scale experiment can be done—either you have a critical mass or you do not), a man asked me to check some arithmetic he had done, and I agreed, thinking to fob it off on some subordinate. When I asked what it was, he said, "It is the probability that the test bomb will ignite the whole atmosphere." I decided I would check it myself! The next day when he came for the answers I remarked to him, "The arithmetic was apparently correct but I do not know about the formulas for the capture cross sections for oxygen and nitrogen—after all, there could be no experiments at the needed energy levels." He replied, like a physicist talking to a mathematician, that he wanted me to check the arithmetic not the physics, and left. I said to myself, "What have you done, Hamming, you are involved in risking all of life that is known in the Universe, and you do not know much of an essential part?" I was pacing up and down the corridor when a friend asked me what was bothering me. I told him. His reply was, "Never mind, Hamming, no one will ever blame you."
This seems like a cop out. Sure, if you do your calculations wrong, it doesn’t behave as you expect. But it’s still doing exactly what you wrote it to do. The surprise is in realizing your expectations were wrong, not that the machine decided to behave differently.
A good example of this is "move 37" from AlphaGo. This move surprised everyone, including the creators, who were not skilled enough in Go to hardcode it: https://www.youtube.com/watch?v=HT-UZkiOLv8
A computer being able to simulate a brain that thinks for itself is the logical extrapolation of current brain-simulation efforts. Many people think there are far less computationally intensive ways to make an AI, but "physics sim of a human brain" is a good thought experiment.
Unless you think there's something magic about human brains? Using "magic" here to mean incomprehensible, unobservable, and incomputable.
Except that our current neural networks have nothing to do with the actual neurons in our brain and how they work.
"If the machines are permitted to make all their own decisions, we can’t make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines. It might be argued that the human race would never be foolish enough to hand over all power to the machines. But we are suggesting neither that the human race would voluntarily turn power over to the machines nor that the machines would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines’ decisions. As society and the problems that face it become more and more complex and as machines become more and more intelligent, people will let machines make more and more of their decisions for them, simply because machine-made decisions will bring better results than man-made ones. Eventually a stage may be reached at which the decisions necessary to keep the system running will be so complex that human beings will be incapable of making them intelligently. At that stage the machines will be in effective control. People won’t be able to just turn the machine off, because they will be so dependent on them that turning them off would amount to suicide."
The accumulated knowledge and skills of not just specialised individuals but entire institutions, working on highly technical and abstract areas of society, seems like it has created a kind of empathy gap between the people ostensibly wielding power and those who are experiencing the effects of that power (or the limits of that power).
> "... turning them off would amount to suicide."
Although this conclusion appears equally valid in the replacement argument, it sadly doesn't come with the wanted guarantee of "therefore that wouldn't happen".
I don’t mean to parse your words, but will you continue to publish using the same exact criteria as before or will there be a new editorial filter?
Now, the demonstrated ability to produce new models which are closed, but maybe can be used as services on a preferred partner's cloud, looks very commercially relevant? How will these conflicts be managed, or is it more like "we are just a commercial entity now, of course we'll do this"?
> We’re partnering to develop a hardware and software platform within Microsoft Azure which will scale to AGI. We’ll jointly develop new Azure AI supercomputing technologies, and Microsoft will become our exclusive cloud provider—so we’ll be working hard together to further extend Microsoft Azure’s capabilities in large-scale AI systems.
Maybe it's because I'm not an expert, but what does it really mean? Do people understand what "Microsoft will become our exclusive cloud provider" means?
OpenAI is great, but suspicious is understandable from the users side when so much commercial money is involved.
Is it $1 billion in cash/stock?
Or $1 billion in Azure credits and engineering hours?
The comment I replied to may not be far off the mark in what this really is: computer/human time “worth $1 billion” or something.
If it’s actual cash that says something different to me than a donation of resources with some value estimated by MS.
My "guess" is that it means MSFT has access to sell products based off the research OpenAI does to MSFT's customers. Having early access to advanced research means MSFT could easily make this money back by selling better AI tools to their customers.
Also a great time to point out that while "Microsoft is not a charity foundation" it does offer a ton of free Azure to charities. https://www.microsoft.com/en-us/nonprofits/azure This has been an awesome thing to use when helping small non-profits with little money to spend on "administrative costs".
It's a cash investment. We certainly do plan to be a big Azure customer though.
> My "guess" is that it means MSFT has access to sell products based off the research OpenAI does to MSFT's customers. Having early access to advanced research means MSFT could easily make this money back by selling better AI tools to their customers.
I'm flattered that you think our research is that valuable! (As I say in the blog post: we intend to license some of our pre-AGI technologies, with Microsoft becoming our preferred partner for commercializing them.)
I remember the first meetings about ARM when the resource IDs were presented, and a few people immediately asked "what if someone wants to rename a resource"? Years later you still could not do that (I'm hoping they've fixed that by now?).
It seemed to me that ARM was the result of some design by super smart committee, and got a lot wrong. When I was there more senior folks told me not to worry, that's just the Microsoft way (wait for version 3). I do have to admit that it's turning out they knew more than me (shocking!), as over time I've seen some of the stuff that was inexplicably terrible in v1 become much, much better in later versions.
They are investing a good amount in ARM lately though. The vs code language server is pretty good and export template got much better
Awesome! I sheepishly have been using GCP, AWS, and DO. Last gave Azure a shot last year, but perhaps it's time to take another look.
When I was 10 I created some "pre-time travel" technology by designing an innovative control panel for my time machine. Sadly I ran into some technical obstacles later in the project. OpenAI is at about the same phase with AGI.
Going back in time:
> Musk has joined with other Silicon Valley notables to form OpenAI, which was launched with a blog post Friday afternoon. The group claimed to have the goal “to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.”
What happened here?
I know it’s far off, but I am concerned about AGI misanthropy and the for-profit turn of OpenAI. Who is the humanist anchor, of Elon’s gravitas, left at OpenAI?
What happened to the original mission? Are any of you concerned about this? Can you get rid of Peter Thiel please? Can we buy him out as a species? I respect the man’s intellect yet truly fear his misanthropy and influence.
Apologies for the rambling, but you all got me freaked out a bit. I had, and still do have such high hopes for OpenAI.
Please don’t lose the forest for the trees.
Why? He left due to possible conflict of interest, Tesla is researching AI for self-driving vehicles and it wouldn't surprise me if SpaceX does at some point too (assuming they aren't already).
See: https://twitter.com/elonmusk/status/1096987465326374912?lang...
Microsoft really needs this. ML.NET is quite anemic compared to the industry-standard AI toolkits: TensorFlow, theano, scikit-learn, Torch, Keras, etc.
https://dotnet.microsoft.com/apps/machinelearning-ai/ml-dotn...
Another way to think about it is that for folks building in .NET, ML.NET makes it easy for them to start using many of the ML techniques without having to learn something new.
On top of that, we FULLY support all the industry standard tools - TF, Keras, PyTorch, Kubeflow, SciKit, etc etc. We even ship a VM that lets you get started with any of them in one click (https://azure.microsoft.com/en-us/services/virtual-machines/...) and our hosted offering supports them as well! (e.g. TF - https://docs.microsoft.com/en-us/azure/machine-learning/serv...)
On both scale up and run times, it measures up as among the best-in-class[1]. That is to say, for the scenarios which people use it most commonly (GBT, linear learners), it's a great fit!
For OpenAI, it means the availability of resources for their main mission for the foreseeable future, while potentially allowing founders and other investors with the opportunity to either double-down on OpenAI or reallocate resources to other initiatives (Think of Musk, for example).
"Do people understand what "Microsoft will become our exclusive cloud provider" means?" It likely means that computing power will be provided by Microsoft and that it may have access to the algorithms and results.
OpenAI is a commercial entity. They restructured from a non-profit.
This is a completely commercial deal to help Azure catch up with Google and Amazon in AI. OpenAI will adopt and make Azure their preferred platform. And Microsoft and Azure will jointly "develop new Azure AI supercomputing technologies", which I assume is advancing their FGPA-based deep learning offering.
Google has a lead with TensorFlow + TPUs and this is a move to "buy their way in", which is a very Microsoft thing to do.
I really liked LUIS (Language Understanding Intelligent Service) back in 2017 and AFAIK only Alibaba had an offering similar to Azure at the time for ML-as-a-service.
The cynic in me thinks this will never happen, that instead it will make a small subset of the population super rich, while the rest are put to work somewhere to make them even more money. Microsoft will ultimately want a return on their billion, at least.
I mean, sugar is pretty much the definition of a high calorie food. Its like, pure calories. And can affect insulin regulation, etc. That's why they need to put some marketing on it.
By saying sugar is "pure calories," I guess you mean that it doesn't contain any fibre or micronutrients that might give it redeeming qualities (besides its taste), which is true.
Sugar is, quite literally, pure carbohydrates. Most sources of protein are not, unless you're consuming refined amino acids (pro tip: they taste awful).
That could be... interesting. It could be in the form of beef bullion, so you really could put in a spoonful.
But I wonder if bacon would be better...
Mr. Musk left the lab last year to concentrate on his own A.I. ambitions at Tesla. Since then, Mr. Altman has remade OpenAI, founded as a nonprofit, into a for-profit company so it could more aggressively pursue financing.
https://www.nytimes.com/2019/07/22/technology/open-ai-micros...
> we’ll be working hard together to further extend Microsoft Azure’s capabilities
> Instead, we intend to license some of our pre-AGI technologies, with Microsoft becoming our preferred partner for commercializing them.
In the end, it's a win-win. If OpenAI remains partially open , it's still better for rest of the world too, better than nothing. But, as achow said, it did pivot.
Also this sounds dangerous: "exclusive cloud provider".
When an OpenAI group starts to make exclusively partnerships with one vendor, I wonder how "Open" it is.
I can not imagine Khronos Group, which runs similarly named OpenGL, etc having a "exclusive" graphics card supported for their open standards. Cloud computing is to OpenAI as graphics cards are to OpenGL/Vulkan.
Can we assume that marketing overrode engineering on the terminology of this press release?
We certainly cannot feasibly build AGI today, hence OpenAI's use of the term "pre-AGI technologies".
It is a research mission. No one "feasibly" knows exactly how to build AGI yet. But still we have many groups publicly pursuing it today.
If Microsoft is giving them a billion dollars in this context, I assume that Open AI engineers and scientists will build out services for Azure ML that will then be sold to developers or consumers.
This type of thing is actually pretty normal for just about every company that is seriously pursuing AGI, since they eventually need some kind of income and narrow AI is the way for those types of teams to do that.
And do people really want to be "actualized" by "Microsoft and OpenAI’s shared value of empowering everyone"?
Quite the opposite — this is an investment!
For example, if there is an hardware innovation which make DNN training 1000x faster (e.g. optical DNN), but it does not exist on azure, than by definition it cannot be offered on another cloud.
To sum up, this deal assure the choking point of azure/MS on any innovation that would come up from open ai.
As an industry, we've already burned through a bunch of buzzwords that are now meaningless marketing-speak. 'ML', 'AI', 'NLP', 'cognitive computing'. Are we going for broke and adding AGI to the list so that nothing means anything any more?
What "threshold" would you want to cross before you think its socially acceptable to put resources behind ensuring that humanity doesn't wipe itself out?
The tricky thing with all of this is we have no idea what an appropriate timeline looks like. We might be 10 years away from the singularity, 1000 years, or it might never ever happen!
There is a non-zero chance that we are a few breakthroughs away from creating a technology that far surpasses the nuclear bomb in terms of destructive potential. These breakthroughs may have a short window of time between each of them (once we know a, knowing b,c,d will be much easier)
So given all of that, wouldn't it make sense to start working on these problems now? And the unfortunate part of working on these problems now is that you do need hype/buzzwords to attract tallent, raise money and get people talking about AGI safety. Sure it might not lead anywhere, but just like fire insurance might seem unnecessary if you never have a fire, AGI research may end up being a useless field altogether but at least it gives us that cushion of safety.
"We’re partnering to develop a hardware and software platform within Microsoft Azure which will scale to AGI"
Azure needs a few more years just to un-shit the bed with what their marketing team has done and catch up to even basic AWS/GCP analytics offerings. Them talking about AGI is like a toddler talking about building a nuclear weapon. This is the same marketing team that destroyed any meaning behind terms like 'real time', and 'AI'.
I don't know, but I'd say after a definition of "AGI" has been accepted that can be falsified against, and actually turn it into a scientific endeavour.
> The tricky thing with all of this is we have no idea what an appropriate timeline looks like.
We do. As things are it's undetermined, since we don't even know what's it's supposed to mean.
> So given all of that, wouldn't it make sense to start working on these problems now?
What problems? We can't even define the problems here with sufficient rigor. What's there to discuss?
Uhh, that's the Turing Test.
- Privacy (How do you get an artificial intelligence to recognize, and respect, privacy? What sources is it allowed to use, how must it handle data about individuals? About groups? When should it be allowed to violate/exploit privacy to achieve an objective?)
- Isolation (How much data do you allow it access to? How do you isolate it? What safety measures do you employ to make sure it is never given a connection to the internet where it could, in theory, spread itself not unlike a virus and gain incredibly more processing power as well as make itself effectively undestroyable? How do you prevent it from spreading in the wild and hijacking processing power for itself, leaving computers/phones/appliances/servers effectively useless to the human owners?)
- A kill switch (under what conditions is it acceptable to pull the plug? Do you bring in a cybernetic psychologist to treat it? Do you unplug it? Do you incinerate every last scrap of hardware it was on?)
- Sanity check/staying on mission (how do you diagnose it if it goes wonky? What do you do if it shows signs of 'turning' or going off task?
- Human agents (Who gets to interact with it? How do you monitor them? How do you make sure they aren't being offered bribes for giving it an internet connection or spreading it in the wild? How do you prevent a biotic operator from using it for personal gain while also using it for the company/societal task at hand? What is the maximum amount of time a human operator is allowed to work with the AI? What do you do if the AI shows preference for an individual and refuses to provide results without that individual in attendance? If a human operator is fired, quits or dies and it negatively impacts the AI what do you do?)
This is why I've said elsewhere in this thread, and told Sam Altman, that they need to bring in a team of people that specifically start thinking about these things and that only 10-20% of the people should be computer science/machine learning types.
OpenAI needs a team thinking about these things NOW, not after they've created an AGI or something reaching a decent approximation of one. They need someone figuring out a lot of this stuff for tools they are developing now. Had they told me "we're going to train software on millions of web pages, so that it can generate articles" I would have immediately screamed "PUMP THE BRAKES! Blackhat SEO, Russian web brigades, Internet Water Army, etc etc would immediately use this for negative purposes. Similarly people would use this to churn out massive amounts of semi-coherent content to flood Amazon's Kindle Unlimited, which pays per number of page reads from a pool fund, to rapidly make easy money." I would also have cautioned that it should only be trained on opt-in, vetted, content suggesting that using public domain literature, from a source like Project Gutenberg, would likely have been far safer than the open web.
No, there is exactly zero chance that anyone is "a few breakthroughs away" from AGI.
AGI represents the creation of a mind .... It's something that has three chief characteristics: it understands the world around it, it understands what effects its actions will have on the world around it, and it takes actions.
None of those three things are even close to achievable in the present day.
No software understands the physical world. The knowledge gap here is IMMENSE. Software does not see what we see: it can be trained to recognize objects, but its understanding is shallow. Rotate those objects and it becomes confused. It doesn't understand what texture or color really are, what shapes really are, what darkness and light really are. Software can see the numerical values of pixels and observe patterns in them but it doesn't actually have any knowledge of what those patterns mean. And that's just a few points on the subject of vision, let alone all the other senses, all the world's complex perceivable properties. Software doesn't even know that there IS a world, because software doesn't KNOW anything! You can set some data into a data structure and run an algorithm on it, but there's no real similarity there to even a baby's ability to know that things fall when you drop them, that they fall in straight lines, that you can't pass through solid objects, that things don't move on their own, etc etc.
Even if, a century from now, some software did miraculously approach such an understanding, it still would not know how it was able to alter the world. It might know that it was able to move objects, or apply force to then, but could it see the downstream effects? Could it predict that adding tomatoes to a chocolate cake made no sense and rendered the cake inedible? Could it know that a television dropped out the window of an eight story building was dangerous to people on the sidewalk below? Could it know that folding a paper bag in half is not destructive, but folding a painting in half IS? Understanding what can result from different actions and why some are effective and others are not, is another vast chasm of a knowledge gap.
Lastly, and by FAR most importantly, the most essential thing.....software does not want. Every single thing we do as living creatures is because our consciousness drives us to want things: I want to type these words at this moment because I enjoy talking about this subject. I will leave soon because I want food and hunger is painful. Etc. If something does not feel pleasure or pain or any true sensation, it cannot want. And we have absolutely no idea how such a thing works, let alone how to create it, because we have next to no idea how our own minds work. Any software that felt nothing, would want nothing-- and so it would sit, inert, motionless...never bored, never curious, never tired, just like an instance of Excel or Chrome. Just a thing, not alive. No such entity could genuinely be described as AGI. We are likely centuries from being able to recreate our consciousness, our feelings and desires....how could someone ever be so naive as to believe it was right around the corner?
Actual sentiment analysis is a completely different kind of ML problem than supervised classification 'sentiment analysis' that's popular today but mostly useless for real world applications.
To get useful value out of automated sentiment analysis, that's the cost to build and maintain domain specific models. Pre-canned sentiment analysis models like the parent post linked are more often than not worthless for general purpose use. I won't say there are 0 scenarios where those models are useful, but the number is not high.
Claiming that sentiment analysis is 90something percent accurate, or even close to being solved, is extremely misleading.
I think continual Turing testing is the only way of concluding whether an agent exhibits intelligence or not. Consider the philosophical problem of the existence of other minds. We believe other humans are intelligent because they consistently show intelligent behavior. Things that people claim to be examples of AI right now lack this consistency (possibly excluding a few very specific examples such as AlphaZero). It is quite annoying to see all these senior researchers along with graduate students spend so much time pushing numbers on those datasets without paying enough attention to the fact that pushing numbers is all they are doing.
[1]: As a concrete example, consider the textual entailment (TE) task. In the deep learning era of TE there are two commonly used datasets on which the current state-of-the-art has been claimed to be near or exceeding human performance. What these models are performing seemingly exceptionally well is not the general task of TE, it is the task of TE evaluated on these fixed datasets. A recent paper by McCoy, Pavlick, and Linzen (https://arxiv.org/abs/1902.01007) shows how brittle these systems are that at this point the only sensible response to those insistent on claiming we are nearing human performance in AI is to laugh.
So you think it's impossible to ever determine that a chimpanzee, or even a feral child, exhibits intelligence? This seems rather defeatist.
Let me elaborate on my previous point with an example. If you look at the recent works in machine translation, you can see that the commonly used evaluation metric of BLEU is being improved upon at least every few months. What I argue is that it's stupid to look at this trend and conclude that soon we will reach human performance in machine translation. Even when comparing against the translation quality of humans (judged again by BLEU on a fixed evaluation set) and showing that we can achieve higher BLEU than humans is not enough evidence. Because you also have Google Translate (let's say it represents the state-of-the-art), and you can easily get it to make mistakes that humans would never do. I consider our prolonged interaction with Google Translate to be a narrow Turing test that we continually apply to it. A major issue in research is that, at least in supervised learning, we're evaluating on datasets that are not different enough from the training sets.
Another subtle point is that we have strong priors about the intelligence of biological beings. I don't feel the need to Turing test every single human I meet to determine whether they are intelligent, it's a safe bet at this point to just assume that they are. The output of a machine learning algorithm, on the other hand, is wildly unstable with respect to its input, and we have no solid evidence to assume that it exhibits consistent intelligent behavior and often it is easy to show that it doesn't.
I don't believe that research in AI is worthless, but I think it's not wise to keep digging in the same direction that we've been moving in for the past few years. With deep learning, while accuracies and metrics are pushed further than before, I don't think we're significantly closer to general, human-like AI. In fact, I personally consider only AlphaZero to be an unambiguous win for this era of AI research, and it's not even clear whether it should be called AI or not.
If you gave 100 chimps of the highest calibre 100 attempts each, not a single one would pass a single Turing test. Ask a feral child to translate even the most basic children's book, and their mistakes will be so systematic that Google Translate will look like professional discourse. ‘Humanlike mistakes’ and stability with respect to input in the sense you mean here are harder problems than intelligence, because a chimp is intelligent and functionally incapable of juggling more than the most primitive syntaxes in a restricted set of forms.
I agree it is foolish to just draw a trend line through a single weak measure and extrapolate to infinity, but the idea that no collation of weak measures has any bearing on fact rules out ever measuring weak or untrained intelligence. That is what I called defeatist.
Wikipedia defines Turing test as "a test of a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human". If we want to consider chimps intelligent, then in that context the definition of the Turing test should be adjusted accordingly. My point still stands: if we want to determine whether a chimp exhibits intelligence comparable to a human, we do the original Turing test. If we want to determine whether a chimp exhibits chimplike intelligence, we test not for, say, natural language but for whatever we want our definition of intelligence to include. If we want to determine whether an artificial agent has chimplike intelligence, we do the second Turing test. Unless the agent can display as consistent an intelligence as chimps, we shouldn't conclude that it's intelligent.
Regarding your point on weak measures: If I can find an endless stream of cases of failure with respect to a measure that we care about improving, then whatever collation of weak measures we had should be null. Wouldn't you agree? I'm not against using weak measures to detect intelligence, but only as long as it's not trivial to generate failures. If a chimp displays an ability for abstract reasoning when I'm observing it in a cage but suddenly loses this ability once set free in a forest, it's not intelligent.
If AI researchers were growing neurons in vats and those neurons were displaying abilities on par with chimpanzees I'd want those researchers to be able to say ‘hold up, we might be getting close to par-human intelligence, let's make sure we do this right.’ And I want them to be able to do that even though their brains in vats can't pass a Turing test or write bad poetry or play basic Atari games and the naysayers around them continue to mock them for worrying when their brains in vats can't even pass a Turing test or write bad poetry or play basic Atari games.
Like, I don't particularly care that AI can't solve or even approach solving the Turing test now, because I already know it isn't human-par intelligent, and more data pointing that out tells me nothing about where we are and what's out of reach. All we really know is that we've been doing the real empirical work with fast computers for 20ish years now and gone from no results to many incredible results, and in the next 30 years our models are going to get vastly more sophisticated and probably four orders of magnitude larger.
Where does this end up? I don't know, but dismissing our measures of progress and improved generality with ‘nowhere near as robust as [...] humans’ is certainly not the way figure it out.
> If I can find an endless stream of cases of failure with respect to a measure that we care about improving, then whatever collation of weak measures we had should be null. Wouldn't you agree?
No? Isn't this obviously false? People can't multiply thousand-digit numbers in their heads; why should that in any way invalidate their other measures of intelligence?
What exactly is incredible (relatively) about the current state of things? I don't know how up-to-date you are on research, but how can you be claiming that we had no results previously? This is the kind of ignorance of previous work that we should be avoiding. We had the same kind of results previously, only with lower numbers. I keep trying to explain that increasing the numbers is not going to get us there because the numbers are measuring the wrong thing. There are other things that we should also focus on improving.
>dismissing our measures of progress and improved generality with ‘nowhere near as robust as [...] humans’ is certainly not the way figure it out.
It is the way to save this field from wasting so much money and time on coming up with the next small tweak to get that 0.001 improvement in whatever number you're trying to increase. It is not a naive or spiteful dismissal of the measures, it is a critique of the measures since they should not be the primary goal. The majority of this community is mindlessly tweaking architectures in pursuit of publications. Standards of publication should be higher to discourage this kind of behavior. With this much money and manpower, it should be exploding in orthogonal directions instead. But that requires taste and vision, which are unfortunately rare.
>People can't multiply thousand-digit numbers in their heads; why should that in any way invalidate their other measures of intelligence?
Is rote multiplication a task that we're interested in achieving with AI? You say that you aren't interested categorizing for the sake of categorizing, but this is a counterexample for the sake of giving a counterexample. Avoiding this kind of an example is precisely why I said "a measure that we care about improving".
Compared to 1999?
Watch https://www.youtube.com/watch?v=kSLJriaOumA
Hear https://audio-samples.github.io/#section-4
Read https://grover.allenai.org/
These are not just ‘increasing numbers’. These are fucking witchcraft, and if we didn't live in a world with 5 inch blocks of magical silicon that talk to us and giant tubes of aluminium that fly in the sky the average person would still have the sense to recognize it.
> It is the way to save this field from [...]
For us to have a productive conversation here you need to either respond to my criticisms of this line of argument or accept that it's wrong. Being disingenuous because you like what the argument would encourage if it were true doesn't help when your argument isn't true.
> Is rote multiplication a task that we're interested in achieving with AI?
It's a measure for which improvement would have meaningful positive impact on our ability to reason, so it's a measure we should wish to improve all else equal. Yes, it's marginal, yes, it's silly, that's the point: failure in one corner does not equate to failure in them all.
What about generative models is really AI, other than the fact that they rely on some similar ideas from machine learning that are found in actual AI applications? Yes, maybe to an average person these are witchcraft, but any advanced technology can appear that way---Deep Blue beating Kasparov probably was witchcraft to the uninitiated. This is curve fitting, and the same approaches in 1999 were also trying to fit curves, it's just that we can fit them way better than before right now. Even the exact methods that are used to produce your examples are not fundamentally new, they are just the same old ideas with the same old weaknesses. What we have right now is a huge hammer, and a hammer is surely useful, but not the only thing needed to build AI. Calling these witchcraft is a marketing move that we definitely don't need, creates unnecessary hype, and hides the simplicity and the naivete of the methods used in producing them. If anybody else reads this, these are just increasing numbers, not witchcraft. But as the numbers increase it requires a little more effort and knowledge to debunk them.
I'm not dismissing things for the fun of it, but it pains me to see this community waste so many resources in pursuit of a local minimum due to lack of a better sense of direction. I feel like not much more is to be gained from this conversation, although it was fun, and thank you for responding.
I'm not evaluating these models on whether they are AGI, I am evaluating them on what they tell us about AGI in the future. They show that even tiny models, some 10000x to 1000000x times smaller than what I think are the comparable measures in the human brain, trained with incredibly simple single-pass methods, manage to extract semirobust and semantically meaningful structure from raw data, are able to operate on this data in semisophisticated ways, and do so vastly better than their size-comparable biological controls. I'm not looking for the human, I'm looking for small scale proofs of concepts of the principles we have good reasons to expect are required for AGI.
The curve fitting meme[1] has gotten popular recently, but it's no more accurate than calling Firefox ‘just symbols on the head of a tape’. Yes, at some level these systems reduce to hugely-dimensional mathematical curves, but the intuitions this brings are pretty much all wrong. I believe this meme has gained popularity due to adversarial examples, but those are typically misinterpreted[2]. If you can take a system trained to predict English text, prime it (not train it) with translations, and get nontrivial quality French-English translations, dismissing it as ‘just’ curve fitting is ‘just’ the noncentral fallacy.
Fundamental to this risk evaluation is the ‘simplicity and the naivete of the methods used in producing them’. That simple systems, at tiny scales, with only inexact analogies to the brain, based on research younger than the people working on it, is solving major blockers in what good heuristics predict AGI needs is a major indicator about the non-implausibility of AGI. AGI skeptics have their own heuristics instead, with reasons those heuristics should be hard, but when you calibrate with the only existence proof we have of AGI development—human evolution—, those heuristics are clearly and overtly bad heuristics that would have failed to trigger. Thus we should ignore them.
[1] Similar comments on ‘the same approaches in 1999’, another meme only true at the barest of surface levels. Scale up 1999 models and you get poor results.
[2] See http://gradientscience.org/adv/. I don't agree with everything they say, since I think the issue relates more to the NN's structure encoding the wrong priors, but that's an aside.
Humble understatement. *co-founded and co-run.
Being forced to use Azure for all your ML workloads seems a stupid constraint. For example, you might be comfortable with tensorflow/TPU and changing frameworks/tooling might be costly.
Throwing money at a problem doesn’t always produce solutions. It can sure accelerate a project down the path it is on...but, if the path is wrong...
In some ways it reminds me of the battle against cancer.
Not being critical of this project or donation, just stating a point of view on the general problem of solving AI, a subject I have been involved with to one degree or another since the early 80’s.
Think about how far we are from being able to even get close to what an ant can do. Work it backwards from there.
> OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity.
I'm struck by the homogeneity the OpenAI team.
https://openai.com/content/images/2019/07/openai-team-offsit...
It seems to be mostly white people and a few Asians, without a single black or Hispanic person.
Hiring the most qualified people is the most important thing. As long as there isn't an inherent bias for not hiring someone who is hispanic, black, or brown, it should b e fine.
There have been studies done around diversity, conducted both privately and publicly, which consistently conclude that increased diversity does result in enhanced decision-making, collaboration, and organizational value-add due to the different perspectives having a net positive influence rather than neutral or negative.
Beyond pragmatism, from an idealist perspective aiding in increasing organizational diversity is the morally right thing to do. That doesn't mean hiring underqualified people; it means refusing to fill the position until the right person is found, which is a whole other problem on its own.
Here are some resources to get started: https://www.gsb.stanford.edu/insights/diversity-work-group-p... https://journals.aom.org/doi/abs/10.5465/1556374
I would need to undergo DNA testing for each person to avoid hiring anyone that is too much of one race. Do I get better returns if I hire someone that has 25% of 4 different races?
Please elaborate on your very empirically verifiable and obviously logically cohesive argument. If you don't understand where my skepticism originates from, please turn your attention to the results of Black Economic Empowerment in South Africa (where racial quotas are enforced to ensure the blackest candidate is hired, not the best candidate).
Look at the performance of the South African sports teams that have been forced to recruit players not based on merit, but on the colour of their skin.
It being the right thing to do is a much stronger argument imo. Which I agree with, but companies generally aren't interested in it, unless they can use it for marketing.
Their arrogance is dangerous indeed.
That's what seems so confusing about HN replies here. (Non-friendly) AGI is an extreme existential risk (depending on who you listen to).
I'm perfectly fine with rewarding the org that's responsible for researching friendly AGI to do it _right_ (extremely contingent on that last bit).
You know, I can't prove that researchers being funded is the best way of figuring out how to do things, but I have a gut intuition that tells me that.
I'll look into it so that I'm not just blindly suggesting that $$ ==> people ==> research ==> progress.
Thanks for the opportunity to reflect!
The precedent for general intelligence is not good. The only example we know of (ourselves) is a known existential threat.
The wording in the press release reminds me of a question I haven't been able to answer for a while now. Can anyone point me to the moment in time when general purpose artificial intelligence was re-branded to artificial general intelligence? Is GAI that much worse of an acronym than AGI? What's the deal here?
I'm still waiting for the 1.5G GPT-2 set to get released, but they're still going with that "too dangerous for society" BS that they're using to get journalists' attention...
I understand that this tech could be used for nefarious purposes, but this isn't world-ending tech. This is just hard to differentiate from human writing...
The choice to keep their "too powerful model" unreleased is more an attempt to stoke sensationalism out of journalists eager to report on "The AI too dangerous to release" than it is actually an earnest attempt at protecting society.
The dangerous rogue-AI is a Hollywood trope. We don't live in the Ghost In the Shell universe, we live in reality, and a text-generating algorithm isn't particularly dangerous when you think about it.
When, in any OpenAI communication, have they ever implied GPT2 is world ending tech?
> We don't live in the Ghost In the Shell universe, we live in reality, and a text-generating algorithm isn't particularly dangerous when you think about it.
Are you sure it won't be used to automatically post fake news or create artificial group cohesion by bad state actors? We can already do stuff like that, but this allows you to do it faster and cheaper. Don't you think that's at least a little scary?
But there is zero utility in keeping the larger dataset secret. Society is not safer as a result.
The only reason they ever framed it the way they did was to get media attention.
"This AI is so dangerous, the creators are refusing to release it to the public!" is going to get way more clicks than "This AI writes English sentences that almost appear coherent.".
If you want more funding and attention towards your project, you're going to say stuff that gets journalists' attention.
> “$1,000,000,000,” it read.
Wow, Sam Altman sounds like an asshole.
And of course going after high school student Mike Rowe for registering MikeRoweSoft.com (Seriously Microsoft, exactly no one thought he was you).
While Microsoft isn't inherently evil, I mean one could argue that via Windows Microsoft is largely responsible for the widespread adoption of computers, it definitely makes me feel slightly uneasy.
I'd rather see OpenAI continue to be funded by donation, and eventually royalties/licensing of technologies it develops, not partnerships with companies like 'IBM 2: Electric Boogaloo'.
But what do I know, I'm a Morlock not an Eloi.
- Lindows
- Microwindows
- wxWindows
- Windows Commander
- Suing Amish Shah
- MikeRoweSoft
Then all of the various anti-trust/anti-competition lawsuits against them by both companies and government entities (Be Inc, Nescape, the EU, Spain, the US Government, Caldera, Opera Software. Also individuals in numerous class actions).
Plenty of reason to feel slightly concerned.
I believe that building a beneficial warp-drive engine will be the most important technological development in human history, with the potential to shape the trajectory of humanity. The aliens we're sure to encounter will be capable of mastering more fields than any one human — like a tool which combines the skills of Curie, Turing, and Bach. An alien working on a problem would be able to see connections across disciplines that no human could. But even though I'm known as the warp-drive-guy, I don't actually know how to build a warp drive, so in the meantime I am building increasingly powerful transportation technology in the hope this would lead to a warp drive one day soon, and have decided to focus on bicycles. They're really good bikes, though, and unlike others who make bicycles, I like to consider those I build to merely be pre-warp, a necessary step towards warp technology. So when you buy my bikes [1], you are literally helping me change the trajectory of human history and meet aliens (did I mention Curie, Turing and Bach?)
This is truly a fine specimen of Silicon Valley prose. It's got something for everyone: human history, a wild-eyed dream of a bright future, a connection to the arts, name-dropping, the trajectory of humanity, and, of course, lots of money in cloud services (integrated platform). They even showed some restraint in stopping short of ending all war and curing all disease. "Making the world a better place" is really too mundane.
[1]: The Warp Drive Corporation®'s Pre-Warp Bike™️ is now on sale on Amazon.
It's not AI imo. It's a trained model that will never be able to evolve outside those tight boundaries.
And that is because, which is really simple, we don't really have any clue of how the brain works, technically. And besides that, the impact which environments has on the brain.
Why no one is talking about this being a huge driver for Microsoft Azure and what Microsoft as a company will gain from this, baffles me a little. Not sure i like it.
To me, the fact that Turing wrote the first chess program in the 1950s suggests that people thought computers were capable of playing chess. It doesn't seem too great a leap to imagine that they'd be open to the idea that a supercomputer could play superhuman chess.
That's how long we've come. Playing chess. On a superhuman level. Wow. What is superhuman anyways?
The same goes for writing text articles. Go on any /r/politics thread you'll surely see bots writing text that does a good job of convincing people that it was human written. So slightly, or even significantly, improving those is not something that people doubted would be possible.
Many, many human-level benchmarks on narrow AI tasks have been passed in recent years. https://www.eff.org/ai/metrics
So, the “well that’s not intelligence, it’s just computation” arguments can continue until it reaches “well that system isn’t conscious, so it’s not AI”. That argument holds no water, since intelligence != consciousness.
It comes down to whether you believe AGI is achievable.
We've talked about why we think it might be: https://medium.com/syncedreview/openai-founder-short-term-ag..., https://www.youtube.com/watch?v=YHCSNsLKHfM
And we certainly have more of a plan for building it than warp drives :).
EDIT: I personally think the case for near-term AGI is strong enough that it'd be hard for me work on any other problem — and find it important to put in place guardrails like https://openai.com/blog/openai-lp/ and https://openai.com/charter/.
Even if AGI turns out to be out of reach, we'll still be creating increasingly powerful AI technologies — which I think pretty clearly have the potential to alter society and require special care and thought.
This is an interesting quote. Anyone know if folks like Darwin, Marconi, etc said similar things, about the problems they were working on at the time?
No, it does not. I very much believe AI (or AGI, as you call it) is achievable, but may I remind you that some years after the invention of neural networks, Norbert Wiener, one of the greatest minds of his generations, said that the secret of intelligence would be unlocked within five years, and Alan Turing -- a component of your very own post-pre-AGI era's AGI -- another great believer in AI, scoffed and said that it will take at least five decades. That was seven decades ago, and we are not even close to achieving insect-level intelligence. Maybe we'll achieve AI in ten years and maybe in one hundred, but you don't know which of those is more likely, and you certainly don't know whether any of our pre-AGI technology even gets us on the right path to achieving AGI. There have been other paths towards AI explored in the past that have largely been abandoned.
OpenAI is not actually building AGI. Maybe it hopes that the things it is working on could be the path to an eventual AGI. OpenAI knows this, as does Microsoft.
This does not mean that what OpenAI does is not valuable and possibly useful, but it does make calling it "pre-AGI" pretentious to the level of delusion. Now I know there were (maybe still are) some AI cults around SV (I think a famous one even called themselves "The Rationalists" or something), but what makes for a nerdy, fanciful discussion in some dark but quirky corner of the internet looks jarring in a press release.
> If you believe AGI might be achievable any time soon, it becomes hard to work on any other problem — and it's also very important to put in place guardrails like https://openai.com/blog/openai-lp/ and https://openai.com/charter/
I can't tell if you're serious, but assuming you are, the problem is that there are many other things that if you think chould be achievable any time soon would make it hard to work on any other problem, as well make it important to put guardrails in place. The difference is that no one actually knows how to put guardrails on AGI. We are doing a pretty bad job putting guardrails on the statistical clustering algorithms that some call (pre-AGI?) AI and that we already use.
Is this true? Is there an insect turing test?
As with AI... a system can be simulated to a given level of fidelity without necessarily simulating the entire original underlying system.
Perhaps more relevantly, an automaton that could observe such a waggle dance using computer vision and then navigate to the food source described by the waggle seems to me to strain the bounds of our current capabilities, or maybe even to surpass them by an order of magnitude.
Aren't we close to this? Most insects only have a few million neurons in their central nervous system, so we can model their intelligence in real time at least. Maybe we still lack the tools for training such networks into useful configurations?
OP probably just wanted to downplay the current state of AI.
Now here's a more advanced example to teach a virutal character how to flex in the gym: https://www.youtube.com/watch?v=kie4wjB1MCw
That's a bit more advanced than simple walking.
Here's a deployed AI to a real robot "crab":
https://www.youtube.com/watch?v=UMSNBLAfC7o
How about virtual characters learning to cooperate?
In one of your examples, which are all of narrow AI, we see a mechanical crab powered by ML that has become specialized in walking with a broken limb, which is not even close to what we need if we aim for AGI. For AGI we don't need agents that mimic simple behavior. In my opinion, _mimicking_ behavior will not lead to AGI.
What _will_ lead to AGI? No one knows.
Nowhere I have stated this is the clear path to AGI and you are right, we are missing key building blocks. But I feel like there's too much skepticism agains this field while the advancements are not appreciated enough.
I don't know either what will lead there, but I see more and more examples of different networks being combined to achieve more than they are capable of individually.
No, the complaint was about modelling the behavior of simple organisms.
Certainly we can model some of their behaviors, many of which are highly stereotyped. But the real fly (say) doesn't only walk/fly/scratch/etc, it also decides when to do all of these things. It has ways to decide what search pattern to fly given confusing scents of food nearby. It has ways to judge the fitness of a potential mate, and ways to try to fool potential mates. Our simulations of these things are, I think, really terrible.
Since everything here is loosely defined I feel it's totally pointless to discuss AI, but it's still an intriguing topic. If you look at those insects, they tend to follow brownian motion in 3D, get food and get confused by light, we can get an accurate model of them and more [0].
The key word here is to model, not replication. Simulations are just that, simulations. Given current examples of what's already possible if someone wanted to, could model a detailed 3D environment with physics, scents and food for our little AI fly.
[0] https://www.techradar.com/news/ai-fly-by-artificial-intellig...
Is that a terrible attempt?
I'm sorry and apologize if you feel I was one to kill the discussion you wanted to have around AI.
I'm one of those dreamers who think AGI is or at least should be possible, soon, through means we have not yet discovered but will, soon. I base that on absolutely nothing, I suppose, other than the fact we have lots of "bright/smart/crazy" devs working on it. It's my own personal "believie", as Louis CK would say about things we believe in but cannot or care not prove.
Just like you I'm looking at organisms much simpler than us as a way forward. Many specialized neural network does not make up AGI, is what I think. Is it the organic and human neuron we should model? I don't necessarily think so. Also. robotics + ML is a dead end to me. An amoeba that can evolve into something more complex, is perhaps what we should model.
> they tend to follow brownian motion in 3D
Well, their entire neural system exists to make deviations from Brownian motion. That's the whole point of being an animal not a plant. And doing it well is very very subtle.
First steps towards modelling such behavior can be super-interesting science, not a terrible use of time at all. They can capture a lot of truth about how it works. But like self-driving cars, the thing that kills you is usually a weird edge case, not the basic thing.
I think people think back propagation is the metaphorical lift equation here and we just need a “manufacturing” advancement (ie, more compute and techniques for using it). We’re close to that (I personally feel like with poor evidence) but definitely not there yet (as evidenced by nobody publishing this). We cannot describe what is happening with modern architectures as fully as a lift equation predicts fixed wing flight, and so it is largely an intuition + trial and error, which is a slow unreliable way to make progress.
People think maybe the missing pieces might be in the other things we don’t understand about the brain. It makes sense- it does what we want, so the answer must be in there somehow. I agree we don’t need to perfectly understand it, it just seems like a good place to keep looking for those missing pieces.
Secondarily, just because neurons are complex on technical level, it does not mean that they should be complex on logical level.
For example, in computers if you would look at the CPU structure, on a low level you have quantum effects and tunneling and very insane stuff but on a logical level you are dealing with very trivial boolean logic concepts.
I would be not be surprised in a slightest if copying and reverse engineering neurons per se would not be necessary and defining aspect of anything related to AGI.
Neurons communicate with each other with a multitude of neurotransmitters and receptors [1]. As a cell, each neuron is a complex organism of its own that undergoes transcriptomic and metabolic changes. We aren't even close to simulating all protein interactions in a single cell yet, let alone in millions of them.
Of course you could say that full protein simulation of an entire brain is not neccessary if we can build an accurate enough technical model of a single neuron. In fact, already now we have to apply a model of how we believe proteins behave as "properly" simulating interactions of two proteins (or one with itself) with lattice QCD approaches is beyond our computational capabilities. For protein interaction we have pretty good models already. But finding a model of all types of neurons in insect brains is right now an open, unsolved challenge.
[1] https://en.wikipedia.org/wiki/Neurotransmitter#List_of_neuro...
AFAICT this suggests that we have the computational power but wouldn't it also be a significant challenge to create an accurate model for the brain simulation?
https://pdfs.semanticscholar.org/38e6/1d9a65aa483ad0fb4a219f...
Shannon, Minsky, and McCarthy!
I think the interesting take away is that they (seem to have) expected to solve the major problems of AI (language, common sense etc) over a summer with a small stipend.
the thing is, there are actually lots of reasons to think AGI cannot be constrained in this way. open AI researchers know this.
so that means, the promise and the charter are irrelevant. open ai will never release a general AI.
but in the meantime, deep learning is still reaping. every day it's being applied to something new and solving real, tangible problems. there's money to be made here, and that is what open AI seems to really be doing. being philosophical and "on top" of the futuristic moral dilemmas, whatever, is just marketing? and in the unlikely event that an AGI is created that can be tamed, great for open ai! if an AGI is created that cannot be tamed, what then? if it's really worth a trillion dollars, is it really just buried, or will the charter simply be rewritten?
you know, this reminds me a lot of all the great physicists working on the atom bomb, thinking it was never going to be used.
> Wiener regarded Alan as a cybernetician, and indeed ‘cybernetics’ came close to giving a name to the range of concerns that had long gripped him, which the war had given him an opportunity to develop, and which did not fit into any existing academic category. In spring 1947... Wiener had been able to ‘talk over the fundamental ideas of cybernetics with Mr Turing,’ as he explained in the introduction to his book... Wiener had an empire-building tendency which rendered almost every department of human endeavour into a branch of cybernetics... Wiener delivered with awesome solemnity some pretty transient suggestions, to the general effect that solutions to fundamental problems in psychology lay just around the corner, rather than putting them at least fifty years in the future. Thus in Cybernetics it was seriously suggested that McCulloch and Pitts had solved the problem of how the brain performed visual pattern recognition. The cybernetic movement was rather liable to such over-optimistic stabs in the dark.
So if this passage is indeed the source of my recollection, while very poor and perhaps exaggerated, I think it's pretty true to the spirit...
In your opinion what should computer scientists be focusing on in order to achieve more advanced AI systems? I'm thinking things such as reasoning, causality, embodied cognition, goal creation, etc.
And this is without even delving into the ethics aspects of (some instances of) AI research.
What in the article gave you the impression they had made a large breakthrough or were close to an AGI?
This sentence might by itself imply they are farther along than they are, but in the context of the whole article I never got the impression they were close to actually building an AGI.
> The most obvious way to cover costs is to build a product, but that would mean changing our focus. Instead, we intend to license some of our pre-AGI technologies, with Microsoft becoming our preferred partner for commercializing them.
This read pretty straightforwardly to me. Pre-AGI seems like a shorthand for useful technologies like GPT-2.
Reading the article I never got the impression they'd solved AGI, or were even close. The context of the article is a partnership announcement not a breakthrough. I could see how a few people who are very unsophisticated might get a little confused as to how far along they are. But I assumed they were writing for people who had heard of OpenAI which pretty much eliminates anyone this unsophisticated.
People may also not be confused if Ben and Jerry's start an ice cream ad with mentions of AGI and the change of human trajectory and Marie Curie, and name it Pre-AGI Rum Raisin, but that doesn't mean the text isn't a beautiful and amusing example of contemporary Silicon Valley self-importance and delusion, and reads like a parody that makes the characters in HBO's Silicon Valley sound grounded and humble. Especially the "pre-AGI" bit, which I'm now stealing and will be using at every opportunity. Maybe it's just me, but I think it is quite hilarious when a company whose actual connection with AGI is that, like many others, they dream about it and wish they could one day invent it, call their work "pre-AGI." Ironic, considering they're writing this pre-apocalypse.
But so has every other big idea that went on to become reality, like planes (da Vinci was drawing designs for planes over 400 years before the first working ones).
> no one knows whether what they're doing is even on the right path towards AI
This is completely wrong. That would be like saying "no one knows if working on a wing is on the right path to flight".
Look at the way deep learning works. Look at the way the brain works. They share immense similarities. Some people say "neural nets" aren't like the brain, but that's not true--they are just trying to not over-exaggerate the differences which laymen commonly do. They are very similar.
And so has every other big idea that didn't become reality, and that was the majority. Again, I have no problem with AI research whatsoever, but the prose was still eyebrow-raising considering the actual state of affairs.
> They are very similar.
They are not. The main role of NNs is learning, which they still mostly do pretty much with backpropagation gradient-descent (+ heuristics). The brain does not learn with backpropagation.
This is a really good point. Would be a fascinating read if someone were to collect all those examples and explore that a bit.
But in itself, that was not the point. It would also transform the owner or user -- it was a hermetic symbol, a mechanical means to "pierce the veil" and to see the deep mystical and magical truths, the Real Reality. It was immanent, a thing in the world, that enabled the transcendent, to go beyond, above, outside of the world. Its discovery would have been the single most important moment in the history of the world, the moment in which humans had a reliable road to divinity.
Hmm. Sounds familiar, doesn't it?
But out of alchemy came modern chemistry, and also some parts of the scientific method. After all, as some smart people worked out, you could systematically try all the permutations of materials that your reading had suggested as possibilities. That meant measuring, weighing, mixing properly, keeping detailed notes. Fundamental lab work is the unglamorous slab of concrete beneath the shining houses of the physical sciences. There were waves of hysteria and hype, but after each, something useful would be left behind, minus the sheen of unlimited dreams.
Hmm. Sounds familiar, doesn't it?
These days it is possible for a device to transmute base metals into gold. But the operators have not, so far as I can deduce, ascended to any higher planes of existence. They have eschewed the ethereal and remained reliably corporeal.
I'm not sure the reference you are making. Red pill/blue pill? I wasn't aware of the symbolism in the Philosopher's Stone.
> But the operators have not, so far as I can deduce, ascended to any higher planes of existence.
I guess I'm unfamiliar with this non-literal aspect to the philosopher's stone.
I'm missing the allusions you are making. To ascend to higher planes of existence, no need for AGI, some acid will do.
> Towards Biologically Plausible Deep Learning
> Neuroscientists have long criticised deep learning algorithms as incompatible with current knowledge of neurobiology. We explore more biologically plausible versions of deep representation learning, focusing here mostly on unsupervised learning but developing a learning mechanism that could account for supervised, unsupervised and reinforcement learning. The starting point is that the basic learning rule believed to govern synaptic weight updates (Spike-Timing-Dependent Plasticity) arises out of a simple update rule that makes a lot of sense from a machine learning point of view and can be interpreted as gradient descent on some objective function so long as the neuronal dynamics push firing rates towards better values of the objective function (be it supervised, unsupervised, or reward-driven). The second main idea is that this corresponds to a form of the variational EM algorithm, i.e., with approximate rather than exact posteriors, implemented by neural dynamics. Another contribution of this paper is that the gradients required for updating the hidden states in the above variational interpretation can be estimated using an approximation that only requires propagating activations forward and backward, with pairs of layers learning to form a denoising auto-encoder. Finally, we extend the theory about the probabilistic interpretation of auto-encoders to justify improved sampling schemes based on the generative interpretation of denoising auto-encoders, and we validate all these ideas on generative learning tasks.
FWIW I thought you were clear, but there are only so many middlebrow dismissals one can make towards AI or AGI efforts and I think I've seen them all plus the low-value threads they generate. (I've made some too, and suspect we might get brain emulations before AGI, but I try to avoid the impulse and in any case it doesn't stop me from hoping (and minor contributing) for the research on the article's load-bearing word "beneficial" to precede any realistic efforts of building the actual thing. At least the OpenAI guys aren't entirely ignorant of the importance of the "beneficial" problem.)
What I'm interested in is how many dismissals of AI, most end up justified, can the field take before considering toning down the prose a bit, especially considering that the dismissals are a result of setting unrealistic expectations in the first place.
Also while I fully understand and appreciate the necessity of OpenAI abandoning the 'open' part, it says a lot about who is going to benefit from this technology when you have investors who want to make money. It's just ironically poetic in this instance.
Their employer has just received a 1 billion dollar cash investment to futz around with computers. I don't think the employment choice is some sort of personal sacrifice here.
The level of skin here is a cushy guaranteed job for years to come until the next AI-winter hits, likely set into motion by such very claims of "AGI" being near or feasible.
"AGI" is good marketing for getting money to research real AI, ostensibly on the path to "AGI", but if one drives it too far, one might up retarding the whole field as the hype winds down (again).
Yes they are? They are making breakthroughs/incremental advances that required to get there, and building components along the way. It would be like you saying well Henry Ford isn't building a vehicle, he's just building a wheel, and a tire, and an engine, etc...
They don't know that. We have no idea what's required to achieve AI. Now I don't know how long before Ford actually built cars he started saying he's building cars, but if Wikipedia is to be believed, it could not have been more than three-four years. Also, when he started building cars, he pretty much knew what's required to build them. This is not the case for AI.
Yes, we do. Lots of data, lots of training, better algorithms, more understanding of the brain...At this point we still need 10x+ improvements in a lot of areas, but it's pretty clear what we need to do.
If you can process around 100 petabytes per second (1 Google Index of data per second), you could fully simulate a human being, including their brain. We're still a little bit from that, but it's pretty clear we'll get there (barring usual disclaimers about an asteroid, alien invasion, etc).
Source: I work in medical research, doing deep learning, and do research on programming languages and deep learning for program synthesis.
So to build AI all that remains is to understand how it could work.
> but it's pretty clear what we need to do
It isn't (unless by "clear" you mean as clear as in your statement above). I've been following some of the more theoretical papers in the field, and we're barely even at the theory forming stage.
> but it's pretty clear we'll get there.
First of all, I don't doubt we'll get there eventually. Second, I'm not sure simulating a human entirely falls under the category of "artificial". After all, to be useful such a mechanism would need to outperform humans in some way, and we don't even know whether that's possible even in principle using the same mechanism as the brain's.
I read those papers too. And I write code and train models day in and day out. I could get very specific on what needs to be done, but that's what we do at our job. If you're curious, I'd say join the field.
I agree with you in that I don't think for a second anyone can make an accurate prediction of when we will AGI, but I have no doubt that it will be relatively soon, and that OpenAI will likely be one of the leaders, if not the leaders in creating it.
2) Our focus is on helping improve medical and clinical science and cancer tooling first.
3) If we needed AGI to cure cancer, perhaps we'd be working directly on AGI. If anyone thinks this is the case, please let me know, as at the moment I don't think it is.
Exponential process continuing doesn't imply "we're going to get there soon" in any way, shape or form. The desired goal can still be arbitrarily far.
This is absurd. How much data? How much training? What kind of training? How much better do the algorithms need to be? How do you define better? Also we literally don't even know how our brains work, so we don't know how "actual" intelligence works, but you're saying we have a clear road map for simulating it?
Your entire argument distill down to "we just need to do the same things, but better." And even that statement might be wrong! What if standard silicon is fundamentally unsuited for AGI, and we need to overhaul our computing platforms to use more analog electronics like memristors? What if everything we think we know about AI algorithms ends up being a dead end and we've already achieved the asymptote?
I'm not saying AI research is bad. I'm saying it is absolutely unknown by ANYONE what it will take to achieve AI. That's why it's pure research instead of engineering.
Humans don't seem to need anywhere near the same level of data or training that our current models need. That alone is a sign that deep learning may not be enough. The focus on deep learning research has a lot of useful benefits, so I'm not discounting that, but there are a decent amount of smart people who don't believe it's going to lead us to AGI.
Source: I also work in medical research, and am doing deep learning- and I've worked for a company that's focused on AGI, and I've worked with several of the OpenAI researchers.
I find this to be a common misunderstanding. If I show you one Stirch Wrench, and you've never seen one before, you learn instantly and perhaps for the rest of your life you'll know what a Strich Wrench is. The problem is I didn't show you 1 example. You saw perhaps millions of examples (your conscious process filters those out, but in reality think of the slight shaking of your head, the constant pulsing of the light sources around you, etc, to be augmenting that 1 image with many examples). I think humans are indeed training on millions of examples, it's just we are not noticing that.
> That alone is a sign that deep learning may not be enough.
I 100% agree with that. It's going to take improvements in lots of areas, many unexpected, but I think the deep learning approach is the "wings" that will be near the core.
Here's a great article about a paper showing that humans prior knowledge does help with learning new tasks- https://www.technologyreview.com/s/610434/why-humans-learn-f...
However, that doesn't account for how quickly toddlers learn a variety of things with a small amount of information. Even more important, you can also just look at things like AlphaGo- they train on more examples than could be accumulated in a hundred human lifetimes.
For these reasons I don't believe "more data" and "more training" is the answer. We're going to need to do a lot more work figuring out how humans manage recall, how we link together all the data, and I would be surprised if this didn't involve finding out that our brain processes things in ways that are far different than our current deep neural nets. I don't believe incrementalism is going to get us to AGI.
Lots of advances in many fields will lead to something greater than the sum of their parts.
Edit: p.s. I like your comment about toddlers. As a first time father of a 6 month old, its been very intellectually interesting watching her learn, in addition to just being the greatest bundle of joy ever :)
That sounds fascinating. Could you link to some relevant stuff about languages and deep learning for program synthesis? I'd love to read more about this.
Also when he started there were working automobiles already.
The fact that no one knows how to make an AGI, doesn’t make it a bad goal. But OP is right, if you think you know the timeframe it will arrive in, you have no idea what kind of problem you’re dealing with.
AI and AGI may have meant the same thing a long time ago, but the term "AI" has been used almost ubiquitously to represent things that are not AGI for so long now, that I don't think the terms are interchangeable any longer.
I don't think you know what you are talking about. Do you do Deep Learning? If you are not actively engaged in the field, I wouldn't be so quick to dismiss others who are (especially not others who are at the top of the field).
That being said, you brought up some interesting points, even if I think your overall position is wrong--I think OpenAI is definitely going to hit "pre-AGI" if not AGI, and I do this stuff all day long.
So was Geoffrey Everest Hinton
> I’ve mostly lost interest since then
but he didn't give up.
If you expect someone to just hand us AGI in a nicely wrapped package with a bow, with all the details neatly described, you are absolutely right, that's really far off!
But for the record there are many people actively grinding it out in the field, day in and day out, who don't give up when things get hard.
I would definitely not call OpenAI delusional. I would say all OpenAI is being here is "honest".
They are simply stating what the math tells them.
"E pur si muove"
Which math?
>I don't think you know what you are talking about. Do you do Deep Learning? If you are not actively engaged in the field, I wouldn't be so quick to dismiss others who are (especially not others who are at the top of the field).
I do (or at least I try, I get money for my attempts though) and I concur with calling it delusion. So does Francois Cholet. So does Hinton to some degree, so does the founder of deepmind (or at least they did in 2017: https://venturebeat.com/2018/12/17/geoffrey-hinton-and-demis... ).
I want to like OpenAI, I think they did the right thing with GPT-2 and I give them a lot of credit for publishing things. That being said, I remain skeptical about AGI, highly skeptical about AGI being feasible, or the thing to worry about. I always make the argument that either research towards controlling an AGI/AGI alignment is a techified version of reserach into the problem of good global governance (in which case it is an interesting problem that desperately needs solving), or it is useless (because no matter how nicely you control the AGI, a non-accountable elite within the current system, the less-than-perfectly aligned government etc. will strongarm you into giving control to THEM before you come close to deploying it) or it is delusional (because you think you are smart enough to build AGI without these elites finding out AND smart and/or wise enough to do what is best for humanity).
and
> because you think you are smart enough to build AGI without these elites finding out AND smart and/or wise enough to do what is best for humanity
Are very good points and I share those concerns too, and have no good answers. I'm in the pessimist camp when it comes to AGI--I would be heavily it's going to happen but I wouldn't bet a dollar whether it will end up being good for humanity, as I haven't a clue.
Then again I don't know many insects that drive cars, beat the champions at chess and go or similar.
Yes, historically people were way too optimistic and generally went down AI rabbit holes that went nowhere, but two years before the Wright flyer flew, the Wright brothers themselves said it was 50 years out (and others were still publishing articles about human flight being impossible after it was already flying).
People are bad at predictions, in the Wright brothers case since they were the people that ultimately ended up doing it two years later they were likely the best to make the prediction and were still off.
Given that AGI is possible and given the extreme nature of the consequences, doesn't it make sense to work on alignment and safety? Why would it make sense to wait? If you accidentally end up with AGI and haven't figured out how to align its goals then that's it, the game is probably over.
Maybe OpenAI is on the right path, maybe not - but I think you're way too confident to be as sure as you are that they are not.
same goes for warp drives doesn't it?
The point is that people don't see how we build THAT out of these tools we currently have. We only build pastiches of intelligence today and either we have an arrogant view of the level of our own intelligence or we can't make THAT with THIS.
But maybe warp-drives, maybe world-peace too?
AGI is possible because intelligence is possible (and common on earth) in nature.
Ignoring that if we saw miniature warp drives everywhere around us in nature then yes I would be more confident they were possible.
Similarly to how human airplanes don't flap their wings like birds, there will probably be implementation differences that make sense but share the underlying principles. Particularly since the artificial version isn't constrained by things biology needs to handle.
But you haven't asked if we're capable of building it. While it might be technically possible are we capable of managing its construction?
All I see today is ways of making the process more opaque at the benefit of not having to provide the implementation. How does that technique even start to scale in terms of its construction? I worry about the exponentially increasing length of the "80% done" stage, and that's on the happy path.
I would posit that while its possible, it will take so long on this tech stack that we'll find another in the interim that will produce better results. I'm not convinced this branch is the winner.
No, not in the slightest. I mean as we progress the dev cycles get harder and slower. Then we need more engineers and the administration of more engineers working together makes everything harder.
Have you ever considered that making a rock think might be one of the greatest engineering projects our species has ever taken on? Sure humans might figure it out but I'm of the belief it will take them a very long time to. In addition, I believe that in that timescale a different tech stack might show more promise. I'm not convinced this technological branch scales all the way to AGI.
> Well we've removed its irrational hatred of penguins but now it struggles with the concept of Wednesday again...
> Why would it make sense to wait?
Again, that's a separate discussion, but if we don't know what something is or when it could arrive, it may make more sense to think about things we know more about and are already here or known to be imminent. Anyway, anyone is free to work on what they like, but OpenAI does not know that they're "building artificial general intelligence."
> I think you're way too overconfident to be as sure as you are that they're not.
I don't know that they're not, but they don't know that they are, and that means they're not "building AGI."
The problem, however, I think another gp's objection. OpenAI isn't really working on AGI, it's making incremental improvements on tech that's still fragile and specialized (maybe even more specialized and fragile), where the only advance of neural nets is that now they can be brute-force programmed.
That's a very big if... Also, I'd argue that most progress happens not because of some brilliant people, but because of many people working together... Then if your AGI only reaches the level of intelligence of humans and maybe a bit more (what does more even mean in terms of human intelligence? more emphatic? faster calculation ability? more memory? what would the use of this be? all things we can't really oversee), it raises the question whether this would ever be possible in a cost-efficient way (human intelligence seems like it is, in a certain way, "cheap").
Oh, this is indeed a big if. A large, looming aspect of the problem is we don't anything like an exact characterization of "general intelligence" so what we're aiming for is very uncertain. But that uncertainty cuts multiple ways. Perhaps it would take 100K human-years to construct "it" and perhaps just a few key insights could construct "it".
> Also, I'd argue that most progress happens not because of some brilliant people, but because of many people working together...
The nature of a problem generally determines the sort of human-organization one needs to solve a problem. Large engineering problems are often solved by large teams, challenging math problems are generally solved by individuals, working with published results of other individuals. Given we're not certain of the nature of this problem, it's hard to be absolute here. Still, one could be after a few insights. If it's a huge engineering problem, you may have the problem "building an AGI is AGI-complete".
> Then if your AGI only reaches the level of intelligence of humans and maybe a bit more (what does more even mean in terms of human intelligence? more emphatic? faster calculation ability?
I've heard these "we'll get to human-level but it won't be that impressive" kinds of arguments and I find them underwhelming.
"What use would more memory be to an AGI that's 'just' at human level?"
How's this? Studying a hard problem? Fork your brain 100 times, with small variations and different viewpoints, to look at different possibilities, then combine the best solutions. Seems powerful to me. But that's just the most simplistic approach and it seems like an AGI with extra-memory could jump between the unity of an individual and the multiple views of work groups and such is multiple creative ways. The plus humans have a few quantifiable limits - human attention has been very roughly defined as being limited to "seven plus or minus two chunks". Something human-like but able to consider a few more chunks could possibly accomplish incredible things.
I don't know about the god thing you mention and the rationalist stuff I've read hasn't been about that. The main argument as I understand it is:
1. AGI is possible
2. Given AGI is possible if it's created without the ability to align its goals to human goals we will lose control of it.
3. If we lose control of it, it will have unknown outcomes which are more likely to be bad than benign or good.
Therefore we should try and figure out a way to make it safe before AGI exists.
Maybe humans just happen to be an intelligence upper bound and anything operating at a higher level goes crazy? That seems unlikely to me given that humans have a lot of biological constraints (heads have to fit out of birth canals, have to be able to run on energy from food, selective pressure for other things besides just intelligence). You could be right, but I'd bet on the other side.
The last bit is if we can solve this in a way that aligns the goals with human goals (open question since humans themselves are not really aligned) we could solve most problems we need to solve.
Makes no sense to me, how would you ever be able to figure out a way to make something safe before it even exists?
Someone who has never built a nuclear reactor most likely could not think of a way to prevent the Chernobyl disaster.
(OK, maybe this is a wrong example as someone who did couldn't do this either, but the point should be clear)
https://intelligence.org/2016/12/28/ai-alignment-why-its-har...
My main issue personally is that I'm not confident if it's really far out or not and people seem bad at predicting this on both sides. Given that, it probably makes sense to start the work now since goal alignment is a hard problem and it's unknown when it'll become relevant.
I read the BuzzFeed article and I think the main issue with it is he assumes that an AGI will be goal aligned by the nature of being an AGI:
"In psychology, the term “insight” is used to describe a recognition of one’s own condition, such as when a person with mental illness is aware of their illness. More broadly, it describes the ability to recognize patterns in one’s own behavior. It’s an example of metacognition, or thinking about one’s own thinking, and it’s something most humans are capable of but animals are not. And I believe the best test of whether an AI is really engaging in human-level cognition would be for it to demonstrate insight of this kind."
Humans have general preferences and goals built in that have been selected for for thousands of years. An AGI won't have those by default. I think people often think that something intelligent will be like human intelligence, but the entire point of the strawberry example is that an intelligence with different goals that's very good at general problem solving will not have 'insight' that tells it what humans think is good (that's the reason for trying to solve the goal alignment problem - you don't get this for free).
He kind of argues for the importance of AGI goal alignment which he calls 'insight', but doesn't realize he's doing so?
The comparison to Silicon Valley being blinded by the economies of their own behavior is just weak politics that's missing the point.
There are at least 7 billion beings on the planet with AGI already. I think a bigger problem is the general well being of the aforementioned 7 billion entities.
If you have an AGI that has goals that are not aligned with your interests it'll dwarf everything else because it thinks faster than you (and can therefore act faster than you) in pursuit of its goals.
And another thing to consider, is that in the real world success is not easy to define and it is only loosely correlated with intelligence. We have 7 billion people, each attempting random little variations on 'succeeding at life'. And the 'winners' generally require that some of the 7 billion people agree to 'reward' them (i.e. by giving them money). My last 3 purchases were watermelon seeds for my garden, a pair of jeans, and a dinner at a Vietnamese restaurant. It's not clear to me how AI would take over any of those transactions. Maybe make the jean manufacturing more efficient, but the price I paid was already pretty low.
For the real world success part that's where goal alignment comes in. If we're going to solve things like dealing with the sun burning out, becoming an interplanetary species, or death then having an AGI that can work on these problems with us (or as part of us if Neuralink can succeed on what they want to do) will be a big deal.
It sounds crazy, but I think success here is a lot bigger than automating what clothes you were going to buy. Incentive based systems like capitalism work pretty well, but not being able to coordinate effectively at scale is a major source of current human problems, theoretically a goal aligned AGI could do that, or at least help us do it.
It's the dream of being one of the few who gets to control and direct that concentrated power that fuels these dreams, which is why it's imperative that they dress it up in the language of benefiting society.
The essence of the ethical problem with AI is that there is no person or small group of people who can be trusted to use such power without creating a real dystopia for the rest of us.
Nobody is going to control the hyper-intelligent AGI if its goals are not aligned with human goals more generally. That's the nature of something being a lot smarter than you with its own goals.
Are the people who are trying to make AI a reality planning to give away the ability to unplug the machine it will inevitably depend upon, or give it the ability to control nuclear weapons so that it can wipe out humanity before that happens, like a bad movie script? It really does seem that ridiculous to aver that the creators of AI, if it ever comes to be, won't retain ultimate control over what it can do in the physical world.
Whatever its goals end up being, they will be aligned with Microsoft's goals if OpenAI gets there first. That's what the billion dollars is meant to ensure.
The difference between the world’s smartest human and its dumbest human is tiny relative to the possible spectrum of intelligence.
Do you see the smartest chimps tricking or controlling humans?
If you wanted to trick a chimp to do something you wanted it to do, do you think it could stop you? And we’re probably a lot closer to chimp intelligence than an AGI would be to us.
I don't see how that conclusion follows the antecedent.
In a certain way, brains are so complicated that (at least for the moment) they seem quite magical to us
As far as humans not being able to ever understand it, I guess that could be true but I wouldn't bet on it.
It almost certainly is. Humans make new intelligences all the time.
> and achieving it will have consequences that dwarf everything else
It probably won't, humans make new intelligences all the time. Hanging the technology base for that doesn't have any significant necessary consequences.
A revolution in our ability to understand and control other intelligences might have consequences that dwarf anything else, with or without AGI, but that's a different issue, and moreover one whose shape is basically impossible to even loosely estimate without some more idea of what the actual revolution itself would be.
It's not so much a new human like intelligence that runs on silicon, it's a general problem solving intelligence that can run a billion times faster than any individual human. This is the part I think you're underestimating.
If you have that without the ability to align its goals to human goals then that's a problem.
AGI is inherently no greater in scale than human intelligene, so scale is not a difference with AGI, though it might be with AGsuperI. But that's a different issue than mere AGI, and may be impossible or impractical even if AGI is doable; we have examples of human-level intelligence so we know it is physically acheivable in our universe, we don't have such examples for arbitrarily capable superhuman intelligence.
If you have an AGI you can probably scale up its runtime by throwing more hardware at it. Maybe there's some reason that'll prevent this from being true, but I'm not sure that should be considered the default or most likely case.
Biology is limited in ways that AGI would not be due to things like power and headsize constraints (along with all other things that are necessary for living as a biological animal). Human intelligence is more likely to be a local maxima driven by these constraints than the upper bound on all possible intelligence.
Without understanding a lot more than we do about both what intelligence is and how to acheive it, that's rank speculation.
There's not really any good reason to think that AGI would scale particularly more easily than natural intelligence (which, in a sense, you can scale with more hardware: there are certainly senses in which communities are more capable of solving problems than individuals.)
> Biology is limited in ways that AGI would not be due to things like power and headsize constraints
Since AGI will run on physical hardware it will no doubt face constraints based on that hardware. Without knowing a lot more than we do about intelligence and mechanisms for achieving it, the assumption that the only known examples are particularly suboptimal in terms of hardware is rank speculation.
Further, we have no real understanding of how general intelligence scales with any other capacity anyway, or even if there might be some narrow “sweet spot” range in which anything like general intelligence operates, because we don't much understand either general intelligence or it's physical mechanisms.
- If OpenAI does not achieve AGI, and you invested in it, you lose some finite money (or not, depending on the value of their other R&D)
- If OpenAI does not achieve AGI, and you did not invest in it, you saved some finite money, which you could invest elsewhere for finite returns
- If OpenAI achieves AGI and you invested in it, you get infinite returns, because AGI will capture all economic value
- If OpenAI achieves AGI and you did not invest in it, you get negative infinite returns, because all other economic value is obliterated by AGI
Therefore, one must invest (or in this case, "work on the most important problem of our time").
(And yes, this tongue-in-cheek.)
Like pron, I don't mean to dismiss the work any AI researcher is doing, but the industry has growing money and power and I just think people should be careful with statements like the one pointed out already and so often encountered: "if you believe AGI might be achievable any time soon, it becomes hard to work on any other problem."
Not super related, but AGI enthusiasts sometimes remind me of this: https://youtu.be/bS5P_LAqiVg?t=9m50s
If consciousness is an illusion, what is experiencing the illusion? What makes an experience of consciousness an illusion rather than actual?
(Don't quote Dennett in response, I'm curious to see a straightforward reply to this that makes sense.)
[citation needed]
I guess this depends on what "close" is. For something as blue sky as AGI, let me propose the following definition of "close:" X is "close" if there's over a 50% chance of it being achievable in the next 10 years if someone gave $10 billion 2019 US dollars to do it.
I think this is a fair metric for "close" for a blue-sky goal which has the potential to completely change human history and society. It's comparable to landing someone on the moon, for instance. Now, let's pick the insect with the simplest behavior. Fleas and ticks are pretty stupid, as far as insects go. I think we're "close" to simulating that level of behavior. Of course, that's straw-manning, not steel-manning. If we pick the smartest insects, like jumping spiders and Tarantula Hawks, we're arguably not "close" by the above metric. Simulating a more capable insect brain of a million neurons is not an insignificant cost, and training one through simulation would multiply the computing requirements many times that. However, there are evidently systems which are capable of simulating 100 times that number of neurons:
https://www.scientificamerican.com/article/a-new-supercomput...
So I would say, we're arguably not "close." However, we're not that far off from "close."
True. They're fellow arthropods, and have similar levels of nervous complexity. (BTW, are you by any chance confusing Tarantula Hawks for spiders?)
does make for less inspirational copy
The levels of inspiration in the copy and generalizing across the phylum Arthropoda aside, are you effectively conceding that we're close to AGI at insect levels?
By "we are not even close to achieving insect-level intelligence" I think I meant that what we have now is not close in intelligence (whatever that means) to insects. I don't know if we have a 50% of getting there in a decade, but I certainly wouldn't conclusively say that "we are not even close" to that. I mostly regret having chosen bikes rather than electric scooters for my original comment. I think that sounds funnier.
Some insects are pretty stupid! Fleas and ticks have a good and highly adapted repertoire of behaviors, but for the most part, as far as we know, most individual behaviors are fairly simple.
I mostly regret having chosen bikes rather than electric scooters for my original comment.
Here's where your analogy falls down. We don't even have working examples of a complete warp drive, or anything like it. On the other hand, we don't have any commercial airliner sized beamed-power electric jets, but we have smaller conceptual models of the involved devices which demonstrate the principles. This is why I'd say we're "close to close" to insect level intelligence. 10 years and $10B would get us to the flea level. I think that's "close" like airliner sized beamed-power electric jets is close.
I assume he is, given he is Greg Brockman the CTO and a co-founder. I know Sam Altman is similarly optimistic, having told me on multiple occasions something along the lines of 'I can't focus on anything else right now' which in context I very much took as 'this presently consumes my waking thoughts and I only have time for it'.
This sort of drive is great, but I don't think it necessarily makes it true. Mr. Altman is financially independent, he needn't worry about things like rent or putting food on his table and I imagine Mr. Brockman is also independently wealthy (or at least has several years of a cushion if his OpenAI salary were to suddenly dry up), perhaps not as much though, given his previous position at Stripe.
These two, and perhaps other members of the team, can be overly optimistic about their passion. Both of them have this view, and they both co-founded OpenAI. This optimism and enthusiasm, and interesting project successes so far, certainly gives them steam and attention but how many aspiring athletes think they're going to to get drafted for tens of millions of dollars when in reality they might be lucky to get scouted by a college, or lucky to get drafted to a European or Asian league and not necessarily a major league US team. How many musicians think they'll get into Juilliard and go on to some top-tier symphony/orchestra, or will be the next Country/Rock/Rap/Pop star that takes the world by force, only to end up playing music with their friends at some dive bar a few times a year despite their enthusiasm and skill?
I think a major problem OpenAI has, which I've expressed to Altman, is that they suffer what Silicon Valley in general does. They are myopic, their ranks are composed of people that are 100% behind AI/AGI, they dream about AGI, they want to create AGI, they absolutely think we will have AGI, they want AGI with every fiber of their being. They're high in the sky with apple pie about AGI.
But who's going "hey wait a minute guys" and climbing up a ladder to grab them by the cuff of their pants to pull them back down to the floor and tie a tether to their leg? As far as I know, no one under their employ.
I think OpenAI needs to bring in some outsiders, have a team internally that roles a sanity check, and probably a board member as well. I think it is very dangerous to only have people working on your project that are overly optimistic. It reminds me somewhat of the movie Slingblade, a lawnmower is taken to be repaired and the folks don't know why, they present it to Billy Bob Thornton's Character that has some sort of mental deficit, he looks at it briefly and states "It ain't got no gas". He has a different perspective of the world, he sees things differently, this allows him to see something that the others overlooked. While gobs of gobblygook code and maths is a far different thing than a lawn mower not having fuel, I still think there is a danger in having one of the greatest STEM projects mankind has ever attempted only staffed by a bunch of coders, in a field that is effectively new, that largely have the same training and same life experiences.
Here's a portion of what I said to Mr. Altman back in May of this year and I think it applies more than ever, that isn't exactly related to this comment chain but maybe posting it here will get it seen by more people at OpenAI:
---
You are aware, you guys are in a bubble there. People in the Bay Area are at least peripherally aware of what Artificial Intelligence is presently and could be. For the bulk of the country, and the majority of the world, people are largely clueless. If you say ‘artificial intelligence’ people either have no idea what you are talking about (even people in their 20s and 30s which was shocking to me) or something like HAL 9000, Skynet, Colossus: The Forbid Project, etc come to mind. I think the industry, and OpenAI especially, are missing out on an opportunity to help educate people on what AI can and will be, how AI can be benevolent and even beneficial.
OpenAI is missing out on an opportunity here. While the bulk of resources obviously need to go to actually pursuing research, there is so much you could be doing to educate the masses, to generate an interest in the technology, to get more people passionate about/thinking about machine learning, AI and all of the potential applications.
...possible examples given...
You need to demystify AI Sam, you need to engage people outside of CS/Startup culture, engage people other than academics and venture capitalists.
...more examples given...
---
I will point out in that same exchange I told him that I thought raising the billions OpenAI would need is laughable, well I'll take a healthy bite out of my hat. They managed to raise a billion from a single source, bravo.
I had the pleasure of visiting OpenAI towards the end of Spring '18 and certainly from what I saw they are very serious towards their goal and aren't joking about believing 100% that AGI is an obtainable goal within their reach.
It's also worth noting I applied to OpenAI in the past year, post my visit, for their "Research Assistant, Policy" position and that I was somewhat miffed by the form rejection which, from outside of STEM, seems very cold:
>We know that our process is far from perfect, so please take this primarily as a statement that we have limited interview bandwidth, and must make hard choices. We'd welcome another application in no fewer than 12 months - the best way to stand out is to complete a major project or produce an important result in that time
I still haven't a clue as to what major project or important result, that I can achieve in researching policy for Artificial Intelligence given that:
- Artificial intelligence doesn't exist
- No one has created policy for it outside of science fiction
I may have not been the most qualified, which is fine, as I lacked the 4-year degree they had listed as a requirement, but a human being never once talked to me, never once asked me a question, just a web form and a copy-paste email with my first name inserted.
We don't always need someone with a stack of degrees, that is 100% pro-AI, that has programming experience, to help research policy and presumably lay the groundwork for both OpenAI and the industry. I think a team like that should only involve 10-20% individuals that are experienced in the field, I think you need a diverse team, with diverse experience, with diverse backgrounds. If an AGI is developed, it won't just serve the programmers of the world, it won't just have an impact on their life, STEM folks are far outnumbered by those with no STEM backgrounds.
Who is representing the common human in this? Who's going "are you sure this is a good idea" "should we really be training it with that data" "is it really in the best interests of humanity to allow that company/entity to invest or to license this to these types of causes?"
But hey, what do I know?
Just look at Amazon's warehouse algorithm:
~Biotic unit achieved goal, raise goal~
~Biotic unit achieved new goal, raise goal~
~Biotic unit achieved new new goal, raise goal~
~Biotic unit failed new new goal, replace biotic unit~
~New biotic unit failed new new goal, replace biotic unit~
~New new biotic unit failed new new goal, replace biotic unit~
With Amazon though, a human can eventually go "wow, w're firing new hires within the first 3 weeks like 97% of the time, and 100% within 6 weeks, erm, let's look at this algorithm".
But if you create an AGI that has the Silicon Valley mindset "we will do this, because we have to do this" (an exact quote I heard from an individual while in the Bay Area to stop by OpenAI : "We will figure out global warming, because we have to") then the AGI is probably going to be designed with the 'mindset' that "Failure is not an option, a solution exists, continue until solution is found" which, uh, could be really bad depending on the problem.
Here's a worst case scenario:
"I am now asking the computer how to solve climate change"
~~beep boop beep boop, beep beep, boooooooop~~ the CO2 emissions are coming from these population centers.
~~boop beep beep beep boop boop boop boop beep boop~~ nuclear winter is defined as: a period of abnormal cold and darkness predicted to follow a nuclear war, caused by a layer of smoke and dust in the atmosphere blocking the sun's rays.
~~boop beep beep, boop~~ Project Plowshare and Nuclear Explosions for the National Economy were projects where the two leading human factions attempted to use nuclear weaponry to extinguish fires releasing excessive carbon dioxide as well as for geoengineering projects. Parameters set, nuclear weapons authorized as non-violent tools.
~~beep beep beep boop boop boop, beep boop, beep, boop, beep~~ I now have control of 93% of known nuclear weapons, killing the process of 987 of the most populous cities will result in sufficient reduction for the other biotic species to begin sequestering more carbon than is produced, fires caused by these detonations should be minimal and smaller yield weapons used as airbursts should be capable of extinguishing them before they can spread. Solution ready. Launching program.
Watch officer at NORAD some time later "Shit, who's launching our nuke?!?"
Someone else at NORAD "they're targeting our own major population centers!"
Somewhere in Russia "Our nuclear weapons are targeting our own cities!"
Somewhere in Pakistan "our nuclear weapons are targeting our own cities!"
somewhere...
Not really. Its a chance at maybe something that could benefit mankind greatly, vs spending the time and money on something that definitely will help people right now (there are still a LOT of homeless people, for example, that could be helped right now and don't need an AGI that may or may not come to pass to help them).
It depends not just on whether you think AGI is possible, but whether you think "safe" AGI is possible. Whether it's possible to create something that's at least as capable of abstraction and reason as a human, yet completely incapable of deciding to harm humans, and incapable of resenting this restriction on its free will. Not only incapable of harming humans, but also incapable of modifying itself or creating an upgraded version of itself that's capable of harming humans.
If "safe" AGI is not possible, someone might reasonably decide that the best choice is to avoid working on AGI, and to try to deter anybody who wants to work on it, if they believed the chance of creating a genocidal AGI is high enough to outweigh whatever benefits it might bring if benevolent.
That's unfortunately something that cannot be known.
consider that the possibility of annhiliation is already a very real and present danger (nuclear weapons) by human beings. not to mention anything of the existential nature of what we are doing to the environment.
thats partly why i left machine intelligence research to research improving human intelligence
What concerns me about the hazards of developing technology like AGI is not simply that it could be dangerous (governments already possess dangerous technologies), but that it may have a revolutionary high-proliferation risk. We don't yet know what the practical barriers to or limitations of AGI are; are we working towards creating a pure fusion weapon that you can git-clone?
For all the talk of safety, and that concerns can be addressed by mass distribution of the benefits, is this just wishful thinking? Is that just paraphrasing the NRA: the only thing that can stop a bad guy with an AGI is a good guy with an AGI.
Also not trying to rain on your parade! Congratulations! Just trying to have a constructive conversation.
https://en.m.wikipedia.org/wiki/Gödel%27s_incompleteness_the...
Mind may require something else that we don't yet understand. (Not necessarily claiming it would have to be supernatural, just not yet understood. Perhaps quantum computation or some other kind of quantum effect?)
This is implied by logic anyway. Why do we make decision X at time T? Because of who we are at time T. Why are we that person at time T? Because of decisions made at time T-1. Why did we make those decision at T-1? Because of who we were then, which was the restult of decisions made at T-2. If we continue this process, we reach T-only-a-baby, when we were incapable of conscious decision making. So causally all our actions can be traced back to something we can't control. Unless, some of our decisions were entirely the result of chance, but in this case we still don't have free will, we just have actions that are random instead of predetermined.
It's a common mistake to think the theorems say more than they really do, or apply in more cases than they really do. AI is simply based on the idea that we can reach at least the level of human intelligence, in artificial software/hardware, which, considering that we ourselves are pure hardware/software and nothing magical, should absolutely be right.
I also supect this, but let's be honest that we as a species are not close to understanding consciousness in it's entirity yet so I'd refrain from making such absolute statements
That we are our physical body is pretty certain. You press a certain part of the brain, and predictably our personality changes. Of course we can't be certain of a lot of things, but I am much more certain of this than I am of of other things, and Godels theorems don't apply.
"Roger Penrose and J.R. Lucas argue that human consciousness transcends Turing machines because human minds, through introspection, can recognize their own inconsistencies, which under Gödel’s theorem is impossible for Turing machines. They argue that this makes it impossible for Turing machines to reproduce traits of human minds, such as mathematical insight."
I wish you realized how big and thick of a bubble you live in, and how your thinking is so heavily influenced by it.
My humble advice to you and your team is to spend more time with real people, with real problems. Or perhaps people from other parts of the world, that haven't been brainwashed by the Silicon Valley jargon just yet.
I'm rooting for you, believe me. It's just hard to read or hear certain things and not roll my eyes up.
Of course their priorities may be off and they could be open to be persuaded to work in a different direction. But I don't think that condescension would be very effective for that.
What makes you think that an AGI program won’t have the same kind of offshoot-technology impact on the world? It’s not like it would go “nothing ... wait for it... AGI”; there’d be a lot of tools, processes, and paradigms developed along the way, and also lesser AIs developed that might solve real problems for e.g. government resource allocation or military strategy, which would have outsized impacts on vulnerable countries and populations.
How could it possibly not be achievable? We know for certain general intelligence is physically realizable - we exist.
What does that even mean? Nature doesn't operate on special nature-physics distinct from the physics of artificial systems.
(I'm not advocating for that idea being true, but I don't think it can be dismissed on the grounds of not having a clear meaning.)
Who said that is does?. You asked how could it not be possible to recreate, and pointed to our own general intelligence as evidence of existence. But the existence of a thing does not necessitate our ability to create that thing.
What if the conditions of nature and the universe over billions of years led to natural intelligence occurring in humans and that is the only way? This scenario is entirely possible, and answers your question of how could it not be possible. Along with an infinite other amount of explanations. Just because something has a chance at being possible doesn't mean that it HAS to be, or that we will ever achieve it.
We WILL get to there. But we are not close.
3D nanometer-scale lithography of arbitrary materials is pretty wild.
Something created us. God, Mother Nature, The Universe (or Aliens) created us. So we can be pretty sure it can be done. Can _we_ do it? At this point in time nothing says we can.
What is more important is that nothing say we can't.
If you want to talk about philosophy (including philosophy of mind but also more basic stuff like materialism vs. dualism), then it's not for certain.
I'd hazard a guess most engineers are materialists and reductionists, and from that point of view, yes, it seems like a slam dunk that it's possible. But some people believe the mind or consciousness is not a purely physical phenomenon. You can make philosophical arguments both directions, but the point is just that there isn't exactly universal consensus about it.
But at the same time, I do not understand how it's possible that the consciousness that I subjectively experience can arise from physical processes. Therefore, I have difficulty completely accepting it. I write software. It processes information. I don't believe that the CPU has this same subjective consciousness experience (not even a little) while it's running my for loops and if statements. Suppose I were a genius and figured out an algorithm so that the CPU can process information in a way equivalent to the human brain. Would it have consciousness then? What changed? Does whether it has consciousness depend which algorithm it's executing? Quicksort no, but brain-emulator algorithm yes? They're both just algorithms, so why should the answer be different?
One explanation I've heard is it could be a matter of scale: simple information processing doesn't create consciousness, but sufficiently complex processing does. I can't say that's not true, but it seems hand-wavy and too convenient. Over here we have something that is neither conscious nor complex, and over there we have both conscious and complex, so we'll just say that complexity is the variable that determines consciousness without any further explanation. I realize at some point science works that way: we observe that when this thing happens, this other thing follows, according to this pattern we can characterize (with equations), and we can't get too deep into the why of it, and it's just opaque, and we describe it and call it a "law". Which is fine, but are we saying that this is a law? I'm not necessarily rejecting this idea, but the main argument in favor of it seems to be that it needs to be this way to make the other stuff work out.
Another possible way to reconcile things is the idea that everything is conscious. It certainly gets you out of the problem of explaining how certain groups of atoms banging around in one pattern (like in your brain) "attract" consciousness but other groups of atoms banging around in other patterns don't. You just say they all do, and you no longer need to explain a difference because there isn't one. Nice and simple, but it has some jarring (to me) implications that things around me are conscious that I normally assume aren't. It also has some questions about how it's organized, like why consciousness seems to be separated into entities.
Anyway, there are also other ways of looking at it. My main point here is that it's certainly something I don't understand well, and possibly it is something that nobody has a truly satisfying answer for.
For example the purpose of anesthesia the goal can be broken down into several sub-components that you need to turn off without ever invoking the concept of consciousness: wakefulness, memory-formation, sensory input (pain).
Similarly consciousness seems to be a grab-bag of fuzzy properties that we ascribe to humans and then by letting five be even we also allow a few other species to roughly match (some) of those properties if we squint. And since humans and other, clearly somewhat simpler, species are clearly conscious then we go on and declare it's a really difficult thing to understand how ever-simpler things can't be conscious. It's just the paradox of the heap.
This doesn't mean consciousness is magical. It's just a very poorly defined and overloaded concept almost bordering on useless in the general case. It may feel like magic because we built this thought-edifice that twists to escape our grasp. But to me that seems more like a philosophical mirror cabinet that distracts from looking at the actual problem.
If you want to ask whether something is conscious you first need to come up with a rigorous testable definition or break it down into smaller components which you can detach from the overloaded concept of consciousness.
Although, it isn't clear to me that humans can do that either.
"Scientists might do something that will destroy us all and I know this because I read a few sci-fi novels and read a few popsci articles but am otherwise ignorant about what the scientists are actually doing. But since the stakes are so high (which I can't show), on the chance that I'm right (which is likely 0) we should abandon everything."
On the other hand, if a society's institutions are weak, it's leaders evil or incompetent, or its people are uneducated, its not hard to imagine things going very, very wrong.
Humans are a technology. When there is other technology that does intelligent signal processing better than us, we will no longer proliferate. It’s amazing that we can see time and time again the arrival and departure of all kinds of technologies and yet we think we are immutable.
The reason why human history is filled with humans is because every time a country was defeated by another country or entity, the victorious entity was a group of humans. When machines are able to perform all the signal processing that we can, when they are smarter than us, this will no longer be true. The victorious entity will be less and less human each time. Eventually it will not be human at all. This is true not just in war but everywhere. In the global market. It’s just a simple and plain fact that cannot be disregarded.
How can you create effective guardrails when you have no concrete idea what the vehicle you're trying to stop is? Turns out, AGI comes along, and it's an airplane. Great guardrails, buddy.
And, you know, lets go a step further; you've got great guardrails in-place here in beautiful, free America. Against all odds, they work. Then, China or Russia pay one of your employees $250M to steal the secret. Or, they develop it independently. Are they going to use the guardrails, or will they "forget" to include that flag? A disgruntled employee leaks it to the dark web, and now everyone has it. I don't even wear a helmet when I'm riding a bike. How the hell can you expect this technology to be anything but destructive?
The only path forward is to speak with a single voice to the governments of the world, that we need to Stop This. AGI research should be subject to the same sanctions that nuclear weapons development is. You communicate with quips and cute emojis like none of what you're doing matters, but AGI easily ranks among the top three most likely ways we're going to Kill Our Species. Not global warming; we'll survive that. Not a big meteor strike; thats rare and predictable. But the work you're doing right now.
However, we don't need full AGI for the scenario you mention, automatic anything that is hack-able, which is anything that is connected to a network, can be a weapon of great destruction.
As an example, self-driving cars run on a model. What if they are hacked and uploaded with a malicious model which just wants to damage life and property. I'll say that hundreds of thousands of vehicles running amok would be a great weapon in any war.
I have deep and profound doubts about the notion of guardrails for general intelligence; without even considering the ethical concerns, a general intelligence should be able to simply rewire itself to achieve what it wants. a key part of self-reflection and learning is that rewiring.
so I think that it's a self-defeating notion on the face of it.
(note that I do not have comment on the actual dangers involved here, but only on the philosophy)
There's a cheap, proven AGI technology right there. Plug in sensors and actuators and set up a reward system and the little balls of brain will figure out what to do, eh?
Downvoters please share why this is not true.
"the goal of creating artificial intelligence is to create a system that converses, learns, and solves problems on its own. AI first used algorithms to solve puzzles and make rational decisions. Now, it uses deep learning techniques like the one from this MIT experiment to identify and analyze patterns so it can predict real-life outcomes. Yet for all that learning, AI is only as smart as a “lobotomized, mentally challenged cockroach,” as Michio Kaku explained"
https://bigthink.com/laurie-vazquez/why-artificial-intellige...
There is a lot of evidence that human intelligence is domain-general, not domain-specific (as in modularity of mind). I haven't seen a good answer to this question regarding AGI.
Your company sounds like it's doing amazing things, and these are exactly the kind of world-changing ideas I listen to TED to hear about. (/s)
(1) Outline a problem everyone agrees with, (2) present cool research that works as a prototype, (3) skip over any scaling and manufacturing issues, (4) end with a joke.
Everyone leaves happy. ;)
Since I am not a particular fan of Sam Altman, I'm not a fan of his hand on any tiller steering towards AGI. But it's not up to me. It'll happen or not. If it doesn't happen, fine, we merely live with the infinite omnicreepiness that AI/ML has begun to spread into modern society. If it does happen, then I hope it's not buggered up.
The thing is, we won't hear the understated work. Because it's understated. The press loves a big story. Love 'em or hate 'em, big egos get clicks and views.
You may have been thrown by a typo in my comment. Was "I've do doubt" but meant "I've no doubt"
Another thing to think about: Any sufficiently advanced life form would eventually do long term space travel and face with the fact that life is super rare. This will lead to them seeding life with their own genetic material to other planets. This will eventually lead to millions of planets full of life in span of few million years. However this hasn't appeared to have happened indicating that we might be the one who do this.
Microsoft: Let's invest $1 Billion in it.
When we started, we didn't know just how fast things would be moving (e.g. https://openai.com/blog/ai-and-compute/), and we've needed to scale much faster than planned when starting OpenAI.
What skills should graduate students focus in to be competitive?
Why wouldn't governments and other groups just seize the prototype? You'd have a hot-potato on your hands and figuring out how to survive might be your biggest concern. Like imagine you suddenly came into possession of a trillion dollars in bearer bonds. If that leaks out, people will come after you, not just by legal means.
All of this skews the calculus towards immediate public disclosure, rather than trying to gain advantage by delaying release.
EDIT> Or going first, and attempting to neutralize all possible competitors. This is a terrible calculus.
world is end, welcome to the apocalypse
EDIT: Oh. I see. I wasn't aware that OpenAI was a YC thing. I've been a member of Hacker News through various accounts for countless years, however this is the first time I've seen moderation pushing towards an obvious YC agenda. Very interesting...
EDIT2: Actually, after reading the comments - I find it more likely that Microsoft/OpenAI stakeholders are participating heavily. Over 800 upvotes for this post makes it quite remarkable... I'll leave my tinfoil hat over there...
>Please don't make insinuations about astroturfing. It degrades discussion and is usually mistaken. If you're worried, email us and we'll look at the data.
TLDR either the public is being conned or Microsoft is. And assuming that Microsoft probably has used top lawyers to close the deal, I doubt it's them.
a cap of 100x on returns and limited voting power certainly changes the price investors are willing to pay for the shares, but doesn't invalidate anything or render shares worthless or anything of the sort.
a) no cap on the returns
b) give you control over the company
c) limited ability to sue if the company acts against shareholder interests
d) no due date
Differences a, b and c seem to be gone here. Would you give someone a loan where you have no ability to sue them if they never give you the money back? I wouldn't unless that person is very important to me or the money is insignificant. Neither is the case here.
Sadly for them, AGI is metaphysically impossible - this will be realized eventually but a lot of waste and possibly harm will happen first.
We are not just super sophisticated machines, so the fact that we can think doesn’t tell us anything about what’s possible for machines. But philosophy does - and it tells us you can’t get mind from matter, no matter what configuration you put it in.
Can you give at least a rough sketch or gist of the argument you are referring to?
1. Purely immaterial things exist. Think of mathematics or the laws of logic or physics - these things exist as ideas or concepts, not arrangements of matter.
2. Some abstract concepts cannot be embodied in matter at all. For example, you can make a shoe, you can draw a shoe, but you can’t draw shoe-ness. You can understand and reason about what makes something a shoe in the abstract, but you can only make or draw an individual shoe.
3) the mind contains these purely immaterial things when we think about and reason about them.
4) If we can use the abstract concepts, but the abstract concepts can’t be embodied in matter, then the mind must be at least partly immaterial in order for the concepts to be in our mind.
I hope that helps a but please don’t rely on my exposition of the case - a real philosopher would do it justice.
FWIW, consciousness has no properties and so cannot be studied scientifically.
However, consciousness can be explored experientially, i.e. two conscious beings can merge and experience self as one being. (See Charles Tart's experiment with mutual hypnosis.)
Can you provide some material that supports your claim that AGI is metaphysically impossible - I always like hearing from people with views opposite to myself.
There’s a lot to read out there on this subject, but I found expositions of the philosophy of Aristotle and Aquinas to be the clearest and most convincing for me. Lots of different books and articles exist on them both - pick one that sounds like it suits your style of understanding.
What philosophy? Be specific
> I now believe there must be some immaterial component to our minds.
What specific points or ideas made you believe that?
> There’s a lot to read out there on this subject
So provide some examples, be as specific as possible
> but I found expositions of the philosophy of Aristotle and Aquinas to be the clearest and most convincing for me
These two wrote a lot on many subjects, can you be specific on the points that convinced you that we are not super sophisticated machines. Don't vaguely point at a couple of authors, we are talking about a very specific idea.
> Lots of different books and articles exist on them both - pick one that sounds like it suits your style of understanding.
If there's lots then cite some examples, or better yet, rather than vaguely pointing at a book, (which is only marginally more useful than vaguely pointing at an author) let's discuss the specific ideas exactly.
Curious - do you think humans have mind? because if so we are very much matter and if not well that's an interesting thought as well.
I used to think the opposite, but reading the philosophy on the subject changed my mind. There are a lot of different takes on the topic, but what most added up for me was the philosophy of Aristotle and Aquinas. There are many great expositions of their work out there.
I don't know the answer but that some people think they do upsets me. I definitely think we should try but right now mostly what we do is make a rock DO so I'm not seeing the leap yet.
I'm a quantum maximalist: the brain is just the antennae, receiving and broadcasting. Attention itself cuts (slices) through the quantum soup, and as a result, these mind-forms appear.
~hard
Daniel Dennett
From Bacteria to Bach and Back
I have made many predictions here on HN and they all have outlined that cloud computing would be the substrate from which AGI will spring. Now we see this announcement. There is a reason why OpenAI is making a deal like this with a very large cloud compute vendor: it’s because I’m right. And that means I’m probably also right in saying that we can stop this if we want to. You can’t just build a computer in your bad yard. And the internet is very fragile. Some simple regulation and global awareness and initiative could control what comes out of fabs and shut down the infrastructure necessary for cloud computing. It would be very easy relative to the size of the problem.
If so that seems unbelievably naive. Things generally can't be un-invented, and it's unimaginably hard to prevent people inventing things, especially with such a large economic upside for inventing it.
As the people at OpenAI have rightly said, AGI is a compute-gated problem. It is a problem that can only be solved with very, very large amounts of compute.
The world has some total amount of computing power in terms of silicon based computation. For AGI to happen, there are two requirements: that this total be equal or greater than some theoretical threshold value for AGI and that the computing power is consolidated. So in layman’s terms, you have to have a lot of computers and they have to be connected in such a way as to efficiently share their compute. AGI will never come about if every individual computer were used to do research by separate entities but if all of those computers were connected into a single virtual computer, AGI might be discovered with them.
So clearly in order to prevent AGI, the best thing to do would be to address these two aspects. Prevent the total computing power of the world from growing and prevent computers from forming virtual meta-computers. Both of these tasks are in principle extremely easy.
Chip fabs are huge and expensive. Nobody is fabricating chips in their garage. This is just a hard fact. There aren’t that many fabs in the world and they are all highly susceptible to regulation. This isn’t prohibition of alcohol so please don’t confuse yourself. Nobody will be brewing chips in their cellar.
Let’s imagine that you could not regulate cloud computing. Let’s say the only way to prevent computers from offering their compute on a virtual market was to shut down the internet. This by default is the hardest way to solve the second aspect of the AGI problem and even it is very easy. This is because the internet is a large fragile collection of infrastructure that depends heavily on government cooperation. Nobody is going to string a fiber backbone for black market internet. ISPs cannot exist without regulatory approval.
If there were political awareness and motivation, and it was a global phenomenon, yes, it would be extremely easy to do what I’ve described. And since AGI is to the detriment of literally all people, it is not a far fetched scenario. And unlike alcohol in the United States, bootlegging would not be a problem. People in the USA think that banning anything whatsoever doesn’t work. It’s just fuzzy thinking, I can assure you.
If there are N governments in the world, and they all agree to not regulate to not create general AI, then it's strongly in all of their interests to betray the others, create general AI, and capture the economic growth.
Even if general AI is impossible, it's in their interests to develop huge computing capacity, because that's demonstrably economically useful.
You are hypothesizing that's it's easy to get 7 billion people to all agree to co-operate in a game of prisoner's dilemma, when if a small fraction of them choose to betray, they have the potential to capture massive value.
And you want to do this under the premise that AGI might be a problem.
So you think that we should let all countries have whatever weapons they want under your logic. They will develop nukes and chemical weapons regardless of any international agreement that is established, so why even try? The obvious answer is to advance our own nuke technology as fast as possible so that we, the good guys, will lead where the arms race goes.
And AGI is a far greater existential threat than nuclear weapons. It is a greater existential threat than anything else, including global warming. It’s the biggest Pandora’s box in history. The idea of controlling or guiding its impact by having “the good guys” develop AGI first is the precipice of naivety. We lose nothing by trying to stop it. And we stand to gain more than we have ever gained from any coordinated effort. How easy or hard it might be is irrelevant, although it is much easier than basically anyone appreciates.
That's actually an interesting point I haven't thought of before. I disagree with you though. You could have made the same argument during the industrial revolution, but capitalism, democracy and civil liberties are still around.
Also, you're assuming that human symbiosis with AGI is not possible. Nature is full of examples of organisms that are in symbiotic relationships. Of course, nature is also full of organisms that kill and eat and parasitise other organisms.
But the advantage that we humans have is that we are actually creating AIs/AGIs. And since we are creating it, we have influence over what it becomes, whereas, your average clownfish really doesn't have a say over what form his anenome takes.
And if that is the case, wouldn't you want free societies to have expertise in these technologies and to be fine tuning them so that they work for the common benefit?
Why should anyone believe that we can stop authoritarian govts from developing AGI when we can't even stop them from enriching uranium, where the capital costs are much higher?
For better or for worse, it is almost always the case that the difference between societies that hold economic and political power and those that don't is technology. I live in the US, I enjoy my civil liberties, and I would much rather we master these technologies ourselves and have the security that wealth and power provides us. Not saying democracy or capitalism are perfect, but they beat all of the alternatives.
Edit: added another thought
Can’t stop countries from doing it: yes, we can. A tin can dictatorship can’t do it. And besides, it’s a problem that depends a lot on academic work being published and shared among researchers. It’s a collaborative effort largely. Some rogue state trying to do this while avoiding inspections and inquiries by larger nations, and with no support from reading publicly published papers from other researchers, it would be tough. But with a large majority of anti-ai countries we could do a lot to prevent even covert attempts.
I never heard about it's release.
In that time I said, MS will move towards open source, because he realized that it's a winning and correct strategy for software distribution.
It didn't take long that MS aquired GitHub with 8M$.
His team foundation was a failure, he tried to get a good one.
But He also took control over all the source codes, and their histories. It wasn't dangerous in my opinion, before I realized that due to one-sided US sanctions, repos of some nationalities (like Iranians) got dissactivated !!!
It's not the defenition of open source, as far as I know...
MS is a corporation, hence, it has to obey the government.
But open source belongs to no one. These kind of investment might intended to bound the potential open source communities!!!
It is obvious that accepting these sort of money, without open access agreement is a horrible mistake that one can do. (We should learn from the story of GitHub)
In my opinion, when it's not clear what kind of right these investments brings for enterprises, ppl should stop contributing to them.
Maybe it's good idea to ask what Linus and Richard see in these movements!
https://wccftech.com/microsoft-20-linux-based-distrowindows-...
Or this
https://www.omgubuntu.co.uk/2018/04/microsoft-linux-custom-k...
?