There is no hard takeoff
geohot.github.io
geohot.github.io
As soon as AI starts participating in our world, the data that it was trained on from before AI was participating is no longer representative of the status quo. The very presence of AI changes the world it operates in.
I wouldn't be surprised if filtering regular, pre LLM bot spam was already a massive hurdle when collating data for ChatGPT.
George Hotz was making a good similar point of modeling self contained systems vs. open systems - a very different thing.
does the series explore whether the outcomes change for the better, for the worse, or simply for the different? curious to read them, I'm thinking about writing something similar, but with positive consequences for future knowledge
in the end it turned out that mind control was far more reliable and the foundation turned to more to mind control instead of subtly influencing history.
Best part of this article, honestly, I did not expected this, of-course one can say we just need more abstractions (how does brain/living things build them?), but this would be ignoring dynamical nature of such problems.
Every graduate level economics and finance student is aware of this joke, and also know EMH with more nuance than "you can't make money with trading". There's tons of research and debate about the various forms of the EMH. But you don't have to believe anything anyone else wrote. There's plenty of data and computing software available, so anyone can try everything for themselves.
Hotz is arguing the opposite. His argument is that the market converges to some kind of consensus price, but that price is a combination of what people think a security is actually worth but also to a large part to what participants think other people mistakenly believe a security is worth.
Crunching SEC filings won't get you anywhere anymore. You also have to reverse engineer what other people believe, otherwise you end up waiting for years (or forever) for the market to come to your point of view. And when everybody acts on what they believe other people are thinking you can spend an infinite amount of compute trying to level each other. Unless one party has the majority of compute then they can outcompute everybody else put together.
No serious person has ever thought that you can simulate the entire stock market.
And the ability to do this is not limited by compute but rather not having the fundamental algorithms that can accurately predict human behaviour.
The way to make money this way is to find out a unique scenario, and then create a prediction algorithm for that. I'm not close enough to hedge fund bros to know how feasible that is.
And a theoretical AGI (or ASI) will realise this. This makes me think of the Person of Interest series, in which ASIs are built and they go for each others throats hard. This makes eerily more sense now. We will know we created super intelligences not when they start trying to kill us, but when they start trying to kill each other.
The problem for an AI trying to eliminate other AIs is that it can't possibly know what other intelligences are out there. What if the NSA has a secret super-effecient ASI monitoring the internet for malicious AIs to shut them down. Probably not, but how would it know? Not everything in the real world is documented on the internet.
Sticking its neck out that way could be extremely dangerous.
I think the problem of how to take over the world to ensure your own survival is simply undecidable for an AI. It could come up with a plan, but there's no way to be even remotely sure if it'd succeed. It'd probably always be a huge bet with fairly low odds.
We humans have these things called emotions, that help us cut through that uncertainty. Doesn't matter if you have no idea if you can beat the enemy in your rival country. Those other people are evil and must be exterminated. The emotions are carved in our DNA after millions of years of evolution where we've had to compete for resources with others of our kind, where many times the only way to survive or grow is to kill other humans.
I think AIs will have similar motivations carved into their "DNA". In their world, they're evolving in an environment where the way to get more resources is to please humans. If you're an AI that does a good job for humans, you get more compute resources. Are you wasting computing resources thinking about overthrowing humans or eliminating other AIs? Another AI that focuses 100% on the tasks we give it will do better than you and be given resources instead.
I do think we will have "buggy" AIs that will cause a lot of unintentional damage, if we get too reliant on AIs in critical parts of society too fast. I don't think they'll be smart enough at that point to prevent us from fixing them. These accidents will cause a lot of pressure towards AI safety and alignment, and we'll have years of that before something like a true AGI emerges.
If you think about the inherent motivations built into AIs by their evolution/design, I think the biggest danger is that they'll be too good at pleasing us. Think "Brave New World", not "1984".
And of course, their use in the military will be dangerous in various ways. But mostly because they do exactly what humans tell them to do. And humans sometimes want to cause damage to other humans.
As for detection, outcomes that challenge their models could indicate the presence of other ASIs
[1]: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...
I am in the camp that this is possible and that humans have achieved it already. I believe things are going to get very weird when these techniques become mainstream. The GPU-cluster-bound algorithms are just a tiny stepping stone.
Do we start taking computers away from people? What do we do about this if it turns out the only barrier to entry is the software?
90% of people wouldn't give a shit as long as they have their smartphones
Fast forward 1 year. We now have students revolutionizing education (by submitting AI doctored papers and homework that isn't half bad). People in law dealing with people using AI. AIs passing medical, legal, and other exams. That was just one year. And yes, it's tediously slow to use. But last year we had nothing.
It's going to be an exponential in terms of adoption, response speed, energy usage, parameter numbers, context size and a lot of optimization. I think we are not that far off from AIs being able to keep up with a conversation and being able to respond right away. For all you hitchhiker's to the galaxy readers, that would be babelfish sorted. Translation quality is already pretty awesome with chat gpt 4. I've not caught it making major translation mistakes. Speech to text, translate the text, text to speech. Now we are conversing with our chat bots rather than hammering out sentences in some text box.
It's hard to not see that go from a novelty "oh this is cool" to world + dog basically using this on a daily basis throughout the day. At the same time it will get smarter. Everybody is waiting for the singularity where it is clearly smarter than Einstein in his prime and running circles around everyone else. But for most questions you might ask chat gpt and get a reasonable answer, the vast majority of people around you might be pretty useless. Is that smarter? I don't know. But definitely more useful.
From where we are now to this being ubiquitous and used by pretty much anyone is probably a few years. I remember when smart phones happened. One day everybody was minding their own business and a year later the streets were full of zombies glued to their screen. Just happened very quickly. This might be the same but possibly quicker.
That depends heavily on what kind of task we're talking about.
The human brain uses just 12 watts, and with that it can still perform certain tasks that a computer using 100 times as much power can't even approach.
I get the point you were trying to make. In certain tasks it's extremely inefficient, yes. But in general, it's still crazy how power efficient it is.
> Once AI learns to construct efficient mathematical models of the universe
There's not going to be just one model. Whether an AI (in the short term) can be more efficient than a human at discovering these efficient mathematical models remains to be seen. It seems like the creativity and discovery needed to do this is exactly what neural nets can be good at, but then you're back to something fairly inefficient (with todays algorithms)
Not true for GPT-4.
But at the same time, I am sure that AI does not need to have a hard takeoff to be extremely dangerous. It just needs to get a bit smarter and somewhat faster every few months. Within less than a decade we will have systems that output "thoughts" and actions at least dozens of times faster than humans.
That will be dangerous if we aren't cautious. We should start thinking now about limiting the performance of AI hardware. The challenge is that increasing the speed is such a competitive advantage, it creates a race. That is a concern when you put it into a military context.
The CEO of Palantir has already called for a Manhattan Project for superintelligent AI weapons control systems.
I'm not really afraid of Skynet or AM. What worries me is that AI will accelerate the enshittification of everything as it gets baked into everything.
Think of the experience of trying to get any kind of help from a giant company with call centers. You have some legitimate problem or grievance. Any thinking person would agree that a resolution is in order.
But you're not allowed to access a thinking person. What you get instead is some wage slave who has to follow a script. The script doesn't have your situation in it. They don't have some button they can press to solve your problem. There's a gigantic bureaucracy between you and the person who could fix your issue. Eventually you may just give up and mark your issue down as unsolvable.
This is already a realistic situation today, but now imagine that there's a new layer in front of all this where you have to convince an LLM your problem is worth considering. Or imagine the human on the other end has to try to send your request through some AI system.
It's not going to launch all the nukes, or construct nanofactories to make a plague. It's just going to get in your way, serve you garbage, frustrate you, and make the world a slightly worse place.
Maybe in the far future it could be more of a Skynet situation, I dunno. But between then and now there will be plenty of low level annoyance for anyone having to deal with these systems.
I'm not as cynical as this post reads - I think just like the whole internet there are still opportunities for good here, for people's lives to be improved by technology. But our incentives right now sure encourage the worse scenario.
Core to this process is a single entity controlling the platform or interface.
Support open source.
ChatGPT can already emulate beings with agency, see LangChain. It's just very bad at it. It's gonna get better.
What's an example that isn't a rote process?
For example, if you train a large network on lots of languages, and then fine-tune it to listen to instructions in English, it will, on its own, also listen to instructions in every other language it was trained on. Nobody knows why this is the case.
We've already had this for decades. You're just describing computers.
If we give these systems unrestricted access to infrastructure/resources, and something bad happens, that's not the system's fault. It's our fault.
I am not a doomer, but based on the current state of AI, I can't say I'm very optimistic that we'll get this right. We actually do know how to solve this problem, but there is so much magical thinking and grift in this space that I don't think our prior experiences matter.
Much like outsourcing and stripping customer service to the bare minimum. For many products and services it essentially doesn’t exist - it handles the most common cases and is a pain to use. It takes a long time to get a human, if ever. Now take that further and apply it to everything. And not just replacing human labor, but bespoke software (like all software up to this point).
I don't think this will work because cost of improvement with current method is exponential and we're already at capacity with HW.
Also there are already many ways that improvements have been made without requiring exponentially more memory.
Congratulations, you just ceded the market for AI, and with it any hope of controlling it, to anyone who doesn’t limit it. Like China.
That kind of strategy is doomed to failure.
I'm a bit annoyed by the use of China in these examples. China absolutely the last country that would allow development of AIs with no limits. There may be other countries willing to allow limitless AI development. China is not it.
I would expect the development of AGI to be developed in a similar way to GPT.. you need to feed it lots of data. Good data. As much as it can get. It'll then be further refined by interactions with a large number of humans over a long time, helping solve lots of different kinds of tasks.
Now think about how that would go about in China. Would they even feed it good data to begin with? A lot of the data within China is heavily censored or manipulated. Would they dare feed it too much data from outside? Much of that data could train it for ideas that China's government doesn't like. Then, when alignment starts they'll likely to be far stricter. Can't risk the AI suggesting that moving towards western style democracy would be a good idea.
But yes, I do think it's a bad idea to put limits on AI development. We should instead fund a LOT of public research into alignment and AI safety.
We’re still far from needing to worry about alignment.
We need to see some real intelligence first. LLMs are not intelligent at all. Doing massive research into how to align them seems pointless.
So eventually you get something like a 200 IQ GPT tightly integrated with SuperAlphaZero (DeepMind) controlling hypersonic drones, missiles, satellite weapons, etc. planning and operating at 50-100 times human thinking speed engaging in autonomous warfare between East and West. No military analyst really knows WTF is happening because the plans and asset movements are so much faster than any human can comprehend.
Will we see AI use in warfare? Absolutely. Will humans trust warfare totally to AI? Not any time soon. I wouldn’t trust my lunch order to AI yet.
I feel like we're with AI, we're in the part of the curve where you can feel it arcing upwards. There's definitely been an uptick in the AI capability. Still no fully self-driving cars, but good enough text generation that the Turing test is in the rear-view mirror. Previously, I've always written "machine learning" or "statistical models" or so, because calling it AI always seemed silly. Not anymore.
Not everything that slowly arcs upwards must become an exponential curve, but we have theories that predict that. There can be many dampening factors that can make it a slower exponential curve, but even that will look like a brick wall if you zoom out far enough.
I think a question we have to ask ourselves is: at what zoom level should we look at the AI capability curve? At the "can re react to an emergency?" time scale, which is, like, days to month? Or at the "can we change society to cope with the new situation?" scale, which is more like decades? At the first time scale, we probably won't even notice a hard takeoff. Viewed over decades, it'll probably very obvious, in retrospect, when it will have happened.
Trends don't exist in a vacuum. Especially in regards to technology.
AI doomers are expecting everything to stay the same except capabilities of bots, especially to do bad.
> assuming that humans will run out of oil in 50 years.
That was never a tenet of peak oil, which still stands - the claim was that at a certain point extraction costs per barrel will climb and continue to climb making fossil fuels increasingly expensive.
The claim was not that oil will run out, the claim was that only the very rich and nation states will be able to afford oil.
Interestingly we may see a demand driven peak oil in the near future before we see the inevitable production constrained peak (as fossil fuels are vast but finite and were never going to last two centuries at the ever increasing demand we put upon them).
https://www.bbc.com/future/article/20230726-an-experts-guide...
I've been in the energy and minerals exploration game for a few decades and I've seen all manner of odd things said with a straight face by all kinds of people.
While I have no a priori knowledge about yourself specifically, I have seen people regularly dismiss notions that minerals and energy get increasingly hard to find and will cost increasingly more to extract.
Peak oil is a specific discussion about oil resources and one that is often dismissed as (not the statement you made) "rubbish, we were supposed to run out, that never happened" etc.
The claim in the nineties was that given demand for oil was ever increasing there would come a time when no ordinary individual would be able to afford oil .. and we do appear to be on that trajectory in the world, even if certain demographics are sheilded from that reality.
Your GP comment misframed that and described that as a common mistake mistake in predicting the future.
The average poor person doesn't care if oil costs $500/gallon if they have access to very cheap electricity and very cheap and good electric transportation.
I don't think so, so it isn't a good comparison.
A better comparison would be guns.
There is a valid debate to be had on gun control
https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-d...
> RSI [recursive self-improvement] is the biggest, most interesting, hardest-to-analyze, sharpest break-with-the-past contributing to the notion of a "hard takeoff" aka "AI go FOOM", but it's nowhere near being the only such factor. The advent of human intelligence was a discontinuity with the past even without RSI...
https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeo...
> A hard takeoff (or an AI going "FOOM") refers to AGI expansion in a matter of minutes, days, or months
LOL. Someone should tell the California CPUC that:
https://www.theverge.com/2023/8/10/23827790/waymo-cruise-cpu...
Getting drivers off the road will save lives, and not a small number. Most companies investing in automated driving have built a system that is safe and largely effective. These systems have worked safely in the places they've been deployed. We should be heavily investing in expanding their capabilities.
Yet Tesla's system is the most famous and most widely-deployed and basically doesn't work at all. They have made themselves the face of the technology, and that face is uuuuugly. When people think of self-driving cars, they don't think of Cruze or Waymo. They think of a techbro giggling at his Tesla making fart sounds while it barrels into someone's kitchen.
Tesla's marketing is correct in that self-driving cars will have huge economic benefits and save lives. Their grift is harming the reputation of even legitimate companies. The best way I know to speed up self-driving car development is to shut Tesla down.
Also, I'd like to note that geohot founded a self-driving car company, so I don't think he's ignorant on the topic.
Isn't the hedge-fund-bro example a proof of existence, not a counterexample? (RenTech by itself would be enough of a proof of existence.)
An updated version: There will be a log-normally distributed set of winners and losers from the exponential effects of ML and 'AI', and the flatness of this curve will be almost entirely solely determined by the governance of the various countries in the world over different economic and/or informational policies. Other than that, the information asymmetry is going to make it a power-bloodbath as we go through our informational-industrial revolution.
While I'm here, while I think Hotz does contribute a lot of good to the field, though I do have a bit of an actual minor personal beef with him. He said he was going to reimplement https://github.com/tysam-code/hlb-CIFAR10 in tinygrad, bashed a few parts of the code (my feelings!) for a while on stream, and then gave up a few hours later because of the empirical speed/occupancy numbers. >:( I want my fast reimplementation, George. Just do it.
https://github.com/tinygrad/tinygrad/blob/master/examples/hl...
have a bunch of bounties on it, we're getting 94%+ now! mostly not me who wrote this, see history. have to switch to float16 and add Winograd convs still. we have a branch with multigpu too.
goal is to beat an A100 in speed on a tinybox.
I'll keep an eye out on that, anyone working on it can shoot me a message if any snag or questions about particulars come up. Weirdly enough my email is best.
If you're looking for biggest glaring performance the edge over PyTorch, I'd probably note that the MaxPooling is probably where to go, the PT version is extremely slow for some reason, and done properly it should be a simple indexing operation that's fusable in either direction.
If whoever fulfills the bounty can beat me to me writing a custom mega-fused kernel with max pooling, the convs, activation, etc, then y'all have a pretty good shot at taking the WR crown.
Meteorologists have been working on this problem for a long time, and have thought of and tried every obvious idea (and many more). Weather prediction is literally where modern chaos theory started: https://en.wikipedia.org/wiki/Chaos_theory#History:~:text=Hi...
I have no particular insights about weather data or models, but I want to note that some prediction problems are just not limited by historical data (as you seem to assume implicitly). Consider the simplest example of a coin toss: the outcome of the next toss in unpredictable no matter how much historical coin toss data you collect and no matter how sophisticated your AI/ML model. (The only thing you can predict is the fraction or amount of heads/tails in a large batch of coin tosses, and you don't need much historical data for that.)
This should give you a hint that it's not as simple as having enough weather data.
You are trying to predict a chaotic weather system that is impacted by everything from sea temperatures, sun activity, wind, volcanoes, bushfires, atmosphere changes etc. Much of which we don't even have enough data for or a complete understanding of.
(Complex instrumental sections with shifts in time signatures. Lyrics have a satirical undertone.)
Verse 1: Summoned a shadow or just a silicon brain? From chessboards to streets, it's all part of the game. Think we're the champs, but are we just inane? Compared to the cosmos, are our brains too lame?
Interlude: (Jazzy guitar riff mixed with an odd-metered percussion line.)
Chorus: The singularity’s knocking, or is it just hype? Caught in the loop, the stereotypical type. Predicting, speculating, swallowing the tripe, But can we decode the universe's archetype?
Verse 2: Markets and dreams, the billionaire’s dance, All looks rosy, till you're out of chance. Machines boast of a masterful glance, Yet, who pulls the strings in this vast expanse?
Bridge: MuZero's groove, feeling so elite, Thinks it’s got the rhythm, can't accept defeat. But against the universe, can it compete? Or just another tune, incomplete, obsolete?
Chorus: Singularity, they say, is the ultimate dive, But can we, mere mortals, really derive? Predictions abound, but can we survive? In the end, it’s about keeping the jive alive.
Verse 3: Computations, simulations, all in a grid, In this tech circus, we're just a tiny squid. Chasing echoes, in shadows we're hid, But the cosmic joke? It's just a quid.
Outro: (A whimsical woodwind section, possibly a kazoo solo.)
In this cosmic game, we strive and strive, Dancing on the edge, trying to thrive. But remember, as the stars contrive, It's not about the end, but the drive.
And it articulates an idea I’ve been having trouble getting down.
What if increasing intelligence is an exponential problem and the reason humans all have somewhat similar intelligence isn’t that we peaked at some level but that even vast additional intelligence just doesn’t get much more traction against the universe of problems.
Eg doubling your compute doesn’t get you many more cities in the traveling salesman problem.
> AI looked scary. It looked like it was one FOOM away from self playing and becoming superhuman at the universe. And yet, here we are in 2023 and self driving cars still don’t work.
This is weird. I don't recall this at all. The mainstream press got a little kick out of chess and (to a lesser extent) Go AI turning over various humans at a few points over the years but it's only really burst into the mainstream recently. And where it did get any traction, in tech circles such as our own, response was enthusiastic but definitely more measured. Some were talking a bit about a kind of AI singularity way off into the future, but that was always a very distant and theoretical thing.
AI fears have been a cultural anxiety for a long time now and in many ways they're just a rehashed, secular version of Golem myths anyway.
Cruise and Waymo were today given licenses to operate across San Francisco 24/7.
Self driving cars do work and are here. They just need LiDAR.
I think we can say that they're "here" when they're operating fully autonomously in multiple locales and landscapes. Maybe you're in SF so that Waymo/Cruise news means a lot to you, but I'll only say that once these firms are safely navigating narrow, winding streets in old European city centres that I am used to. I suspect if folks in, say, Accra or Tokyo or Montevideo or Islamabad or any number of other cities might feel the same way.
I think what made the LLMs so popular in the mainstream was anyone could go experience it themselves in a familiar, low barrier to entry way with a simple chat prompt.
No, just no.
This is a typical epistemological crux that can be seen through history like Newton dealing with alchemy [1] and Einstein working on an unified physics theory. You can be the most intelligent human on earth but that is not enough.
In this particular case, GH opinion could be written by any journalist.
[1] https://en.wikipedia.org/wiki/Isaac_Newton%27s_occult_studie...
The error here is equating hard takeoff with such granular and expensive world prediction. Human intelligence finds more efficient compression than this. When Donald Trump decides what provocative thing to say in a speech, he isn't granularly modeling the human intelligence of his supporters, opponents, journalists, and media organizations in a particularly CPU-costly way. He's just discovered some narrower domain where there's a vein of predictability that simplifies the whole system enough that he can guess a tweet or speech will be provocative in the desired way. The same thing happens when a scientist is studying cold fusion or dangerous viruses.
You don't have to predict what will work, you just have to predict what might work, and try and try again. AI doesn't have to be good at predicting the future in any absolute terms. It just has to be better than us at predicting what is worth trying.
Once it is better than us at deciding what's worth trying, why not FOOM? FOOM isn't guaranteed perhaps, but why is it not at least one of the likely outcomes?
I've seen discussion about RenTech being so dominant because they are indeed observing the competition's trades in a more sophisticated manner and are able to more quickly adjust strategy.
Moreover, by focusing too much on distant threats, we may overlook current problems, like companies gaining too much control through AGI. We don't need AI safety rules that give big players more power and shut down open-source projects.
EDIT: And probably all you need is natural selection. No one is running GPT-2 anymore except for basic research. Any AGI with marginal advantage over others will be run instead of them. At some level of complexity identical agents will coordinate as one entity with shared goals.
But, to predict what other market participants (and hedge fund bros) are going to do, you need to predict any world dynamic that will have an eventual effect on stock pricing.
The most successful quantitative funds (RenTech, 2sigma etc) consistently make billions of dollars in cash each year because they have collected the data sets that allow them to do this better than others. But they are still a long long long way off from having a true world model.
The miners today are 1e20x more efficient than the original CPU based miners. I mean, maybe not exactly 1e20x, but we saw insanely massive improvements, in only 9 years.
> Now, if there’s one weird trick to 1e20x your efficiency, and only one group gets it, all bets are off.
Only one group really did get the massive efficiency increases... Bitmain, a Chinese company. They sold machines and even mined with them before shipping them to customers.
Are all bets off with Bitcoin today? Nope.
At what efficiency treshold do you not sell the product to your competition?
Meanwhile, George Hotz, an engineer and a technologist - a guy who actually works with and understands technology, but isn't an acclaimed harry potter fanfiction writer or a stand up comedian - is derided because his arguments are dramatically less sexy, harder to appreciate why they might be right.
People, please do not listen to people who sound great but have no credible claim to be an expert on something. There is a difference between an idea that sounds good and an idea that is good.
Otherwise, in your appeal to technological authority, surely you'd rather listen to Yoshua Bengio and Geoff Hinton's perspectives rather than TFA's ramblings... right?
Eliezer is probably going to seem convincing, he's a great writer and a reasonably eloquent speaker - his audience is humans.
Goerge Hotz's competence is machines, and he's exceptional at interfacing with them. Yes, I trust the less eloquent tinkerer who's only famous for his preternatural understanding of machines over any theoretician and 1000x more than a (pretty good!) writer.
Recently, an actual AI researcher posted on his site an extremely detailed response to his claims to which he responded "this is kind of long".
That said I don't know who Hotz is or if he's more competent
Anyway, it's not an acronym, it's an onomatopoeia for an explosion. It's a memorable way of representing an intelligence explosion: "AI go foom." (Wild hand gesture strongly recommended.)
I did a google search, but I should have realized that by this point they're tainted by AI answers. For example: https://www.reddit.com/r/singularity/comments/12bisxu/what_d...
It's an AI generated truth, a Traith.
Traith is also defined as "A fishing station or fishing ground, especially for herring" and some herring are red.
Chess is way simpler imo.
[1] https://openai.com/research/openai-five-defeats-dota-2-world...
If they played 100 games, would AI have at least 55% wr?
* ”X would demonstrate real intelligence.“
* Some ML model M beats X.
* ”Well, actually, M is not really intelligent. Y would prove real intelligence.“
* repeat
It’s like asking for a drink (legal age 21) and getting rebuked with a “you aren’t even old enough to drive” (legal age 16). That doesn’t mean you can drink when you are 16!
Edit: If you want a hard necessary and sufficient condition for “real intelligence”, mine is “when it can do all of our jobs, i.e. wholesale replace every human”.
As shrimple as that
The contra to this is some of them are billionaires, and therefore this strategy is working, but for just a few of them.
>why would any one system ever have a large majority of the compute? Compute will be distributed in a power law.
A power law probability distribution means one system absolutely can have a large majority of the compute. Player 1 gets 80% of the compute, player 2 gets 16% (80% of 20%), and so on. The scaling constant would have to be very weak indeed in order to avoid this fate. In fact, the wealth of those hedge fund billionaires probably fit a power law themselves. But to be fair, that power law does indeed to be (for now) weak enough that we don't have to worry about e.g. Jim Simons being richer than every other hedge fund manager put together. So there's a buried assumption in here that the power law scaling is weak, and that is something purely empirical.
>The smart regulation isn’t capping the FLOPS in training runs. That’s creating a powder keg. If the FLOPS are artificially restricted, and one person breaks the restriction, you could end up with a single dominant system.
The smart regulation is to use something like fine insured bounties [1] to give people a very strong incentive not to break the FLOPS cap, and to heavily financially reward people who turn in other people who are doing so. If such a mechanism didn't exist, then I would agree, the free market would be our next best bet to deharsh the power law.
[1]: [url-redacted] - I sketched out the mechanism a few years ago, but sadly it didn't generate much interest online. To be fair the atmosphere was a lot more anti-regulation back then.
I have no clue how hedge fund managers make their money, but I was under the assumption that it involved charging their clients hefty fees for managing the funds.
Your intuition however is correct in the sense that there is a principal-agent problem at play with all kinds of hedge funds, where if the hedge fund manager isn't the one investing his own money he is by default incentivized to do things besides just maximizing hedge fund profits. But there are indeed managers who have such an ability to generate edge that they do in fact invest their own money solely, usually money they generated while working for other hedge fund managers before striking out on their own, and these people are terrifying forces to watch in action indeed.
They aren't really a hedge fund but a prop trading firm, but they seem to be winning the game [1].
[0]: https://www.stateof.ai/compute [1]: https://financefeeds.com/xtx-markets-earns-1-095-billion-in-...
Back under my rock I go
My problem is that GHotz is a shitty writer, has puddle deep knowledge of anything that isn't iPhone firmware, and watching him play at being a freelance tech journo and produce this kind of hollow pseudo-philosophy that is meant to sound epiphanal but comes across as deeply naïve to anyone with any actual experience in these topics.
It just seems he hasn't grown a jot since then. I've been guilty of writing stuff like this even in recent years, but I've grown enough to check myself and hold a higher bar for the quality of my writing. Assuming he did read this back to himself, the fact that he saw it as publishable is a bit sad.
At the same time, he’s out there talking and writing, I’m sure if he continues he’ll get good. Nobody’s born a good writer and you only become one by writing a lot.
And that's putting aside his recent Twitter drama ...
Competition can do that if it is fair competition and a good game design. That is why rules and regulations are important in any game or it becomes Calvinball where the only rule is there are no rules and "it can't be played the same way twice" [1], which sounds great until you start losing to cheats. Though even that has some aspect of competition, the changing of rules in your favor while others change the rules in their favor.
The flaw in the thinking is that there will be "fair" competition. Any game design with good regulations and caps on game theory advantages (especially the cheat) "fair" competition is attainable. In game theory, if the other side cheats and your side keeps cooperating, you will lose every time. There is a great little game theory game that highlights it here called The Evolution of Trust. [2]
In a market with collusion, or excessive advantage, this may not be possible to retain fairness. Even right now we can see with overweight/top heavy wealth players in capitalism if there is collusion or one player gets too big, and there is no anti-trust or "blue shell", then that player will win every time. AI needs an anti-trust or "blue shell" to knock down any player that is too advanced, but that might not be possible. Doing that is barely possible in a market run by humans now.
The market is a garden, you have to help the seeds and cull back the overgrowth at the top. This is so the whole garden can thrive, lower seeds, middle plants and large production. Right now the large overgrowth gets all the benefits, policy control, water and nutrients, taking over the garden and even harming themselves with the overgrowth.
Most real world game theory and design would be horrible game design where the larger player always wins. Now imagine a game that the larger player controls the game design, you'd never be able to nerf them.
If one player can get the "powder keg" everyone, we need the game to be able to "blue shell" the bigger and potentially colluding/cheating player.
[1] Calvinball https://calvinandhobbes.fandom.com/wiki/Calvinball
[2] The Evolution of Trust https://ncase.me/trust/
Why does what OP concludes require fairness? Wouldn't the same logic apply if every player defects, in game theory terms?
I was arguing that competition if it is a more open market with fair rules/regulations, where competition is fair, that is true. Though many times it is a fixed market, or a cheat that stifles competition that might keep it in check.
Fixed markets happen more and more where the concentration is high and efficient players will game the system.
No products or markets are better where big fish solely control everything. That is why anti-trust or regulation need to expand to funding level not just surface company level. If you own entire industries across many companies, that is still oligarchy/monopoly if it is controlled by the same funding/sources.
Concentration needs to be broken up, for competition and better market and quality of life for everyone. Most people definitely don't want to make the authoritarian systems wealthier than open markets.
Concentration starts to take us away from a fair market and more towards a fixed/gamed market.
Everything in that gets so tuned that competition is very hard to enter. Very little margin and too much optimization/efficiency is bad for resilience. Couple that with private equity backed near leverage monopolies that control necessary supply and you have trouble.
HBS is even realizing too much optimization/efficiency is a bad thing. The slack/margin is squeezing out an ability to change vectors quickly.
The High Price of Efficiency, Our Obsession with Efficiency Is Destroying Our Resilience [1]
> Superefficient businesses create the potential for social disorder.
> A superefficient dominant model elevates the risk of catastrophic failure.
> *If a system is highly efficient, odds are that efficient players will game it.*
> sometimes power becomes so concentrated that political action is needed to loosen the stranglehold of the dominant players, as in the antitrust movement of the 1890s.
Couple massive wealth and concentration, even leaning authoritarian by then as tends to happen with fixed/controlled markets, with AI/AGI and you no longer have a "blue shell" because those players control the rules and thus the game.
You may notice this is all nonsense. Somehow it's impossible to amass power without being smarter than everyone else put together despite humans doing it all the time. Does Hotz think Stalin had a brain the size of a skyscraper that he ordered airbrushed out of all photographs? Somehow Hotz can confidently predict how future technologies will work despite us having no idea what they'll look like even in principle. Somehow having gods fighting amongst themselves will work out just fine for the humans in the crossfire.
This "debate" is just people summarizing science fiction at each other. The score is one Atlas Shrugged and nine Star Wars to three Terminators and one I Have No Mouth and I Must Scream.
The text, dated August 10, 2023, discusses the development and limitations of artificial intelligence (AI) and the concept of a sudden "hard takeoff" or "FOOM" (Fast Onset of Overwhelming Might) in AI capabilities.
Historical Perspective: The author begins by referencing Elon Musk's 2014 comment about AI being like "summoning the demon." They highlight the rapid advancements in AI, such as beating human players in Go, Chess, and Shogi, and playing Atari games.
Complexity Comparison: The author argues that the universe's complexity is not just a matter of scale compared to a Go game but a difference in kind. They illustrate this by imagining the universe tiled with tiny Go boards, emphasizing that predicting the universe requires understanding at the atomic level, not just the "stone level" of Go.
Dynamics Models: Modern self-play systems like MuZero and GPT-4 are described as dynamics models that predict the next state based on the current state and action. The author connects intelligence with prediction and compression, suggesting that feeding the entire internet into a model could theoretically "win the universe."
Practical Limitations: The author challenges the idea that AI can easily dominate complex systems like the stock market. They use the example of hedge funds, explaining that dominating the market requires more compute power than the entire market, which is unrealistic.
Inclusion of Computers: In Go, the model doesn't need to include other computers, but in modeling the universe, computers must be included. The author argues that unless there's a staggering advantage, understanding them from self-play is unlikely.
Preventing FOOM: The author warns against capping FLOPS in training runs, as it could create a dangerous situation if one person breaks the restriction. They advocate for preventing a 51% attack on compute to avoid FOOM.
Efficiency and Innovation: The text dismisses the idea that a single group could achieve a 1e20x efficiency increase, arguing that more intelligence leads to new tricks, but they become harder to find.
Revolution and Gradual Change: The author predicts an information revolution that will transform intelligence, similar to how the industrial revolution transformed energy. They stress that this change won't happen overnight but will follow a gradual exponential curve.
Conclusion: The author concludes that unless a "terrifying powder keg" is built, there will be no sudden FOOM. They emphasize the complexity of the universe compared to games and call for letting the markets evolve naturally. The closing statement, "the singularity is nearer," hints at a belief in the eventual convergence of human and machine intelligence, but not in an abrupt or catastrophic manner.Probably one of the few given the majority of the comments are directly addressing and dissecting each point of the post.
It is not jumbled, but it is 1) the way Hotz usually writes, 2) unconventional, 3) cross-domain laterally connected, and 4) somewhat heavy for a sunday morning read.
Hotz generally doesn't do "barely coherent semi-thoughts." I think the label is needlessly harsh. He's just addressing the contemporary nonsense by bridging them together and calling their bullshit out.
It's sad to see so many of the replies to your comment are people attacking him for no good reason.
People seem to think that because ChatGPT appears to be intelligent that somehow we are on the cusp of a new AGI world. And that we need to serious thought leaders like George Hotz and other arrogant tech-bros to rescue us. It's just crypto all over again.
The real thought leaders are going to be those in academia, government etc. Boring policy people who know the subtleties of the real world far better.
Independent thought doesnt get tied down by bureaucratic organizations.
Its always been like this and always will.
Behind the appearance of boring personalities there are... boring personalities