We’re constrained in all sorts of ways because of biology, natural selection, energy, etc. I find it unlikely we just happen to be close to the max threshold.
If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
The speed of light puts limits on how far information can move within a given latency threshold. As you expand a computational system's capacity you face unavoidable trade-offs between interconnection throughput, latency, and computational capacity.
We don't know how close to this maximum the human brain is. However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely.
> If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.
Artificial human scale intelligence would already lead to massive shifts. However, the growth past that point could be incremental.
As the article points out - a motorbike is much faster than a cheetah, and a supersonic aircraft is much faster again, and a hypersonic missile faster again, and a satellite in orbit is much faster again. A bulldozer can push harder than a bull, and a big hydraulic ram much harder. A metal plate is more damage resistant than rhino skin, and a bomb shelter or an aircraft carrier or an underground vault even moreso. It could be a very high limit; Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation
> "We don't know how close to this maximum the human brain is."
We don't, but we do know human eyes are not close to telescopes or microscopes, humans cannot sense radio waves directly at all, human speech is not close to the loudest noise, human memory can't compete with computer storage, human calculation ability can't compete with a scientific calculator, etc. Why would we assume that intelligence has any closer limits?
> "However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely."
Nature doesn't care if we have poor eyesight after age 40, we still make glasses - as far as nature is concerned there are diminishing returns, as far as we are concerned we like clear vision. We also like sunglasses, polarising lenses, swimming googles, safety goggles, magnifying glasses, loupes, night vision goggles, x-rays, thermal imaging, millimeter wave scanners, head-up displays, tele-vision; we haven't stopped trying to enhance our vision. Why rule out wanting to improve intelligence at least a lot further?
Yes, there are many criteria where engineering has trumped what evolution has produced. However there are many others where evolution has developed efficiency or finesse that we struggle to match. So far, intelligence falls in that later category.
> Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation
Those theoretical limits are interesting, but not really relevant as they are intended to find a value that can't be theoretically exceeded, not a practical uper bound.
So far, our understanding of intelligence requires significant communication between different regions of compute. As you try to scale this, you need to dedicate more and more volume to that communication and your average latency between compute regions goes up. Then comes the problem of heat dispersion, which also starts to consume more and more volume as the system scales.
These mean that if latency matters to intelligence (and our understanding of intelligence seems to indicate that it does), then there are real, practical limits on the scaling of intelligence.
> Why rule out wanting to improve intelligence at least a lot further?
I'm not ruling out the desire. I'm not even ruling out the possibility.
I am pointing out that designing intelligence seems to be a lot harder than launching satellites or building a telescope. Intelligence is hard and I've presented good reason to believe that it gets harder the more you try to scale it.
Thus it seems likely that iterative improvements in intelligence will become progressively harder in a way limits the potential for runaway growth.
This doesn't rule out the possibility of a paradigm shift in technology that significantly increases capacity but such a possibility also isn't guaranteed.
> "average latency between compute regions goes up."
This Google'd article[1] says a brain could have 20mS of latency from front to back. We can ping over a hundred miles in that time with today's packet switched public networks, and light can travel 3,750 miles in that time. That's enough space to make a big 'brain' computer.
> "Then comes the problem of heat dispersion"
Biological brains have to stay alive, and have to be energy efficient because food can be scarce. Computers can be cooled with liquid nitrogen[2] without dying, or be under the ocean, in the Arctic, in space, even assuming a superintelligence couldn't come up with new architectures or new cooling methods. Infinite growth, or growth up to the upper bounds of theoretical physics is unlikely, I grant you that, but (assuming it can be done with computers at all) there seems to be a lot of room for a lot of growth.
[1] https://discoverysedge.mayo.edu/2023/03/09/understanding-the...
[2] https://www.pcgamer.com/overclocking-a-cpu-to-7-ghz-with-the...
Let’s assume we can build something 3x as intelligent as a single person. What exactly can it do that a single person can’t? The thing is the world is filled with super human intelligence, groups of people can create things that are beyond any single person but they are still constrained by physical reality.
Also humans don't make trees, humans stand watching while trees make themselves. Humans cannot make a plant or animal cell in a lab starting from atoms; nature can so it isn't a physical limitation. It's a matter of limited understanding of both how they are made, and the techniques to make them. Limited understanding is the thing more intelligence would attack.
Saying we can do something with cells isn’t convincing because we can already make arbitrary changes. I can email a fairly arbitrary DNA sequence and turn that into a viable organism. The existing cellular machinery is a tool to leverage just as other peoples work is a tool to leverage. There is plenty of work to be done, but there is no work in cellular biology that’s both physically possible and outside of the capability for groups of humans and their tools to do.
You can email an arbitrary DNA sequence, but we know that some humans are more intelligent than others, you can't email a more intelligent human DNA sequence because you don't know enough about how DNA codes for human intelligence (AFAIK nobody does). So how can you you say it's not outside the capability for groups of humans to do that, when the problem is the lack of understanding at an organizational level - something that more intelligence could help with? Even practically, it's physically possible for new nerves to be grown, but no groups of humans have cured quadraplegics and no tools exist which can do so - what supports your claim that such a thing is inside our capability?
We don’t have DNA sequences for more intelligent humans because we haven’t tried to find them. It’s easy to point to things not done and say it’s yet to happen, but that doesn’t mean such things are beyond our comprehension.
As to math advancing via individual effort, it seems that way because we lump success to the individual. Fermat’s last theorem wasn’t some major effort but there was real progress in the community and the “lone wolf person” actually benefited from both collaboration (people pointed out a problem with his initial proof etc) and advances that didn’t exist until quite recently. That’s the thing problems become easier when you have access to the correct tools.
Having not done something cannot be used as evidence that it is within our comprehension. It might be, it might not be, but "we haven't tried" is no evidence at all. Whereas "our intelligence must have finite limits" is evidence that somethings will be beyond us, even if we don't know exactly what.
But would ten thousand idiots have been able to use those mathematical advances to prove Fermat's Last Theorem? If not then there are limits to the "groups of people can do things one person can't".