One thought that occurred to me as I was reading this — and I appreciated the logical rigor — is it seems there's generally an implicit assumption that you are comparing the cognitive abilities of a single typical human with a single typical AI. In the very least it's unclear what's being assumed. That is, we talk about the "abilities" of an AI system surpassing human intelligence, but it's a little unclear what's meant by the latter.

Are we comparing an AI system against a single person, or against humans collectively? Usually there's a statement like Hypothesis 2 — "A computer with human-level learning abilities would generally surpass human intelligence because of additional technological advantages" — but then also "although the capacity of a human brain is huge, its input/output channels are bandwidth-limited". But is the relevant comparison a single human brain, or a sort of collective system of humans?

It seems a little misleading in a logical-comparative sense to talk about an AI system built on a collective human system like wikipedia, or the internet, in compared to a single person.

I don't actually mean to question whether a superintelligent system surpassing all human abilities, individually or collectively, could be built. But it seems to be that the concerns about bandwidth and whatnot are a function of the problem definition, and not necessarily a single human system, when a task-solving exercise involves a collective output in both cases.

I think going forward we're going to need much more clarity about what "superintelligence" really means. Currently it seems like AI is essentially mimicking collective human intelligence, which means it is probably constrained at that level, as that level is the source.

If/when AI starts focusing more on reasoning per se, it will probably also be necessary to better define whether there are some limits to discovery and reasoning, in the sense of bounds due to information about a problem domain versus reasoning ability limits. That is, sometimes we know the possible options but just don't have enough empirical information to decide between them; other times we lack explanatory options. Maybe that endeavor also is constrained by human abilities but it seems worthwhile to explore.

Lots of times I suspect the fears of superintelligence have more to do with the experimental nature of how AI was developed than anything else. That is, AI sort of proceeded in a more experimental way without mathematical theory driving it, so we have some catching up to do.