The human cortex use less than 0.2 watts

BTW if so this has a huge implication on consciousness.

premise 1) we know that reduced consciousness is a continuu, indicating a percentage of inhibited (deactivated) neurons, which can be gradually increased via e.g. benzodiazepines up to a coma or cerbral death. Many depressants reports as a symptom the intermediate feeling of reduced consciousness.

Now if we assume only ~ten percent of neurons are active at a given time, and as shown in premise 1) consciousness is a result of neuron activation and is physically located as the acitvated neurons. Then time passe and for some reason/stimulus, the newly activated neurons do not overlap with the previous 10%. In this experience of thought, since there is no overlap, the previous consciousness location has totally shut down like a brain death, and the new consciousness reside in newly activated neurons. Are you the same person? Since the physical location is disjoint, if you suppose you are, you have zero criterion for discriminating you from newly made isomorphic clones which is a paradox assuming there can't be multiple you.

Also yes they speak about the cortex, it is true that the majority (~70%) of the neurons are not in the cortex but in the forgotten cerebellum, however while mysterious, the cerebellum is not necessary for language processing, there are some humans on earth that don't have a cerebellum and while less intelligent, rumors are that they are still human. Although this should be double checked.

also while I think about it, there's probably a huge chunk (how much) of energy spent in building/catabolizing new axons vs executing the runtime.

Fair warning, you will see some pretty disturbing image results if you look it up.

How much carbohydrates did you expect you should consume?

A tangeant would be: do highly intellectual people have shorter lifespan because their body would trade-off energy for their brain thus limiting the supply on other organs and hence creating deficit in e.g. mitochondria bioenergetics?

Poor availability of healthy + low calories + palatable + inexpensive food in developed countries.

The obesity crisis comes from that.

EDIT: That said there is valid criticism of their focus on very large models at the expense of more efficient algorithm research, so maybe that’s what you were getting at.

Unless you are John Von Neumann, you are either approximately correct or very slow in solving novel tasks. On the other hand, we have the capacity to generalize much easier than our DL models.

I think that that relates to our memory. That is, we compress information in a way that allows us to search for the structure, map to the target domain, and then stitch it all together.

Hofstadter argues about it a lot in his books, that is, that we use analogies to learn stuff. In the same vein and in a very handwave-y manner, we can think of analogies as functors across categories. That is, as mappings between one domain to another where the relationship (morphism) between the objects is preserved.

For whatever reason we are capable of identifying those mappings quickly and applying them everywhere. This relates to symmetries where we have little to no trouble identifying objects that underwent certain transformations but our DL have trouble with.

A user in HN suggested that with respect to the visual transformations, we can do that because our optic nerve travels a long distance, from our eyes to our visual cortex, and that allows our brains to perform all kinds of data cleaning and signal processing.

The ideas about compression that I mentioned here is essentially what we saw in deepmind's GATO. More precisely, creating multimodal embeddings in that manner seems to enable good multi-task performance.

Sorry if this comes across as rambling.