I agree. Nothing I wrote above disagrees with your statement.
Can't they just add more?
(I can't edit find the button to edit my comment)
Is memory really an expected bottleneck in the long term?
It sounds like you're viewing "OOM" as "Out of Memory", when the parent meant "Order of Magnitude" - meaning that a baby's "training data" is several Orders of Magnitude more than what the DenseAV model from the paper used.
To massively anthropomorphize convnets, perhaps this serves the purpose of training up shape / color recognition first before fine detail assessment happens.