Humans have had 4 billion years of natural selection and then 4 years of input from all senses before they start identifying digits. I've seen studies suggesting that we're already born with an area of our brain for recognizing letters and words.
Seems at least fair in comparison to allow MAML/pretraining to find a good starting model (e.g: can recognize lines and shapes) by utilizing data other than the classes of interest.
> It's like the Aesop's fable where the sparrow hid in the eagle's feathers and jumped up at the last moment to claim "I'm the bird that flies the highest!".
> you should not need a lot of data for anything
Is choosing suitable starting weights/architecture/"predicates" by hand-designing based on our own built up information qualitatively any different? It still seems like "hiding" utilization of a huge amount of background knowledge about digits/symbols/images/reality.
Arguably harder to expand that way too. I think techniques such as unsupervised learning are probably going to be a more feasible way to utilize the increasing amount of data we're collecting about the universe.
At our current stage, both seem useful. Broad strokes like moving from dense networks to convolutional networks to add locality and translational invariance based on our knowledge that this is an appropriate search space for vision tasks, and then automated methods like NAS and pretraining to determine relevance on a finer level.
We definitely haven't exhausted ways for us to use our intuition to guide networks in the right direction, such as transformers with their attention mechanisms or say a network inherently agnostic to horizontal flips rather than teaching that with data augmentation, but I'm skeptical about what sounds like stepping back into hand-crafted feature extraction which automated techniques have been far more effective at.
> but the reliance on big data for pre-training suggests that the models are still not learning good representations that generalise well - so they still need big data to make up for it.
Wouldn't it be lack of generalization to new tasks after the fact which indicates poor predicates? I don't see why good feature detectors should necessarily themselves be discoverable by hand or with low data, as that doesn't appear to have been the case for organic intelligence.