Thanks for introducing me to this blog! This is the money quote for me:
There’s a mass influx of newcomers to our field and we’re equipping them with little more than folklore and pre-trained deep nets, then asking them to innovate.
There’s a mass influx of newcomers to our field and we’re equipping them with little more than folklore and pre-trained deep nets, then asking them to innovate.
The message I've gotten is "try things out". Innovation isn't necessarily improving specific techniques, but applying them to new fields. To apply techniques to things that are more mundane like data processing in non-AI-focused companies, you're gonna need bodies who know how to apply these newer programming techniques to solve problems.
Not every electrician has to understand electrical engineering.
i think this is especially important if you purely want to do applications. we have a bag of tricks (dropout, batchnorm, different optimizers and learning rate schedules). we have no real theory for why any of this should work; often a proposed explanation will later turn out not to make sense.
so the choice of how to train things comes down to "folklore", the community's collective experience. and there's no guarantee that folklore will generalize to your new architecture or dataset, and no way to know whether it even should.
the presentation seems to have struck a nerve and there's papers and talks floating around now examining the performance of common architectures in very simple settings. it's probably worth paying attention to these at least in the background, as it will hopefully crystallize into a body of knowledge that will be useful for someone trying to decide on architectures and optimization techniques.